Creating an AI, lifelike interviewer—Clara, Hack Princeton '24

CLARA (https://claraprep.co)

TLDR:

• We built Clara in 36 hours (2 all-nighters in a row 🥂)
• **The App: **https://claraprep.co
• **Github: **https://github.com/aoztanir/clara
• Our Buildl: https://dorahacks.io/buidl/19500

As a team of 4, from immigrant families and as immigrants ourselves, we’ve felt the difficulties of coming to a new place up close, and we strongly believe that immigration shouldn’t be scary or expensive. Clara provides immigrants with interactive, comprehensive coaching to improve their interview skills and speed up the immigration process, at zero cost. We believe that Clara enables the immigration process as it should be – straightforward, transparent, and free.

The Problem:

Every year, there are over 16 million US Visa applications from across the world, but 45% of them end up failing. Arriving in the US, whether as an immigrant or a businessperson, is no easy feat, and there are a variety of roadblocks that impede the path towards securing a visa.

First, the immigration process is time-consuming and overwhelming. Immigrants are bombarded with hundreds of forms and papers to fill out, often with little understanding of how to fill the forms out and what they’re agreeing to. A single missed deadline or tiny mistake in any one of these documents could lead to a rejection, sending the immigrant’s application back to the bottom of the pile.

To help manage the process, many turn to lawyers for help. While immigration attorneys are a valuable resource to many, their cost per hour is much too high for many immigrants to use, with the average cost per immigration hovering between $3,000 and $7,000. The lawyers who take pro bono cases are few and far between. Rather than becoming a beacon of clarity to newcomers, lawyers are a source of financial stress.

Those who turn to the internet find themselves navigating a cobweb of government websites, immigration chat boards, and fragmented advice all over the internet. Immigrants don’t know who, and what, to believe. In a new place that is already full of its own stressors, newcomers shouldn’t have to worry about maintaining their status, but on creating a new life.

How We Solve This Problem + Our Feature Set:

Frustrated by the issue, our team set out to build Clara. We envisioned Clara as a ‘one-stop shop’ for everything related to immigration, from visa interview preparation, practicing English skills, and learning more about the immigration process and resources. We wanted the platform to be interactive, simple to use, yet informative, and this ideology influenced our team throughout the creation process.

Our keystone feature is Clara, our huma-like virtual immigration coach. Using voice, immigrants can speak face to face with Clara, practicing communication skills and preparing for visa interviews. Clara is open, kind and creates a sense of excitement toward the immigration process that immigrants need to thrive. Clara can help out with finding resources, giving tailored feedback based on interview performance, and challenging users to become more comfortable with speaking. Clara is able to converse with users in over 40 different languages, offering the comfort of home that many immigrants are missing, while also helping them learn more about their new home.

After each conversation, users’ responses are analyzed by GPT-3.5-turbo and GPT-4O and are ranked on four key metrics, assigning a score to each, as well as an overall score based on the conversation as a whole. These conversations are categorized and tracked, and users can measure their progress from their starting point.

If users aren’t in the talking mood, they have access to an interactive research guide and resource library, where they can find the most up-to-date information on immigration, like language learning classes and building connections in their neighborhoods. Rather than scouring through the internet to find opportunities, all of that potential is wrapped in one place.

Moreover, our app offers immigrants with an AI text chat platform specifically catered towards immigration and the resources necessary to make it a seamless process.

Tech We Used:

To create the frontend platform, we used Next.js/React along with Mantine as our component library, which made creating custom components that look great very simple. We also took advantage of mantine’s prebuilt hooks to speed up our development process.

On the backend, we used Next.js edge functions which we subsequently deployed to Vercel. We were excited to take advantage of the app router, and SSR, along with the new RSC (React Server Components) and their implementation in NextJS. This allowed us to build a full stack app in one codebase which made things infinitely more simple for our team given linking API endpoints to a frontend was not a laborious challenge.

To create Clara, we used HeyGen’s new Streaming API, which was released just a few weeks ago, which was exciting to work with. We utilized webRTC and websockets to communicate with the API, streaming voice, and video back and forth. It was difficult to set up the API given the lack of documentation amidst the novelty, but we managed to get it done in the end. Navigating this and various obscure tutorials/examples made this our biggest challenge.

In order to work with speech recognition we used the browser’s inbuilt speech recognition library created by google which allowed us to transcribe our voices and convert them to text which could subsequently be processed by our backend functions and OpenAI queries. This enabled us to get real time metrics and performance reports on interviews within mere seconds after completing one!

Additionally, for the interactive resource library, we used the Vercel AI SDK and its useChat functionality for Next.JS to query GPT-3.5-TURBO with select 4o queries. We opted for GPT-3.5-Turbo as it was much quicker in sending responses than 4o and met our needs.

For login, we used Google OAuth, and used Supabase and its Postgres instance for our database. With this, we could harness Supabase’s real time features while also taking advantage of a relational SQL database.

Lastly, we deployed our code to production using Vercel!