Breaking the Silence.
Designing and engineering a real-time sign language interpreter for video calls that translates ASL gestures into natural English sentences, so deaf users can finally have a flowing conversation with anyone.
Executive Summary
This project started from a simple question that kept nagging at me. If a deaf person wants to video call their mother, why should they have to stop and type every single thought into a chat box? That is not a conversation. That is messaging with a camera on.
I wanted to build something where the deaf user could just sign naturally, and the hearing person on the other side would see those signs appear as readable subtitles and chat messages in real time. And the reverse too. When the hearing person speaks, the deaf user gets instant captions without relying on lip reading.
What came out of it is a full-stack real-time system built on top of MediaPipe Holistic for body tracking, a multi-model AI inference pipeline running locally on the user's machine with zero cloud API calls, and a local Qwen 2.5 LLM that restructures raw ASL vocabulary into grammatically correct English. The app also includes AR facial expression reactions and a mid-air drawing mode that lets you paint with your fingers through the webcam.
Product
ASL Sign Language Interpreter
Video Call Application
My Role
Product Designer and Developer. End-to-end ownership of research, UX architecture, visual design, frontend development (React), backend engineering (Python/FastAPI), and ML model training.
Objective
Build a working real-time prototype that proves sign language can be translated into natural conversation during a live video call, entirely on-device, with no cloud dependency.
The Challenge
Video calling platforms were built for hearing people. Every single one of them assumes both sides of the call can speak and listen. For a deaf user, the experience completely falls apart.
Typing Kills Conversation
Every existing solution forces the deaf user to stop, look away from the camera, and type their thoughts. That breaks eye contact, kills the flow of conversation, and turns a video call into a glorified chat window.
ASL is Not English
Sign language has its own grammar and sentence structure that does not map one to one with spoken English. A direct word-for-word translation produces broken, robotic text that the hearing person struggles to parse.
Latency is the Enemy
Any delay between signing and seeing the translated text makes the conversation feel disjointed. If it takes more than a second, the hearing person has already moved on to their next thought.
Emotional Expression is Flattened
On a regular video call, you laugh, you nod, you react. But when the deaf user is focused on signing, subtle facial expressions that carry emotion in ASL get lost because the other person does not know how to read them.
Cloud Dependency Adds Risk
Sending live webcam footage to a remote server for processing introduces privacy concerns, network latency, and a hard dependency on internet quality. For something as personal as a family call, that felt unacceptable.
What We Wanted To Build
- ✓Gesture-First InputThe user signs naturally with their hands in front of the webcam. No typing, no special hardware, no pausing to write things out.
- ✓Natural Sentence OutputRaw sign vocabulary gets restructured into proper English grammar automatically, so the hearing person reads a normal sentence, not a jumbled list of words.
- ✓Sub-Second FeedbackThe translation loop needed to feel instant. The user signs, and within roughly 600 milliseconds of resting their hands, a finished sentence appears.
- ✓Emotional LayerFacial expressions should trigger visible AR reactions on screen, like hearts when kissing the air, so the deaf user can express emotion beyond just words.
- ✓Fully Local ProcessingThe entire AI pipeline runs on the user's own machine. No webcam footage leaves the device. No cloud APIs. No subscriptions.
Research and Discovery
Before writing a single line of code, I needed to understand the actual shape of the problem. Four core insights from studying ASL communication patterns and existing accessibility tools drove every decision in this project.
ASL Has Its Own Grammar
Hover to reveal insightASL follows a topic-comment structure, not the subject-verb-object order of English. 'You want food' in ASL could mean 'Do you want to eat?' in English. A direct word lookup produces nonsense. The system needs an actual language model to bridge this gap.
Existing Tools Are Text-First
Hover to reveal insightMost accessibility apps for deaf users focus on speech-to-text transcription for the hearing person's side. Almost nothing exists that goes the other direction, from gesture to text, in real time during a live call.
Latency Destroys Natural Flow
Hover to reveal insightResearch on conversational turn-taking shows that humans tolerate roughly 200 to 400 milliseconds of silence before a pause feels awkward. Any sign language translation system that takes longer than that will feel broken to both parties.
Emotion Carries Meaning in ASL
Hover to reveal insightIn ASL, facial expressions are not just emotional cues. They are grammatical markers. A raised eyebrow can turn a statement into a question. Ignoring the face means losing critical parts of the message.
How We Built It
The full system is a pipeline of specialized AI agents, each responsible for one stage of the translation process. Here is how each piece works.
MediaPipe Holistic Landmark Extraction
Every webcam frame runs through MediaPipe Holistic which extracts 543 3D landmarks covering the face (468 points), both hands (21 each), and upper body pose (33 points). These landmarks get buffered into 30-frame sequences and sent to the Python backend over WebSocket.
Multi-Model Inference Pipeline
The backend runs three models in a strict priority hierarchy. First, a custom PyTorch LSTM model trained on WLASL data checks for dynamic multi-frame signs like 'how are you' and 'want'. If it does not trigger, a pre-trained Kaggle TFLite model handles 250 single-word ASL signs. Both use temporal smoothing that requires 3 consecutive confident predictions before committing.
Resting Detection and Sentence Trigger
The system continuously measures movement across hands and arms. When the maximum displacement drops below 0.02 for 6 consecutive frames, roughly 600 milliseconds, it understands the user has finished signing. That silence window triggers the NLP agent to assemble the buffered words into a sentence.
Local NLP Grammar Translation
Qwen 2.5 0.5B runs entirely on the user's CPU with zero API calls. It takes raw ASL gloss sequences like 'you want food icecream' and restructures them into natural English like 'Do you want to eat ice cream?' using few-shot prompt engineering calibrated to ASL-specific grammar patterns.
AR Facial Expression Reactions
A custom ExpressionDetector analyzes face mesh landmarks every frame to detect three expressions: kiss (mouth width ratio below 0.336), mad (inner brow distance below 0.220), and cry (frown drop above -0.006). Each triggers CSS particle effects like floating hearts or falling tears. A head-turn safety lock prevents false triggers when the face is angled.
Air Paint Mode
Pinching thumb and index finger on the right hand draws glowing neon lines in mid-air through the webcam feed. The left hand pinch acts as an eraser. After 2 seconds of inactivity, the entire drawing crumbles into particle ash that drifts upward with simulated wind turbulence physics, then fades out naturally.
Roadblocks and How We Solved Them
Every interesting technical project has moments where the thing you built confidently just does not work the way you expected. These are the walls we hit and the specific decisions that got us past them.
The Custom Model Kept Triggering 'Want' During Hand Transitions
When the user moved their hands between signs, the brief transition motion was consistently being classified as the sign for 'want' by the LSTM model. The confidence was high enough to pass the threshold. We solved this by adding a motion magnitude veto. Before trusting the custom model's output, the system now checks that the total hand displacement across the 30-frame window exceeds 0.5. If the hands are not moving enough to constitute an actual sign, the prediction gets thrown out regardless of confidence.
Hello and Bye Were Indistinguishable
Both 'hello' and 'bye' in ASL are waving gestures that produce nearly identical landmark trajectories. The Kaggle model could not reliably separate them. We added a handedness heuristic. If the right hand is dominant in the sequence, the system outputs 'hello'. If the left hand is dominant, it outputs 'bye'. Simple, but it matched how native signers actually perform these gestures in practice.
The Entire UI Froze at 20 FPS
The backend was emitting debug telemetry strings 20 times per second. Each string contained fluctuating movement math data. On the frontend, calling setDebugStats with each message forced React to re-render the entire glassmorphism-heavy UI tree 20 times per second. The main thread was completely saturated. The fix was surgical. We removed the React state setter from the debug event handler entirely. The debug data still flows through the WebSocket, but it is consumed only by console.log, never touching the React state tree.
Two-Handed Signs Were Triggering from One Hand
Signs like 'how are you' and 'want' require both hands to be actively visible. But the model was sometimes triggering these predictions when only one hand was in frame, because the landmark tensor fills missing hands with zeros, and the model had learned to associate certain zero patterns with certain signs. We added explicit hand-presence validation. Both hands must be visible for at least 10 out of 30 frames before the system will accept a two-handed sign prediction.
The Kaggle Model Kept Overriding Custom Gestures Mid-Formation
While the user was in the middle of performing a multi-frame custom sign like 'how are you', the Kaggle model would sometimes detect a brief static hand shape within those frames and emit a single-word prediction, stomping over the forming gesture. We introduced a suppression flag. If the custom model is actively detecting a forming gesture with moderate confidence, the Kaggle model's output is suppressed for that frame.
The LLM Blocked the WebSocket Event Loop
The Qwen 2.5 model runs on CPU and takes roughly 200 to 500 milliseconds per inference call. While it was generating a sentence, the entire FastAPI backend was blocked, meaning no new WebSocket frames could be received or processed. We moved the LLM inference into asyncio.to_thread, which runs the CPU-heavy generation in a threadpool worker without blocking the async event loop. New frames continue streaming in while the sentence is being constructed.
What We Learned
Accessibility Is a Design Constraint, Not an Afterthought
Designing for deaf users forced every UX decision to be grounded in real communication needs. The split-screen layout, the subtitle positioning, the chat history panel, none of it was decorative. It all served a specific accessibility function.
Local-First AI Changes the Privacy Equation
Running the entire ML pipeline on-device eliminated the privacy conversation entirely. No webcam footage leaves the machine. That is not just a feature. For something as intimate as a family video call, that is a requirement.
Multi-Model Orchestration Is Harder Than Single-Model Accuracy
Getting one model to work is straightforward. Getting three models to coexist without stepping on each other's predictions required building an entire priority system, temporal smoothing, suppression flags, and handedness heuristics. The integration layer was more complex than any individual model.
Real-Time UX Has No Tolerance for Jank
A translation delay that looks fine in a recorded demo becomes unbearable in a live conversation. Every millisecond of latency compounds. We learned that perceived speed is not about raw model inference time. It is about how quickly the user sees their first piece of feedback.
The Gap Between Demo and Daily Use Is Enormous
The system works well for controlled signing in good lighting with a stable webcam angle. Real-world usage involves variable lighting, partial hand occlusion, users signing at the edge of the frame, and background movement. Bridging that gap is where the next phase of work lives.
"Technology should never be the reason a mother cannot understand her child. This project is a proof of concept that it does not have to be."
Design Reflection
See It In Action
A quick demonstration of the real-time ASL translation pipeline.
Open Source
The complete source code for the Sign Language Interpreter is available on GitHub.
View Repositoryhttps://github.com/Yeshwanthgopal/Sign-language-interpreter