OmniSign

Accessibility · Computer vision · Active R&D

Sign-language recognition built from its data upward.

OmniSign is an actively developed research and engineering platform for Lebanese Sign Language and basic American Sign Language recognition. The data we needed did not exist, so we built the dataset, models, and application together.

Watch the 2:30 demo Technical implementation RHU award announcement Read the dataset and evaluation note
40K+sign samples captured before augmentation
50K+training samples after augmentation
98%narrow internal evaluation under controlled conditions
21teaching traditions observed during collection

National FYP Demo Day 2025

First Prize, Public Choice

We won the Public Choice award at LAU on May 26, 2025. For us, it was recognition that the problem mattered just as much as the technology we built around it.

My contribution

I led the AI and data system from collection to application.

As OmniSign’s AI & ML Lead, I owned the technical path that turned community-contributed recordings into models the product could use. My work covered the collection strategy, processing and cleaning, augmentation, landmark extraction, model training and evaluation, combining isolated and temporal models, and application integration.

I was also involved in community outreach and event planning. That mattered technically: the quality and breadth of the system depended on earning participation from people and organizations across Lebanon, not only tuning a model after the data arrived.

  1. 01Data collection

    Structured collection across communities and different Lebanese teaching traditions.

  2. 02Processing

    Reviewed usable captures, augmented the retained material, and converted recordings into landmark-based training records.

  3. 03Model training

    Built and evaluated computer-vision paths for isolated signs and bounded temporal sequences.

  4. 04Multimodel integration

    Combined isolated-sign and bounded-sequence inference while keeping the two model outputs distinguishable.

  5. 05Integration

    Connected inference to the product experience for live recognition and translated output.

  6. 06Outreach

    Supported community collection, events, and public demonstrations that made the work possible.

The working system

Full project demo

I’ve kept the complete 2 minute 30 second recording here so you can see the recognition flow in context, without a highlight edit hiding the rough edges.

Why it started

It started with a joke nobody laughed at.

During COVID, I met a deaf man at a clinic. He wasn’t struggling to be understood—the staff eventually figured out what he needed through context and gestures.

What I noticed was different. He was trying to tell a joke. You could see it in his face, his timing, and the way he paused for a reaction that never came. Nobody understood him enough to laugh with him.

He had every right to share that moment with the people around him—easier than that. That was when OmniSign became more than an idea.

Our university gave us the option to choose from faculty-approved projects. We chose to find our own challenge instead. It felt overwhelming. But it felt right.

Recognition scope

Several modes, one experience.

Built from the ground up

A language with no map.

We entered this project assuming data existed. It didn’t. We assumed sign language was universal. It wasn’t. Beqaa differed from the South, the South differed from Beirut, and we documented variation associated with 21 teaching traditions observed during collection. That figure describes variation, not formal school partnerships or standardized coverage.

Captured40,000+

sign samples before augmentation

Training set50,000+

samples after augmentation

Vocabulary30+

daily expressions plus the alphabet

We built it by hand. We held data-collection campaigns across Lebanon. Volunteers signed at our homes—family members we taught ourselves—and NGOs and Deaf communities opened their doors to us. It was a messy process. If it wasn’t for the team’s chemistry, it never would have happened.

Community collection

Community participation changed the project’s stakes.

Deaf individuals in Lebanon still reach out and ask how they can help. They don’t see a university assignment; they see the beginning of something that could change their daily lives. A small clinical vocabulary is experimental today; broader medical and clinical coverage remains planned work.

Public engineering record

Three artifacts, each with a different scope.

The main OmniSign application and its community-collected Lebanese Sign Language dataset are not published as open source. These repositories preserve narrower parts of the early engineering work.

Technical implementation

One captured sign is not yet a training example.

The difficult part of OmniSign was maintaining a defensible path from a community-contributed capture to a model input and then to a bounded result. The public artifacts expose that path at prototype level: session-aware collection, explicit rejection of unusable landmark records, separate spatial and temporal representations, held-out evaluation for the isolated-sign baseline, and parallel inference through two model families.

Trace: capture, identity, and the first quality gate

The early collection tool organizes a session by signer, class label, session name, and timestamp, then stores the camera frame separately from its landmark record. A landmark file is written only when MediaPipe detects a hand. That separation matters: the image remains reviewable, while failed landmark extraction does not silently become a normal training vector.

The retained preprocessing code makes the next gate concrete. It rejects unreadable images, samples with no detected hand, and records that do not resolve to exactly 21 hand landmarks with x, y, and z coordinates. It also reports the surviving count per class before splitting. This is mechanical quality control, not proof of linguistic label review; the full community-labeling protocol and private dataset are not public.

Augmentation expands the project corpus from 40,000+ captured samples to 50,000+ training samples. The public prototype demonstrates brightness and contrast changes, hue and saturation changes, motion blur, translation, scaling, and rotation. Those are derived observations, not new participants or independent captures, so the two public counts remain deliberately separate.

The representation forks before the model

Isolated signs: one 63-value hand vector
The isolated-sign path flattens 21 MediaPipe hand landmarks into 63 coordinates. The combined prototype mirrors the horizontal coordinate for a detected right hand to present a common orientation; this does not normalize geometry or scale, or prove handedness invariance. Position and scale therefore remain possible sources of variance.
Temporal signs: thirty frames of holistic state
The sequence path keeps a rolling window of 30 frames. Each frame concatenates pose, face, left-hand, and right-hand features into 1,662 values. A bounded LSTM model receives a tensor shaped as one sequence of 30 such vectors; this is a fixed observation window, not continuous-language segmentation.
Missing landmarks: fixed shape through zero filling
If MediaPipe misses a pose, face, or hand group, the temporal path substitutes a zero vector of the exact expected length. That keeps every frame schema-compatible, but it also makes landmark absence part of the signal. Occlusion and detector failure can therefore influence the model rather than being neutral missing data.
Images remain evidence; landmarks become features
Landmarks lower the dimensionality seen by the classifiers and reduce direct dependence on raw backgrounds. They do not remove the upstream detector dependency: if the hand is outside the frame, occluded, or poorly detected, the classifier never receives a faithful representation.

Training and evaluation: what can be reconstructed

The public ASL alphabet baseline is a separate experiment on a third-party 87,000-image dataset. After landmark extraction, it performs a seeded, class-stratified 80/10/10 image-level split. A PyTorch classifier maps the 63-value vector through two batch-normalized hidden layers with dropout, uses cross-entropy loss and Adam, selects the checkpoint with the lowest validation loss, and evaluates that checkpoint once on the held-out test partition.

Stratification preserves the observed label proportions; it does not establish signer independence. No class-weighted loss or resampling is implemented in this baseline, and the retained evaluation reports aggregate loss and accuracy rather than per-class recall or a confusion matrix. The split is also image-level, so it is not evidence of performance on an unseen signer.

The temporal public artifact retains an LSTM model and its 30-frame inference contract, but not the training notebook, data lineage, or evaluation split that produced it. The project’s separate 98% figure is therefore kept only as a narrow controlled internal result. The surviving record is insufficient to reproduce its split, determine whether sessions or signers were isolated across partitions, or verify that augmented derivatives stayed with their source capture.

Multimodel inference is parallel dispatch, not a universal translator

  1. Frame inputExtract one shared landmark state

    MediaPipe Holistic produces the pose, face, and hand groups used by both inference branches.

  2. Isolated branchClassify the current hand

    The PyTorch path runs only when one hand is available, suppresses results below its configured confidence threshold, and includes an explicit “nothing” class.

  3. Temporal branchClassify the rolling window

    Once 30 frames are available, the TensorFlow LSTM path scores its supported sequence labels on every subsequent frame.

  4. PresentationKeep the branch results distinguishable

    The early integration renders both predictions rather than using a learned router or claiming that one calibrated score can arbitrate between two model families.

The sequence branch in the public prototype is closed-set: it chooses among three supported expressions and does not implement the isolated branch’s confidence rejection. That is useful integration evidence, but it is also why live camera processing must not be described as unrestricted continuous translation. The complete current application and its text-and-speech integration are not public.

Failure analysis and the controls that actually exist

  • No detected hand

    The isolated path does not classify. In the temporal path, an absent landmark group becomes zeros so the window shape remains valid; this preserves execution, not semantic certainty.

  • Out-of-vocabulary signing

    The isolated baseline can abstain through a confidence threshold or “nothing” class. The early sequence prototype cannot; it still returns the highest of its supported labels.

  • Boundary drift

    A sliding 30-frame window can begin or end inside a gesture, include a transition, or mix adjacent movement. No retained public artifact implements explicit start/end segmentation.

  • Evaluation leakage risk

    A defensible evaluation would group source captures and their augmented derivatives before splitting, and isolate signers or sessions when measuring generalization. The retained 98% record does not prove those controls, so the result is not broadened.

  • Variation across teaching traditions

    The 21 figure describes teaching traditions observed during collection, not partner schools or a standardized lexicon. Variation therefore belongs in collection design and evaluation, not as a marketing coverage count.

Ownership and evidence boundary

I led OmniSign’s collection strategy, processing and cleaning, augmentation, landmark extraction, training and evaluation, multimodel integration, and connection of inference to the application. The public Git histories provide direct implementation evidence for my early collector, preprocessing and isolated-sign training path, bounded-sequence inference, and combined-model prototype.

OmniSign is still a team project. The original student team and later contributor are credited below, and the award belongs to that team. The public repositories support my technical ownership of those artifacts; they do not imply sole ownership of the complete application.

Current boundary: OmniSign remains active R&D for isolated signs and bounded temporal sequences. Its small clinical vocabulary is experimental. It is not unrestricted continuous translation, a general benchmark across unfamiliar people and environments, or a replacement for a human interpreter. The community-collected Lebanese Sign Language dataset and complete application are not public.

Development

From final-year project to an ongoing platform.

  1. Started as our FYP

    The original student team—Layth Ayache, Tayseer Laz, Abou Baker Hussien Al Khatib, and Nour El Hariri—chose to define its own challenge with academic supervision from Dr. Oussama Mustapha.

  2. Ongoing

    Active R&D

    OmniSign remains actively developed as a self-funded, community-supported research and engineering platform.

  3. Next

    Broader clinical vocabulary

    A small experimental set exists; broader medical and clinical coverage remains planned.

  4. Future direction

    Scalable sequence modelling

    Transformer-based approaches remain a future research direction rather than demonstrated current functionality.

The people behind OmniSign

One product, distinct responsibilities.

I didn’t build OmniSign alone. Our team brought together AI, product direction, application development, research, and community work.

Project notes

Useful context

Which sign languages do you support?

We currently support Lebanese Sign Language and basic American Sign Language. LSL is our primary focus, and we built its project dataset from scratch with samples from Beqaa, South Lebanon, and Beirut.

Is this unrestricted real-time translation?

No. The application processes live camera input, but the current models recognize isolated signs and bounded temporal sequences within the project vocabulary. Unrestricted continuous translation remains a research goal.

How accurate is it?

A narrow controlled internal evaluation reached 98%. That figure applies only to its defined evaluation conditions; it is not a claim of general accuracy across unfamiliar signers, environments, or unrestricted signing.

How did you collect and label the data?

We collected 40,000+ samples before augmentation through community events, host organizations, families, and volunteers. We extracted landmark features with MediaPipe, labeled the data with community guidance, and expanded the training set to 50,000+ samples through augmentation.

What model architecture did you use?

We used an LSTM network for recognition within bounded temporal sequences. Transformer-based approaches remain a future research direction.

Who is this for?

OmniSign explores tools that can support communication between Deaf and hearing people. It is not presented as a replacement for human interpreters. A small clinical vocabulary is experimental, with broader clinical coverage planned.

What this case study does not claim

I present OmniSign as an accessibility research and engineering platform, not as a replacement for human interpreters or proof that Lebanese Sign Language has been standardized. We earned the award as a team, and I keep every performance number tied to the conditions in which we measured it.