Semi-Automated System to Recognize Handshapes from Indian Sign Language Videos & Convert into HamNoSys Sequence

Research Project • Variable Energy Cyclotron Centre (VECC), Kolkata (Dept. of Atomic Energy, Govt. of India)

Undergraduate Thesis: Carried out under the guidance of Dr. Tapas Samanta (Head of Artificial Intelligence Section / Computer Division, VECC, Kolkata). Achieved Rank 1 in a batch of 200 students in B.Tech Computer Science & Engineering, securing the highest marks in the 8th semester.

Project Overview & Key Objectives

Sign language is a natural, visual-spatial language developed within deaf communities. However, communication barriers between signers and non-signers remain a major hurdle in education, healthcare, and legal domains. Because sign language has no universal written form, phonetic transcription systems like the Hamburg Notation System (HamNoSys) and Signing Gesture Markup Language (SigML) are used to represent lexical gesture parameters (handshape, orientation, location, movement) to drive 3D avatar animations.

Annotating SigML sequences manually is extremely complex and labor-intensive. This project engineered an end-to-end, semi-automated AI pipeline that takes camera feed of Indian Sign Language (ISL), extracts spatial hand landmarks, and accurately classifies fundamental handshapes into standardized HamNoSys notations.

HamNoSystoSigML Annotation Software Interface

HamNoSystoSigML software interface with ISLRTC video player, HamNoSys virtual keyboard, and animated avatar.

MediaPipe 21 Hand Landmark Topology

21 3D knuckle landmarks extracted per frame by MediaPipe Hands.

Key Technical Contributions & Measurable Results

  • End-to-End AI System: Created a fully functional deep learning system to recognize handshapes across 60 ISLRTC videos and transcribe them into the Hamburg Notation System.
  • High-Accuracy Classification: Achieved 94.50% validation accuracy using a custom 1D Convolutional Neural Network (1D CNN) architecture with batch normalization, max pooling, dropout regularization, and early stopping.
  • Interactive Annotation Platform (HamNoSystoSigML): Built a web platform utilizing Node.js for the frontend, Python/Flask for backend processing, and PostgreSQL for gesture persistence, incorporating a virtual HamNoSys keyboard and animated avatar for real-time validation.
  • Dataset Engineering & 9x Augmentation: Captured still-image datasets for 6 fundamental HamNoSys handshape classes (hamfist, hamfinger2, hamfinger23, hamfinger23spread, hamfinger2345, hamflathand) via OpenCV, expanding volume 9x through multi-angle spatial transformations.
  • Real-Time 3D Landmark Extraction: Leveraged Google MediaPipe Hands to extract 21 3D hand landmarks (63 spatial coordinate features per frame), enabling fast, lightweight inference on standard hardware.
Block diagram of AI Engine workflow

End-to-end workflow: Image acquisition, MediaPipe feature extraction, 1D CNN model training, and real-time validation.

Tech Stack: Python, TensorFlow/Keras, OpenCV, Google MediaPipe, NumPy, Pandas, Matplotlib, Node.js, Flask, PostgreSQL.

ClimateActionAI

AI-Enabled Climate Action Platform • BeVisioneers: The Mercedes-Benz Fellowship

Developed as part of the BeVisioneers Fellowship (supported by Mercedes-Benz), selected for a global cohort from a pool of 10,000+ applicants.

  • Launched ClimateActionAI, an AI-enabled web application engineered to monitor, calculate, and systematically optimize carbon footprint reduction and save community wealth.
  • Mentored by the Lead, Analytics and Digital of Mercedes-Benz India on data-driven sustainability strategies and product scalability.
  • Attended the fully funded regional summit in Bangalore (2025) and shortlisted for project scholarships.

Focus Areas: Artificial Intelligence, Sustainability Analytics, Web Application Development, Climate Impact Modeling.

Annual Campaign Strategy & Grant Acquisition

Pro-Bono Consultant • Cancer Support Center, Illinois (via Taproot) [2025]

  • Spearheaded the annual survey campaign for the non-profit organization, restructuring survey instruments and outreach channels.
  • Expanded respondent participation by 130%, generating actionable stakeholder data to maximize grant-acquisition opportunities.