Shuwa Gesture Toolkit

Shuwa (手話) is Japanese for "Sign Language"

Shuwa Gesture Toolkit is a framework that detects and classifies arbitrary gestures in short videos. It is particularly useful for recognizing basic words in sign language. We collected thousands of example videos of people signing Japanese Sign Language (JSL) and Hong Kong Sign Language (HKSL) to train the baseline model for recognizing gestures and facial expressions.

The Shuwa Gesture Toolkit also allows you to train new gestures, so it can be trained to recognize any sign from any sign language in the world.

How it works

By combining pose, face, and hand detector results over multiple frames we can acquire a fairly requirement for sign language understanding includes body movement, facial movement, and hand gesture. After that we use DD-Net as a recognitor to predict sign features represented in the 832D vector. Finally using use K-Nearest Neighbor classification to output the class prediction.

All related models listed below.

  • PoseNet: Pose detector model.

  • FaceMesh : Face keypoints detector model.

  • HandLandmarks : Hand keypoints detector model.

  • DD-Net : Skeleton-based action recognition model.

Installation

  • For MacOS user
    Install python 3.7 from official python.org for tkinter support.

  • Install dependencies

    pip3 install -r requirements.txt

Run Python Demo

python3 webcam_demo_knn.py

  • Use record mode to add more sign.
    record_mode

  • Play mode.
    play_mode

Run Detector demo

You can try each detector individually by using these scripts.

  • FaceMesh
cd face_landmark
python3 webcam_demo_face.py
  • PoseNet
cd posenet
python3 webcam_demo_pose.py
  • HandLandmarks
cd hand_landmark
python3 webcam_demo_hand.py

Deploy on the Web using Tensorflow.js

Instructions here

Train classifier from scratch

You can add a custom sign by using Record mode in the full demo program.
But if you want to train the classifier from scratch you can check out the process here

GitHub

https://github.com/google/shuwa