Projekt-Seminar Deep Learing (Prof. Dr. Jörn Hees)

Group: Leon Schirra, Alexander Kolb, Dominik Zoric, Henrik Theisen

Topic: Finger Counter

Our group focused on AI-based hand gesture recognition, employing four distinct approaches, all rooted in the same preparation process. As part of the preparation, each member captured 100 images for various hand poses, including "0,1,2,3,4,5,Spock,Other." You count along the hand starting with the thumb and ending with the little finger. These images were then mixed and divided into three sets: 80% for training, 10% for testing, and 10% for validation. For each of these three sets, a Mediapipe hand recognition AI was applied, generating CSV files. Each CSV file contains the image names and 21*3 landmark coordinates (21 landmarks with x, y, and z coordinates each). The code for taking the images, creating the 3 CSVs and cleaning up the data can be found in the TestDataGenerator notebook.
The images and the CSVs can be found in our Dataset Repository: https://github.com/henthe/DatasetForFingerCounter.git
Based on this shared preprocessing, we divided our team's focus into the following topics: Leon used the 3D coordinates with Tensorflow. Alexander went with ScikitLearn. Henrik used LightGBM. Dominik used Mediapipe.
At the end of this project we combined our models to create the ultimate hand gesture recognizer "TLS".

As a second project, Dominik went with a different preprocessing algorithm, he used two-dimensional normalized coordinates and trained his model with the Tensorflow Framework.

The Mediapipe code to generate the landmarks is based on this Repository https://github.com/nicknochnack/MediaPipeHandPose

Requirements

• Python 3.10.x
• All other required packages can be installed via the Jupyter Notebooks
• Further requirements for 2D_approach in 2D_approach/2DapproachDocu.md

2D approach with Tensorflow

To run the "2D_approach" a detailed instruction can be found in "2D_approach/2approachDocu.md". The folder contains a jupyter notebook file, where you can find the background of the models building, training and Hyperparameter-tuning and as well as an exported TensorFlow model(model.keras), which can be tested in real time, capturing your webcam. In this approach, Dominik created his "own" preprocessing algorithm, inspired by an already existing GitHub repository(detailed information in 2approachDocu.md). He generated two-dimensional landmarks from the pictures, which the team split beforehand uniformly in train, test and validation. The model is trained to classify hand gestures into our 8 predefined classes, and the jupyter notebook provides functionality for one-time model building, hyperparameter tuning, model evaluation, and visualization of misclassified samples.

Sources

• original respository , used as tutorial
• Youtube Tutorial, original repository explained
• GitHub: Google Research - Tuning Playbook
• Keras Documentation

Mediapipe

The "mediapipe" folder contains a Google colab notebook file, where you can find the background of the models building, training and an exported model. Keep in mind for the use of the model you need the pictures located somewhere, where they can be referenced. For further information of the specific folder structure visit: https://developers.google.com/mediapipe/solutions/customization/gesture_recognizer. After setting up your environment and the pictures, you can execute the cells one by one. The model is trained to classify hand gestures into our 8 predefined classes, and the jupyter notebook provides functionality for one time model building and model evaluation.

Sources

• Mediapipe
• Mediapipe: Customize Handgestures

LightGBM

LightGBM is a tree based Neural Network Model. This notebook is based on the following notebook: https://www.kaggle.com/code/bhavinmoriya/lightgbm-classifier-in-python

Guide

1. Clone the DataSet repository to your local machine.

2. Update the path variable with the path to the cloned repository.

3. Execute all the cells in the notebook except the last two.

4. To try a single image, run the second-to-last cell. Ensure that the image_path variable in line 10 contains the correct path to the image you want to test.

5. For a live demo, run the last cell.

Sources

• Kaggle: Tutorial
• Flaml: Documentation
• LightGBM: Documentation

Scikit-Learn

1. Pull the DataSet Repo somewhere

2. Put path to repo in path variable

3. Run everything

4. If you only want to execute certain classifiers, only execute the corresponding cells. To use Pipeline then change the parameters of the classifiers to be used.

Sources

• Scikit-Learn
• Kaggle: k-Nearest Neighbors
• GitHub: Pipeline & Other Classifier
• GitHub: param-grids

TensorFlow

The Folder "TensorFlow" contains a jupyter notebook file and a pre-trained TensorFlow model (.keras). The Python script implements a deep learning model for hand gesture recognition based on landmark data extracted from images. The model is trained to classify hand gestures into predefined classes, and the script provides functionality for hyperparameter tuning, model evaluation, and visualization of misclassified samples.

Usage

1. Clone the dataset repository if you want to work with our data.
2. Clone this repository and create a new virtual environment to run the Jupyter Notebook.
3. In the first cell of the Jupyter Notebook, adjust the path variables.
4. Run the cells sequentially (most cells require the execution of the previous cell to run).

Sources

• YouTube: TensorFlow 2.0 Complete Course - Python Neural Networks for Beginners Tutorial
• YouTube: Hand Gesture Recognition using TensorFlow | Hand Pose Estimation | MediaPipe
• GitHub: Google Research - Tuning Playbook
• Keras Documentation

Combined Model TLS

The combined model (TLS) of our models from TensorFlow, LightGBM and Scikit-Learn can be found in Combined_Models.