An application that allows user to draw on air, recognizing the text written on air and projecting the text on an AR environment.
Live tracking of hands is done using mediapipe, a cross platform library developed by Google that provides ready-to-use ML solutions. Classified finger joints of hand into 21 landmark positions, thus by manipulating the landmark points, made possible to virtually draw on air.
Used Neural Networks (NN) to recognize the text written on air. Used IAM dataset for training the model. Model was trained using Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN). 5 layers of CNN were used and 2 layers of RNN were used. LSTM of RNN was used to train the model. A Connectionist Temporal Classification (CTC) was used to decode and find loss value of trained texts. The recognized text was projected on user's screens and the text was uploaded to Firebase.
Developed AR application which would read the text from Firebase and project the corresponding text on a user's AR environment.
To run the program, clone the entire repository. Install the packages in requirements.txt and run air_draw.py. Use the index finger to draw, index and middle finger to erase and open palm to clear any drawings. Use the pinky finger (finger up) only to send the drawn text to the deep learning model and predict the text. The predicted text is uploaded into Firebase from where the AR application loads the text.
The directory Air_Board is for the AR application. No need of this directory for writing and prediction purposes. The directory model contains the snapshots of the trained model with which the text is predicted upon.
In firebase_uploader.py, use your own Firebase storage details in self.conig. Also create a new serviceAccountKey.json file for the private keys and other related information. Refer the video below for more information
