Gesture Recognition in Indonesian Sign Language Using Hybrid Deep Learning Models

Sign language comprises unique hand gestures used to communicate between the hearing and hearing-impaired communities, with variations based on regions or countries. Due to the increasing global prevalence of hearing loss, there is a growing need for efficient sign language recognition systems. This paper presents the development of a hybrid CNN-LSTM model specifically designed to recognize static alphabetical gestures in Indonesian Sign Language (BISINDO). Primary data was collected to address the issue of publicly available dataset. These self-collected images underwent augmentation and preprocessing stages prior to model training. The architecture and hyperparameter configuration of the hybrid model were fine-tuned using the Randomized Search CV method. Performance analysis was conducted on both the hybrid model, its constituent models, and state-of-the-art model. The experimental results revealed remarkable performance by the hybrid model with excellent accuracies of 99.60%, 84.87%, and 98.00% on the training, validation, and testing sets respectively, along with a macro average score of 0.98, indicating high precision, recall, and F1-score across all the classes. However, it was observed that the VGG-16 and CNN exhibited slightly superior performance compared to the hybrid model, while the LSTM demonstrated the least favorable performance among the models.