Implementation of Hand Movement Tracking for Real-Time Sign Language Translation
DOI:
https://doi.org/10.28932/ice.v7i2.12818Keywords:
alphabet, attention mechanism, BiLSTM, deep learning, mediapipeAbstract
This study developed a real-time American Sign Language (ASL) sign language identification and interpretation system based on deep learning. The system used two data sources: the ASL alphabet dataset for individual character recognition and the WLASL dataset for vocabulary recognition. The WLASL dataset was chosen as the benchmark for evaluating complex word gestures because it encompasses a wide range of users and extensive movement dynamics. Data processing involved extracting hand-gesture and body-posture markers using MediaPipe, followed by preprocessing and augmentation. Two learning architectures were implemented: a Feedforward Neural Network for alphabet classification and a BiLSTM integrated with an Attention Mechanism for vocabulary recognition. The system was evaluated using accuracy, precision, recall, F1 Score, and K-fold cross-validation. The results demonstrated promising performance: 99% accuracy for alphabet recognition and 78% for vocabulary recognition, with the Attention Mechanism contributing substantially to vocabulary recognition. The system operates in real time at 15-20 FPS and is efficient on mid-range devices, potentially becoming an inclusive communication alternative for the sign language community.References
Abdulhamied, R. M., Nasr, M. M., & Abdulkader, S. N. (2023). Real-time recognition of American sign language using long-short term memory neural network and hand detection. Indonesian Journal of Electrical Engineering and Computer Science, 30(1), 545–556. https://doi.org/10.11591/ijeecs.v30.i1.pp545-556
Al Abdullah, B. A., Amoudi, G. A., & Alghamdi, H. S. (2024). Advancements in sign language recognition: A comprehensive review and future prospects. IEEE Access, 12, 128871–128895. https://doi.org/10.1109/ACCESS.2024.3457692
Al-Qurishi, M., Khalid, T., & Souissi, R. (2021). Deep learning for sign language recognition: Current techniques, benchmarks, and open issues. IEEE Access, 9, 126917–126951. https://doi.org/10.1109/ACCESS.2021.3110912
Alsharif, B., Alalwany, E., Ibrahim, A., Mahgoub, I., & Ilyas, M. (2025). Real-time American sign language interpretation using deep learning and keypoint tracking. Sensors, 25(7), 2138. https://doi.org/10.3390/s25072138
Bora, J., Dehingia, S., Boruah, A., Chetia, A. A., & Gogoi, D. (2023). Real-time Assamese sign language recognition using MediaPipe and deep learning. Procedia Computer Science, 218, 1384–1393. https://doi.org/10.1016/j.procs.2023.01.117
Gu, Y., Sherrine, Wei, W., Li, X., Yuan, J., & Todoh, M. (2022). American sign language alphabet recognition using inertial motion capture system with deep learning. Inventions, 7(4), 112. https://doi.org/10.3390/inventions7040112
Kazbekova, G., Ismagulova, Z., Ibrayeva, G., Sundetova, A., Abdrazakh, Y., & Baimurzayev, B. (2025). Real-time lightweight sign language recognition on hybrid deep CNN-BiLSTM neural network with attention mechanism. International Journal of Advanced Computer Science and Applications (IJACSA), 16(4). https://doi.org/10.14569/IJACSA.2025.0160452
Li, D., Rodriguez Opazo, C., Yu, X., & Li, H. (2020). Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison (arXiv:1910.11006). 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). https://doi.org/10.1109/WACV45572.2020.9093512
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., Chang, W.-T., Hua, W., Georg, M., & Grundmann, M. (2019). MediaPipe: A framework for building perception pipelines (arXiv:1906.08172). arXiv. https://arxiv.org/abs/1906.08172
Nagaraj, A. (2018). ASL Alphabet. Kaggle. https://doi.org/10.34740/KAGGLE/DSV/29550
Orovwode, H., Oduntan, I. D., & Abubakar, J. (2023). Development of a sign language recognition system using machine learning. 2023 International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD) (pp. 1–8). https://doi.org/10.1109/icABCD59051.2023.10220456
Samaan, G. H., Wadie, A. R., Attia, A. K., Asaad, A. M., Kamel, A. E., Slim, S. O., Abdallah, M. S., & Cho, Y.-I. (2022). MediaPipe’s landmarks with RNN for dynamic sign language recognition. Electronics, 11(19), 3228. https://doi.org/10.3390/electronics11193228
Sherstinsky, A. (2020). Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306. https://doi.org/10.1016/j.physd.2019.132306
Sundar, B., & Bagyammal, T. (2022). American sign language recognition for alphabets using MediaPipe and LSTM. Procedia Computer Science, 215, 642–651. https://doi.org/10.1016/j.procs.2022.12.066
Zhang, J., Bu, X., Wang, Y., et al. (2024). Sign language recognition based on dual-path background erasure convolutional neural network. Scientific Reports, 14, 11360. https://doi.org/10.1038/s41598-024-62008-z
Zhang, Z., Feng, F., & Huang, T. (2022). FNNS: An effective feedforward neural network scheme with random weights for processing large-scale datasets. Applied Sciences, 12(23), 12478. https://doi.org/10.3390/app122312478
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Bernardus Anggodho Aryudhawan Hadi, Wilfridus Bambang Triadi Handaya, Suyoto Suyoto
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
![](https://live.staticflickr.com/65535/51110863223_e27aabda19_t.jpg")
