Klasifikasi Arah Pengendali Karakter Permainan Berbasis Gestur Badan Menggunakan Hybrid Convolutional Neural Network dan Long Short-Term Memory

Arsyapradana Fadlanabil Bahri; Oddy Virgantara Putra; Dihin Muriyatmoko

doi:10.61132/prosemnasproit.v2i2.197

Authors

Arsyapradana Fadlanabil Bahri Universitas Darussalam Gontor
Oddy Virgantara Putra Universitas Darussalam Gontor
Dihin Muriyatmoko Universitas Darussalam Gontor

DOI:

https://doi.org/10.61132/prosemnasproit.v2i2.197

Keywords:

Gesture recognition, hybrid CNN-LSTM, deep learning, real-time interaction, game character control

Abstract

The increasing sedentary lifestyle in the digital era has the potential to cause various health problems due to lack of physical activity. One approach that can be taken to encourage physical activity is through the use of digital games with body movement-based control mechanisms. This study aims to develop a body gesture-based game character control system using a hybrid Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) model. CNN is used to extract spatial features from each video frame, while LSTM serves to model the temporal relationship between frames so that movement patterns can be recognized sequentially. The research method used refers to the Machine Learning Lifecycle stages, starting from data collection, preprocessing, model development, to implementation in the endless runner game genre. Testing results show that the CNN–LSTM model is capable of classifying body gestures and generating outputs that can be used as commands to control game characters. The implementation of this system enables more natural and interactive game interactions without conventional input devices, and has the potential to encourage players to lead a more active lifestyle.

References

Baloch, Zartasha, Faisal Karim Shaikh, and Mukhtiar Ali Unar. “CNN-LSTM-Based Late Sensor Fusion for Human Activity Recognition in Big Data Networks.” Wireless Communications and Mobile Computing 2022, no. Ml (2022). https://doi.org/10.1155/2022/3434100.

Barona López, Lorena Isabel, Francis M. Ferri, Jonathan Zea, Ángel Leonardo Valdivieso Caraguay, and Marco E. Benalcázar. “CNN-LSTM and Post-Processing for EMG-Based Hand Gesture Recognition.” Intelligent Systems with Applications 22, no. February (2024): 200352. https://doi.org/10.1016/j.iswa.2024.200352.

Blas, Héctor Sánchez San, André Sales Mendes, Daniel H.de la Iglesia, Luís Augusto Silva, and Gabriel Villarrubia González. “A Multiagent Platform for Promoting Physical Activity and Learning through Interactive Educational Games Using the Depth Camera Recognition System.” Entertainment Computing 49, no. December 2023 (2024): 100629. https://doi.org/10.1016/j.entcom.2023.100629.

Bourke, Emily, Jonathan Rawstorn, Ralph Maddison, and Tony Blakely. “The Effects of Physical Inactivity on Other Risk Factors for Chronic Disease: A Systematic Review of Reviews.” Preventive Medicine Reports 46, no. August (2024): 102866. https://doi.org/10.1016/j.pmedr.2024.102866.

Dang, Tuan Linh, Sy Dat Tran, Thuy Hang Nguyen, Suntae Kim, and Nicolas Monet. “An Improved Hand Gesture Recognition System Using Keypoints and Hand Bounding Boxes.” Array 16, no. February (2022): 100251. https://doi.org/10.1016/j.array.2022.100251.

Liu, Ying, Jiahao Xue, Daxiang Li, Weidong Zhang, Tuan Kiang Chiew, and Zhijie Xu. “Image Recognition Based on Lightweight Convolutional Neural Network: Recent Advances.” Image and Vision Computing 146, no. January (2024): 105037. https://doi.org/10.1016/j.imavis.2024.105037.

Prananta, Gidion B, Hagi A Azzikri, and Chaerur Rozikin. “Deteksi Dan Pengenalan Gesture Tangan Secara Real-Time Menggunakan Jaringan Saraf Tiruan Konvolusional.” Methodika 9, no. 2 (2023): 30–34. https://ejurnal.methodist.ac.id/index.php/methodika/article/view/1911/1578.

Sabzipour, Behmard, Richard Arsenault, Magali Troin, Jean-luc Martel, and Juliane Mai. “Comparing a Long Short-Term Memory ( LSTM ) Neural Network with a Physically-Based Hydrological Model for Streamflow Forecasting over a Canadian Catchment” 627, no. February (2023). https://doi.org/10.1016/j.jhydrol.2023.130380.

Silveira, Erika Aparecida, Carolina Rodrigues Mendonça, Felipe Mendes Delpino, Guilherme Vinícius Elias Souza, Lorena Pereira de Souza Rosa, Cesar de Oliveira, and Matias Noll. “Sedentary Behavior, Physical Inactivity, Abdominal Obesity and Obesity in Adults and Older Adults: A Systematic Review and Meta-Analysis.” Clinical Nutrition ESPEN 50 (2022): 63–73. https://doi.org/10.1016/j.clnesp.2022.06.001.

Wang, Xiaojuan, Yun Zhong, Lei Jin, and Yabo Xiao. “Scale Adaptive Graph Convolutional Network for Skeleton-Based Action Recognition.” Tianjin Daxue Xuebao (Ziran Kexue Yu Gongcheng Jishu Ban)/Journal of Tianjin University Science and Technology 55, no. 3 (2022): 306–12. https://doi.org/10.11784/tdxbz202012073.

Yadav, Anita, C. K. Jha, and Aditi Sharan. “Optimizing LSTM for Time Series Prediction in Indian Stock Market.” Procedia Computer Science 167, no. 2019 (2020): 2091–2100. https://doi.org/10.1016/j.procs.2020.03.257.