Método para identificar estados cognitivos a partir de datos multimodales de comportamiento aplicando técnicas de visión artificial y algoritmos de aprendizaje automático supervisado

Maricarmen Toribio; Gabriel González; Andrea  Magadán; Nimrod  González; Máximo  López

doi:10.29057/xikua.v12i23.11795

Maricarmen Toribio Departamento de Ciencias Computacionales, TecNM/CENIDET, Cuernavaca, Morelos, México https://orcid.org/0009-0004-9089-0058
Gabriel González Departamento de Ciencias Computacionales, TecNM/CENIDET, Cuernavaca, Morelos, México https://orcid.org/0000-0002-1874-9402
Andrea Magadán Departamento de Ciencias Computacionales, TecNM/CENIDET, Cuernavaca, Morelos, México https://orcid.org/0000-0001-5474-4150
Nimrod González Departamento de Ciencias Computacionales, TecNM/CENIDET, Cuernavaca, Morelos, México https://orcid.org/0000-0002-1051-1379
Máximo López Departamento de Ciencias Computacionales, TecNM/CENIDET, Cuernavaca, Morelos, México https://orcid.org/0000-0002-4130-3564

DOI: https://doi.org/10.29057/xikua.v12i23.11795

Keywords: Cognitive States, Facial recognition, UX, Machine Learning, Multimodal data

Abstract

Human intelligence is a psychological quality that encompasses learning from experience, adapting to new circumstances, processing complex abstract concepts, and using knowledge to interact with and modify the environment. These cognitive states are reflected in responses that influence human behavior, manifesting themselves through facial expressions, body movement, and emotional reactions to situations that impact cognitive stability. The inclusion of cognitive state detection during user experience (UX) assessment represents a valuable opportunity to improve the efficiency and quality of products or services. The multimodal extraction strategy includes the detection of 46 points related to head movement, hand position, and facial expressions, three supervised machine learning algorithms Random Forest, KNN, and SVM were analyzed. Two image datasets were used for training, Cam3D and Pandora, obtaining an accuracy of 98% with Random Forest, 97% with KNN, and 95% with SVM, for the detection of three cognitive estates, attention, concentration, and distraction.

Downloads

Download data is not yet available.

References

Ardhendu Behera, P. M. (2020). Associating Facial Expressions and Upper-Body Gestures with Learning Tasks for Enhancing Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education, 236-270 https://doi.org/10.1007/s40593-020-00195-2.

Ashwin T. S., R. M. (2019). Automatic detection of students’ affective states in classroom environment using hybrid convolutional neural networks. Springer Science+Business Media, Doi : s10639-019-10004-6.

DataScientest. (enero de 2023). https://datascientest.com/es/random-forest-bosque-aleatorio-definicion-y-funcionamiento.
Ekman, P. &. (1978). Facial Action Codign System (FACS). https://doi.org/10.1037/t27734-000.

Francesca D’Erricoa, M. P. (2018). Cognitive Emotions in E-Learning Processes and Their Potential Relationship with Students. International Journal of Emotional Education, 89-111 ISSN 2073-7629.

Gama Velasco, A. (2018). Desarrollo de un Sistema interactivo basado en vision artificial para la rehabilitacion del tobillo. Morelos, Cuernavaca: CENIDET.

H.S Gunavathi, M. e. (2018). Towards Cognitive State Detection Using Facial Expression And Hand-Over-Face Gesture. IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology. Bangalore, India: IEE.

https://datascientest.com/es/que-es-el-algoritmo-knn. (Diciembre de 2022). DataScientest.

https://monkeylearn.com/blog/introduction-to-support-vector-machines-svm/. (junio de 2017). monkeylearn.

https://psy.takelab.fer.hr/datasets/all/pandora/. (s.f.). PSY.TXT.

Hu, G. L. (2018). Deep Multi-Task Learning to Recognise Subtle Facial Expressions of Mental States. Conference on Computer Vision (págs. 106 - 123). European: https://doi.org/10.1007/978-3-030-01258-8_7.

Indhumathi R, G. A. (2019). Emotional Interfaces for Effective E-Reading using Machine Learning Techniques. International Journal of Recent Technology and Engineering (IJRTE), 4443-4449 ISSN: 2277-3878 (Online) DOI:10.35940/ijrte.D8391.118419.

King, D. (2015). dlib C++ Library. Obtenido de www.Dlib.Net

Mahdi Pourmirzaei, G. A. (2021). Customizing an Affective Tutoring System Based on Facial Expression and Head Pose Estimation. Human Computer Interaction Cornell university, 17.

MediaPipe. (2020). Obtenido de https://developers.google.com/mediapipe

MediaPipe. (s.f.). https://developers.google.com/mediapipe/solutions/vision/face_landmarker.

Shan Li, S. P. (2020). Automated Detection of Cognitive Engagement to Inform the Art of Staying Engaged in Problem-solving. Journal Pre-proof, DOI: https://doi.org/10.1016/j.compedu.2020.104114.

Shivalila Hangaragi, T. S. (2023). Face Detection and Recognition Using Face Mesh and Deep Neural Network. Department of Electronics and Communication Engineering, Amrita School of Engineering, Bengaluru, Amrita Vishwa Vidyapeetham, India, 741-723; Doi: 10.1016/j.procs.2023.01.054.

Tzvi Michelson, H. A. (2022). A Peek at Peak Emotion Recognition. https://arxiv.org/abs/2205.09791.

Ullari, J. (2013). Procesos Metacognitivo del Pensamiento Lógico Matemático:Razonamiendo Hipotético. Ecuador: Universidad de Cuenca.

Universidad de Cambridge, R. U. (2011). Corpus 3D de estados mentales complejos espontáneos. (Computación Afectiva e Interacción Inteligente) Obtenido de https://www.cl.cam.ac.uk/research/rainbow/projects/cam3d/.

Valderrama Cárdenas, W. (2019). Reconocimiento automático del rostro para verificación de identidad para evaluación en línea. Tesis de Maestía. Cuernavaca, Morelos., México: CENIDET.