Assessing learners’ English public speaking anxiety with multimodal deep learning technologies

Public speaking anxiety (PSA) is a common phenomenon for language learners involving both psychological and physiological symptoms. Timely and effective PSA assessment can help diagnose learners’ speaking anxiety, offer learners feedback to alleviate their anxiety, and improve their public speaking competence. However, it is still a challenging issue to achieve accurate automated assessment of learners’ PSA due to the lack of large-scale and open-source multimodal datasets based on real classroom settings. This study collected the public speaking videos of English as a foreign language (EFL) learners in public speaking courses, and constructed a large-scale multimodal dataset named Speaking Anxiety in Real Classrooms (SARC) with three modalities of acoustic, visual, and textual data (including 1,158 manually-annotated speech videos and corresponding speech drafts of 382 participants). A multimodal deep learning model for automated assessment of learners’ speaking anxiety was proposed, and an online formative assessment platform was then developed to realize the automated assessment of PSA for classroom teaching. A pilot survey study involving 78 participants was then conducted to investigate learners’ acceptance of the platform. Experimental results verified the validity of the deep learning model and the consistency between automated assessment and teacher assessment. Learners’ acceptance data further indicated that collaboration between automated and human assessment provided them with the most satisfactory experience of using the platform to improve their English public speaking.