Multimodal communication in instructional settings: An investigation of the functional roles of gestures and arrows

In daily communication and instructional settings, arrows and spontaneous gestures usually convey similar spatial information to interlocutors.They can visualize spatial information in 2D or 3D space based on shared conceptual and spatial representations.The present study investigates the relationship between spontaneous gestures and arrows from the perspective of generation and comprehension in instructional settings.We introduce a corpus of arrows in written multimodal text and a corpus of gestures generated during communication through multimodal instructional material.We also report an experimental investigation that used corpus material to study the impact of gestures on comprehension learning.A comparison of the corpora reveals functional similarities and differences between arrows and gestures and their complementary role in multimodal communication.The findings suggest that arrows and gestures may share a common conceptual space during communication, having a significant impact on comprehension and learning.