Unpacking Emotional Expressions in Live Streaming Commerce: A Multimodal Machine Learning Analysis Integrating Visual, Vocal, and Verbal Cues

This study examines the correlation between multimodal emotional expressions and sales in live streaming e-commerce, drawing on social presence theory. Leveraging deep learning (CNNs, Bi-LSTM, and BERT), we measured visual (smile, eye gaze, and body motion), vocal (vocal valence and arousal), and verbal (emotional intensity, empathy, and interaction) emotional expressions from 4925 live streaming video clips. The analysis proceeds in three stages: stage 1 leverages interpretable machine learning (XGBoost and SHAP) to evaluate the relative importance of emotional expressions across three channels, revealing the visual channel as the most significant. Stage 2 employs econometric methods to examine the correlation between emotional expressions and sales. The results suggest that all emotional expressions correlate positively with sales, with stronger effects for hedonic than utilitarian products, and customer engagement (likes and comments) mediates these relationships. This study offers theory- and data-driven insights into how multimodal cues drive marketing outcomes in live streaming e-commerce.