The Allocation of Visual Attention in Multimedia Search Interfaces

Multimedia analysts are challenged by the massive numbers of unconstrained video clips generated daily. Such clips can include any possible scene and events, and generally have limited quality control. Analysts who must work with such data are overwhelmed by its volume and lack of computational tools to probe it effectively. Even with advances made in machine-learning, humans are still needed to review and assess multimedia clips to determine if the video contains an event of interest or if it provides an actionable insight. This study seeks to provide a better understanding of the influence of the user interface of a multimedia player on the performance of tasks in terms of accuracy and the allocation of visual attention, especially whether the customizations that users make to the interface can enhance their performance of the tasks. The results provide insights that reinforce the human ability – and preference – to discern “gist” from static images as surrogates to viewing video clips, and identifies characteristics of user interface and interaction design of multimedia players to leverage this human ability. Although the findings do not indicate strong correlations between accuracy and false alarm rates with how subjects allocate their visual attention, the qualitative results reinforce that different event types, especially event types that are more difficult to semantically define, may require a greater reliance on the video playback, and therefore an overall different strategy to the triage and video search process. Qualitative analysis reinforces this finding, and provides specific suggestions for interface design and visual search strategies that can strengthen accuracy while maintaining a high rate of throughput.