ColorPoetry: Multi-Sensory Experience of Color with Poetry in Visual Arts Appreciation of Persons with Visual Impairment

Visually impaired visitors experience many limitations when visiting museum exhibits, such as a lack of cognitive and sensory access to exhibits or replicas. Contemporary art is evolving in the direction of appreciation beyond simply looking at works, and the development of various sensory technologies has had a great influence on culture and art. Thus, opportunities for people with visual impairments to appreciate visual artworks through various senses such as hearing, touch, and smell are expanding. However, it is uncommon to provide a multi-sensory interactive interface for color recognition, such as integrating patterns, sounds, temperature, and scents. This paper attempts to convey a color cognition to the visually impaired, taking advantage of multisensory coding color. In our previous works, musical melodies with different combinations of pitch, timbre, velocity, and tempo were used to distinguish vivid (i.e., saturated), light, and dark colors. However, it was rather difficult to distinguish among warm/cool/light/dark colors with using sound cues only. Therefore, in this paper, we aim to build a multisensory color-coding system with combining sound and poem such that poem leads to represent more color dimensions, such as including warm and cool colors for red, orange, yellow, green, blue, and purple. To do this, we first performed an implicit association test to identify the most suitable poem among the candidate poems to represent colors in artwork by finding the common semantic directivity between the given candidate poem with voice modulation and the artwork in terms of light/dark/warm/color dimensions. Finally, we conducted a system usability test on the proposed color-coding system, confirming that poem will be an effective supplement for distinguishing between vivid, light, and dark colors with different color appearance dimensions, such as warm and cold colors. The user experience score of 15 college students was 75.1%, that was comparable with the color-music coding system that received a user experience rating of 74.1%. with proven usability.