CM-AVAE: Cross-Modal Adversarial Variational Autoencoder for Visual-to-Tactile Data Generation

Vibration acceleration signals allow humans to perceive the surface characteristics of textures during tool-surface interactions. However, acquiring acceleration signals requires a specialized system, which is relatively expensive. Conversely, visual images are more accessible than acceleration signals, and generative models can convert visual images into vibration acceleration signals. Utilizing generative models to generate vibration acceleration signals from visual data circumvents the need for time-consuming actual measurements. Furthermore, this approach can be applied to robot-related tasks. This paper presents a cross-modal adversarial variational autoencoder (CM-AVAE) for visual-to-tactile data generation. Our model incorporates latent space learning from variational autoencoders (VAEs) into generative adversarial networks (GANs) and maps the generator's decoder feature vectors to the discriminator. In addition, a public dataset is chosen to train the model, and relevant evaluation metrics are established to evaluate the model's generated results. The results generated by the CM-AVAE model show significant improvement in objective experiments compared to the baseline models. Furthermore, subjective experimental outcomes also surpass those of the baseline models. Ablation study shows that CM-AVAE introduces latent space learning and maps the decoder feature vectors in the generator to the discriminator, significantly improving the quality of cross-modal data generation.