Surface Material Classification of Robot Perception Using Trimodal Information with Multiloss Ensemble

Surface material classification (SMC) is a critical component of robot perception system for understanding the external environment. However, traditional SMC methods that rely solely on visual information are susceptible to the effects of lighting and distance, and have limitations in classification accuracy and robustness. Drawing inspiration from the human perception system, we propose a Trimodal Fusion Neural Network (TMFNN) based on multiloss ensemble. Our model first extracts visual, auditory, and haptic features using CNN and GRU, and then efficiently fuses them using self-attention mechanism. We also employ an improved multiloss ensemble to enhance model performance. Our classification results on the TUM Dataset reveal that the proposed method significantly outperforms state-of-the-art methods, achieving the highest classification accuracy of 96.56%. Furthermore, we experimentally demonstrate that our proposed multimodal fusion and multiloss ensemble approaches can significantly enhance model accuracy. Overall, our TMFNN is capable of handling more complex SMC tasks and has considerable promise for improving robot perception in a variety of environments.