Triple-Channel Time--Frequency Fusion Custom Neural Network With OpenMax for Open-Set Recognition of Communication Signal Modulations

With the continuous advancement of wireless communication technologies and the growing diversity of modulation schemes, Automatic Modulation Recognition (AMR) plays a crucial role in spectrum sensing and signal analysis. However, traditional deep learning models with fixed architectures often face challenges in simultaneously achieving high-precision classification of known modulation types and effectively rejecting unknown modulated signals. To address this, we propose a novel framework combining a custom neural network with OpenMax. First, we introduce a time-frequency diagram channel alongside the I/Q channels to capture both time-domain waveform details and spectral structure, which enhances the model's ability to distinguish between complex modulation schemes and detect unknown signals in low-SNR environments. The network input consists of the time-frequency diagram, I component, and Q component. Moreover, the OpenMax open-set recognition model is incorporated to perform tail probability fitting and re-calibration of the neural network's activation vectors, enabling precise classification of known categories and dynamic rejection of unknown signals. Extensive simulations under challenging conditions—including Rayleigh fading, carrier frequency offset, and AWGN with SNR ranging from ‑10 dB to 20 dB—demonstrate that the proposed method achieves a closed-set classification accuracy of 92.21% and an unknown signal detection rate of 84.38%. Furthermore, computational complexity and runtime analysis show that the proposed framework achieves end-to-end inference latencies of approximately 2.1 ms on GPU and 3.7 ms on CPU, including time-frequency preprocessing, confirming its practical feasibility for real-time deployment. Compared to conventional methods like dual-channel CNN, dual-channel/triple-channel ResNet, and CNN+LSTM, the proposed approach demonstrates quite good performance in both closed-set and open-set tasks, effectively validating the effectiveness of the time-frequency fusion strategy and uncertainty calibration mechanism in improving model robustness and generalization.