Intelligent AutoEncoder-Based Modulation: Optimizing Transmission Performance over Non-Ideal Wireless Channels

De Tien Lai, Nghia Xuan Pham, Trung Duc Tran

Abstract


This paper presents an End-to-End wireless transceiver architecture based on deep AutoEncoder (AE) networks that jointly optimizes the transmitter and receiver as a single differentiable system, replacing the conventional cascade of independently designed signal processing blocks. The channel model incorporates three concurrent non-ideal impairments: nonlinear distortion from the power amplifier (PA) characterized by the Rapp model, progressive carrier frequency offset (CFO), and flat Rayleigh fading. Through the training process and testing scenarios across three progressively evolving architectures, namely single-symbol constellation shaping, multi-symbol blind CFO compensation, and implicit neural forward error correction (Neural FEC), the obtained results confirm that the AE is capable of autonomously learning PAresilient signal constellations, performing blind CFO estimation without pilot signals, and unifying the modulation and channel coding processes into a single optimal system representation. Monte Carlo BER simulations show that the proposed architecture achieves 3–5 dB SNR gain over conventional 16-QAM with ZF equalization, provides 2–3 dB gain relative to ideally CFO-compensated 16-QAM, and the Neural FEC configuration successfully performs the channel coding function, exhibiting effective error correction performance.

References


K. Hornik, “Approximation capabilities of multilayer feedforward networks,” Neural Networks, vol. 4, no. 2, pp. 251–257, 1991.

J. G. Proakis and M. Salehi, Digital Communications, 5th ed. New York: McGraw-Hill, 2008.

T. O’Shea and J. Hoydis, “An introduction to deep learning for the physical layer,” IEEE Trans. Cogn. Commun. Netw., vol. 3, no. 4, pp. 563–575, Dec. 2017.

S. Dörner, S. Cammerer, J. Hoydis, and S. ten Brink, “Deep learning based communication over the air,” IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 132–143, Feb. 2018.

F. A. Aoudia and J. Hoydis, “Model-free training of end-to-end communication systems,” IEEE J. Sel. Areas Commun., vol. 37, no. 11, pp. 2503–2516, Nov. 2019.

H. Ye, G. Y. Li, and B.-H. Juang, “Power of deep learning for channel estimation and signal detection in OFDM systems,” IEEE Wireless Commun. Lett., vol. 7, no. 1, pp. 114–117, Feb. 2018.

T. Gruber, S. Cammerer, J. Hoydis, and S. ten Brink, “On deep learning-based channel decoding,” in Proc. CISS, 2017, pp. 1–6.

T. O’Shea, K. Karra, and T. C. Clancy, “Learning to communicate: Channel auto-encoders, domain specific regularizers, and attention,” in Proc. IEEE SPAWC, 2016, pp. 1–5.

C. Rapp, “Effects of HPA-nonlinearity on a 4- DPSK/OFDM signal for a digital sound broadcasting system,” in Proc. ESA, 1991, pp. 179–184.

M. Stark, F. A. Aoudia, and J. Hoydis, “Joint learning of geometric and probabilistic constellation shaping,” in Proc. IEEE GC Wkshps, 2019, pp. 1–6.




DOI: http://dx.doi.org/10.21553/rev-jec.457

Copyright (c) 2026 REV Journal on Electronics and Communications

ISSN: 1859-378X

Copyright © 2011-2026
Radio and Electronics Association of Vietnam
All rights reserved