A Hybrid Genetic Algorithm with Tabu Search Using a Layered Process for High-Order QAM in MIMO Detection

In this paper, we propose a hybrid genetic algorithm (HGA) that embeds the tabu search mechanism into the genetic algorithm (GA) for multiple-input multiple-output (MIMO) detection. We modified the selection and crossover operation to maintain the diverse and wide exploration areas, which is an advantage of the GA, and the mutation operation to perform a local search for a specific region. In the mutation process, the ’tabu’ concept is also employed to prevent the repeated search of the same area. In addition, a layered detection process is applied simultaneously with the proposed algorithm, which not only improves the bit error rate performance but also reduces the computational complexity. We apply the layered HGA (LHGA) to the MIMO system with very high modulation order such as 64-quadrature amplitude modulation (QAM), 256-QAM, and 1024-QAM. Simulation results show that the LHGA outperforms conventional detection approaches. Especially, in the 1024-QAM MIMO system, the LHGA has less than 10% of computational complexity but a 6 dB signal-to-noise ratio (SNR) gain compared to the conventional GA-based MIMO detection scheme.