MAE-NIR: A masked autoencoder that enhances near-infrared spectral data to predict soil properties

Soil available nutrients are crucial for promoting crop growth, and controlling their content is essential for increasing yield, promoting smart agriculture, and protecting the environment. Near-infrared spectroscopy technology enables the efficient and nondestructive detection of soil nutrient content in real time. However, current near-infrared spectroscopy datasets suffer from data isolation due to the inability to share data feature advantages, necessitating expensive data acquisition. To address this issue, we propose an unsupervised method for near-infrared spectral enhancement to analyse soil samples from the red soil of southern Anhui. Our proposed framework, named MAE-NIR, is a near-infrared spectral masked autoencoder that learns highly robust and generic spectral features from abundantly available public near-infrared spectral datasets. We collected near-infrared spectroscopy data from a depth of 900–1700 nm and utilized the publicly available spectral dataset LUCAS 2009 to reconstruct the spectral waveform details. This method facilitates the capture of both local and global aspects, thereby aiding subsequent downstream tasks. Several renowned regressors, such as partial least squares regression, random forest, and neural networks, are also employed to assess the effectiveness of near-infrared spectral enhancement using MAE-NIR. The spectral enhancement method based on the masked autoencoder significantly outperforms all other spectral preprocessing methods. The coefficient of determination (R2) values of the best models of available nitrogen, phosphorus, and potassium in the soil increased to 0.941, 0.926, and 0.903, respectively, which are on average 22.42%, 11.14%, and 10.35% higher than those obtained from the previous best preprocessing methods. This indicates the efficacy of using MAE-NIR to predict soil nutrients, as it effectively enhances the accuracy of nutrient content measurements.