Performance of soil total nitrogen monitoring model using optimal preprocessing combinations of in-situ and indoor spectra

doi:10.21203/rs.3.rs-4944863/v1

To explore the performance of in-situ spectral monitoring of soil total nitrogen, this study focused on cotton fields with different soil textures and sampled soil from 0–60 cm depth. Three different preprocessing combinations were applied to the indoor and in-situ spectra, and four modeling methods including the Generalized Regression Neural Network (GRNN), Random Forest Regression (RFR), Support Vector Machine Regression (SVR), and Ordinary Least Squares Regression were optimized using the Northern Goshawk Optimization (NGO) algorithm. The goal was to establish and select the best model for monitoring soil total nitrogen content in each soil layer. The results showed that: (1) Compared with no preprocessing, different preprocessing combinations improved the model accuracy by 0.19–0.49. The optimal preprocessing combination for the surface soil was First Derivative (FD) - Standard Normal Variate (SNV) - Z-score - Savitzky-Golay (SG), and for the medium and deep soil was FD - SNV - Continuum Removal (CR) - SG. (2) The optimized NGO-GRNN model outperformed the GRNN model, with an improvement of 60%, 12%, and 19% in R² for the shallow, medium, and deep soil layers, respectively. (3) The model constructed using indoor spectra performed better than the in-situ spectra in monitoring soil total nitrogen content. However, the in-situ spectra-based models for different soil layers had an R² greater than 0.6, indicating good monitoring performance and eliminating the laborious steps of indoor spectral processing. This study provides theoretical and technical support for rapid acquisition of nutrient information in various soil layers of cotton fields using in-situ spectral monitoring, demonstrating feasibility and robustness.

Earth and environmental sciences/Environmental sciences/Environmental impact

Earth and environmental sciences/Ecology/Agri ecology

soil total nitrogen

in-situ spectra

model

soil total nitrogen monitoring in different soil layers

Fast and accurate assessment of soil nutrient content is a necessary prerequisite for precise fertilizer management and a crucial means to drive the rapid development of smart agriculture [1]. Approximately 50–70% of plant nutrients come from the soil during crop growth and development, with nitrogen being one of the important limiting nutrients for plant growth [2]. Total nitrogen (TN) in the soil reflects the capacity and nitrogen-supplying ability of the soil [3–4]. Therefore, obtaining accurate information on soil TN content is of great significance for crop growth, nitrogen fertilizer management, and environmental protection.

Traditional chemical methods for determining soil TN are time-consuming and labor-intensive, especially for large-scale soil sample analysis. The results also suffer from significant lag and cannot meet the requirements of precision fertilizer application in modern smart agriculture [5–7]. Hyperspectral remote sensing technology has shown great potential in rapid monitoring of soil TN content due to its advantages of being fast, efficient, and non-destructive [8]. Currently, research on rapid monitoring models for soil nutrient assessment based on spectral technology can be divided into two categories: laboratory spectroscopy and in-situ spectroscopy. In laboratory spectroscopy research, Zhang et al. [9] used first-order derivatives and Norris' spectral preprocessing methods to estimate soil nutrient information, achieving an R² of 0.84 and an RMSE of 3.64 using partial least squares regression (PLSR). Liu et al. [10] utilized multiple spectral preprocessing methods and a support vector machine (SVM) model based on selected feature wavelengths, obtaining good results with an R² of 0.816 and an RMSE of 0.20. Bai et al. [11] compared the accuracy of different modeling methods in constructing soil TN inversion models using mid-infrared diffuse reflectance spectroscopy. The backpropagation neural network (BPNN) model combined with principal component analysis (PCA) for dimensionality reduction showed the best monitoring capability, with an R² of 0.78 and an RMSE of 0.12. In comparison with laboratory spectroscopy research, the prediction accuracy of in-situ spectroscopy is not as high, with an R² of only 0.45 reported in one study utilizing PLSR. Similar conclusions were drawn in other studies as well [12]. The relatively low accuracy of in-situ spectroscopy may be attributed to the complex composition, heterogeneous texture, and irregular physical features of the soil, as well as factors such as crop cover, sunlight, and soil moisture [13–16]. However, in-situ spectroscopy measurements offer advantages such as speed, real-time monitoring, non-destructiveness, and lack of pollution. They circumvent the issues associated with laboratory spectroscopy, such as sample collection, transportation, drying, and grinding, simplifying the cumbersome sample preparation process [17]. This is more in line with the requirements of precision agriculture and has garnered increasing attention from researchers.

Therefore, the preprocessing of in-situ spectral data using scientific approaches and the selection of appropriate algorithm models become the focus and challenges of research on in-situ spectroscopic techniques for soil nutrient monitoring [18]. In addition to the aforementioned issues, there is also a challenge of weak spectral penetration, making it difficult to monitor deep soil nutrient levels. To address this problem, some research has utilized generalized regression neural network (GRNN), support vector regression (SVR), and extreme learning machine algorithms to construct soil nutrient models at different depths [19]. These studies have provided a theoretical basis and technical support for monitoring deep soil nutrient levels using in-situ spectroscopy. However, they also point out the difficulties and key issues in monitoring soil nutrient levels using in-situ spectroscopy: (1) selection of spectral preprocessing methods and model screening; (2) weak spectral penetration into the soil; (3) lack of consideration for the impact of soil texture types on spectral data, resulting in a lack of model robustness.

The focus and challenges of current in-situ spectral monitoring of soil nutrients, considering the influence of soil texture on spectral reflectance, are addressed in this study. The study focuses on pre-planting soils in cotton fields in the Kekelik-Kashi area of China. Spectral data and soil total nitrogen (TN) content were collected and measured for different soil texture types at different soil depths, including the surface layer (0–3 cm), shallow tillage layer (3–20 cm), mid-tillage layer (20–40 cm), and deep tillage layer (40–60 cm). Correlations between spectral data and soil TN content were analyzed to establish monitoring models for TN in different soil layers based on indoor and in-situ spectral data. A comparative analysis and applicability analysis of the monitoring capabilities of the two models were conducted, aiming to provide a theoretical basis and scientific support for in-situ spectral monitoring of soil TN. The research hypotheses are as follows: (1) Different pre-processing combinations have varying effects on the monitoring performance of soil TN models in different soil layers, and the optimal pre-processing combination should be selected for each soil layer to improve the monitoring performance; (2) The GRNN modeling method optimized by the NGO algorithm performs best in enhancing the monitoring capability of the models; (3) Indoor spectral monitoring outperforms in-situ spectral monitoring in terms of monitoring performance, but in-situ spectral monitoring simplifies the experimental process and is feasible; (4) With the selection of the optimal modeling method, in-situ spectral monitoring can achieve an overall prediction accuracy of over 0.6, demonstrating certain credibility and robustness.

2.1 Research Area Overview

2.1.1 Research Area

Experimental ground soil samples will be collected in 2022 and 2023 from selected cotton fields. Based on the results of the second national soil survey in 2021 and on-site investigations, one cotton field will be selected from each of the following locations: Shihezi University Teaching and Experimental Field (2A), Yaoxi Village in the 3rd Company of the 148th Regiment (148A), 3 Trees Village in the 11th Company of the 147th Regiment (147A), Huangqu Village in the 3rd Company of the 147th Regiment (147C), Weiguo Village in the 17th Company of the 150th Regiment (150A), and Wajianpai Village in the 25th Company of the 150th Regiment (150C). Each selected cotton field will have a size ranging from 0.25 to 0.5 hectares. These six cotton fields will be the research area (Fig. 1). All selected cotton fields have been continuously cultivated for more than five years. The numbers in parentheses represent the field identification numbers.

2.1.2 Soil Texture Types

The soil texture of Shihezi University Teaching and Experimental Field (2A) and the 3rd Company of the 148th Regiment (150A) is loamy sand, with ranges in the content of sand, silt, and clay particles as follows: 84.8% − 89.2%, 2.7% − 7.1%, and 92.9% − 97.3% respectively. The soil texture of the 17th Company of the 150th Regiment (150A) and the 25th Company of the 150th Regiment (150C) is sandy loam, with ranges in the content of sand, silt, and clay particles as follows: 64.3% − 76.6%, 3.3% − 7.8%, and 96.7% − 92.2% respectively. The soil texture of the 11th Company of the 147th Regiment (147A) and the 3rd Company of the 147th Regiment (147C) is sandy clay loam, with ranges in the content of sand, silt, and clay particles as follows: 62.6% − 73.4%, 23.7% − 30.1%, and 76.3% − 69.9% respectively.

2.2 Data Acquisition and Analysis

2.2.1 Soil Sample Collection and Determination

Soil samples were collected from the six research areas using a grid sampling method (4x5 sampling points) in bare soil conditions. Each sampling area was a rectangular plot measuring 21 m x 28 m (588 m2), with a spacing of 7 m between sampling points (Fig. 2a). Samples were taken before cotton planting and irrigation, avoiding the influence of straw mulch. GPS was used to precisely locate and mark each sampling point in the experimental area.

At each sampling point, the soil core sampling method was employed, and soil columns with a depth of 60 cm were collected. Each soil column was divided into three layers: the shallow layer (0–21 cm), the middle layer (21–42 cm), and the deep layer (42–60 cm) (Fig. 2b). The shallow, middle, and deep layers were further divided into 7, 7, and 6 subdivisions respectively, each subdivision was sampled using a quartering method to obtain approximately 500g of soil. The collected soil samples were mixed evenly, stored in self-sealing bags, and brought back to the lab. After natural drying and removing impurities, the soil samples were passed through a 1 mm sieve by grinding, prepared as test samples, and the TN content of the test samples was measured using the Kjeldahl method.

There were 20 sampling points per field, and 20 soil samples were collected from each sampling point at depths of 0–60 cm (every 3 cm as a layer). A total of 2400 soil samples (20×20×6) were collected from 6 cotton fields in 2022. In 2023, the same experimental design was used for sample collection, and the obtained results were used as a completely new dataset for model validation.

2.2.2 Spatial distribution characteristics of total nitrogen content in the soil surface layer (0–3 cm)

As shown in Table 1, the coefficient of variation (CV) of total nitrogen content in the soil surface layer (0–3 cm) in the study area ranged from 24.1–42.3%, indicating a moderate level of variability. Most of the soil samples had a high level of total nitrogen content, accounting for 53% of the total sampling points in the study area, distributed relatively evenly among the six experimental sites. The next most common levels were high and moderate, accounting for 19% and 26% of the total sampling points, respectively. The high-level samples were concentrated in plot 147C (sandy clay loam), while the moderate-level samples were concentrated in plots 2Lian and 150A (sandy loam and loamy sand). Additionally, homogeneous subareas showed stability and consistency in terms of CV, with sandy loam > loamy sand > sandy clay loam. The CV for loamy sand ranged from 31.56–35.33%, for sandy loam it ranged from 24.11–27.58%, and for sandy clay loam it ranged from 38.45–42.32%.

Table 1

Statistics Table of Total Nitrogen Content at Different Levels in 0 ~ 3 cm Soil Surface Layer
Soil type	Study area	Grade	Rich	Extremely Rich	Medium	Low	CV/%
								SD
Loamy sand	3rd Company, 148th Regiment	Number of sampling points	0	9	13	0	35.3	0.15
		Quantity proportion	0	0.42	0.48	0
	Total field 2	Number of sampling points	0	9	11	0	31.5	0.12
		Quantity proportion	0	0.45	0.55	0
Sandy clay loam	147th Regiment, 11th Company	Number of sampling points	6	11	3	0	24.1	0.23
		Quantity proportion	0.3	0.55	0.15	0
	3rd Company, 147th Regiment	Number of sampling points	11	7	2	0	27.6	0.15
		Quantity proportion	0.55	0.35	0.1	0
Sandy loam soil	150 Regiment, 17 company	Number of sampling points	0	8	10	2	42.3	0.19
		Quantity proportion	0	0.4	0.5	0.1
	150 Regiment, 25 company	Number of sampling points	2	18	0	0	38.5	0.14
		Quantity proportion	0.1	0.9	0	0

2.2.3 Field in-situ spectral data

The in-situ spectral measurements were conducted on clear or lightly cloudy days. Three measurements were taken at each sampling point in each research area, and the average value was calculated, resulting in a total of 60 spectral data points used for modeling. Before collecting the in-situ spectra, the vertical surface of the soil was carefully leveled, and precautions were taken to avoid other fragments such as stones, crop roots, and large soil pores in the soil samples. During whiteboard calibration and spectral measurement, the fiber optic probe was positioned 15 cm above the soil surface and perpendicular to the sample. In addition, the measurement personnel faced the direction of sunlight to avoid shadowing on the probe and the measured soil. The SR-3500 spectrometer was used for spectral data acquisition, with the parameters shown in Table 2.

Table 2

Portable ground object spectrometer SR-3500 parameters
Parameters	Parameter values
Dimension/mm	216×279×89
Spectral range/ nm	350 ~ 2500 nm
Spectral resolution/ nm	2.8 nm@700 nm 8 nm@1500 nm 6 nm@2120 nm
Spectrolmeter type/ nm	3 Diffraction grating
λ Repeatability/ nm	0.1 nm
λ precision/ nm	0.5 nm
Output format	ASD

2.2.4 Soil indoor spectral data acquisition

After air-drying and grinding the soil samples, they were sieved through a 1mm mesh and thoroughly mixed. The soil samples were then divided into two parts using the quartering method: one part for indoor spectral collection and the other part for chemical analysis. The soil samples were placed in aluminum dishes with a diameter of 15 cm and a depth of 2 cm, with the interior coated in black. The dishes were filled with soil samples and leveled on the surface. The SR-3500 spectrometer was used to acquire indoor spectral data. The fiber optic probe was positioned 15 cm above the soil sample, with the fiber optic field of view fully filled with soil sample to ensure vertical alignment with the target, and then the fiber optic probe was fixed. Three different positions were randomly selected for measurement in each soil sample, with 20 measurements taken at each position. The average of the 60 spectra for each soil sample was used as the indoor spectrum. Both the indoor and in-situ spectra were visible and near-infrared spectra (vis-NIR).

2.2.5 Combination of preprocessing methods

The experiment divided different preprocessing methods into four groups: ① No preprocessing: The spectral dataset was not preprocessed. ② Combination 1: The spectral dataset was sequentially preprocessed with SG-FD-SNV-MinMax scaling. ③ Combination 2: The spectral dataset was sequentially preprocessed with SG-FD-SNV-Z-score normalization. ④ Combination 3: The spectral dataset was sequentially preprocessed with SG-FD-SNV-CR normalization.

2.3 Model development and validation

The modeling methods chosen are Partial Least Squares Regression (PLSR) [20], Random Forest (RFR) [21–22], General Regression Neural Network (GRNN) [23], and Support Vector Regression (SVR) [13]. PLSR combines the ideas of multiple regression, principal component analysis, and correlation analysis to effectively eliminate multicollinearity issues among variables, making it one of the most commonly used algorithms in the field of soil spectroscopy. RFR is a supervised learning strategy that can handle datasets with mixed categorical and numerical variables and perform multiclass analysis, avoiding the influence of features with different magnitudes. GRNN possesses excellent nonlinear mapping capability and learning speed, and it can determine a maximum likelihood estimate that is closest to the actual value by adjusting the weights of different inputs. The basic principle of SVR is to reduce structural risk of the system and solve quadratic programming problems. Compared to other models, it is more suitable for small sample regression monitoring and has advantages in solving small sample, nonlinear, and high-dimensional pattern recognition problems. Additionally, the Northern Goshawk Optimization (NGO) algorithm is used to optimize and compare the above modeling methods. NGO is an optimization algorithm proposed in 2022 by Mohammad et al. [24], which simulates the behavior of the Northern Goshawk during hunting, including prey recognition and attack, pursuit, and escape behaviors.

The developed models are validated using a new dataset collected in 2023 (referred to as the new dataset). The performance evaluation metrics used are the determination coefficient (R²) and root mean squared error (RMSE) for model evaluation and comparison. A higher R² value indicates a better fit to the data, while a lower RMSE value indicates higher model accuracy [25–26]. The formulas are described as follows:

In the formulas, represents the measured total nitrogen content of cotton field soil sample i, represents the monitored total nitrogen content of cotton field soil sample i, represents the average total nitrogen content of cotton field soil samples, n is the number of cotton field soil samples, and SD represents the standard deviation between the monitored total nitrogen content and the measured total nitrogen content.

3.1 Soil Total Nitrogen Prediction Based on Laboratory Spectra

3.1.1 Correlation between Total Nitrogen Content and Laboratory Spectra

The laboratory spectra of the cotton field soil samples and their correlation with corresponding nitrogen content are shown in Fig. 3. From Fig. 3 (left), it can be observed that higher nitrogen content corresponds to lower reflectance in the laboratory spectra, especially after the 950 nm wavelength. Clear absorption valleys exist in the laboratory spectra near 450 nm, 1350 nm, 1900 nm, and 2250 nm. According to Fig. 3 (right), the laboratory spectra exhibit a negative correlation with the corresponding nitrogen content overall. The absolute values of correlation coefficients around 1550 nm to 1900 nm, 2000 nm to 2200 nm, and 2300 nm are all higher than 0.5.

3.1.2 Analysis of Laboratory Spectra Preprocessing Features

Figure 4 (a-h) shows the spectra obtained by applying eight preprocessing methods to the same dataset. By directly observing the characteristics of the spectral signals, the following results can be concluded: the first derivative and second derivative preprocessing methods effectively eliminate baseline and background interference in the spectra (c, d); SNV and MSC preprocessing methods significantly improve the fitting of the spectra, with the sensitivity wavelength position b performing better than e for soil total nitrogen content prediction, and they reduce the influence of scattering on the original spectra (b, e); the SG convolution smoothing method makes the spectral curves smoother and reduces noise interference in the original spectra (a); scaling, normalization, and standardization preprocessing methods confine the soil laboratory spectra within a specific range, removing the influence of size differences on monitoring results (f, g, h). Therefore, the optimal preprocessing methods selected for the laboratory spectra of the soil dataset are SG convolution smoothing, first derivative, and multiple scattering correction. Scaling preprocessing cannot intuitively distinguish the differences among the methods.

3.1.4 Model Evaluation Based on Laboratory Spectra

In the training set, the best models established using the selected optimal preprocessing combinations are PLSR (combination 2), RFR (combination 2), GRNN (combination 3), and SVR (combination 4) models (Fig. 5). The R² values of the four monitoring models in the training set are all greater than 0.75. Among them, the generalized neural regression network (GNRR) and partial least squares regression (PLSR) models have the highest sensitivity for cotton field total nitrogen monitoring, with R² values of 0.89 and RMSE values of 0.07 and 0.06, respectively. The random forest regression (RFR) model performs second best, with an R² of 0.88 and an RMSE of 0.07. The support vector regression (SVR) model has the lowest accuracy, with an R² of 0.81 and an RMSE of 0.11.

In the validation set, the R² values of the four models are all greater than 0.60. The random forest regression (RFR) model has the highest accuracy, with R² and RMSE values of 0.87 and 0.07, respectively, which is a decrease of 0.01 in accuracy compared to the training set. The generalized neural regression network (GNRR) model performs second best, with an R² of 0.84, an improvement of 0.05 compared to the training set. The support vector regression (SVR) model has the lowest accuracy for cotton field total nitrogen monitoring in the validation set, with an R² of 0.60 and an RMSE of 0.15, indicating a decrease in accuracy by 0.23. However, all four models have an R² value greater than 0.60, indicating good monitoring performance.

3.2 Soil Total Nitrogen Prediction Based on Field Spectra

3.2.1 Comparison and Analysis of Field Spectra and Laboratory Spectra

The spectra mainly respond to the strength of N-H stretching vibration in total nitrogen. The absorption valleys near 1350 nm are related to the polarization of N-H bonds in the soil. The absorption valleys near 1900 nm are associated with the stretching of N-H bonds in the soil and the combination frequency peaks with O-H bonds. The results of this study also indicate that the absorption valley positions in the laboratory spectra and field spectra of the soil surface are similar, occurring around 1350 nm, 1900 nm, 2050 nm, and 2150 nm. However, the field spectra have wider and deeper absorption valleys compared to the laboratory spectra (Fig. 6).

3.2.2 Model Evaluation Based on Field Spectra

Table 3 shows that different modeling methods have varying accuracy in predicting cotton field total nitrogen for the same soil layer. The random forest regression (RFR) model performs well in handling sample outliers and noise tolerance, and it is less prone to overfitting during modeling. The generalized neural regression network (GNRR) model can self-learn and find maximum likelihood estimates. In terms of the entire soil vertical scale, the accuracy of the RFR, PLSR, GRNN, and SVR models based on the modeling dataset follows the order of GNRR > RFR > SVR > PLSR. In terms of different soil layers, the accuracy of total nitrogen monitoring in the cotton field shows a pattern of shallow layer > middle layer > deep layer.

When considering individual soil layers, for the shallow layer, the four models perform in the order of GRNN > RFR > SVR > PLSR, with R² values of 0.72, 0.67, 0.46, and 0.52, respectively. GRNN and RFR outperform other models in monitoring soil surface total nitrogen. For the middle layer, the accuracy of the four regression models is in the order of GRNN > RFR > SVR > PLSR, with R² values of 0.68, 0.65, 0.63, and 0.02, respectively. PLSR shows poor monitoring accuracy. For the deep layer, the accuracy of the four regression models is in the order of GRNN > SVR > RFR > PLSR, with R² values of 0.63, 0.37, 0.32, and 0.04, respectively. PLSR performs poorly, while GRNN performs the best with an R² of 0.63 and RMSE of 0.10. In conclusion, different models have different monitoring capabilities in different soil layers, but the GRNN model consistently exhibits an R² greater than 0.6 in all soil layers, demonstrating stable monitoring ability and being the optimal model for monitoring soil total nitrogen in different layers. On the other hand, the PLSR model performs the worst, showing only limited monitoring ability in the shallow layer, and even having an R² less than 0.1 in the middle and deep layers, making it unsuitable for monitoring total nitrogen in different soil layers.

Table 3

Regression Model of Nitrogen Content and Sensitive Bands in Cotton Field (n = 75)
Topsoil	PLSR		RFR		GRNN		SVR
Topsoil	R²	RMSE	R²	RMSE	R²	RMSE	R²	RMSE
Shallow soil layer	0.46	0.36	0.67	0.03	0.72	0.04	0.52	0.14
Middle soil layer	0.02	1.67	0.65	0.03	0.68	0.02	0.63	0.12
Deep soil layer	0.04	1.53	0.37	0.27	0.63	0.1	0.32	2.27

3.2.3 Optimization of Direct Monitoring Models Based on NGO Algorithm

As shown in Table 4, the NGO-GRNN model optimized by the NGO algorithm outperforms the NGO-RFR and NGO-SVR models, with NGO-GRNN > NGO-RFR > NGO-SVR in terms of R² performance across different soil layers. The R² values of the NGO-GRNN model for the shallow, medium, and deep soil layers are 0.84, 0.79, and 0.82, respectively, with all RMSE values below 0.1. Compared to the best-performing GRNN monitoring model, the optimized NGO-GRNN model achieves better monitoring accuracy, with R² improvements of 60%, 12%, and 19% for the shallow, medium, and deep soil layers, respectively.

Table 4

Regression Model of Cotton Field Nitrogen Content and Sensitive Bands Based on NGO Algorithm Optimization (n = 75)
Topsoil	NGO-RFR		NGO- GRNN		NGO-SVR
Topsoil	R²	RMSE	R²	RMSE	R²	RMSE
Shallow soil layer	0.66	0.08	0.84	0.03	0.65	0.06
Middle soil layer	0.57	0.14	0.79	0.11	0.52	0.1
Deep soil layer	0.59	0.13	0.82	0.08	0.07	0.18

3.2.4 Validation of Direct Monitoring Models based on a New Dataset

The results of the NGO-RFR, NGO-GRNN, and NGO-SVR models based on the validation set are shown in Fig. 7. Among the three models for monitoring total nitrogen content in different soil layers, the NGO-GRNN regression algorithm achieves the highest accuracy and best performance, with R² values of 0.75, 0.62, and 0.66, and RMSE values of 0.04, 0.08, and 0.12, respectively. For the shallow and medium soil layers, the NGO-RFR regression model achieves R² values of 0.74 and 0.66, and RMSE values of 0.02 and 0.04, respectively. The NGO-SVR regression model performs the worst and fails to achieve quantitative monitoring of total nitrogen content. In the deep soil layer, the NGO-GRNN regression model exhibits the best performance, with R² and RMSE values of 0.77 and 0.12, respectively, reaching the level of quantitative monitoring for total nitrogen content. Among the three models, the NGO-SVR regression model based on the NGO algorithm performs the worst in validating the monitoring of total nitrogen content in the deep layer, with an R² value of 0.04 and an RMSE value of 1.07. Its monitoring performance is significantly lower than the other two models and cannot be used for monitoring deep layer total nitrogen content.

4.1 Modeling using laboratory spectroscopy

A significant amount of research has been conducted on the monitoring of soil total nitrogen content using laboratory spectroscopy, which has received wide recognition and application [27–28]. However, due to the complex composition of soil, heterogeneous texture, irregular physical characteristics, and the influence of surface crop cover, rapid monitoring of soil nutrient information through field spectroscopy poses significant challenges [13–16]. Currently, researchers have mainly focused on studying soil spectral feature band selection, model algorithm selection, and improvements in spectroscopic instrumentation to overcome these challenges.

Regarding soil spectral feature band selection, the optimal methods may differ due to factors such as soil texture and vegetation. However, it is widely recognized that combining preprocessing methods can improve the accuracy of feature spectral selection [29–32]. This is consistent with the findings of this study, where the use of combined preprocessing methods improved model prediction accuracy by 19–49% compared to the unprocessed data. However, the study also found that the impact of different combination methods on the accuracy improvement of the same model or different models can vary, and sometimes it can even lead to a decrease in model accuracy. For example, combination 2 had the best effect on improving the accuracy of the Random Forest Regression model (R² = 0.858), but it had the lowest effect on the Support Vector Regression model (R² = 0.501). Therefore, cautious selection of appropriate combinations based on the adaptability of the models is necessary to improve the accuracy of soil total nitrogen models. In terms of model selection, it is generally believed that Random Forest Regression (RFR), General Regression Neural Network (GRNN), and Support Vector Machine (SVM) have good monitoring performance for soil nutrient inversion [33]. By comparing the accuracy (R²) of models established based on GRNN, RFR, SVR, and PLSR, this study found that RFR > GRNN > SVR > PLSR, with RFR achieving the best performance (R² = 0.88), and PLSR achieving the poorest performance (R² < 0.5). Few studies have reported on instrument optimization, primarily focusing on improvements in the spectroscopic instrument probe. For example, Waiser et al. installed a contact reflectance probe on a spectrometer, and the predictive accuracy of the measured spectra approached that of laboratory-measured spectra [17, 34].

4.2 Modeling using in situ spectroscopy

Collecting field soil spectral information faces various challenges. Soil conditions exhibit heterogeneity in time, space, and depth, including surface conditions, soil moisture content, and variations in the total nitrogen content at different depths. These factors pose challenges to accurately collecting in situ soil spectral measurements in the field, making it difficult to extract effective information about soil properties and reducing the accuracy of using in situ spectroscopy to monitor soil total nitrogen content at different depths [35–37]. However, studies have shown that the predictive ability of in situ spectroscopy can be enhanced through spectral preprocessing and model algorithm selection [38–39]. This is consistent with the findings of this study, where different models showed varying levels of accuracy (R²) in monitoring soil TN at different depths. For example, in the shallow layer, the model accuracy order was GRNN (0.72) > RFR (0.67) > SVR (0.46) > PLSR (0.52), while in the middle layer, the order was GRNN (0.63) > SVR (0.37) > RFR (0.32) > PLSR (0.04). However, the GRNN model performed best in terms of monitoring accuracy at all depths, while the PLSR model performed poorly at all depths (with R² values below 0.6). This suggests that the use of neural networks (GRNN) may outperform traditional machine learning models (RFR, SVR, PLSR) in monitoring soil TN accuracy at different depths.

To further select the best monitoring model and improve model accuracy, the PLSR model, which performed the worst, was removed, and the GRNN, RFR, and SVM monitoring models were optimized using the NGO algorithm. Some of the optimized models showed improved accuracy, with NGO-GRNN performing the best. For shallow, middle, and deep layers, the accuracy (R²) of NGO-GRNN improved by 16.7–30.2% compared to GRNN, and all these R2 values exceeded 0.7. However, compared to the best-performing RFR model (R² = 0.88) established using laboratory spectroscopy, the predictive performance of the NGO-GRNN model using in situ spectroscopy still showed some gaps. A study conducted in 2023 confirmed the findings of this study, where the predictive performance of soil nutrients using visible near-infrared spectroscopy (vis-NIR) and mid-infrared spectroscopy (MIR) demonstrated that laboratory MIR > laboratory vis-NIR > in situ vis-NIR > in situ MIR [40]. The higher prediction accuracy of outdoor spectroscopy models compared to field spectroscopy models is attributed to the fact that laboratory soil samples undergo a series of processes such as air-drying and grinding, which reduces the influence of soil moisture on the spectra.

4.3 Feasibility of soil total nitrogen monitoring using in situ spectroscopy models

The study has identified the best in situ spectroscopy model based on the NGO-GRNN algorithm. Although this model still has lower prediction accuracy compared to indoor spectroscopy models, it enables rapid and non-destructive monitoring of soil total nitrogen content across various soil types. When the R² value of a field spectroscopy model for soil TN monitoring exceeds 0.6, it can be considered effective [40]. When verifying the NGO-GRNN model with a new dataset, the average R² value was greater than 0.6, indicating its relevance for soil TN monitoring at different soil depths.

Furthermore, the use of in situ spectroscopy for soil TN monitoring only requires direct measurement of in situ spectra using portable spectrometers, eliminating the need for soil sampling, packaging, transportation, drying, grinding, and sieving processes. This significantly reduces time and labor costs. The collection of in situ spectroscopy data is of great significance for the wider application of soil remote sensing. However, the model is limited to the relatively homogeneous soil texture types in the sampling area of Xinjiang. It is applicable only to sandy loam soil, loamy sand soil, and sandy clay loam soil, while there are at least 10 different soil texture types according to international and domestic standards. Further research is needed to determine if the model is equally feasible and robust for other soil texture types.

This study has identified the optimal spectral preprocessing combinations for different soil layers and improved the prediction accuracy of soil total nitrogen monitoring models using the NGO algorithm. It has also compared the modeling performance of in-situ and laboratory spectra. The results show that: (1) the optimal spectral preprocessing combinations for soil spectra are as follows: surface layer: FD-SNV-Z-score-SG; middle and shallow layers: FD-SNV-CR-SG; (2) NGO can optimize the predictive ability of in-situ spectral models; (3) although the laboratory spectra outperform the in-situ spectra in terms of predictive performance, the in-situ spectra still achieve trustworthy prediction accuracy and save steps in soil total nitrogen monitoring. This study provides a theoretical basis and technical support for rapidly obtaining soil nutrient information before cotton planting in the reclamation area of Shihezi, Xinjiang. It lays a foundation for the development of precision irrigation and fertilization as well as precision agriculture. Further efforts are needed in future research, particularly in improving model robustness and removing noise from in-situ spectra.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data avaliable

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Funding

This project did not receive any funding from external commercial or non-commercial institutions.

Author Contribution

Author Contributions StatementY.D. conducted the experiments, analyzed the data, and wrote the main manuscript text. Y.K.and G.L. contributed to the experimental design and data interpretation. X.L. supervised the project, provided critical feedback on the manuscript, and acted as the corresponding author. All authors reviewed and approved the final manuscript.

Lal, R. Soil carbon sequestration impacts on global climate change and food security[J]. Science, 2004, 304(5677): 1623-1627. https://doi.org/10.1126/science.109736
Belyaev V I , Sokolova L V .The influence of the preceding crops on the soil moisture content under the spring soft wheat fields[J].IOP Conference Series: Earth and Environmental Science, 2021, 624(1):012218 (5pp). https://doi.org/10.1088/1755-1315/624/1/012218.
Iznaga, A. C., Orozco, M. R., Alcantara, E. A., et al. Vis/NIR spectroscopic measurement of selected soil fertility parameters of Cuban agricultural Cambisols[J]. Biosystems engineering, 2014, 125: 105-121. https://xueshu.baidu.com/usercenter/paper/show?paperid=15e871bd582b33c01a13624c1470e760&site=xueshu_se
Luo, X. N, Zhu, Y. J., Zhang, H. Y., et al. Influence of planting system on soil nutrient characteristics in cotton fields in the Tarim Oasis[J]. Agricultural Research in Arid Areas, 2012, 30(03): 114-118+129.
Guillaume, D., Jacek, N., Pecio, A., et al. Effect of the number of calibration samples on the prediction of several soil properties at the farm-scale[J]. Geoderma, 2014, 214-215: 114-125. https://doi.org/10.1016/j.geoderma.2013.09.022
Nocita, M., Stevens, A., Toth, G., et al. Prediction of soil organic carbon content by diffuse reflectance spectroscopy using a local partial least square regression approach[J]. Soil Biology and Biochemistry, 2014, 68: 337-347. https://doi.org/10.1016/j.soilbio.2013.10.022
Wang, J. M., Wang, J. P., Hu, Y. M., et al. Study on spatial variation characteristics of soil nutrient in chestnut calcareous soil farmland and sampling method[J]. Agricultural Research in Arid Areas, 2006(05): 59-63.
Cheng, C. W., Laird, D. A., Mausbach, M. J., et al. Near‐infrared reflectance spectroscopy–principal components regression analyses of soil properties[J]. Soil Science Society of America Journal, 2001, 65(2): 480-490. https://doi.org/10.2136/sssaj2001.652480x
Zhang, J. J., Tian, Y. C., Yao, X., et al. Estimation of soil total nitrogen content based on hyperspectral data[J]. Journal of Natural Resources, 2011, 26(05): 881-890.
Liu, Z. M., Li, X. Q., Chen, G. L., et al. Inversion of soil total nitrogen content in Yunnan rubber plantation based on hyperspectral data[J]. Tropical Agricultural Science and Technology, 2023, 46(01): 35-41
Bai, Z. J., Peng, J., Luo, D. F., et al. Inversion model of soil total nitrogen content in farmland in southern Xinjiang based on mid-infrared spectroscopy[J]. Spectroscopy and Spectral Analysis, 2022, 42(09): 2768-2773
Ji, W. J.. Study on prediction of soil properties based on field vis-NIR hyperspectral data and removal of field moisture influence[D]. Zhejiang University, 2014.
Shi, Z., Guo, Y., Jin, X., et al. Research progress of soil near-ground sensors[J]. Acta Pedologica Sinica, 2011, 48(06): 1274-1281.
Yang, D. H., Li Y. Q., Dao, J., et al. Estimation of major nutrient content in soil based on multi-spectral remote sensing data from unmanned aerial vehicles and ground-based hyperspectral remote sensing[J]. Jiangsu Agricultural Sciences, 2022, 50(02): 178-186
Faber, D. J., Post, A. L., Henricus, J. C. M., et al. Analytical model for diffuse reflectance in single fiber reflectance spectroscopy[J]. Optics Letters, 2020, 45(7): 2078-2081. https://doi.org/ 10.1364/ol.385845
Mutanga, O., Skidmore, A. K., Wieren, S. V. Discriminating tropical grass (Cenchrus ciliaris) canopies grown under different nitrogen treatments using spectroradiometry[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2003, 57(4): 263-272. https://doi.org/ 10.1016/s0924-2716(02)00158-2
Li, C. L. Prediction of soil organic carbon based on field visible and near-infrared spectroscopy and removal of moisture influence[D]. Huazhong Normal University, 2021. (in Chinese)
Qi, H. J., Karnieli, A., Li S. W. Prediction of soil available nitrogen in the field based on Y-gradient generalized least squares weighted correction[J]. Spectroscopy and Spectral Analysis, 2018, 38(01): 171-175. (in Chinese)
Feng, W. Z., Wang, X. T., Han, J., et al. Study on monitoring soil salinization based on scale transformation of satellite and unmanned aerial vehicle remote sensing data[J]. Water-saving Irrigation, 2020, 303(11): 87-93, 104 (in Chinese)
Jiang, G. X. Application of partial least squares regression method (PLS) in short-term climate prediction[D]. Nanjing University of Information Science & Technology, 2007. (in Chinese)
Yang, F. R., Luo, S., Li, Q. Research on Extreme Random Forest Algorithm Based on Feature Selection [J]. Application Research of Computers, 2020, 37(09): 2625-2628+2633. DOI:10.19734/j.issn.1001-3695.2019.04.0122. (in Chinese)
Yuan, X., Mei, X., Huang, J., et al. Feature Selection Based on Random Forest Algorithm and Its Application in fMRI Data [J]. Microelectronics & Computer, 2014, 31(08): 132-135.DOI:10.19304/j.cnki.issn1000-7180.2014.08.029. (in Chinese)
Luo, L., Liu, Z. Finite-time robust stability analysis of uncertain stochastic delay reaction diffusion generalized cell neural networks [J]. Journal of Sichuan Normal University (Natural Science Edition), 2014, 37(02): 208-215. (in Chinese)
Fu, X., Zhu, L. K., Huang J. P., et al. Multi-threshold image segmentation based on improved Northern Goshawk algorithm[J]. Computer Engineering, 2022. (in Chinese)
Xu, X. G. Evaluation method of crop yield estimation accuracy in remote sensing estimation[D]. Graduate University of Chinese Academy of Sciences (Institute of Remote Sensing Applications), 2004 (in Chinese)
Yang, F., Sun, J. L., Zhang, B., et al. Accuracy evaluation of MODIS LAI based on PROSAIL model and TM and measured data[J]. Transactions of the Chinese Society of Agricultural Engineering, 2010, 26(04): 192-197+385. (in Chinese)
Stevens, A.,Wesemael B. V., Bartholomeus, H., et al. Laboratory, field and airborne spectroscopy for monitoring organic carbon content in agricultural soils[J], 2007.https://doi.org/10.1016/j.geoderma.2007.12.009
Kooistra, L., Wanders, J., Epema, G. F., et al. The potential of field spectroscopy for the assessment of sediment properties in river floodplains[J]. Analytica Chimica Acta, 2003, 484(2):189-200. https://doi.org/ 10.1016/s0003-2670(03)00331-3
Yang, S., Wang, C., Yang, W., et al. The influence of preprocessing on the estimation of soil organic matter spectra[J]. Shanxi Agricultural Sciences, 2020, 48(10): 1637-1640, 1645. (in Chinese)
Xu, L., Chen, Y., Hong, Y., et al. Hyperspectral estimation model of soil organic carbon considering roughness [J]. Spectroscopy and Spectral Analysis, 2022, 42(09): 2788-2794. (in Chinese)
Sheng, M., Wang, K., Wang, Y., et al. Rapid discrimination of tea garden soil acidity and alkalinity based on near-infrared spectroscopy[J]. Journal of Anhui Agricultural University, 2018, 45(04): 580-587. (in Chinese)
Peng, Y., Xie, H., Li, J., et al. Research on rapid analysis methods of organic carbon and nitrogen in black soil based on near-infrared spectroscopy [C] // Towards the future of soil science (Volume 1) - Proceedings of the 12th National Congress of the Chinese Society of Soil Science and the 9th Cross-Strait Soil Fertilizer Academic Conference, 2012. (in Chinese)
Song, X., Zhang, M., Zhou, H., et al. Estimation of soil total nitrogen content in the Taihu Lake region based on optimized soil spectral parameters[J]. Journal of Agricultural Resources and Environment, 2020, 37(01): 43-50. (in Chinese)
Waiser, T. H., Morgan, C. L. S., Brown, D. J., et al. In situ characterization of soil clay content with visible near‐infrared diffuse reflectance spectroscopy[J]. Soil Science Society of America Journal, 2007, 71(2): 389-396. https://doi.org/10.2136/sssaj2006.0211
Babaeian, E., Tuller, M.. The Feasibility of Remotely Sensed Near-Infrared Reflectance for Soil Moisture Estimation for Agricultural Water Management[J]. Remote Sensing, 2023. https://doi.org/10.3390/rs15112736
Mutanga, O., Skidmore A. K., Kumar L., et al. Estimating tropical pasture quality at canopy level using band depth analysis with continuum removal in the visible domain[J]. International Journal of Remote Sensing, 2005. https://doi.org/10.1080/01431160512331326738
Munawar, A. A., Zulfahrizal, Z., Hayati, R., et al. Agricultural products quality determination by means of near infrared spectroscopy[J]. Iop Conference Series: Earth and Environmental Science, 2022. https://doi.org/10.1088/1755-1315/951/1/012112
Krishan, G., Bhagwat, A., Sejwal, P., et al. Assessment of groundwater salinity using principal component analysis (PCA): a case study from Mewat (Nuh), Haryana, India[J]. Environmental Monitoring and Assessment, 2022. https://doi.org/10.1007/s10661-022-10555-1
Gholizadeh, A., Borůvka, L., Saberioon, M. M., et al. Comparing different data preprocessing methods for monitoring soil heavy metals based on soil spectral features[J]. Soil and Water Research, 2016. https://doi.org/10.17221/113/2015-swr
Yin, J. X., Shi Z., Li B. G., et al. Prediction of Soil Properties in a Field in Typical Black Soil Areas Using in situ MIR Spectra and Its Comparison with vis-NIR Spectra[J]. Remote Sensing, 2023, 15(8).https://doi.org/10.3390/rs15082053

No competing interests reported.