In this study, DNN is the architecture for downscaling modeling. The steps are as follows: select input variables, determine activation functions, optimizers, DNN model architecture, hyper-parameters, and optimize model parameters. The process is shown in Fig. 4. After normalization, the model’s input data are transformed into the input variables needed to build DNN by using the original variables and KPCA. After determining input variables, activation functions, optimizers, DNN architecture, and parameters are optimized in accordance with performance indicators of assessment models. In this study, RMSE is used as a performance assessment indicator, as shown in Eq. (5).
$$\text{R}\text{M}\text{S}\text{E}=\sqrt{\frac{\sum {({Y}_{i}-{X}_{i})}^{2}}{n}}$$
5
Where \({X}_{i}\) is predicted value; \({Y}_{i}\) is observed value; n is number of data items.
4.1 Predictive variable selection
In this study, original variables (ORG) are selected, and KPCA is used for data conversion. Data should be normalized before conversion to ensure that variables have similar upper and lower limits. The normalization equation is as follows:
$$Z=\frac{X-\overline{X}}{{S}_{x}}$$
6
Where Z is normalized value; \(\overline{X}\) is average; \({S}_{x}\) is standard deviation.
After normalizing variables, the model can iterate fast, the time of gradient descent can be reduced, and features between factors are preserved and scaled to improve the model accuracy.
KPCA converts the normalized variables, and the parameters to be optimized in data conversion by KPCA are \({\gamma }\) and n, i.e., the number of kernel principal components. \(\gamma\) used is 0.167 by Debruyne (2010), 0.3, 0.2, 0.1, 0.01, 0.001, and 0.0001 by Deep et al. (2017), and 0.001, 0.007, 0.01, 1 and 10 by Ezukwoke and Zareian (2019). Fischer and Bunke (2009) suggested that \(\gamma\) should be between 0.01 to 100, Jiang et al. (2011) suggested that \(\gamma\) should be between 0 to 100, and Kaneko (2018) suggested that \(\gamma\) should be between \({2}^{-20} to {2}^{9}\). This indicates their different opinions on \(\gamma\). Therefore, Li et al. (2018) used APS to automatically find the optimal solution for γ, automatically optimize parameters, and automatically find γ and the optimal number of principal components.
The core method of APS is kernel-based class separability, which intends to map radial basis functions into high-dimensional space for classification. The more similar the data are in each category, the better; the less similar the data are in different categories, the better. APS performs gradient descent by this classification method and partially differentiates radial basis functions to find the optimal γ automatically. The optimal value of γ is 0.099 and 0.427 for Taichung and Hualien in ACCESS, 0.082 and 0.667 for Taichung and Hualien in CSMK3, respectively.
After finding the optimal γ, it is used to convert original data into different radial basis functions with parameters (1 to 19 in ACCESS, 1 to 12 in CSMK3). K-nearest neighbor classification (KNN) is used to assess simulated and actual values of converted data to get accuracy. Then, kernel principal components with the highest accuracy are selected as the final number. In Taichung and Hualien, the number of kernel principal components with the highest accuracy is 4 and x. Li (2010) verified that the accuracy could effectively show the pros and cons of radial basis functions, and a high accuracy indicates strong effects.
KNN is used for the classification modeling of data and uses models for data classification. The accuracy equation is as follows:
$$\text{A}\text{c}\text{c}\text{u}\text{r}\text{a}\text{c}\text{y}=\frac{\text{A}\text{c}\text{c}\text{u}\text{r}\text{a}\text{t}\text{e}\text{l}\text{y} \text{c}\text{l}\text{a}\text{s}\text{s}\text{i}\text{f}\text{i}\text{e}\text{d} \text{s}\text{a}\text{m}\text{p}\text{l}\text{e}\text{s}}{\text{A}\text{l}\text{l} \text{s}\text{a}\text{m}\text{p}\text{l}\text{e}\text{s}}$$
7
4.2 Model parameter optimization
It is highly time-consuming and difficult to determine DNN model architecture and optimize corresponding parameters. Therefore, both model architecture and parameter optimization must be considered. This study obtains the optimal solutions of DNN parameters according to the stage optimization proposed by Bai et al. (2016, 2017). Bengio (2012) suggested that 10,000 iterations should be the minimum number for training. We have applied 50000 iterations in this study. It is better to have 1 to 2 hidden layers due to optimization Liu (2017). Bai et al. (2016, 2017) said that there should be 1 to 3 hidden layers, and they should be optimized progressively. Busseti (2012) suggested using 2 to 3 hidden layers for optimization. Dalto (2015) used 3 hidden layers and recommended the same number of nodes for each layer. Therefore, this study sets 1 layer in the beginning and increases the number of layers for optimization. Coppola Jr. et al. (2005) suggested that if the input layer has n factors, there are 2n + 1 nodes in the hidden layer. (Bai et al., 2016) said that there is no mature method to determine the number of node layers in theory. There are 5 to 30 nodes in the hidden layer of our model. (Busseti, 2012) used 10 to 50 nodes for training. In this study, the number of nodes to be measured ranges from 10 to 100.
Figure 4 shows the process of the preferred DNN downscaling model, divided into preferred optimizer, learning rate, batch size, and DNN architecture (number of hidden layers and number of neurons in each hidden layer). Parameters to be optimized include optimizers, activation functions, batch size, and the number of node layers. The optimizers to be selected are AdaGrad and Adam. The activation functions are sigmoid and tanh with batch sizes of 16, 32, and 64. The number of nodes ranges from 10 to 100 at an interval of 5. The research process is as follows: first, optimize optimizers, activation functions, and learning rates. Then, optimize the batch size and finally optimize number of layers (NL) and number of nodes per each layer (NNPL).
Optimizers and activation functions are first determined. The initial batch size is 16, the number of layers is 1, and the number of nodes is 10 and 50 to determine its convergence trend. Its learning rate is adjusted to converge. Optimizers and activation functions are selected to achieve convergence. After selecting optimizers and activation functions, learning rates are adjusted to determine overfitting. If there is overfitting, dropout will be performed on DNN architecture. Finally, according to RMSE, AdaGrad is selected as the optimizer, and sigmoid is selected as the activation function. The learning rate is \({10}^{-3}\) and the batch size is optimized.
The batch size is tested based on the selected optimizer, activation function, learning rate, and the setting of 1 DNN layer and 10 nodes. The batch size is changed from 16 to 512. After comparing RMSE, 64 is selected as the optimal batch size to optimize the number of node layers. First, it is assumed that DNN has 10 nodes per layer, with the number of nodes gradually increasing in units of 5 (10, 15, 20…, and 100). For in-order training, the number of nodes in the first layer is 85, which is the number of nodes at the minimum RMSE. After determining the number of nodes in the first layer, a second layer is added to the DNN architecture. The number of nodes in the second layer starts from 10 and gradually increases in units of 5. The number of nodes (35) at the minimum RMSE is taken as the number of nodes in the second layer. If the RMSE of the optimal number of nodes in the second layer is smaller than that of the optimal number of nodes in the first layer, a third layer will be added to DNN based on the above method. Otherwise, the DNN architecture is determined, and parameters are optimized.
The NL and NNPL are finally optimized to obtain optimal parameters. AdaGrad is selected as the optimizer, and sigmoid is selected as the activation function. The learning rate is \({10}^{-4}\) and the batch size is 64. The first layer has 100 nodes, and the second layer has 35 nodes. The final optimized parameters of GCM at all stations are shown in Table 3. Y and X in Eq. (5) are treated by Eq. (6) before calculation, and KPCA is noticeably larger in dimensionless RMSE (DR) of ACCESS and CSMK3 models. Most optimizers are AdaGrad, and most activation functions are sigmoid. Learning rates mostly fall in between \({10}^{-2}\) and \({10}^{-3}\) and the batch sizes are 16, 32, or 64 with usually 1 layer.
Table 3
Table of the optimization parameters in DNN downscaling model
Models | Station | Type | Quantity | Gamma | Optimizer | Excitation | Rate | Batch size | layer1 | layer2 | D-RMSE |
ACCESS | Taichung | original | 19 | | adagrad | sigmoid | 0.0001 | 16 | 80 | | 0.803 |
KPCA | 4 | 0.099 | adagrad | sigmoid | 0.001 | 64 | 100 | 35 | 0.835 |
Hualien | original | 19 | | adagrad | sigmoid | 0.001 | 64 | 55 | | 1.007 |
KPCA | 13 | 0.427 | adagrad | sigmoid | 0.0001 | 16 | 70 | | 1.105 |
CSMK3 | Taichung | original | 12 | | adagrad | sigmoid | 0.001 | 64 | 40 | | 0.845 |
KPCA | 5 | 0.082 | adagrad | sigmoid | 0.001 | 32 | 10 | | 1.593 |
Hualien | original | 12 | | adagrad | tanh | 0.01 | 16 | 90 | | 1.038 |
KPCA | 12 | 0.667 | adam | tanh | 0.001 | 16 | 10 | | 1.191 |
* D-RMSE is dimensionless | | | | | | | | | |
4.3 Assessment of the effectiveness of historical scenarios
After parameters of the above 4.2 DNN architecture are determined, a DNN statistical downscaling model is built based on the historical scenarios of ACCESS and CSMK3 and the historical records of Taichung and Hualien stations. The data period is from 1950 to 2005, and monthly rainfall and historical atmospheric and oceanic factors are used. There are a total of 672 data items/points. The data are ranked according to their number, and the data period is divided at a ratio of 3:1:1; training (1950/01 to 1983/12); verification (1984/01 to 1994/12); testing (1995/01 to 2005/12).
Figure 5 shows the effectiveness of the DNN downscaling model in simulating monthly rainfall in Taichung and Hualien based on historical scenario data tested by ACCESS and CSKM3 (1995/01 to 2005/12). For Taichung, the test data shows that the simulation results are dispersed and vary greatly. The DNN downscaling model based on original variables is more effective than the model built by KPCA in grasping extreme rainfall. The model built by KPCA has a significantly higher underestimation rate than the original variables. Compared with CSMK3, ACCESS has small data dispersion, and its underestimation and overestimation rates are proportionally equal. For Hualien, the simulation results vary slightly. The original variables and KPCA have roughly the same trend in extreme values. CSMK3 has a higher underestimation rate than ACCESS. Overall, for Taichung and Hualien, in both ACCESS and CSMK3, KPCA has a higher dimensionless RMSE than that of original variables, and KPCA has a higher underestimation rate than that of original variables. At Hualien station, in ACCESS, KPCA is good at grasping extreme values. In rest models, original variables are good at grasping extreme values. Regardless of ACCESS, CSMK3, KPCA, or original variables, Hualien outperforms Taichung overall.