This study aimed to comparatively evaluate the performance and the number of trainable parameters of Convolutional Neural Network (CNN), Dense, Gated Recurrent Unit (GRU), and Long Short-Term Memory (LSTM) networks on benchmark datasets. To accomplish this objective, we devised three different network architectures for each benchmark dataset. These architectures comprised of 3 hidden layers with either 16 or 512 neurons per layer and 10 hidden layers with 256 neurons per layer. The 3-layer models represented shallow neural network architectures, while the 10-layer models represented moderately deep networks. Our aim was to investigate the network efficiency with the smallest number of parameters and with a moderate number of neurons in each layer. The choice of either 16 or 512 neurons per layer in the 3-layer models and 256 neurons per layer in the 10-layer models was deliberate to observe the effect of varying the number of neurons in each layer on the network performance. In order to maintain uniformity in the comparison of different models, we fixed the batch size to 128 for models with 3 hidden layers and 256 for models with 10 hidden layers. After each hidden layer, a 50% dropout rate was applied to reduce overfitting. The input format for Conv2D and Dense networks was (batch_size, width, height, number_of_channels). The structure of each network included the specified number of hidden layers with a pre-determined number of neurons, followed by a flattening operation and an output layer, which was a Dense layer with the same number of neurons as the number of classes in each model. The softmax activation function was used as the classifier. The Conv2D models employed a kernel size of (3, 3) with a stride of (1, 1), while the initializers for the kernel and bias were set to 'glorot_uniform' and 'zeros,' respectively. The recurrent initializer for GRU and LSTM networks was set to 'orthogonal.' The RMSprop optimizer was utilized with a learning rate of 0.0001 and a decay rate of 0.00001. The loss was calculated using the 'categorical_crossentropy' loss function, and accuracy was evaluated using the 'accuracy' metric. The models were trained using 90% of the training data, with 10% reserved for validation. Each model was trained for 30 epochs as part of this internal comparative study.
A) Fashion-MNIST
The comparative analysis of the performance of various models on the Fashion-MNIST dataset is depicted in Table 2. In the initial experiment, where three hidden layers were employed, each with 16 neurons, the Dense model emerged as the best performer with a test accuracy of 84.08% and a test loss of 0.45. The second-best performer was the Convolutional Neural Network (CNN) model, which achieved 83.51% accuracy and a loss of 0.46. However, it should be noted that the number of parameters in the CNN model was significantly lower than that in the Dense model, being approximately 82K as compared to 126K.
In the subsequent experiment, where the number of hidden units was increased to 512, the Gated Recurrent Unit (GRU) model emerged as the clear winner with the highest accuracy of 89.29% and the lowest loss of 0.31. The second-best performer was the Long Short-Term Memory (LSTM) model, which achieved 88.20% accuracy and a loss of 0.32. Meanwhile, the accuracy and loss of the Dense and CNN models were 84.26% and 0.44, and 83.81% and 0.47, respectively. In terms of the number of parameters, the GRU model was the best performer with the lowest number, being nearly 4.1M, as compared to 4.5M for the Dense model, 5.5M for the LSTM model, and 7.1M for the CNN model.
Finally, in the experiment where the number of hidden layers was increased to ten, each layer containing 256 neurons, the GRU model once again emerged as the best performer with a test accuracy of 86.54% and a test loss of 0.39. The Dense and LSTM models had the same loss value of 0.45, but the accuracy of the Dense model was higher at 84% compared to 83.10% for the LSTM model. The accuracy and loss value of the CNN model were 82.68% and 0.60, respectively. In terms of the number of parameters, the Dense model had the lowest number (nearly 2.6M) followed by the GRU model (3.8M). The number of parameters for the LSTM and CNN models were 5.1M and 5.5M, respectively.
In conclusion, our experiments revealed that as the number of hidden neurons increased, the GRU model performed better than other models. Furthermore, increasing the number of neurons instead of the number of layers led to improved performance for all models.
Table 2: Performance comparison among CNN, Dense, LSTM, and GRU for Fashion-MNIST dataset with 50% dropout over 30 epochs. The three most notable performance metrics are emphasized in bold: highest test accuracy, lowest test loss, and the fewest number of parameters used.
Methodology
|
# hidden layers
|
# neurons per layer
|
Batch size
|
Test Acc (%)
|
Test Loss
|
# trainable params
|
CNN
|
3
|
512
|
128
|
83.81
|
0.47
|
7,202,826
|
Dense
|
84.26
|
0.44
|
4,540,426
|
LSTM
|
88.20
|
0.32
|
5,449,738
|
GRU
|
89.29
|
0.31
|
4,127,754
|
CNN
|
3
|
16
|
128
|
83.51
|
0.46
|
82,250
|
Dense
|
84.08
|
0.45
|
126,026
|
LSTM
|
80.97
|
0.52
|
11,594
|
GRU
|
81.72
|
0.50
|
9,962
|
CNN
|
10
|
256
|
256
|
82.68
|
0.60
|
5,477,130
|
Dense
|
83.99
|
0.45
|
2,599,690
|
LSTM
|
83.10
|
0.45
|
5,091,338
|
GRU
|
86.54
|
0.39
|
3,844,106
|
Comparison: The comparative analysis of the Fashion-MNIST dataset indicated that the Dense model exhibited the highest performance in terms of accuracy (84.08%) and loss (0.45) for the smallest implementation scenario with three hidden layers and sixteen neurons per layer. The CNN model, on the other hand, had a slightly lower accuracy (83.51%) and loss (0.46) but had a significantly higher number of parameters (82K) compared to the Dense model (126K).
In comparison, the GRU and LSTM models demonstrated lower accuracy (81.72% and 80.97%, respectively) and higher loss (0.50 and 0.52, respectively), but with a much lower number of parameters (less than 10K and 11.5K, respectively). Despite this trade-off of performance for fewer parameters, it was considered justifiable as the recurrent models performed close to the best performance with only a small fraction of the parameters.
Upon increasing the number of hidden units to 512 in each of the three hidden layers, the GRU model emerged as the best performer with an accuracy of 89.29% and a loss of 0.31. The LSTM model followed closely with 88.20% accuracy and a loss of 0.32, while the Dense and CNN models had lower accuracy (84.26% and 83.81%, respectively) and higher loss (0.44 and 0.47, respectively) along with higher numbers of parameters (4.5M and 7.1M, respectively). The GRU model also had the lowest number of parameters (4.1M) and was considered the best overall model.
Finally, when the number of hidden layers was increased to ten, each layer with 256 neurons, the GRU model again exhibited the best performance with 86.54% accuracy and 0.39 loss. The Dense and LSTM models had the same loss (0.45), but the Dense model had a higher accuracy (84%) compared to LSTM (83.10%). The CNN model had the lowest accuracy (82.68%) and highest loss (0.60), along with the highest number of parameters (5.5M). The Dense model had the lowest number of parameters (2.6M), followed by GRU (3.8M), LSTM (5.1M), and CNN (5.5M).
In conclusion, the results of the analysis showed that increasing the number of hidden neurons improved the performance of all models, with GRU demonstrating the highest performance. Furthermore, increasing the number of neurons instead of the number of layers helped all models attain their best performance.
Discussion: Based on the results of the comparative analysis carried out on the Fashion-MNIST dataset, it can be deduced that the performance of various models can differ depending on the configuration of their hidden layers and neurons. The analysis revealed that the GRU model was the most efficient in terms of both accuracy and lowest loss, while having the fewest parameters among all models, including Dense, LSTM, and CNN.
At the outset, when the number of hidden layers was three and the number of neurons per layer was 16, the Dense model emerged as the top performer with 84.08% accuracy and a loss of 0.45. The CNN model followed closely, with 83.51% accuracy and a loss of 0.46, though it had a considerably higher number of parameters than the Dense model.
In the second scenario, when the number of hidden neurons was increased to 512 in each of the three hidden layers, the GRU model outperformed all others, registering 89.29% accuracy and a loss of 0.31. The LSTM model came in second, with 88.20% accuracy and a loss of 0.32, while the GRU model maintained the lowest number of parameters among all models at 4.1 million.
Finally, when the number of hidden layers was raised to ten and the number of neurons per layer was 256, GRU once again emerged as the top performer, with 86.54% accuracy and a loss of 0.39. The Dense model followed close behind, with 84% accuracy and a loss of 0.45, having the lowest number of parameters among all models at nearly 2.6 million.
In conclusion, the results suggest that the GRU model can be a reliable option for accurate, low-loss predictions, especially when the number of parameters must be kept low. However, it is essential to consider the trade-offs between accuracy and other performance metrics when selecting a model for a particular task.
B) MNIST
The results of the evaluation of different models on the MNIST dataset are presented in Table 3. For the 3-layer model with 16 neurons per layer, the GRU model exhibited the highest accuracy with a score of 94.81%. This model also had the lowest loss at 0.16 and the fewest parameters at nearly 10,000. The LSTM model was the second-best performer with an accuracy of 94.22% and a loss of 0.18, having nearly the same number of parameters at 11,500. The Dense and CNN models had an accuracy of 92.44% and 92.27%, respectively, with similar loss values of 0.27. However, the Dense model had the most parameters at 126,000, while the CNN model was comprised of 82,000 parameters, both notably higher than the GRU and LSTM models. For the 3-layer model with 512 neurons per layer, the GRU model again performed the best with an accuracy of 99% and a loss of 0.03. It also had the minimum number of parameters at 4.1 million. The LSTM model was the second-best performer with an accuracy of 98.73% and a loss of 0.04, and 5.4 million parameters. The Dense model had the second-lowest number of parameters after GRU at 4.5 million, with an accuracy of 92.60% and a loss of 0.27. The CNN model had an accuracy of 92.04% and a loss of 0.28, with the largest number of parameters at 7.2 million. Finally, for the 10-layer model with 256 neurons per layer, the GRU model was once again the best performer with an accuracy of 98.45% and a loss of 0.05, having the second-lowest number of parameters at 3.8 million after Dense, which had 2.6 million parameters. The LSTM model was the second-best performer with an accuracy of 95.13% and a loss of 0.15, having 5.1 million parameters. The Dense and CNN models had an accuracy of 92.54% and 91.35%, respectively, and a loss of 0.27% and 0.31%. The CNN model had the largest number of parameters at 5.4 million. Overall, the GRU model with 3 layers and 512 neurons per layer was the best performing model in all implementations in the evaluation. Additionally, in all three scenarios, the GRU model was the best performing model type, with LSTM being the second-best.
Table 3: Performance comparison among CNN, Dense, LSTM, and GRU for MNIST dataset with 50% dropout over 30 epochs. The three most notable performance metrics are emphasized in bold: highest test accuracy, lowest test loss, and the fewest number of parameters used.
Methodology
|
# layers
|
# neurons per layer
|
Batch size
|
Test Acc (%)
|
Test Loss
|
# trainable params
|
CNN
|
3
|
512
|
128
|
92.04
|
0.28
|
7,202,826
|
Dense
|
92.60
|
0.27
|
4,540,426
|
LSTM
|
98.73
|
0.04
|
5,449,738
|
GRU
|
99.08
|
0.03
|
4,127,754
|
CNN
|
3
|
16
|
128
|
92.27
|
0.27
|
82,250
|
Dense
|
92.44
|
0.27
|
126,026
|
LSTM
|
94.22
|
0.18
|
11,594
|
GRU
|
94.81
|
0.16
|
9,962
|
CNN
|
10
|
256
|
256
|
91.35
|
0.31
|
5,477,130
|
Dense
|
92.54
|
0.27
|
2,599,690
|
LSTM
|
95.13
|
0.15
|
5,091,338
|
GRU
|
98.45
|
0.05
|
3,844,106
|
Comparison: This comprehensive evaluation highlights the superiority of GRU models when it comes to working with the MNIST dataset. Across all three configurations, the GRU model demonstrated the highest accuracy and the lowest loss, surpassing its competitors by a significant margin. Additionally, in two out of three scenarios, the GRU model was characterized by the smallest number of parameters, making it a computationally efficient solution. The LSTM model, while coming in at a close second, had higher loss values compared to the GRU model. The Dense model had a moderate performance, with the second lowest number of parameters in the scenario of 3 layers and 512 neurons per layer, but its accuracy and loss were still not on par with those of GRU and LSTM models. Lastly, the CNN model had the largest number of parameters and was the least successful model in terms of accuracy and loss in all three scenarios. Overall, the results suggest that for the MNIST dataset, GRU models provide a superior performance, with high accuracy and low loss values, as well as being computationally efficient.
Discussion: This comparative analysis presents a noteworthy observation regarding the performance of various deep learning models on the MNIST dataset. Results indicated that GRU models consistently outperformed other model types such as LSTM, Dense, and CNN, in terms of accuracy, loss, and computational efficiency. Specifically, GRU achieved the highest accuracy, lowest loss, and in two out of three scenarios, had the fewest number of parameters. These results emphasize the superiority of GRU when compared to the other models for image classification problems in the MNIST dataset. The information presented in this study may prove to be of great significance for researchers and practitioners alike, as they seek to develop deep learning models for image classification tasks and make informed decisions regarding the type of model to utilize for their specific needs.
C) EMNIST-Letters
This study aims to investigate the performance of various neural network models, including Convolutional Neural Network (CNN), Dense, Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU), on the EMNIST-Letters dataset. A 50% dropout rate was applied over 30 epochs and the models were evaluated using test accuracy and test loss. The number of hidden layers and the number of neurons per layer were adjusted across the models, with the batch size being set to 128 for models with 3 hidden layers and 256 for models with 10 hidden layers. The results of the study demonstrate that LSTM and GRU models exhibit exceptional performance in terms of test accuracy, with the highest values of 93.56% and 94.22%, respectively. Additionally, these models exhibit lower test loss compared to the CNN and Dense models. The analysis also revealed that an increase in the number of hidden layers resulted in a decline in test accuracy for all models, except for the GRU model, which showed a slight improvement. Reducing the number of neurons per layer also caused a decrease in test accuracy for all models. In terms of the number of trainable parameters, LSTM and GRU models require fewer parameters than CNN and Dense models, highlighting their efficiency in achieving good performance on the EMNIST-Letters dataset. In conclusion, this study provides valuable insights into the performance of various neural network models on the EMNIST-Letters dataset using a 50% dropout rate over 30 epochs. These findings can serve as a valuable resource for researchers and practitioners in the field of deep learning who are pursuing similar tasks.
Table 4: Performance comparison among CNN, Dense, LSTM, and GRU for EMNIST-Letters dataset with 50% dropout over 30 epochs. The three most notable performance metrics are emphasized in bold: highest test accuracy, lowest test loss, and the fewest number of parameters used.
Methodology
|
# layers
|
# neurons per layer
|
Batch size
|
Test Acc (%)
|
Test Loss
|
# trainable params
|
CNN
|
3
|
512
|
128
|
71.29
|
1.0301
|
11,167,770
|
Dense
|
72.08
|
1.0064
|
10,962,970
|
LSTM
|
93.56
|
0.2013
|
5,679,130
|
GRU
|
94.22
|
0.1939
|
4,357,146
|
CNN
|
3
|
16
|
128
|
72.11
|
1.0113
|
206,170
|
Dense
|
71.88
|
1.0089
|
326,746
|
LSTM
|
82.69
|
0.5444
|
18,778
|
GRU
|
81.88
|
0.5736
|
17,146
|
CNN
|
10
|
256
|
256
|
69.71
|
1.0781
|
5,739,290
|
Dense
|
72.18
|
1.0101
|
5,810,970
|
LSTM
|
92.41
|
0.2373
|
5,206,042
|
GRU
|
93.61
|
0.2007
|
3,958,810
|
Comparison: The present study conducted a comprehensive comparison of the performance of four prominent neural network models on the EMNIST-Letters dataset. The models under consideration were Convolutional Neural Network (CNN), Dense Network, Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU). The performance evaluation was carried out based on the test accuracy and test loss metrics over a period of 30 epochs, with a 50% dropout rate.
The first set of results, with three layers and 512 neurons per layer, revealed that the LSTM and GRU models displayed a remarkable superiority over the CNN and Dense models, with test accuracy scores of 93.56% and 94.22%, respectively, as opposed to 71.29% and 72.08% for the CNN and Dense models. Furthermore, the LSTM and GRU models exhibited lower test loss scores, at 0.2013 and 0.1939, compared to 1.0301 and 1.0064 for the CNN and Dense models, respectively.
The second set of results, with three layers and 16 neurons per layer, indicated that while the LSTM and GRU models still maintained an edge over the CNN and Dense models, the performance differential was not as pronounced as observed in the first set of results. The LSTM model achieved a test accuracy score of 82.69%, and the GRU model scored 81.88%, in contrast to 72.11% and 71.88% for the CNN and Dense models, respectively. The test loss scores for the LSTM and GRU models were also lower, at 0.5444 and 0.5736, compared to 1.0113 and 1.0089 for the CNN and Dense models, respectively.
The third set of results, with ten layers and 256 neurons per layer, continued to demonstrate the superiority of the LSTM and GRU models over the CNN and Dense models. The LSTM model recorded a test accuracy score of 92.41%, and the GRU model scored 93.61%, compared to 69.71% and 72.18% for the CNN and Dense models, respectively. The test loss scores for the LSTM and GRU models were also lower, at 0.2373 and 0.2007, compared to 1.0781 and 1.0101 for the CNN and Dense models, respectively.
In light of the above results, it can be concluded that the LSTM and GRU models tend to demonstrate superior performance over the CNN and Dense models on the EMNIST-Letters dataset, with a 50% dropout rate over 30 epochs. The differences in performance are more significant with three layers and 512 neurons per layer, but remain perceptible with three layers and 16 neurons per layer and ten layers and 256 neurons per layer.
Discussion: The results obtained from the comparative analysis presented in Table 4 indicate the superiority of LSTM and GRU models over CNN and Dense models in regards to test accuracy and test loss on the EMNIST-Letters dataset. The sequential nature of LSTM and GRU models, which allow them to retain information from previous inputs and selectively propagate relevant information, appear to be better suited to handle sequential data such as the text data in the EMNIST-Letters dataset.
It is noteworthy that the results also demonstrate that a simple network architecture, as evidenced by the models with three layers and 512 neurons per layer, may outperform a more complex one, such as the models with ten layers and 256 neurons per layer. This highlights the critical role of appropriate network design for a specific task and dataset, as over-complexity may not necessarily result in improved performance.
Moreover, the number of trainable parameters, which reflect the number of adjustable values in a model, was found to be significantly lower in LSTM and GRU models compared to CNN and Dense models. This suggests that LSTM and GRU models can learn more compact representations of the data with fewer parameters, thereby contributing to their superior performance.
D) EMNIST-Digits
The evaluation of the performance of various applied models on EMNIST-Digits dataset is provided in Table 5. The models tested were Convolutional Neural Networks (CNNs), Dense Networks, Long Short-Term Memory (LSTM) networks, and Gated Recurrent Unit (GRU) networks. The performance was measured using several evaluation metrics such as the number of layers in the network, the number of neurons per layer, the batch size used during training, the test accuracy, the test loss, and the number of trainable parameters in each model.
The data suggests that the architecture of the models was consistent, with either 3 or 10 layers and either 512 or 256 neurons per layer, and a batch size of 128 or 256. The results indicate that there were differences in the test accuracy, test loss, and number of trainable parameters between the models.
The LSTM and GRU models demonstrated higher test accuracy compared to the CNN and Dense models, achieving test accuracy values of 99.47 and 99.56, respectively, for the models with 3 layers and 512 neurons per layer, and 99.33 and 99.47 for the models with 10 layers and 256 neurons per layer. This suggests that LSTM and GRU models may be better suited for this particular task than the simpler CNN and Dense models.
It should be noted that the number of trainable parameters in the LSTM and GRU models was higher than in the CNN and Dense models. This could result in longer training time and higher computational resources requirements. However, the trade-off in terms of improved performance may make it a worthwhile investment, depending on the specific needs of the task.
In conclusion, the findings of this study suggest that LSTM and GRU models provide higher test accuracy compared to the CNN and Dense models for the task in question. Further research is necessary to gain a deeper understanding of the reasons behind the improved performance and determine the most suitable model for the task at hand.
Table 5: Performance comparison among CNN, Dense, LSTM, and GRU for EMNIST-Digits dataset with 50% dropout over 30 epochs. The three most notable performance metrics are emphasized in bold: highest test accuracy, lowest test loss, and the fewest number of parameters used.
Methodology
|
# layers
|
# neurons per layer
|
Batch size
|
Test Acc (%)
|
Test Loss
|
# trainable params
|
CNN
|
3
|
512
|
128
|
93.90
|
0.2237
|
7,202,826
|
Dense
|
94.15
|
0.2163
|
4,540,426
|
LSTM
|
99.47
|
0.0248
|
5,449,738
|
GRU
|
99.56
|
0.0220
|
4,127,754
|
CNN
|
16
|
93.92
|
0.2221
|
82,250
|
Dense
|
93.95
|
0.2203
|
126,026
|
LSTM
|
98.21
|
0.0605
|
11,594
|
GRU
|
98.20
|
0.0613
|
9,962
|
CNN
|
10
|
256
|
256
|
93.01
|
0.2563
|
5,477,130
|
Dense
|
94.04
|
0.2191
|
2,599,690
|
LSTM
|
99.33
|
0.0257
|
5,091,338
|
GRU
|
99.47
|
0.0215
|
3,844,106
|
Comparison: A comparative analysis of four neural network models, Convolutional Neural Network (CNN), Dense Network, Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU) was conducted on the EMNIST-Digits dataset. The evaluation considered various configurations of the number of layers, number of neurons per layer, and batch size, and was measured based on the test accuracy and test loss over 30 epochs with 50% dropout. The results showed that LSTM and GRU models consistently outperformed CNN and Dense models in terms of test accuracy, with the GRU model achieving the highest test accuracy of 99.56%. The test loss for LSTM and GRU models was also found to be lower compared to the test loss of CNN and Dense models. A higher number of neurons per layer resulted in improved performance for all models, with GRU model still achieving the highest test accuracy of 99.56%, and the LSTM model having the lowest test loss (0.0248). On the other hand, increasing the number of layers to 10 decreased the performance of all models, yet GRU model still achieved the highest test accuracy of 99.47%, and the LSTM model had the lowest test loss (0.0257). In conclusion, the results indicate that both the number of neurons per layer and the number of layers have a significant impact on the performance of the models. Among the four models, the GRU model emerged as the best in terms of test accuracy and test loss across various configurations.
Discussion: The present study delves into the evaluation of four distinct deep learning models on the EMNIST-Digits dataset, including Convolutional Neural Networks (CNN), Dense Networks, Long Short-Term Memory (LSTM) networks, and Gated Recurrent Unit (GRU) networks. The performance of each model was assessed based on the number of layers, number of neurons per layer, and batch size, using test accuracy and test loss as evaluation metrics over a span of 30 epochs with 50% dropout. The analysis revealed that the LSTM and GRU models exhibit superior performance compared to the CNN and Dense models, with GRU demonstrating a slight advantage over LSTM in most cases. Increasing the number of neurons per layer from 16 to 512 resulted in improved performance for all models, with the LSTM and GRU models recording the highest test accuracy and the lowest test loss. Additionally, the study observed that the number of layers also plays a significant role in the performance of the models, with an increase from 3 to 10 layers resulting in improved performance, particularly for the LSTM and GRU models. It is important to note that the number of trainable parameters increases as the number of layers and neurons per layer increases, with the highest number being 7.2 million for the CNN model with 3 layers and 512 neurons per layer. In conclusion, the results provide valuable insight into the suitability of LSTM and GRU models for the EMNIST-Digits dataset, and suggest that increasing the number of layers and neurons per layer can enhance the performance of the models.
E) Kuzushiji-MNIST
The results presented in Table 6 pertain to the application of four distinct neural network models, including CNN, Dense, LSTM, and GRU, to the Kuzushiji-MNIST dataset. The models were trained for 30 epochs and used a 50% dropout rate. For each of the models, several parameters were specified, including the number of layers, the number of neurons per layer, the batch size, the test accuracy, the test loss, and the number of trainable parameters. In the first set of results, the models consisted of three layers and 512 neurons per layer, with a batch size of 128. The LSTM model displayed remarkable performance with a test accuracy of 92.27% and a test loss of 0.3197. The GRU model also displayed exceptional results with a test accuracy of 94.01% and a test loss of 0.2672. The Dense and CNN models had slightly lower test accuracy scores (70.42% and 69.41%, respectively), and higher test loss scores compared to the LSTM and GRU models. The number of trainable parameters for the LSTM and GRU models were relatively similar, while the number of parameters for the CNN and Dense models were significantly higher. In the second set of results, the models consisted of three layers and 16 neurons per layer, with a batch size of 128. Both the LSTM and GRU models showed similar performance with test accuracy scores of 72.10% and 72.52%, respectively. The Dense and CNN models had slightly higher test accuracy scores (70.25% and 69.46%, respectively), but higher test loss scores compared to the LSTM and GRU models. The number of trainable parameters for all models was significantly lower compared to the first set of results. In the third set of results, the models consisted of ten layers and 256 neurons per layer, with a batch size of 256. The GRU model displayed the best performance, with a test accuracy of 90.18% and a test loss of 0.3661. The LSTM model also displayed commendable performance, with a test accuracy of 83.18% and a test loss of 0.5784. The Dense and CNN models had slightly higher test accuracy scores (69.42% and 67.06%, respectively), and higher test loss scores compared to the LSTM and GRU models. The number of trainable parameters for the GRU model was significantly lower compared to the LSTM model, while the number of parameters for the CNN and Dense models was higher compared to both the LSTM and GRU models.
Table 6: Performance comparison among CNN, Dense, LSTM, and GRU for Kuzushiji-MNIST dataset with 50% dropout over 30 epochs. The three most notable performance metrics are emphasized in bold: highest test accuracy, lowest test loss, and the fewest number of parameters used.
Methodology
|
# layers
|
# neurons per layer
|
Batch size
|
Test Acc (%)
|
Test Loss
|
# trainable params
|
CNN
|
3
|
512
|
128
|
69.41
|
1.0453
|
7,202,826
|
Dense
|
70.42
|
1.0075
|
4,540,426
|
LSTM
|
92.27
|
0.3197
|
5,449,738
|
GRU
|
94.01
|
0.2672
|
4,127,754
|
CNN
|
16
|
69.46
|
1.0323
|
82,250
|
Dense
|
70.25
|
1.0058
|
126,026
|
LSTM
|
72.10
|
0.8967
|
11,594
|
GRU
|
72.52
|
0.8972
|
9,962
|
CNN
|
10
|
256
|
256
|
67.06
|
1.0847
|
5,477,130
|
Dense
|
69.42
|
1.0388
|
2,599,690
|
LSTM
|
83.18
|
0.5784
|
5,091,338
|
GRU
|
90.18
|
0.3661
|
3,844,106
|
Comparison: A comparative analysis was conducted to evaluate the performance of four distinct neural network models - Convolutional Neural Network (CNN), Dense Neural Network, Long Short-Term Memory (LSTM) network, and Gated Recurrent Unit (GRU) network - applied to the Kuzushiji-MNIST dataset with a 50% dropout rate, trained over 30 epochs. The evaluation considered various configurations of the models, including the number of layers, the number of neurons per layer, and the batch size, which were used to assess the test accuracy and test loss, as well as the number of trainable parameters. The results indicated that the LSTM and GRU models generally outperformed the CNN and Dense models in terms of both test accuracy and test loss. The highest performance was achieved by the GRU model with three layers and 512 neurons per layer, achieving a test accuracy of 94.01% and a test loss of 0.2672. Conversely, the lowest performance was demonstrated by the CNN model with ten layers and 256 neurons per layer, with a test accuracy of 67.06% and a test loss of 1.0847. Analysis of the data showed that the number of neurons per layer and the number of layers in the model had a significant effect on the performance of the models, with larger models generally yielding lower test accuracy and higher test loss. Additionally, the batch size also had an impact on the results, with larger batch sizes leading to decreased test accuracy and increased test loss.
Discussion: This research provides evidence that Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models outperformed Convolutional Neural Network (CNN) and Dense Neural Network models in terms of test accuracy. The top-performing GRU model was configured with 3 layers, 512 neurons per layer, and a batch size of 128, resulting in a test accuracy of 94.01% and a test loss of 0.2672. The number of trainable parameters also plays a role in the performance and efficiency of a model, and both the LSTM and GRU models had higher numbers of trainable parameters than the CNN and Dense models.
The data suggests that an increase in the number of layers from 3 to 10 does not necessarily result in improved test accuracy, but rather, it complicates the model by adding more trainable parameters and increasing the risk of overfitting. It is worth noting that the optimal model architecture and hyperparameters may vary depending on the specific dataset and task at hand.
In conclusion, the GRU model appears to be the most suitable for the Kuzushiji-MNIST dataset, with a balanced combination of test accuracy and trainable parameters. However, the specific needs of each task and dataset must be taken into consideration when choosing the optimal model architecture.
F) Kuzushiji-49
The performances of the applied modelsare using the Kuzushiji-49 dataset are presented in Table 7. The models consisted of different types of neural networks, including Convolutional Neural Networks (CNN), Dense Neural Networks, Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRU) networks. Each model was evaluated using key parameters such as the number of layers, the number of neurons per layer, the batch size, and the number of trainable parameters. The performance of the models was measured by two metrics: test accuracy and test loss. The first set of models comprised CNN and Dense models with three layers and 512 neurons per layer. A batch size of 128 was used. The CNN model had a test accuracy of 56.56 and a test loss of 1.8580, with 16,867,377 trainable parameters. On the other hand, the Dense model recorded a test accuracy of 57.13 and a test loss of 1.8180, with 20,195,377 trainable parameters. The second set of models consisted of LSTM and GRU models, each with three layers and 512 neurons per layer. Again, a batch size of 128 was utilized. The LSTM model showed a test accuracy of 92.99 and a test loss of 0.3028, with 6,008,881 trainable parameters. The GRU model achieved a test accuracy of 93.50 and a test loss of 0.2933, with 4,686,897 trainable parameters. The third set of models consisted of CNN and Dense models with three layers and 16 neurons per layer, with a batch size of 128. The CNN model recorded a test accuracy of 56.65 and a test loss of 1.8354, with 384,305 trainable parameters. The Dense model achieved a test accuracy of 57.21 and a test loss of 1.8183, with 615,281 trainable parameters. The fourth set of models, LSTM and GRU models with three layers and 16 neurons per layer, recorded a test accuracy of 65.76 and a test loss of 1.3468 for the LSTM model, with 29,105 trainable parameters. The GRU model showed a test accuracy of 63.27 and a test loss of 1.4577, with 27,473 trainable parameters. The fifth set of models consisted of CNN and Dense models with ten layers and 256 neurons per layer, and a batch size of 256. The CNN model recorded a test accuracy of 55.02 and a test loss of 1.9352, with 6,116,145 trainable parameters. The Dense model showed a test accuracy of 57.68 and a test loss of 1.7996, with 10,427,185 trainable parameters. The sixth set of models, LSTM and GRU models with ten layers and 256 neurons per layer, recorded a test accuracy of 87.59 and a test loss of 0.4746 for the LSTM model, with 5,370,929 trainable parameters. The GRU model achieved a test accuracy of 92.48 and a test loss of 0.3021, with 4,123,697 trainable parameters. The results of the study demonstrate that the LSTM and GRU models demonstrated superior performance compared to the CNN and Dense models in terms of test accuracy. The GRU model recorded the highest accuracy among all the models. Additionally, the number of neurons per layer, the number of layers, and the batch size also appear to impact the performance of the models, with higher accuracy being associated with larger batch sizes and more neurons per layer.
Table 7: Performance comparison among CNN, Dense, LSTM, and GRU for Kuzushiji-49 dataset with 50% dropout over 30 epochs. The three most notable performance metrics are emphasized in bold: highest test accuracy, lowest test loss, and the fewest number of parameters used.
Methodology
|
# layers
|
# neurons per layer
|
Batch size
|
Test Acc (%)
|
Test Loss
|
# trainable params
|
CNN
|
3
|
512
|
128
|
56.56
|
1.8580
|
16,867,377
|
Dense
|
57.13
|
1.8180
|
20,195,377
|
LSTM
|
92.99
|
0.3028
|
6,008,881
|
GRU
|
93.50
|
0.2933
|
4,686,897
|
CNN
|
16
|
56.65
|
1.8354
|
384,305
|
Dense
|
57.21
|
1.8183
|
615,281
|
LSTM
|
65.76
|
1.3468
|
29,105
|
GRU
|
63.27
|
1.4577
|
27,473
|
CNN
|
10
|
256
|
256
|
55.02
|
1.9352
|
6,116,145
|
Dense
|
57.68
|
1.7996
|
10,427,185
|
LSTM
|
87.59
|
0.4746
|
5,370,929
|
GRU
|
92.48
|
0.3021
|
4,123,697
|
Comparison: The presented data pertains to the performance evaluation of four distinct models (CNN, Dense, LSTM, GRU) trained on the Kuzushiji-49 dataset. The models vary in their architectural design, such as the number of layers, the number of neurons per layer, and the batch size. The test accuracy and test loss are used as metrics to gauge the effectiveness of the models.
It is clear from the results that the LSTM and GRU models outperform their CNN and Dense counterparts. For instance, the LSTM model with three layers and 512 neurons per layer demonstrates a test accuracy of 92.99% and a test loss of 0.3028, while the CNN model with the same configuration exhibits a test accuracy of 56.56% and a test loss of 1.8580.
Furthermore, the results suggest that simply increasing the number of layers and neurons per layer is not a guarantee of improved performance. For example, the CNN model with ten layers and 256 neurons per layer, which has 6,116,145 trainable parameters, shows a lower test accuracy of 55.02% compared to the model with three layers and 512 neurons per layer.
In conclusion, the selection of a suitable model and its configuration must be based on the specific task and problem at hand. The results indicate that for the Kuzushiji-49 task, the LSTM and GRU models are more effective than their CNN and Dense counterparts.
Discussion: The findings from the study indicate that GRU and LSTM models achieve the highest accuracy, with 93.50% and 92.99% respectively, compared to the lower accuracy of CNN and Dense models, which reach 57.68% and 57.21% respectively. The results also suggest that increasing the number of neurons per layer and the number of layers in a model generally leads to a higher accuracy and lower loss, but it can also result in a higher number of trainable parameters and the potential for overfitting. Increasing the batch size in the training process can result in a lower loss and higher accuracy, but it also increases the computational requirements. The complexity of the model, as measured by the number of trainable parameters, influences the accuracy and loss, with more complex models having a higher accuracy and lower loss but also a higher risk of overfitting. In conclusion, the optimal model configuration will depend on the specific task and data, and a good model will balance accuracy, loss, and complexity while considering the trade-offs between these factors.