The influence of network structure on different gradient descent optimization algorithms
DOI:
https://doi.org/10.62051/k2z93r32Keywords:
optimization algorithm; network structure; model performance.Abstract
The gradient optimization algorithms and network architectures play pivotal roles in the field of artificial intelligence. However, there is limited research comparing multiple optimization algorithms across different network structures. In this paper, the effectiveness of several optimization techniques in image processing tasks is investigated, along with an investigation of their effects on different neural network architectures. LeNet, AlexNet, and the backpropagation neural network are the three popular neural network architectures included in the selection, along with three different optimization algorithms. An extensive assessment of training loss, test accuracy, convergence time, and other metrics was carried out to determine how well these algorithms worked in various network designs through rigorous experimentation on image datasets. The results show complex differences in the impact of optimization techniques across various neural network configurations, providing essential information for the choice of the best algorithms and network structures.
Downloads
References
M. Jordan, T. Mitchell, Machine learning: Trends, perspectives, and prospects. Science 349,255-260, 2015.
Y. Xue Y, Y. Wang, J. Liang, A self-adaptive gradient descent search algorithm for fully-connected neural networks, Neurocomputing, Volume 478, 2022, Pages 70-80.
Y. Sun, B. Xue, M. Zhang, et al., "Automatically Designing CNN Architectures Using the Genetic Algorithm for Image Classification," IEEE Transactions on Cybernetics, vol. 50, no. 9, pp. 3840-3854, Sept. 2020.
F. Mignacco, P. Urbani, The effective noise of stochastic gradient descent. J. Stat. Mech: Theory Exp. 2022, 083405, 2022.
S. Haji, A. Abdulazeez, Comparison of Optimization Techniques Based on Gradient Descent Algorithm: A Review. PalArch's J. Archaeol. Egypt/Egyptol, 18(4), 2715-2743, 2021.
Y. Tian, Y. Zhang, H. Zhang, Recent Advances in Stochastic Gradient Descent in Deep Learning. Mathematics 2023, 11, 682.
K. Chen, S. Yang, C. Batur, "Effect of multi-hidden-layer structure on performance of BP neural network: Probe," ICNC 2012, China, 2012, pp. 1-5.
J. Zhang, X. Yu, X. Lei, et al. A novel deep LeNet-5 convolutional neural network model for image recognition. Comput. Sci. Inf. Syst. 2022, 19, 1463–1480.
F. Hu, G. Xia, J. Hu, et al. Transferring Deep Convolutional Neural Networks for the Scene Classification of High-Resolution Remote Sensing Imagery. Remote Sens. 2015, 7, 14680-1470.
O. Nocentini, J. Kim, M. Bashir, et al. Image Classification Using Multiple Convolutional Neural Networks on the Fashion-MNIST Dataset. Sensors 2022, 22, 9544.
Downloads
Published
Conference Proceedings Volume
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.