Mitigating Neural Network Training Challenges Through Effective Optimizers and Activation Function Selection

Abstract

Training deep neural networks is often hindered by problems such as vanishing gradients (Sigmoid), neuronal instability (ReLU), slow convergence (SGDM), and poor generalization (Adam). This research systematically investigates how the choice of optimizers and activation functions mitigates these challenges. Using aConvolutional Neural Network(CNN)trainedon the MNISTdataset, five activation functions (Sigmoid, ReLU, LeakyReLU, GeLU, SiLU) and four optimizers (SGDM, Adam, AdamW, RMSProp) were compared in 20 unique configurations. Performance was evaluated on the basis of validation accuracy, convergence speed, and gradient stability.