Activation Functions & Non Linearity Questions

Know common activation functions: ReLU, sigmoid, tanh, softmax, GELU, and Swish. Understand why non-linearity is necessary: stacking any number of purely linear layers collapses to a single linear transformation, eliminating the network's ability to model complex functions. Know the advantages and disadvantages of each activation function (vanishing gradients, dead neurons/dying ReLU, computational cost, output range, saturation). Understand why ReLU remains the default choice in modern architectures despite its simplicity, and when smoother alternatives (GELU, Swish) are preferred in transformer-style models.