Activation Function

An activation function is a non-linear function applied elementwise inside a Neural Network so that stacked layers don’t collapse to a single linear map.

Why does nonlinearity matter?

Without an activation, a 2-layer network $f=W_2{W}_{1}x$ is just $f=W_{3}x$ with $W_3=W_2{W}_1$, a linear classifier. The activation is what lets the network compose hierarchical, non-linear feature transforms.

Intuition

Each activation is a “gate” with some shape. A linear combination picks a direction in input space, and the activation decides what to do with signals pointing that way: sigmoid says “pass through if positive-ish, else suppress”, ReLU says “pass through unchanged or kill entirely.” Saturation (sigmoid/tanh at extreme inputs) is the enemy of learning because the derivative goes to zero, so gradients can’t flow back through that neuron.

Common activations:

• Sigmoid Function: see the page for drawbacks, no longer used
• $\tanh (x)$: still kills gradients when saturated, but zero-centered (vs sigmoid)
• Rectified Linear Unit (ReLU), $f(x)=\max (0,x)$
  • Does not saturate in the positive region
  • Very computationally efficient
  • Converges much faster than sigmoid/tanh in practice
  • More biologically plausible than sigmoid
  • Drawback: not zero-centered, leading to “dead ReLU” units where gradient stays 0
• Leaky ReLU, $f(x)=\max (0.01x,x)$
• Parametric ReLU, $f(x)=\max (\alpha x,x)$
• Exponential Linear Unit (ELU): $f(x)=\{\begin{array}{ll}x& x>0\\ \alpha (e^x-1)& x\le 0\end{array}$
  • All benefits of ReLU
  • Closer to zero-mean outputs
  • Negative saturation regime adds robustness to noise vs Leaky ReLU
  • ELU/Leaky ReLU exist because “dead ReLU” is a real failure mode: once a neuron’s pre-activation is always negative, gradient is zero forever and that neuron never recovers. Letting a little negative signal through keeps the neuron alive
• Maxout Neuron
• Softmax Function

In general, just use ReLU and be careful with learning rates.

• From CS231n (?)

Parametric ReLU makes alpha a learnable parameter: https://datascience.stackexchange.com/questions/18583/leakyrelu-vs-prelu

There’s also Mish.

Modern defaults

From the CS231n Lec 4 slides:

• ReLU: safe default for FC/CNN layers, cheap, doesn’t saturate on positives
• GELU, $f(x)=x\cdot \Phi (x)$ (where $\Phi$ is the standard normal CDF): default in Transformers (BERT, GPT), smooth, slightly negative-permitting near the origin
• SiLU / Swish, $f(x)=x\cdot \sigma (x)$: used in newer vision models and Llama-style LLMs, smooth like GELU but cheaper

CS231n’s blunt rule of thumb: “ReLU is a good default choice for most problems.”