Regularization

Dropout

Dropout is a regularization technique that randomly zeroes activations during training so no single neuron can be relied on.

Why would randomly breaking the network help?

It forces redundant representations and prevents co-adaptation (neuron A can’t rely on neuron B always being on). Another view: dropout trains a huge ensemble of weight-sharing subnetworks, one per sampled mask.

• When an activation is dropped, its output and thus its gradient in Backprop are both 0
• I was asked about this in my Intact interview

Train / Test

From the CS231n Lec 6 slides:

• Train: at each forward pass, sample a random binary mask $z$ and zero out activations with probability $1-p$, common setting $p=0.5$
• Test: don’t drop, instead scale all activations by $p$ so the expected output matches training

This is the general regularization recipe: $y=f_W(x,z)$ at train time (random $z$), and at test we want

$y={\mathbb{E}}_z[f(x,z)]=\int p(z)f(x,z)dz$

where:

• $z$ is the random binary mask
• $p(z)$ is its distribution

Intuition

At train time, each neuron fires with probability $p$, so its downstream expected contribution is $p\cdot \text{activation}$. At test time nothing drops, so activations are larger on average by a factor of $1/p$. Scaling by $p$ at test time (or equivalently $1/p$ at train time, “inverted dropout”) lines the two up so the network sees the same expected input distribution in both phases.

Dropout’s “scale by $p$” is the cheap deterministic approximation to this integral.

Ensemble Interpretation

A net with $N$ activations has $2^N$ possible binary masks. For a 4096-wide layer that’s $2^{4096}\approx 10^{1233}$ subnetworks sharing weights. Each forward pass trains one of them on one minibatch; test-time scaling averages the ensemble.

Intuition

You can’t rely on any specific teammate showing up, so every neuron has to be individually useful. No “this feature works only if that other feature co-activates” hacks, because at any given step the other feature might be dropped. It’s ensemble learning disguised as a regularizer: you train an exponential number of subnetworks with shared weights, and at test time you implicitly average them by scaling.