Inductive Bias

Was really interested in this when examining LLM Scaling Laws.

Inductive bias is the set of assumptions your learning method builds in so it can generalize from limited data.

Without assumptions, lots of functions could fit the training set equally well — the bias is what makes the model prefer some solutions over others.

Simple examples

• Linear regression has an inductive bias that the relationship is roughly linear.
• L2 regularization biases toward smaller-norm weights (simpler / less sensitive functions).
• CNNs bias toward local patterns and translation invariance (a cat is still a cat if it moves a bit in the image).
• Data augmentation adds an inductive bias like: “the label shouldn’t change under small crops/rotations/noise.”