Instance Normalization

Resources

• https://paperswithcode.com/method/instance-normalization

Instance Normalization is a normalization technique commonly used in tasks like style transfer and image generation. It normalizes the features independently for each sample and each channel.

Instance normalization is applied per-instance per-channel(copy-pasted from chatgpt):

$${\mu }_{n,c}=\frac{1}{HW}\sum _{h=1}^H\sum _{w=1}^W{x}_{n,c,h,w}$$ $${\sigma }_{n,c}^2=\frac{1}{HW}\sum _{h=1}^H\sum _{w=1}^W(x_{n,c,h,w}-{\mu }_{n,c}{)}^2$$ $${\hat{x}}_{n,c,h,w}=\frac{x_{n,c,h,w}-{\mu }_{n,c}}{\sqrt{{\sigma }_{n,c}^2+ϵ}}$$

Then optionally apply learned affine parameters:

$$y_{n,c,h,w}={\gamma }_c\cdot {\hat{x}}_{n,c,h,w}+{\beta }_c$$

Where:

• $ϵ$ is a small number to avoid division by zero.
• ${\gamma }_c$, ${\beta }_c$ are learnable scale and shift parameters (per channel).

Key differences from other normalizations:

• Compared to BatchNorm, instance norm doesn’t compute statistics across the batch.
• It improves style invariance and is often used in generative networks.

# PyTorch example
import torch
import torch.nn as nn
 
norm = nn.InstanceNorm2d(num_features=3)  # for 3-channel images
out = norm(torch.randn(8, 3, 32, 32))     # input: (batch, channel, height, width)

Related

• Batch Normalization
• Layer Normalization