nn.parameter

Parameter or buffer?

• https://docs.pytorch.org/docs/2.14/generated/torch.nn.parameter.Parameter.html
• https://docs.pytorch.org/docs/2.14/generated/torch.nn.parameter.Buffer.html

The difference is that parameters are learnable, but buffer should not be. Buffers can still change over time, but they are not updated during the backward pass through gradient descent.

Parameters:
    weight          # γ, learnable
    bias            # β, learnable

Buffers:
    running_mean    # updated during **forward**
    running_var     # updated during **forward**
    num_batches_tracked