Nonlocal Block

The Nonlocal Block drops self-attention into a CNN so any spatio-temporal location can attend to any other.

Starting from a 3D feature map $f\in {\mathbb{R}}^{C\times T\times H\times W}$:

1. Use three $1\times 1\times 1$ convolutions to produce $Q,K,V$
2. Flatten the $(T,H,W)$ dimensions into tokens
3. Compute attention weights across all locations
4. Apply those weights to $V$
5. Project back to $C$ channels and add the residual

Structurally this is the same idea as self-attention, just applied over video or feature-map positions instead of text tokens, and using convolutions for the projections.

Related:

• Action Classification
• SlowFast
• VideoMAE