PyTorch

Tensor

A tensor is a fundamental data structure that holds n-dimensional data of scalar types (int, float, double, etc.).

Resources

• PyTorch internals

A tensor contains:

• Address of storage that it’s pointing to (physical layout)
• Metadata
  • Shape (logical layout)
  • Stride (tells how to)
  • Device

Stride

This is super fundamental and important that your

a = torch.randn(5,4,3)
a.stride() # (12, 3, 1)

• By default, it is stored Row-Major Layout

To go from logical to physical layout, we use the following formula

data = storage_offset + idx[0] * stride[0] + idx[1] * stride[1] + ...

When we take a transpose, the underlying data storage is the same:

b = a.T
b.stride() # (1, 3, 12) # flipped!
 
assert b.data_ptr() == a.data_ptr()

More generally, for any view transformation on the tensor, the underlying storage is not the same. This is how we can get really good speedup, we don’t have to shuffle data around, we use strides to tell us how to interpret that data.

Though, sometimes, it may be faster to copy the data.

If we take a slice, do you think the stride is the same?

c = a[:, 1:, :]
c.storage_offset() # 0
c.shape() # (5,3,3)
c.stride() (12,3,1)

What about a stride of 0?

Play around and see different storage offsets.

How about in NumPy or PyTorch?

You have the same d

Kajanan made it click for me:

• A vector is a collection of scalars
• A matrix is a group of vectors
• A tensor is a group of matrices!