PyTorch

https://bnikolic.co.uk/blog/fast-nonlinear-optimisation.html

Resources

• https://www.learnpytorch.io/
• https://github.com/hahnec/torchimize

Replacing Tensorflow as the state of the art DL model.

Tips:

• ALWAYS use torch.compile

Torch

torch.cat() # This is INEFFICIENT
torch.unbind

• torch.as_tensor

Indexing

I swear this indexing messes with my brain. Consider the example below

X = torch.randn((32,3))
C = torch.randn((27,2))
 
C[X] # is valid
C[X].shape # torch.Size([32, 3, 2])

Reshaping

I think torch can reshape by doing

X.view(-1, 6) # Reshape this, without changing it in view
X.reshape(-1) # this does it in place, it changes it permanently

Tensors

Tensors are the fundamental building block of machine learning. Their job is to represent data in a numerical way.

There is a difference between torch.tensor and torch.Tensor

torch.tensor # infers the type
torch.Tensor # doesn't infer the type, casts to float32

import torch
 
# Scalars
scalar = torch.tensor(7)
scalar # tensor(7)
scalar.ndim # 0
scalar.items() # 7
 
vector = torch.tensor([7, 7])
 
# Initializing values
tensor = torch.rand((3, 4))
zeros = torch.zeros((3, 4))
ones = torch.ones((3, 4))
 
# Arithmetic
torch.matmul(tensor, tensor)
torch.mm(tensor, tensor) # Alternative syntax
tensor @ tensor # Same thing, actually faster
 
 
# Other functions
zero_to_ten = torch.arange(start=0, end=10, step=1)
ten_zeros = torch.zeros_like(input=zero_to_ten) # will have same shape
torch.max(x), torch.min(x)
 
tensor = torch.arange(10., 100., 10.)
tensor_float16 = tensor.type(torch.float16)

The same operations can be done with NumPy

Tensor data types: https://pytorch.org/docs/stable/tensors.html#data-types You can cast as float by using .float()

Dimensions of tensors

For Mac

Although we don’t have an Nvidia GPU, there has been added support. Use device = torch.device("mps")