https://stevengong.co Last 10 notes on 🛠️ Steven Gong Quartz -- quartz.jzhao.xyz https://stevengong.co/notes/Activation-Checkpointing https://stevengong.co/notes/Activation-Checkpointing Activation Checkpointing The notes below are written by AI slop. Please fix when you spend time deep diving. Training trades memory for compute by recomputing intermediate activations during the backward pass instead of storing them from the forward pass. Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Context-Parallelism https://stevengong.co/notes/Context-Parallelism Parallelism (ML Training) Context Parallelism Context Parallelism (CP) parallelizes a Transformer across the sequence dimension. Each GPU gets a slice of the tokens instead of a full sequence. Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Data-Parallelism https://stevengong.co/notes/Data-Parallelism Distributed Parallel Training Data Parallelism This is the lowest hanging fruit to enable Distributed Parallel Training, where we just shard across the batch dimension. Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Fully-Sharded-Data-Parallel https://stevengong.co/notes/Fully-Sharded-Data-Parallel Data Parallelism Fully Sharded Data Parallel (FSDP) FSDP is just more fancy Data Parallel, where we shard many other things (parameters, gradients, optimizer state). Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Gradient-Checkpointing https://stevengong.co/notes/Gradient-Checkpointing Gradient Checkpointing Resources Arsh’s summer project: https://blog.janestreet.com/trading-off-compute-for-memory-with-activation-checkpointing/ https://huggingface. Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Model-FLOPs-Utilization https://stevengong.co/notes/Model-FLOPs-Utilization Model FLOPs Utilization (MFU) The ratio of the theoretical matmul FLOPs required to train a model over the actual wall-clock FLOPs delivered by the hardware. Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Model-Parallelism https://stevengong.co/notes/Model-Parallelism Distributed Parallel Training Model Parallelism. Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Parallelism-(ML-Training) https://stevengong.co/notes/Parallelism-(ML-Training) Parallelism (ML Training) See this for the different paradigms: https://colossalai.org/docs/concepts/paradigms_of_parallelism/ Read this guide here, and watch the CS336 videos Ultrascale playbook by HuggingFace A Transformer operates on tensors of shape (Batch,Seq,Dim). Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Pipeline-Parallelism https://stevengong.co/notes/Pipeline-Parallelism Parallelism (ML Training) Pipeline Parallelism Use of Pipelining for GPUs to do training. https://colossalai.org/docs/concepts/paradigms_of_parallelism/ Tensor Parallelism GPipe paper: https://arxiv. Sun, 11 Oct 2026 18:02:12 GMT https://stevengong.co/notes/Sequence-Parallelism https://stevengong.co/notes/Sequence-Parallelism Parallelism (ML Training) Sequence Parallelism There seems like a few way to do this. This technique is a bit overloaded. Learned from the huggingface tutorial. Sun, 11 Oct 2026 18:02:12 GMT