Semianalysis
Has some really good articles to learn about GPUs
- https://newsletter.semianalysis.com/p/nvidia-tensor-core-evolution-from-volta-to-blackwell
- https://newsletter.semianalysis.com/p/dissecting-nvidia-blackwell-tensor
Some notes from tensor core article:
- Issuing an instruction takes ~30pj, whereas the floating point op only takes 1.5pj
- so thereโs a lot of overhead in just issuing instructions, maybe we can combine that into a single instruction at the hardware level