🛠️ Steven Gong

Search

SearchSearch
Oct 05, 2026 · 1 min read · 30 views · 3 in the last 30 days

Where readers are reading from

FijiTanzaniaWestern SaharaCanada · 0 readersUnited States of America · 0 readersKazakhstanUzbekistanPapua New GuineaIndonesiaArgentinaChileDemocratic Republic of the CongoSomaliaKenyaSudanChadHaitiDominican RepublicRussiaThe BahamasFalkland IslandsNorwayGreenlandFrench Southern and Antarctic LandsEast TimorSouth AfricaLesothoMexicoUruguayBrazilBoliviaPeruColombiaPanamaCosta RicaNicaraguaHondurasEl SalvadorGuatemalaBelizeVenezuelaGuyanaSurinameFrance · 0 readersEcuadorPuerto RicoJamaicaCubaZimbabweBotswanaNamibiaSenegalMaliMauritaniaBeninNigerNigeriaCameroonTogoGhanaIvory CoastGuineaGuinea-BissauLiberiaSierra LeoneBurkina FasoCentral African RepublicRepublic of the CongoGabonEquatorial GuineaZambiaMalawiMozambiqueEswatiniAngolaBurundiIsraelLebanonMadagascarPalestineThe GambiaTunisiaAlgeriaJordanUnited Arab Emirates · 0 readersQatarKuwaitIraq · 0 readersOmanVanuatuCambodiaThailandLaosMyanmarVietnamNorth KoreaSouth KoreaMongoliaIndiaBangladeshBhutanNepalPakistanAfghanistanTajikistanKyrgyzstanTurkmenistanIranSyriaArmeniaSwedenBelarusUkrainePolandAustriaHungaryMoldovaRomaniaLithuaniaLatviaEstoniaGermany · 1 readersBulgariaGreeceTurkeyAlbaniaCroatiaSwitzerlandLuxembourgBelgiumNetherlandsPortugalSpainIrelandNew CaledoniaSolomon IslandsNew ZealandAustraliaSri LankaPeople's Republic of China · 1 readersTaiwanItalyDenmarkUnited KingdomIcelandAzerbaijanGeorgiaPhilippinesMalaysiaBruneiSloveniaFinlandSlovakiaCzech RepublicEritreaJapanParaguayYemenSaudi ArabiaTurkish Republic of Northern CyprusCyprusMoroccoEgyptLibyaEthiopiaDjiboutiSomalilandUgandaRwandaBosnia and HerzegovinaNorth MacedoniaSerbiaMontenegroKosovoTrinidad and TobagoSouth SudanSelect a country to explore · Last 30 days
China1
Germany1
Singapore0
United States0
Canada0
United Arab Emirates0
Traffic · last 30 days012Oct 21Oct 31Nov 10Nov 19
views
Traffic · all time146Mar 2Mar 5Mar 8Mar 11Mar 13
views

Collective Operations

AllReduce

All-Reduce = ReduceScatter + AllGather

https://tech.preferred.jp/en/blog/technologies-behind-distributed-deep-learning-allreduce/ is really good

Mental Model of All-Reduce

Usually, when I think of AllReduce, in my head it’s more of a Reduce + Broadcast

Graph View

Backlinks

  • Collective Operations
  • ReduceScatter
  • Torch Distributed

Created with Quartz, © 2026

  • Blog
  • LinkedIn
  • Twitter
  • GitHub