Cluster Computing
Running a workload across a group of loosely-coupled machines connected by a fast network. The logical step up from a single multicore box when one machine can’t hold the data or the compute.
Spectrum:
- HPC cluster: homogeneous nodes, MPI for message passing, job scheduler (Slurm, PBS). Tightly synchronized scientific workloads (weather, CFD, molecular dynamics).
- Data-parallel cluster: Hadoop/Spark/Dask. Sharded data, coarse-grained tasks, fault tolerance via re-execution.
- Service clusters: Kubernetes, Nomad. Long-running services, request-level parallelism.
- Beowulf-style: commodity Linux boxes on Ethernet. Original “Google idea.”
Why not a bigger single machine?
- Cost scaling: a single 2-socket server tops out at something like 200 cores and 8 TB RAM; past that, a small cluster is cheaper per core than exotic SMP.
- Availability: a cluster survives node failure. A single machine doesn’t.
- Data locality: data often arrives already distributed (logs, IoT); ship compute to it.
Costs a single machine doesn’t have:
- Network latency (µs vs ns for local memory).
- Partial failures: nodes drop, networks partition.
- Coordination overhead: consensus, leader election, distributed locks.
- Serialization: data has to cross the wire.
Rule of thumb: a good laptop beats a poorly-tuned 10-node cluster for anything that fits in one machine’s RAM. Clusters earn their keep when the workload genuinely doesn’t fit.
From ECE459 L30
Two buy-more-hardware scenarios: (1) one computer can’t keep up, or (2) two cheap computers beat one expensive one. The same splitting tricks (threads, processes) scale to multiple boxes. Focus is on calculations, not general distributed-systems coordination.
Message passing
Shared memory is off the table across machines. Network communication is way more expensive than intra-machine, so be careful about how much you send. Rust already nudges you this way.
MPI used to be the go-to. Adoption has since slid in favour of other stacks [Dur15]. Course moves on.
REST
Synchronous is the default but callbacks / polling let you approximate async. Still requires the remote to be up each call; for more decoupling, use a broker.
Kafka [Kur20]
Distributed streaming platform. Producers write records to topics; consumers read. Records live for a fixed retention and can be replayed.
Internally: immutable log, split into partitions (more partitions = more parallelism). Producer picks a partition; consumers track their own offset per partition (commit offset = I’ve read up to here). Broker logic stays simple; consumers move at their own speed; no coordination required; retention, not consumer-drain, cleans the log.
Contrast with a classic queue: classic queues pressure you to drain quickly (full queue drops/charges). Taking items out then processing later is only safe if you’ve persisted them first; else a crash eats the work.
AWS alternatives
- SNS: pub/sub fan-out, non-persistent. Push notifications, alerts, “update your cache” broadcasts. Miss it, miss it.
- SQS: work queue. Items deleted after consumption; ordering guarantees may or may not exist; limited retention.