Message Broker
Need to understand how this relates to Pub-Sub?
Broker
A “broker” is a person who or entity which arranges transactions between a buyer and a seller.
A message broker is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver.
Does the message broker transmit the data or does it just establish the connection?
In most common cases, a message broker actively transmits the data. Ex:
Producers and consumers do not communicate directly with each other. Instead, they communicate through the broker, which facilitates the exchange of data.
But this doesn't seem to scale...
It seems that all new processes will talk to this message broker to discover other processes.
I think Hemal Shah was the first to introduce me to this idea. (the DB image)
.png)
But how does this scale?
You can scale it by deploying a cluster of brokers. See below.
Broker Clustering
Many modern message brokers (e.g., RabbitMQ, Kafka) support clustering or replication. In a cluster, multiple broker instances work together, providing redundancy. If one broker goes down, another broker in the cluster can take over the message routing. This helps mitigate the single point of failure risk.
Some systems use Load Balancing to distribute the traffic across multiple broker instances. This approach prevents any single broker from being overwhelmed by too many messages.