When dealing with Microservices, you will often come across Messaging Queues or some of its implementation. As an Architect, you will often have to choose a service or platform for the same. We often come across this questions : "Should I go for Kafka or RabbitMQ ?".
While they both may seem like enabling your micro services to communicate asynchronously and get the job done but the way these two work is very different from one another.
Asynchronous Messaging Patterns
Asynchronous messaging is a messaging scheme where message production by a producer is decoupled from its processing by a consumer. This allows each producer and consumer to scale independently allowing a boost in the application systems.
We typically identify two main messaging patterns — message queuing and publish/subscribe.
- Message Queueing
In the message-queuing communication pattern, queues temporally decouple producers from consumers. Multiple producers can send messages to the same queue; however, when a consumer processes a message, it’s locked or removed from the queue and is no longer available. Only a single consumer consumes a specific message.
If consumer fails to process some message, messaging platform typically return back the message to queue to retry processing from other consumers.
- Publish/Subscribe also well known as Pub/Sub
In Pub/Sub model, a single message can be received and processed by multiple subscribers concurrently.
This pattern allows a publisher, for example, to notify all subscribers that something has happened in the system. Many queuing platforms often associate pub/sub with the term topics.
Generally there are two types of subscriptions:
A. Ephemeral subscription: Subscriptions to topics that are active only when the consumer is active/running. When the consumer stops running, then the subscription and any unprocessed messages are lost.
B. Durable subscription: Subscriptions to topics are maintained until they are specifically deleted. The subscription is maintained even if a consumer shuts down. When the consumer is running again, the message processing is resumed.
RabbitMQ is an implementation of a message broker — often referred to as a service bus. Simply said; it is software where queues are defined, to which applications connect in order to transfer a message or messages.
- Queues: RabbitMQ supports classic message queuing out of the box. We define named queues, and then publishers can send messages to that named queue. Consumers, in turn, use the same queue to retrieve messages to process them.
- Message Exchange: RabbitMQ implements pub/sub via the use of message exchanges. A publisher publishes its messages to a message exchange without knowing who the subscribers of these messages are. The exchange then pushed the message to a queue. The consumers listen to this queues and processes messages. Exchange decides which queues to send the messages based on rules.
Unlike RabbitMQ, Apache Kafka is an open-source distributed event streaming platform. It is not an implementation of message broker. Event streaming thus ensures a continuous flow and interpretation of data so that the right information is at the right place, at the right time.
Kafka's storage layer is not queue based but consists of partitioned transaction log. It provides Stream APIs to process data real time which makes it very popular.
- Topics : Kafka doesn’t implement the notion of a queue. Instead, Kafka stores collections of records in categories called topics. For each topic, Kafka maintains a partitioned log of messages. Each partition is an ordered, immutable sequence of records, where messages are continually appended.
- Consumer Groups: Kafka allows pub/sub model by the convention of consumer groups. A producer can send messages to a specific topic, and multiple consumer groups can consume the same message. Each consumer group can scale individually to handle the load.
Kafka doesn't have a real classic messaging queue pattern. However, it can be replicated by creating a topic with single consumer group consuming the messages.
While RabbitMQ and Kafka are sometimes interchangeable, their implementations are very different from each other. One is a message broker(RabbitMQ), and the other is a distributed streaming platform(Kafka).
I will cover more on this in next topic.
Check Part II
Hope you liked this one. Cheers.