In the light of recent engineering conversations I have had about whether Kafka is a good candidate to replace a messaging solution like RabbitMQ, SQS, ActiveMQ etc, I thought I’d share some thoughts on the basics of Kafka, and in the process, highlight some differences between Kafka and other popular messaging services.
What exactly is Kafka?
At it’s core, Kafka is a distributed append-only commit log offering certain message semantics and guarantees.
Wow, that’s a mouthful, but I thought Kafka was a Message Queue? Where is the Queue part?
Expanding on the earlier definition, Kafka provides constructs for writing/publishing (to) and reading from a distributed append-only log while offering some guarantees on how messages are written and consumed from the logs. The strict ordering of messages being published to the underlying Kafka logs resembles a queue like behavior - that’s the Queue part.
Hmmm, Kafka is really not a Message Queue?
Kafka is not a Message Queue(MQ) in the traditional sense like RabbitMQ, ActiveMQ or SQS. Message queues like ActiveMQ, SQS etc., typically implement some sort of a queuing protocol like AMPQ, STOMP etc. Kafka does not implement any of the well known standard queuing protocols. The creators of Kafka decided to make some engineering trade-offs at the outset to focus on scale, message durability, and to some extent, simplicity. See Kafka Vs the Rest for info on how Kafka differs from other messaging solutions.
The other major difference is, messages in a standard MQ are not durable; the messages are deemed transitionary i.e. they disappear once consumed. For this reason, they are categorized as fire and forget systems. Of course, you can always devise strategies to make the message more durable, but such systems treat messages as ephemeral out-of-the-box. In case of Kafka, all messages are persisted in logs, and hence they are durable unless and until the logs are explicitly cleaned up. For a more in-depth difference between Kafka and other messaging systems, see the reference section.
More accurately, Kafka can be classified as a streaming platform with high throughput and message durability.
What about using Kafka as a Pub-Sub system?
Kafka’s ability to support subscriptions to parallel data streams through topics/partitions makes it a viable candidate for a high throughput Pub-Sub system.
When should I consider Kafka vs X,Y,Z messaging solutions?
If you are looking for a point-to-point messaging solution and don’t care much about message durability, a standard message queue like RabbitMQ, ActiveMQ or SQS will suffice. On the other hand, if you are looking for a high scale Pub-Sub messaging platform with message durability, then Kafka is a good choice.