Skip to main content

Kafka Consumer: Why Group ID

Unlike  JMS consumers, Kafka consumers need group id. Why? Let's start the analysis from JMS consumers. JMS supports both queue and topic as follows,  
  • point-to-point queue with multiple consumers, each of which receives a subset of the messages in the queue.
  • publisher subscriber topic with multiple consumers, each of which receives a full copy of all the messages in the topic.
JMS queue obviously has the advantage of load balancing in message consumption, while a topic has the advantage of supporting multiple subscribers. Now the question is how we combine JMS queue and topic into a single message model(without a separate queue and topic) with the advantage of both load balancing and multiple subscribers. With the introduction of group id, this objective is achieved in kafka.


Specifically, a kafka consumer group is composed of one or more consumers with the same group id, and each consumes a subset of the messages based on kafka topic partition. Moreover, in Kafka, multiple groups of consumers can be created with a topic. To conclude, a single kakfa group of consumers works like a JMS queue and multiple groups, each is composed of multiple consumers, work like JMS topic in the way that each group gets a full copy of messages..

It is important to note that the group id of a kafka consumer group must be unique. If two subscribers from two different teams subscribe to the same topic with the same group id, it may ends up with one subscriber getting a subset of the messages and another subscriber receiving the remaining messages. One way to solve the uniqueness issue is to control group id through boarding process.


The following are diagrams that show how the consumers of JMS topic and queue and Kafka topic work.
JMS Queue and Consumer

JMS Topic and Consumer

Kafka Consumer



Comments

Post a Comment

Popular posts from this blog

Apache Nifi and System Integration

Apache Nifi and System Integration Introduction Apache Nifi is a distributed data platform based on enterprise integration pattern(EIP). It is a very powerful tool to build data pipeline with its large number of built-in processors.In today's service orientated architecture or a system composed of micro services, the flow of data among systems is fundamental in building enterprise applications. Among integration tools (Mule ESB, Apache Camel, Apache Nifi), Nifi is my favorite due to its built-in processors, ease to use and dynamic/hot redeployment, all leading to high productivity. When it is used together with Kafka connector, we can build real time CDC(change data capture) bi-direction integration system. This feature is especially useful in application re-engineering and migration. Nifi Basics Unlike camel, Nifi is a web based integration tool where you configure your processors, the building block of Nifi, to pilepine your data from your source system to your target ...
Post a Comment
Read more

Kafka Messages and Data Consistency

Kafka Message and Data Consistency With its scalability and fault tolerance features, Kafak has been becoming more an more popular in large scale,  real time  enterprise applications. Kafka messages are published to partitions that are usually located on different nodes and consumed by multiple consumers, each of which read messages from a single partition. This raises a data consistency issue due to multiple partitions and consumers. For example, if a security in a trading system is modified twice within a very short time and the messages could be published to two different partitions. As a result, the two messages are processed by two consumers and there is no guarantee that the last message ends up in your application or your data storage. How can this issue be resolved? Kafka Key With Single Threaded Consumer Kafka message is published with a key and payload. The messages with the same key are published to the same partition that will be consumed by the same c...
Post a Comment
Read more