Unlike JMS consumers, Kafka consumers need group id. Why? Let's start the analysis from JMS consumers. JMS supports both queue and topic as follows, point-to-point queue with multiple consumers, each of which receives a subset of the messages in the queue. publisher subscriber topic with multiple consumers, each of which receives a full copy of all the messages in the topic. JMS queue obviously has the advantage of load balancing in message consumption, while a topic has the advantage of supporting multiple subscribers. Now the question is how we combine JMS queue and topic into a single message model(without a separate queue and topic) with the advantage of both load balancing and multiple subscribers. With the introduction of group id , this objective is achieved in kafka. Specifically, a kafka consumer group is composed of one or more consumers with the same group id , and each consumes a subset of the messages based on kafka topic partition. Moreover...
Kafka Message and Data Consistency With its scalability and fault tolerance features, Kafak has been becoming more an more popular in large scale, real time enterprise applications. Kafka messages are published to partitions that are usually located on different nodes and consumed by multiple consumers, each of which read messages from a single partition. This raises a data consistency issue due to multiple partitions and consumers. For example, if a security in a trading system is modified twice within a very short time and the messages could be published to two different partitions. As a result, the two messages are processed by two consumers and there is no guarantee that the last message ends up in your application or your data storage. How can this issue be resolved? Kafka Key With Single Threaded Consumer Kafka message is published with a key and payload. The messages with the same key are published to the same partition that will be consumed by the same c...