• What is it?
• Zookeeper
• How Kafka uses Zookeeper?

What is it?

Leader election is a process that is used to choose a leader or coordinator from among a group of nodes in a distributed system.

The leader is responsible for coordinating the actions of the other nodes and ensuring that the group functions correctly.

Zookeeper

Apache Zookeeper is a distributed coordination service that is used to manage distributed systems.

It provides a centralized service for:

• Storing and accessing config data
• Leader election (ZooKeeper Leader Election Protocol)
• It allows nodes to join or leave the group at any time.
• Distributed sync

It is different from Kubernetes, because K8s is focused on automating the deployment, scaling, and management of containerized applications.

How Kafka uses Zookeeper?

Kafka relies on Apache Zookeeper to store and manage the metadata for Kafka topics and brokers.

Zookeeper is used to store metadata about the Kafka cluster, including the list of brokers in the cluster, the number of replicas of each topic, and the mapping of partition topics & replicas to brokers.

When a Kafka producer publishes a record to a topic, it first determines the set of brokers that are responsible for the topic by querying Zookeeper. The producer then sends the record to one of the brokers in the set, which is responsible for forwarding the record to the appropriate partition within the topic.

Similarly, when a Kafka consumer subscribes to a topic, it first determines the set of brokers that are responsible for the topic by querying Zookeeper. The consumer then reads the records for the topic by fetching them from the appropriate broker and partition.

When a Kafka producer publishes a record to a topic, it relies on the broker to which it is connected to retrieve the metadata for the topic from Zookeeper and determine the set of brokers that are responsible for the topic. Same with consumer.

Overall, Zookeeper plays a critical role in the operation of a Kafka cluster by storing and managing the metadata that is needed to coordinate the actions of the producers, consumers, and brokers.