How Kafka workflow works from 1000 feet view ?

Here is the high level workflow explaining how Kafka cluster works at 1000 feet

Producer Side

 1. Say Kafka cluster have one Topic with three partition(P1,P2,P3) on three nodes.
 2. Each partition has two replica(R11, R12),(R21, R22),(R31, R32) on other nodes.
 3. Cluster has a zookeeper.
 4. On kafka startup, kafka elects the leader for each partition say out of P1, R11, R12 , P1 is the leader whereas R11 and R12 are followers.
 5. Producer connects to Kafka cluster which can be any node in cluster and  get all metadata info on client side through kafka client side library.
 6. Producer connects to leader partition  as leader is responsible for all read and writes.
 7. Partition can be either specified/or selected at run time
based on message key and no of partition/or round robin fashion.
 8. Producer can go for any acknowledgement level i.e. fire and forget/leader acknowledgement/all insynch replica(ISR) acknowledgement. As explained here



 There are two common strategies for keeping replicas in sync, primary-backup replication and quorum-based replication.In primary-backup replication, the leader waits until the write completes
 on every replica in the group before acknowledging the client. If one of the replicas is down, the leader drops it from the current group and continues to write to the remaining replicas. A failed replica is allowed to rejoin the group if it comes back and catches up with the leader. With f replicas, primary-backup replication can tolerate f-1 failures.


In the quorum-based approach, the leader waits until a write completes on a majority of the replicas. The size of the replica group  doesn’t change even when some replicas are down. If there are 2f+1
 replicas, quorum-based replication can tolerate f replica failures. If  the leader fails, it needs at least f+1 replicas to elect a new  leader.

  9. In case leader fails, zookeeper will select new leader from remaining ISR here (after comparing logs either based on primary-backup replication and quorum-based replication) . See Kafka and Quorum section


Consumer side

 1. Say I have nine consumers all belonging to single consumer group.
 2. Each consumer will be assigned each specific partition out of nine partition.
 3. In case any consumer or partition dies, consumer group coordinator will trigger the re balance as explained here


What is Quorum

"Quorum" is generally applied at the time of writing and leader selection in distributed systems.
It is designed to provide the high availability with possibility of network partition.

Quorum at the time of writing

"Quorum" refers to the minimum number of nodes that must agree on a transaction before it is
considered committed from
http://zookeeper-user.578899.n2.nabble.com/Meaning-of-quorum-and-ensemble-td7581147.html  .
It generally n/2+1(3/2+1 = 2) writes. For 8 nodes it will be 5 writes.

Quorum at the time of leader selection

As long as a majority of the ensemble are up, the service will be available.
Because Zookeeper(or any other distributed system) requires a majority, it is best to use an
odd(n/2+1) number of machines. Also noting with an even number of nodes you also run the risk
of a split brain, say you had 8 nodes, and the network partitioned into 2 parts with 4 nodes on
each side. Each (4) node side would not be able to continue as they do not have quorum.

It is in conjunction with Quorum at the time of writing. As we see above that Each (4) node side
would not be able to continue . It mean at least one side

should have 5 modes which internally means at least 5 node side partition will have one latest write.