What are Topics and Partitions in Kafka?

What is a Topic?

A Topic is Kafka's fundamental building block for organizing messages. It's essentially a feed or channel where messages flow through. If Kafka were a post office, Topics would be like different mailboxes, each dedicated to a specific type of message.

What is a Partition?

Each Topic can be divided into multiple Partitions, which is a key feature for scalability. Think of it as splitting a busy highway into multiple lanes. Here's why Partitions are important:

1. Parallel Processing - Each Partition operates independently, similar to multiple CPU cores

2. Load Distribution - Data is spread across your cluster, preventing single-server bottlenecks

3. High Throughput - Multiple Partitions enable concurrent operations for better performance

Partition Storage Model

Topic: "Order Messages"
├── Partition 0: [Order1] -> [Order2] -> [Order3]
├── Partition 1: [Order4] -> [Order5] -> [Order6]
└── Partition 2: [Order7] -> [Order8] -> [Order9]

Each message in a Partition receives a unique offset number, which serves as its sequential identifier within that Partition.

Partition Replication Mechanism

For fault tolerance, Kafka maintains multiple copies of each Partition:

• Leader Replica - The primary copy that handles all read/write operations

• Follower Replicas - Backup copies that maintain synchronization and provide failover capability

Partition 0
├── Leader (Server 1)
├── Follower (Server 2)
└── Follower (Server 3)

Producer Assignment Strategies

Producers use several strategies to distribute messages across Partitions:

1. Round-Robin - Distributes messages evenly across Partitions

2. Key-Based - Routes messages with the same key to the same Partition

3. Custom Logic - Implements specific routing rules based on business requirements

Consumer Reading Patterns

Consumer groups coordinate Partition reading through different assignment strategies:

1. Range Assignment - Allocates continuous Partition ranges to consumers

2. Round-Robin Assignment - Distributes Partitions evenly across consumers

3. Sticky Assignment - Maintains stable assignments to minimize rebalancing overhead

Practical Recommendations

1. Partition Sizing Guidelines

   • Calculate your expected message volume

   • Consider your infrastructure capacity

   • Formula: Partition count = (Target throughput/sec) ÷ (Single partition throughput)

2. Important Considerations

   • Each Partition requires system resources

   • Adding Partitions is straightforward, but removal is complex

   • Excessive Partitions can impact cluster stability

3. Key Metrics to Watch

   • Consumer lag measurements

   • Replica synchronization status

   • Partition load distribution

Summary

Proper Topic and Partition design is fundamental to a well-performing Kafka deployment. Consider your specific use case, plan your capacity requirements, and choose configurations that align with your performance needs.

Visit Message Queue Essentials to actively practice more Kafka interview questions.