⚡ Event-Driven Architecture
Apache Kafka Architecture Deep Dive
Apache Kafka's distributed log architecture — Topics, Partitions, Consumer Groups, and Offsets
Kafka is not a traditional message queue — it's a distributed commit log. Messages are stored persistently (not deleted on consume), ordered within partitions, and replayable. This makes it the backbone of Event Sourcing, Stream Processing, and Cross-system Data Synchronization.
Key Concepts
Persistent Distributed Log
Messages are appended to a partitioned log and retained for days/weeks (configurable). Any consumer can replay from any offset — rebuilt failed microservices can re-process all historical events.
Partitions = Parallelism
A Topic has N partitions. Each partition is consumed by exactly one consumer in a Consumer Group. 10 partitions → 10 consumers process in parallel. This is how Kafka achieves millions of messages/second.
Consumer Group Offsets
Each Consumer Group maintains its own read offset per partition. Multiple independent consumer groups read the same topic without interfering. Order Service and Email Service both read OrderCreated independently.
Partition Key for Ordering
Messages with the same key always go to the same partition = guaranteed ordering. Order events for order #1001 all go to partition 3, ensuring they're processed in sequence.
System Architecture Blueprint
Consumer Group Sequence
Deep Dive: Offsets and Partitions
Unlike a standard Queue (like RabbitMQ) which deletes a message after it's acknowledged, Kafka stores messages durably for a retention period. This fundamental difference unlocks enormous power:
- **Consumer Independence:** Consumer Group A (e.g. Orders) and Consumer Group B (e.g. Analytics) both read the same Topic but maintain their own independent Offsets (red pointers in the diagram).
- **Replayability:** If the Analytics service has a bug and corrupts its database, you can simply reset its Offset to 0. It will replay the entire history of events from the beginning of time and rebuild the database.
- **Scaling (Partitions):** A Topic is split into Partitions. The number of partitions dictates your maximum concurrency. If a topic has 3 partitions, a consumer group can have at most 3 consumers reading in parallel (as shown above).
Kafka vs RabbitMQ
| Feature | Apache Kafka | RabbitMQ |
|---|---|---|
| Architecture | Pull-based distributed log | Push-based smart broker |
| Message retention | Days to weeks (configurable) | Deleted after consumer acks |
| Replay | Yes — rewind to any offset | No — messages are consumed and gone |
| Throughput | 1M+ messages/sec per node | ~50k messages/sec |
| Consumer model | Consumer groups pull at own pace | Broker pushes to registered consumers |
| Use case | Event sourcing, stream processing, audit log | Task queues, RPC patterns, complex routing |
| Learning curve | High (Zookeeper/KRaft, partition tuning) | Medium (AMQP, exchanges, bindings) |
NestJS Kafka Producer
order.service.tstypescript
1@Injectable()
2export class OrderService {
3 constructor(
4 @Inject('KAFKA_SERVICE')
5 private readonly kafka: ClientKafka,
6 ) {}
7
8 async placeOrder(dto: CreateOrderDto): Promise<Order> {
9 // 1. Write to DB (local transaction) - Database ထဲ ရေးသွင်းခြင်း
10 const order = await this.orderRepo.create(dto);
11
12 // 2. Publish event — fire and forget - Event ကို လွှတ်တင်ခြင်း
13 this.kafka.emit('order.created', {
14 key: order.id, // Partition key — အော်ဒါတစ်ခုတည်း၏ Event များ အားလုံး Partition တစ်ခုတည်းသို့ သွားစေရန်
15 value: {
16 orderId: order.id,
17 customerId: dto.customerId,
18 items: dto.items,
19 totalAmount: order.totalAmount,
20 timestamp: new Date().toISOString(),
21 }
22 });
23
24 return order; // Return immediately — don't wait for downstream (အဖြေ ချက်ချင်းပြန်ပေးသည်၊ ဆက်စောင့်မနေပါ)
25 }
26}
27
28// Consumer in a different microservice - အခြား Microservice မှ Consumer
29@Controller()
30export class InventoryConsumer {
31 @EventPattern('order.created')
32 async handleOrderCreated(@Payload() event: OrderCreatedEvent) {
33 await this.inventoryService.reserveStock(event.items);
34 }
35}💡
Senior Architect Insight: Partition count is a one-time decision for a topic — you can only increase, never decrease. Start with 2-3x your expected max consumer parallelism. 12 partitions lets you scale to 12 consumer instances per group. Always set replication.factor >= 2 in production.