📨

Apache Kafka

Apache Kafka — Complete Event Streaming Platform PRODUCERS .NET Producer Confluent.Kafka Async Produce await Produce Idempotence enable.idempotence Partitioning Strategies Round-robin · Key-based · Custom KAFKA CLUSTER Broker Nodes Broker 1 (Leader) · Broker 2 (Follower) · Broker 3 (Follower) · Controller (KRaft) Topic: orders Partitions: 0,1,2,3 Replication: 3 Topic: payments Partitions: 0,1,2 Replication: 3 Topic: events Partitions: 0,1 Replication: 2 Internal Topics: __consumer_offsets · __transaction_state Stored in log · Compacted · Single partition Schema Registry Avro / JSON / Protobuf KRaft / ZooKeeper Controller · Quorum · Leader Election STREAM PROCESSING Kafka Streams DSL · Processor API Windowing · Joins KSQL / ksqlDB SQL on Streams Pull / Push Queries KAFKA CONNECT Source Connectors Database · MQ · File Sink Connectors S3 · Elastic · JDBC CONSUMERS Consumer Groups Group coordination · Rebalancing Offset Management auto.offset.reset · commit Serialization Avro · JSON · Protobuf Error Handling Dead Letter · Retry · Poison Pill EVENT-DRIVEN ARCHITECTURE & OPERATIONS Event Sourcing Append-only log State reconstruction CQRS Command / Query split Separate read/write models Monitoring JMX · Cruise Control Lag · Throughput · Alerts