jsdvcsdhjbcsdhjbcjhd

• What is a Message Broker?

• What is Event-Driven Architecture (EDA)?

• Why Are REST APIs Not Always Enough?

• What is Apache Kafka, Why we use Kafka and Why is it Popular?

  • How scalability work in Kafka cluster?

  • How Messages Are Distributed by the Producer

  • Key Takeaways:

• Our list of brokers in ABB:

  • 📌 General Cluster Information

  • 📌 Partitions

  • 📌 Key Insights

• Topic configs:

  • Main Topic Metrics

  • Field Descriptions

• When does Kafka delete messages in a topic?

  • Scenario 1: retention.ms = 7 days, segment.ms = 14 days, segment.bytes = 1GB

  • Scenario 2: retention.ms = 7 days, segment.ms = 3 days, segment.bytes = 500MB

  • Key Benefits in This Scenario:

• Consumer Group Model in Apache Kafka: Partition Distribution Among Consumers

• What is it Offset?

  • Why Does a Consumer Commit Offsets in Kafka?

  • Committing Offsets in Kafka and __consumer_offsets

  • How It Works?

• Why Kafka?

  • Real-Time Data Streaming

  • Kafka vs Traditional Message Brokers (RabbitMQ, ActiveMQ, JMS)

  • What Does Scalability Mean in the Context of Kafka and Other Message Brokers?

• What is a Kafka Connector and Why we Use Kafka Connector?

•  How Does Kafka MongoDB Connector Work?

  • How Does the Connector Prevent Resending Old Messages After a Restart?

    • How It Works:

    • Where Is the Resume Token Stored?

• Meeting Record:

What is a Message Broker?

A Message Broker is software that:

·       Receives, stores, and transmits messages between different services.

·       Decouples producers and consumers, enabling asynchronous communication.

·       Guarantees message delivery (depending on the configuration).

·       Allows systems to scale without tightly coupling components.

·       Is a key component in Event-Driven Architecture (EDA), facilitating event transmission between services.

What is Event-Driven Architecture (EDA)?

EDA is an architectural approach where systems react to events rather than follow predefined workflows.

🔹 Event – A change in system state (e.g., "User placed an order").
🔹 Producers – Services that generate events.
🔹 Message Broker – Transmits events to other services.
🔹 Consumers – Services that react to events.

Open maxresdefault-20250207-194912.jpg

Why Are REST APIs Not Always Enough?

Traditional HTTP-based communication creates several problems:

✅ High service dependency – if one service fails, others may crash or slow down.
✅ Scalability issues – increasing load requires API load balancing.
✅ Processing speed limitations – clients have to wait for responses if the server is overloaded.

Message Brokers solve these issues by enabling asynchronous processing – services send messages, and consumers process them when ready.

Open image-20250210-102341.png

What is Apache Kafka, Why we use Kafka and Why is it Popular?

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, and scalable data processing in real time.

Originally developed at LinkedIn, Kafka has become one of the most popular solutions for event-driven architectures due to its performance, and scalability.

Open kafka-clusters-20250207-201243.webp

How scalability work in Kafka cluster?

How Kafka Balances Load Between Brokers

Open Screenshot 2025-02-10 at 14.27.55.png

✅    Kafka efficiently distributes load by using partitions within a topic. Each partition has a leader broker that handles both reads and writes, while followers replicate data for fault tolerance.

How Messages Are Distributed by the Producer

1. When a Producer sends messages to a Kafka cluster:

   • If a key is provided, Kafka applies a hashing function to determine which partition will receive the message. This ensures that messages with the same key always land in the same partition.

   • If no key is specified, Kafka distributes messages in a round-robin manner across available partitions to balance the load.

2. In this example (from the diagram):

   • The first message for Partition 0 goes to Broker 1, as it is the leader for this partition.

   • The second message for Partition 1 goes to Broker 2, where it is the leader.

3. After the leader writes the message, it is replicated to follower brokers for redundancy:

   • Partition 0 is replicated to Broker 2.

   • Partition 1 is replicated to Broker 1.

Key Takeaways:

• Kafka balances load by distributing partitions a  mong brokers.

• The Producer does not directly choose a broker; it sends data to partitions, and Kafka ensures even distribution.

• If the number of partitions exceeds the number of brokers, some brokers will handle multiple leader partitions.

• Consumers always read from leader partitions to maintain data consistency.

This architecture ensures high availability, fault tolerance, and efficient load balancing in a Kafka cluster.