Create Next App

An Interactive Introduction to Apache Kafka

This guide transforms the standard presentation on Kafka into an interactive experience. Explore the fundamental concepts, architecture, and use cases of the powerful distributed streaming platform at your own pace.

What is Kafka?

Definition: A distributed streaming platform that allows you to publish, subscribe to, store, and process streams of records in real-time.

Origin: Originally developed at LinkedIn and later open-sourced as a project of the Apache Software Foundation.

Core Idea: It functions like a supercharged messaging system, enabling high-throughput, fault-tolerant data pipelines and streaming applications.

Core Concepts

Kafka's power comes from a few simple but powerful ideas. Click on each card below to learn about the key components that make up the Kafka ecosystem. Understanding these concepts is the first step to mastering Kafka.

How It Works: The Data Flow

At its heart, Kafka facilitates a simple flow of data from producers to consumers through a central, durable log. This section provides an interactive visualization of that process. Click on each element in the diagram to see its role in the system.

📬

Producer

🗂️

Broker (Topic)

💻

Consumer

Click an element above to learn more about its function.

Features & Use Cases

Kafka's unique architecture provides a powerful set of features, making it suitable for a wide variety of applications. Use the tabs below to explore its key characteristics and see how companies are using it to build modern data systems.

Getting Started

Ready to try Kafka yourself? This section provides a high-level overview of the setup process and conceptual code examples for producing and consuming messages. Use the tabs to navigate between the setup guide and the code samples.

Basic Environment Setup

Prerequisites: Ensure you have a recent version of the Java Development Kit (JDK) installed on your system.
Download Kafka: Get the latest binary release from the official Apache Kafka website.
Start ZooKeeper: Kafka uses ZooKeeper for cluster management. Start the ZooKeeper server using the script provided with Kafka.
Start Kafka Broker: In a separate terminal, start the Kafka broker server.
Create a Topic: Use the command-line tools to create your first topic. This is the channel your messages will be sent to.
Ready to Go: Your single-node Kafka cluster is now running and ready to send and receive messages!

// Configure producer properties
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// Create a producer
Producer<String, String> producer = new KafkaProducer<>(props);

// Send a message
producer.send(new ProducerRecord<>("my-topic", "key", "Hello, Kafka!"));

// Close the producer
producer.close();

// Configure consumer properties
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "my-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

// Create a consumer
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

// Subscribe to a topic
consumer.subscribe(Collections.singletonList("my-topic"));

// Poll for new messages
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        System.out.printf("Received message: value = %s%n", record.value());
    }
}