The Apache Kafka is an open source stream processing platform developed by the Apache Software Foundation written in Scala and Java Programming. And It is designed to handle real time data feeds with high throughput, low latency, and durability. The Kafka is used to build real time data pipelines and streaming applications that can process data stream at scale. Running Kafka on Docker allows us to quickly set up a scalable Kafka environment, isolating it from our local machines environment and ensuring consistency across different development environments.

Terminologies

Here we explain some Terminologies for understanding the Apache Kafka.

1. Kafka Broker: A Kafka server that stores and serves data.
2. Topic: A category to which records are published.
3. Partition: A division of topics data. The Kafka splits each topic into one or more partitions.
4. Producer: AN application that sends records to a Kafka topic.
5. Consumer: An application that reads records from a Kafka topic.
6. Zookeeper: A centralized service for maintain configuration information naming providing distributed synchronization and providing group services.

Step by Step Process to Set up Kafka on Docker

Here we explain a step by step process to set up Kafka on Docker with examples and outputs for your reference.

Step 1: Install Docker

First ensure the Docker is installed on your machine. Otherwise you can download Docker from official website and follow the installation instructions for your operating system.

Step 2: Create a Docker Network

Create a Docker network to allow the Kafka and Zookeeper container to communicate with each other. Below we provide the docker command for this.

docker-compose.yml

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Run below command

docker compose -f docker-compose.yml up

docker network create kafka-network

docker

Step 3: Start Zookeeper

Run a Zookeeper container using Docker and The Zookeeper is a prerequisite for running Kafka.

docker run -d --name zookeeper --network kafka-network -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:latest

Step 4: Start Kafka

Run a Kafka container and link it to the Zookeeper container.

docker run -d --name kafka --network kafka-network -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 -e KAFKA_BROKER_ID=1 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentinc/cp-kafka:latest

Step 5: Verify Kafka Installation

To ensure Kafka is running correctly and list all topics.

docker exec -it kafka kafka-topics --list --zookeeper zookeeper:2181

Example

Here we provide a example for your understanding purpose. Now creating a topic and sending a message.

Create a Topic

docker exec -it kafka kafka-topics --create --topic test-topic --partitions 1 --replication-factor 1 --if-not-exists --zookeeper zookeeper:2181

Send a Message

docker exec -it kafka kafka-console-producer --broker-list localhost:9092 --topic test-topic

Consume the Message

docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic test-topic --from-beginning

Output

Frequently Asked Questions

1. Why do we need Zookeeper for Kafka?

Answer: Zookeeper manages and coordinates the Kafka brokers. It helps in leader election, configuration management and synchronizing the state between brokers.

2. Can I run Kafka without Docker?

Answer: Yes, you can run Kafka directly on your machine by downloading the Kafka binaries and setting it up manually. Docker simplifies the setup process and ensures consistency across environments.

3. What are the default ports used by Kafka and Zookeeper?

Answer: Kafka uses port 9092 by default while Zookeeper uses port 2181.

4. How can I scale Kafka using Docker?

Answer: You can scale Kafka by running multiple Kafka broker container. Each broker should have a unique broker ID and advertised listener.

5. What is the purpose of the kafka-network in the setup process?

Answer: The kafka-network Docker network allows the Kafka and Zookeeper containers to communicate with each other ensuring they are on the sane virtual network.

Conclusion

Setting up Kafka on Docker is a streamlined and efficient way to build a scalable and consistent Kafka environment by leveraging Dockers containerization capabilities and you can isolate your Kafka setup from the local machine. Ensuring consistency across different environments and simplifying the management of dependencies and configuration. Implementing Kafka on Docker not only saves time in setup but also offers the benefits of containerized applications such as easy scalability, portability and maintainability making it an excellent choice for developers.