Running Apache Kafka and Zookeeper on Your Server

Prerequisites

A server running a Linux-based operating system (e.g., Ubuntu, CentOS).
Access to the terminal with sudo privileges.
Basic understanding of terminal commands.

Installing Java

Apache Kafka requires Java to run. Make sure you have Java installed:

Update your package index:
```
sudo apt update
```
Install OpenJDK:
```
sudo apt install openjdk-17-jdk
```
Verify the installation:
```
java -version
```

Installing Zookeeper

Zookeeper is required for managing Kafka brokers. Here’s how to install it:

Download Zookeeper:

wget https://downloads.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz

Extract the downloaded file:

tar -xzf apache-zookeeper-3.8.0-bin.tar.gz

Move to /opt:

sudo mv apache-zookeeper-3.8.0-bin /opt/zookeeper

Create a configuration file:

sudo nano /opt/zookeeper/conf/zoo.cfg

Add the following configuration:

tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=60

Create the data directory:
```
sudo mkdir /var/lib/zookeeper
```

Installing Kafka

Download Kafka:

wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz

Extract the downloaded file:
```
tar -xzf kafka_2.13-3.4.0.tgz
```
Move to /opt:
```
sudo mv kafka_2.13-3.4.0 /opt/kafka
```

Configuring Zookeeper

Make sure Zookeeper is configured correctly. The configuration file was created earlier. You may edit it if needed.

Configuring Kafka

Create a configuration file for Kafka:

sudo nano /opt/kafka/config/server.properties

Add or modify the following configurations:

broker.id=0
listeners=PLAINTEXT://:9092
log.dirs=/var/lib/kafka/logs
zookeeper.connect=localhost:2181

Create the logs directory:
```
sudo mkdir -p /var/lib/kafka/logs
```

Setting Up Systemd Services

Create a Zookeeper Service

Create the service file:

sudo nano /etc/systemd/system/zookeeper.service

Add the following content:

[Unit]
Description=Apache Zookeeper
After=network.target
 
[Service]
User=your_username
ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg
ExecStop=/opt/zookeeper/bin/zkServer.sh stop
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

Create a Kafka Service

Create the service file:

sudo nano /etc/systemd/system/kafka.service

Add the following content:

[Unit]
Description=Apache Kafka
After=zookeeper.service
 
[Service]
User=your_username
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

Starting Services

Reload systemd to recognize the new services:
```
sudo systemctl daemon-reload
```
Start Zookeeper:
```
sudo systemctl start zookeeper
```
Start Kafka:
```
sudo systemctl start kafka
```

Enable both services to start on boot:

sudo systemctl enable zookeeper
sudo systemctl enable kafka

Verifying the Installation

Check the status of Zookeeper:
```
sudo systemctl status zookeeper
```
Check the status of Kafka:
```
sudo systemctl status kafka
```

Create a test topic:

/opt/kafka/bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

List topics to confirm:

/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Troubleshooting

Check logs if Kafka or Zookeeper fail to start:
- Zookeeper logs: /opt/zookeeper/logs/zookeeper.out
- Kafka logs: /opt/kafka/logs/server.log
Ensure the correct version of Java is installed.
Verify that no other services are using the same ports (2181 for Zookeeper, 9092 for Kafka).

Conclusion

You have successfully installed and configured Apache Kafka and Zookeeper on your server. Both services are set up to run automatically at startup, ensuring reliable operation. You can now start producing and consuming messages using Kafka.

Readme Docker Compose.yaml