Previous Next JavaScript must be enabled to correctly display this content

## 2 Kafka Middleware Setup

This topic provides the information about the kafta middleware setup.

- Zookeeper Setup
  This topic provides the systematic instructions to setup the Zookeeper.
- Kafka Setup
  This topic provides the systematic instruction to setup kafka.

### 2.1 Zookeeper Setup

This topic provides the systematic instructions to setup the Zookeeper.

Kafka uses ZooKeeper to manage the cluster. ZooKeeper is used to coordinate the brokers/cluster topology. ZooKeeper is a consistent file system for configuration information. ZooKeeper gets used for leadership election for Broker Topic Partition Leaders. Here we are going to start a node of 3 zookeeper ensemble on 3 servers each.

1. Extract the kafka installation files in `/tools/kafka` on both the servers.
2. Navigate to config folder in `/tools/kafka/conf`.
3. Duplicate the zookeeper.properties and rename it to zookeeper1.properties.
4. Open zookeeper1.properties and modify the following properties.

       DataDir= <kafka home directory>/data
       tickTime=2000
       clientPort= Zookeeper client Port value (2181)
       initLimit=10
       syncLimit=5

       server.1=<hostname>:<peer port>:<leader port>
       #1 is the id that we put in myid file.

       server.2=<hostname>:<peer port>:<leader port>
       #2 is the id that we will put in myid file of second node.

       server.3=<hostname>:<peer port>:<leader port>
       #3 is the id that we will put in myid file of third.

   Example:

   `tickTime=2000`

   `initLimit=5`

   `syncLimit=2`

   `clientPort=2181`

   `dataDir=/tmp/zookeeper-oblm/zookeeper-node1`

   `server.1=server1-IP:2666:3666`

   `server.2=server2-IP:2667:3667`

   `server.3=server3-IP:2668:3668`

   Note:

   Update the IP value with the respective server IP.
5. Duplicate the zookeeper.properties file and rename it as zookeeper2.properties and zookeeper3.properties in the same directory on Server 2 and Server 3 respectively, (other names can also be used). These configuration files are used for each of the zookeeper nodes.
6. Open zookeeper2.properties and modify the following properties.

   `clientPort=2182`

   `dataDir=/tmp/zookeeper-oblm/zookeeper-node2`

   `server.1=server1-IP:2666:3666`

   `server.2=server2-IP:2667:3667`

   `server.3=server3-IP:2668:3668`

   Note:

   Update the IP value with the respective server IP.
7. Open zookeeper3.properties and modify the following properties.

   `clientPort=2183`

   `dataDir=/tmp/zookeeper-oblm/zookeeper-node3`

   `server.1=server1-IP:2666:3666`

   `server.2=server2-IP:2667:3667`

   `server.3=server3-IP:2668:3668`

   Note:

   Update the IP value with the respective server IP.
8. Copy the zookeeper1.properties, zookeeper2.properties and zookeeper3.properties and paste it in the local.
9. Open the directory /tmp/zookeeper-oblm/zookeeper-node1 on server 1 and create a file named myid, open with text editor and write 1, save and close.
10. Open the directory /tmp/zookeeper-oblm/zookeeper-node2 on server 2 and create a file named myid, open with text editor and write 2, save and close.
11. Open the directory /tmp/zookeeper-oblm/zookeeper-node3 on server 3 and create a file named myid, open with text editor and write 3, save and close.
12. Run the command to start the zookeeper nodes.
    On Server 1:

    `./bin/zookeeper-server-start.sh ./config/zookeeper1.properties`

    On Server 2:

    `./bin/zookeeper-server-start.sh ./config/zookeeper2.properties`

    On Server 3:

    `./bin/zookeeper-server-start.sh ./config/zookeeper3.properties`

Parent topic: Kafka Middleware Setup

### 2.2 Kafka Setup

This topic provides the systematic instruction to setup kafka.

1. Extract the kafka installation file in `/tools/kafka` on both the servers.
2. Navigate to config folder in Apache Kafka (`/tools/kafka/config`).
3. Duplicate the server.properties from config folder and rename it to server1.properties.
4. Open server1.properties and modify the following properties.

       broker.id= (Unique Integer which identifies the kafka broker in the cluster.
       listeners=PLAINTEXT://<hostname>:<Kafka broker listen port(9092)>
       log.dirs=<Kafka home directory>/logs
       log.retention.hours= <The number of hours to keep a log file
       before deleting it (in hours), tertiary to log.retention.ms property>
       log.retention.bytes= <The maximum size of the log before deleting it>
       log.segement.bytes= <The maximum size of a single log file>
       log.retention.check.interval.ms= <The frequency in milliseconds that
       the log cleaner checks whether any log is eligible for deletion>
       zookeeper.connect=<zookeeper_hostname_1>:<zookeeper_client_port>,
       <zookeeper_hostname_2>:<zookeeper_client_port>,<zookeeper_hostname_3>:<zookeeper_client_port>

   Example:

   `broker.id=0`

   `port=9092`

   `log.dirs=/tmp/kafka-oblm/logs-node1`

   `zookeeper.connect=server1-IP:2181,server2-IP:2182,
   server3-IP:2183`

   `num.partitions=1`

   `min.insync.replicas=2`

   `default.replication.factor=3`

   `offsets.topic.min.insync.replicas=2`

   `offsets.topic.replication.factor=3`

   `transaction.state.log.replication.factor=3`

   `transaction.state.log.min.isr=2`

   Note:

   If the Apache Zookeeper is on different server, then change the zookeeper.connect property. i.e., update the highlighted value for the respective server IPs. min.insync.replicas: A typical configuration is replication-factor minus 1.
5. Duplicate the server.properties into the same directory and rename it to server2.properties on server 2 and server3.properties on server 3.
6. Open server2.properties and modify the following properties.

   `broker.id=1`

   `log.dirs=/tmp/kafka-oblm/logs-node2`

   `listeners=PLAINTEXT://<hostname>:<Kafka broker listen port(9093)>`

   Note:

   By default, Apache Kafka will run on port 9092 and Apache Zookeeper will run on port 2181.
7. Open server3.properties and modify the following properties.

   `broker.id=2`

   `log.dirs=/tmp/kafka-oblm/logs-node3`

   `listeners=PLAINTEXT://<hostname>:<Kafka broker listen port(9094)>`

   Note:

   By default, Apache Kafka will run on port 9092 and Apache Zookeeper will run on port 2181.
8. Copy the server1.properties, server2.properties, server3.properties and paste it in local.
9. To run Kafka brokers, change path to `/tools/kafka` directory and run the following command in separate terminals.
   On Server 1:

   `nohup ./bin/kafka-server-start.sh config/server1.properties`

   On Server 2:

   `nohup ./bin/kafka-server-start.sh config/server2.properties`

   On Server 3:

   `nohup ./bin/kafka-server-start.sh config/server3.properties`

10. The values set for Logs is under the segment: “Log Retention Policy” in server*.properties file attached in the document. The values set under this segment are defaults from Apache.
11. At present, kafka takes the default value for message size as: `message.max.bytes=1000012`.
12. Add and update this field in server*.properties for increasing based on requirement.
13. To add compression type for all data generated by the producer, add the following property in server*.properties file.
    `compression.type=none`

    Note:

    The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd.

Parent topic: Kafka Middleware Setup