Previous Next JavaScript must be enabled to correctly display this content

## 2.2 Kafka Setup

This topic provides the systematic instruction to install and setup kafka.

1. Extract the kafka installation file in `/tools/kafka` on both the servers.
2. Navigate to config folder in Apache Kafka (`/tools/kafka/config`).
3. Duplicate the server.properties from config folder and rename it to server1.properties.
4. Open server1.properties and modify the following properties.

       broker.id= (Unique Integer which identifies the kafka broker in the cluster.
       listeners=PLAINTEXT://<hostname>:<Kafka broker listen port(9092)>
       log.dirs=<Kafka home directory>/logs
       log.retention.hours= <The number of hours to keep a log file
       before deleting it (in hours), tertiary to log.retention.ms property>
       log.retention.bytes= <The maximum size of the log before deleting it>
       log.segement.bytes= <The maximum size of a single log file>
       log.retention.check.interval.ms= <The frequency in milliseconds that
       the log cleaner checks whether any log is eligible for deletion>
       zookeeper.connect=<zookeeper_hostname_1>:<zookeeper_client_port>,
       <zookeeper_hostname_2>:<zookeeper_client_port>,<zookeeper_hostname_3>:<zookeeper_client_port>

   Example:

   `broker.id=0`

   `port=9092`

   `log.dirs=/tmp/kafka-oblm/logs-node1`

   `zookeeper.connect=server1-IP:2181,server2-IP:2182`

   `num.partitions=2`

   `min.insync.replicas=1`

   `default.replication.factor=2`

   `offsets.topic.replication.factor=2`

   `transaction.state.log.replication.factor=2`

   `transaction.state.log.min.isr=1`

   Note:

   If the Apache Zookeeper is on different server, then change the zookeeper.connect property. i.e., update the highlighted value for the respective server IPs. min.insync.replicas: A typical configuration is replication-factor minus 1.
5. Duplicate the server.properties into the same directory and rename it to server2.properties on server 2.
6. Open server2.properties and modify the following properties.

   `broker.id=1`

   `broker.id=1`

   `log.dirs=/tmp/kafka-oblm/logs-node2`

   Note:

   By default, Apache Kafka will run on port 9092 and Apache Zookeeper will run on port 2181.
7. Copy the server1.properties, server2.properties, server3.properties and paste it in local.
8. To run Kafka brokers, change path to `/tools/kafka` directory and run the following command in separate terminals.
   On Server 1:

   `nohup ./bin/kafka-server-start.sh config/server1.properties`

   On Server 2:

   `nohup ./bin/kafka-server-start.sh config/server2.properties`

9. The values set for Logs is under the segment: “Log Retention Policy” in server*.properties file attached in the document. The values set under this segment are defaults from Apache.
10. At present, kafka takes the default value for message size as: `message.max.bytes=1000012`.
11. Add and update this field in server*.properties for increasing based on requirement.
12. To add compression type for all data generated by the producer, add the following property in server*.properties file.
    `compression.type=none`

    Note:

    The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd.

Parent topic: Kafka Middleware Setup