This topic provides the systematic instructions for Kafka cluster setup.

1. Untar/unzip the kafka binary and move them to the folder that will be the kafka home directory.
2. User can create three folders in the directory where kafka folder is present. User can name them as node1, node2, node3. These folders are for storing kafka and zookeeper logs. Within each of these folders, create two folders called kafka-logs and zookeeper.

   Note:

   Please skip the above two steps if already completed during Zookeeper setup.

3. Navigate to <kafka home directory>/kafka_2.13-3.7.0/Inside kafka/config folder, there would be a file named server.properties, copy that and create three similar files server1.properties, server2.properties, server3.properties.Edit the below lines in the <kafka home directory>/kafka_2.13-3.7.0/config/server1.properties,server2.properties,server3.properties

   broker.id= 0 for node1, broker.id= 1 for node2, broker.id= 2 for node3(Unique Integer which identifies the kafka broker in the
   cluster.)
   listeners=PLAINTEXT://<hostname>:<Kafka broker listen port for server1 use port as 9092,server2 use port as 9093 and server3 use 9094 port.
   log.dirs=<kafka home directory>/nodeX/kafka-logs
   e.g. node1/kafka-logs
   log.retention.hours= <The number of hours to keep a log file before
   deleting it (in hours),tertiary to log.retention.ms property>
   log.retention.bytes= <The maximum size of the log before deleting it>
   log.segment.bytes= <The maximum size of a single log file>
   log.retention.check.interval.ms= <The frequency in milliseconds that
   the log cleaner checks whether any log is eligible for deletion>
   zookeeper.connect=<zookeeper_hostname_1>:<zookeeper_client_port>,<zook
   eeper_hostname_2>:<zookeeper_client_port>,<zookeeper_hostname_3>:<zook
   eeper_client_port>,
   offsets.topic.replication.factor=3
   offsets.topic.min.insync.replicas=2
   transaction.state.log.replication.factor=3
   transaction.state.log.min.isr=2
   min.insync.replicas=2

   Note:

   For the setup to work properly, atleast two kafka nodes should be up and running at every point of time.
4. To start the kafka, navigate to <kafka home directory>/kafka_2.13-3.7.0/ folder and run the below commands on each node.

   export JMX_PORT=[PORT VALUE1]
   nohup bin/kafka-server-start.sh config/server1.properties &
   

   export JMX_PORT=[PORT VALUE2]
   nohup bin/kafka-server-start.sh config/server2.properties &

   export JMX_PORT=[PORT VALUE3]
   nohup bin/kafka-server-start.sh config/server3.properties &

   The Default value of JMX Port is 9999.

   Tail the log for server status.

5. To create topic, navigate to <kafka home directory>/kafka_2.13-3.7.0/ folder and run the below command.

   /bin/kafka-topics.sh --create --bootstrap-server <hostname>:<client port>, <hostname>:<client port>, <hostname>:<client port> --replication-factor 3 --partitions 3 --topic <topic name>

6. To list the available topic on kafka server, navigate to <kafka home directory>/kafka_2.13-3.7.0/ folder and run the below command.

   ./bin/kafka-topics.sh --list --bootstrap-server <hostname>:<port>, <hostname>:<port>, <hostname>:<port>

7. To describe the topic, navigate to <kafka home directory>/kafka_2.13-3.7.0/ folder and run the below command.

   ./bin/kafka-topics.sh --describe --topic <topic name> --bootstrap-server <hostname>:<port>,<hostname>:<port>,<hostname>:<port>

8. To start a producer, navigate to <kafka home directory>/kafka_2.13-3.7.0/ folder and run the below command.

   export JMX_PORT=[PORT VALUE]//Different Value from the server
   JMX port
   
   ./bin/kafka-console-producer.sh --broker-list
   <hostname>:<port>, <hostname>:<port>, <hostname>:<port> topic <topic name>

   Note:

   By default, port is taken as 9092 for the producer.
9. To start a consumer console for viewing the received messages sent by the producer, use the following command.

   export JMX_PORT=[PORT VALUE]//Different Value from the server
   JMX port
   
   ./bin/kafka-console-consumer.sh --bootstrap-server
   <hostname>:<port>,<hostname>:<port>,<hostname>:<port> --topic <topic_name> --
   from-beginning

Configure a Standalone Kafka Instance in Cluster Mode

If there is already a standalone Kafka instance with Oracle Banking Microservices Architecture services running on it, it is expected the topics are already created in the Kafka instances. In this case, use the below steps to enable replication of messages between Kafka brokers:

10. Download and edit the increase.json to have all the topics and their replication confirmation updated.
    Example: If you have a 3 node setup, the json will look like the attached sample. It is ideal to have the number of replicas equal to the number of brokers.

    {"version":1,
      "partitions":[
         {"topic":"<Topic Name>","partition":0<if there is just one partition, else there has to be a 
         different record for each partition per topic>,"replicas":[comma separated list of broker ids]}
    ]}

11. Run the following command
    .\bin\windows\kafka-reassign-partitions.bat --bootstrap-server localhost:9092 --reassignment-json-file increase.json --execute