Previous Next JavaScript must be enabled to correctly display this content

## Deploying an Arbiter Node Enabled Topology

An `Arbiter Node` is a service that supports write availability when the store replication factor is two and a single Replication Node becomes unavailable. The role of an `Arbiter Node` is to participate in elections and respond to acknowledge requests if one of the two Replication Nodes in a shard becomes unavailable.

Arbiter Nodes are automatically configured in a topology if the store replication factor is two and a primary zone is configured to host `Arbiter Nodes`.

For example, suppose a store consists of a primary zone, "Manhattan" with two Storage Nodes deployed in the same shard. In this example, an Arbiter Node is deployed in the third Storage Node (capacity = 0) in order to provide write availability even if one of the two Replication Nodes in the shard becomes unavailable.

Note:

Durability.ReplicaAckPolicy must be set to SIMPLE_MAJORITY, so that writes can succeed if a Replication Node becomes unavailable in a shard. For more information on `ReplicaAckPolicy`, see this Javadoc page.

1. Create, start, and configure the store. Note that a Storage Node with capacity equal to zero is deployed, which will host the Arbiter Node.

   - Create the store:

         java -Xmx64m -Xms64m \
         -jar kv/lib/kvstore.jar makebootconfig \
         -root KVROOT \
         -host node01 \
         -port 8000 \
         -harange 8010,8020 \
         -capacity 1

         java -Xmx64m -Xms64m \
         -jar kv/lib/kvstore.jar makebootconfig \
         -root KVROOT \
         -host node02 \
         -port 9000 \
         -harange 9010,9020 \
         -capacity 1

         java -Xmx64m -Xms64m \
         -jar kv/lib/kvstore.jar makebootconfig \
         -root KVROOT \
         -host node03 \
         -port 10000 \
         -harange 1000,10020 \
         -capacity 0 \

   - Create and copy the security directories:

         java -Xmx64m -Xms64m \
         -jar kv/lib/kvstore.jar
         securityconfig \
         config create -root KVROOT -kspwd password
         Created files
         KVROOT/security/security.xml
         KVROOT/security/store.keys
         KVROOT/security/store.trust
         KVROOT/security/client.trust
         KVROOT/security/client.security
         KVROOT/security/store.passwd (Generated in CE version)
         KVROOT/security/store.wallet/cwallet.sso (Generated in EE version)

         Created
         scp -r KVROOT/security node02:KVROOT/
         scp -r KVROOT/security node03:KVROOT/

   - Start the store by running the following command on each Storage Node:

         java -Xmx64m -Xms64m -jar KVHOME/lib/kvstore.jar \
         start -root KVROOT &

2. Load the following script `conf.txt` to deploy the zone, admin and Storage Nodes. To host an Arbiter Node, the zone must be primary and should have the `-arbiters` flag set.

       ssh node01
       java -Xmx64m -Xms64m -jar KVHOME/lib/kvstore.jar runadmin \
       -port 8000 -host node01 load -file conf.txt \
       -security KVROOT/security/client.security

   The file, `conf.txt`, would then contain content like this:

       ### Begin Script ###
       plan deploy-zone -name "Manhattan" -type primary -arbiters -rf 2 -wait
       plan deploy-sn -zn zn1 -host node01 -port 8000 -wait
       pool create -name SNs
       pool join -name SNs -sn sn1
       plan deploy-admin -sn sn1 -port 8001 -wait
       plan deploy-sn -zn zn1 -host node02 -port 9000 -wait
       pool join -name SNs -sn sn2
       plan deploy-sn -zn zn1 -host node03 -port 10000 -wait
       pool join -name SNs -sn sn3
       ### End Script ###

3. Create a topology, preview it, and then deploy it:

       kv-> topology create -name arbTopo -pool SNs -partitions 300

   Output:

       Created: arbTopo

       kv-> topology preview -name arbTopo

   Output:

       Topology transformation from current deployed topology to arbTopo:
       Create 1 shard
       Create 2 RNs
       Create 300 partitions
       Create 1 AN

       shard rg1
         2 new RNs : rg1-rn1 rg1-rn2
         1 new AN : rg1-an1
         300 new partitions

       kv-> plan deploy-topology -name arbTopo -wait

   Output:

       Executed plan 6, waiting for completion...
       Plan 6 ended successfully

4. Verify that the Arbiter Node is running.

       kv-> verify configuration

   Output:

       Verify: starting verification of store mystore
       based upon topology sequence #308
       300 partitions and 3 storage nodes
       Time: 2024-04-05 06:57:10 UTC   Version: 24.1.11
       See node01:KVROOT/mystore/log/mystore_{0..N}.log
       for progress messages
       Verify: Shard Status: healthy:1 writable-degraded:0
                                                    read-only:0 offline:0
       Verify: Admin Status: healthy
       Verify: Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false]
       RN Status: online:2 offline:0 maxDelayMillis:6 maxCatchupTimeSecs:0
       Verify: == checking storage node sn1 ==
       Verify: Storage Node [sn1] on node01:8000
       Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false]
       Status: RUNNING
       Ver: 24.1.11 2024-04-05 09:33:45 UTC  Build id: a72484b8b33c
       Verify:         Admin [admin1]          Status: RUNNING,MASTER
       Verify:         Rep Node [rg1-rn1]
       Status: RUNNING,MASTER sequenceNumber:635 haPort:8011 available storage size:11 GB
       Verify: == checking storage node sn2 ==
       Verify: Storage Node [sn2] on node02:9000
       Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false]
       Status: RUNNING
       Ver: 24.1.11 2024-04-05 09:33:45 UTC  Build id: a72484b8b33c
       Verify:         Rep Node [rg1-rn2]
       Status: RUNNING,REPLICA
       sequenceNumber:635 haPort:9010 available storage size:12 GB delayMillis:6 catchupTimeSecs:0
       Verify: == checking storage node sn3 ==
       Verify: Storage Node [sn3] on node03:10000
       Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false]
       Status: RUNNING
       Ver: 24.1.11 2024-04-05 09:33:45 UTC  Build id: a72484b8b33c
       Verify:         Arb Node [rg1-an1]
       Status: RUNNING,REPLICA sequenceNumber:0 haPort:node03:10010
       ...

5. Now suppose node02 is unreachable. Verify this by using `verify configuration`:

       kv-> verify configuration

   Output:

       Verify: starting verification of store mystore
       based upon topology sequence #308
       300 partitions and 3 storage nodes
       Time: 2024-04-05 06:57:10 UTC   Version: 24.1.11
       See node01:KVROOT/mystore/log/mystore_{0..N}.log
       for progress messages
       Verify: Shard Status: healthy:0 writable-degraded:1
                                                      read-only:0 offline:0
       Verify: Admin Status: healthy
       Verify:
             Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false]
       RN Status: online:1 offline:1
       Verify: == checking storage node sn1 ==
       Verify: Storage Node [sn1] on node01:8000
       Zone:
           [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false]
        Status: RUNNING
        Ver: 24.1.11 2024-04-05 09:33:45 UTC  Build id: a72484b8b33c
       Verify:         Admin [admin1]          Status: RUNNING,MASTER
       Verify:         Rep Node [rg1-rn1]
       Status: RUNNING,MASTER sequenceNumber:901 haPort:8011 available storage size:12 GB
       Verify: == checking storage node sn2 ==
       Verify:         sn2: ping() failed for sn2 : Unable to connect to
       the storage node agent at host node02, port 9000, which may not be
       running; nested exception is:
               java.rmi.ConnectException: Connection refused to
               host: node02; nested exception is:
               java.net.ConnectException: Connection refused
       Verify: Storage Node [sn2] on node02:9000
       Zone:
           [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false] UNREACHABLE
       Verify:         rg1-rn2: ping() failed for rg1-rn2 : Unable to connect
       to the storage node agent at host node02, port 9000, which may not
       be running; nested exception is:
               java.rmi.ConnectException: Connection refused to host: node02;
               nested exception is:
               java.net.ConnectException: Connection refused
       Verify:         Rep Node [rg1-rn2]      Status: UNREACHABLE
       Verify: == checking storage node sn3 ==
       Verify: Storage Node [sn3] on node03:10000
        Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=true masterAffinity=false]
        Status: RUNNING
       Ver: 24.1.11 2024-04-05 09:33:45 UTC  Build id: a72484b8b33c
       Verify:         Arb Node [rg1-an1]
       Status: RUNNING,REPLICA sequenceNumber:901 haPort:node03:10010 available storage size:16 GB delayMillis:? catchupTimeSecs:?
       Verification complete, 3 violations, 0 notes found.
       Verification violation: [rg1-rn2]
       ping() failed for rg1-rn2 : Unable to connect to the storage node
       agent at host node02, port 9000, which may not be running;
       nested exception is:
               java.rmi.ConnectException: Connection refused to
               host: node02; nested exception is:
               java.net.ConnectException: Connection refused
       Verification violation: [sn2]   ping() failed for sn2 : Unable to
       connect to the storage node agent at host node02, port 9000, which
       may not be running; nested exception is:
               java.rmi.ConnectException: Connection refused to host: node02;
       nested exception is:
               java.net.ConnectException: Connection refused
       ...

   In this case the Arbiter Node supports write availability so you can still perform write operations while node02 is repaired or replaced. Once node02 is restored, any written data will be migrated.

6. Test that you can still write to the store with the help of the Arbiter Node. For example, run the script file `test.kvsql` (see below for test.kvsql) using the Oracle NoSQL Database Shell utility (see below example). To do this, use the `load` command in the Query Shell:

       > java -jar KVHOME/lib/sql.jar -helper-hosts node01:8000 \
       -store mystore -security USER/security/admin.security

       kvsql-> load -file ./test.kvsql

   Output:

       Statement completed successfully.
       Statement completed successfully.
       Loaded 3 rows to users.

   Note:

   For the Enterprise Edition (EE) installation, make sure the kvstore-ee.jar is added in the classpath.

   The following commands are collected in `test.kvsql`:

       ### Begin Script ###
       load -file test.ddl
       import -table users -file users.json
       ### End Script ###

   Where the file `test.ddl` would contain content like this:

       DROP TABLE IF EXISTS users;
       CREATE TABLE users(id INTEGER, firstname STRING, lastname STRING,
       age INTEGER, primary key (id));

   And the file `users.json` would contain content like this:

       {"id":1,"firstname":"Dean","lastname":"Morrison","age":51}
       {"id":2,"firstname":"Idona","lastname":"Roman","age":36}
       {"id":3,"firstname":"Bruno","lastname":"Nunez","age":49}