Previous Next JavaScript must be enabled to correctly display this content

## Performing a Switchover

To continue from the example of the previous section, after performing the failover, you can return the old nodes to service by performing the following switchover procedure:

1. After the failed zones are repaired, restart all the Storage Nodes of the failed zones without starting any services (avoids hard rollback):

       java -Xmx64m -Xms64m \
       -jar KVHOME/lib/kvstore.jar restart -disable-services \
       -root nyc1/KVROOT &

   Note:

   When performing planned maintenance, there is no need to isolate nodes or disable services prior to bringing nodes back online.

2. Reestablish network connectivity or reenable the standard startup sequence of the previously failed zones.

3. Repair the topology so that the topology for the newly restarted Storage Nodes can be updated with changes made by the failover.

       java -Xmx64m -Xms64m -jar KVHOME/lib/kvstore.jar runadmin \
       -host jersey1 -port 5000 \
       -security USER/security/admin.security

       kv-> plan repair-topology -wait

   Output:

       Executed plan 10, waiting for completion...
       Plan 10 ended successfully

   Note:

   This assumes that you must have followed the steps as mentioned in Create users and configure security with remote access.

   Note:

   This command will also restart services on the previously failed nodes.

   Use the `verify configuration` command to confirm that there are no configuration problems.

4. Run the ping command. The "maxCatchupTimeSecs" value will be used for the -timeout flag of the `await-consistency` command.

   Use the timeout flag to specify an estimate of how long the switchover will take. For example, if the nodes have been offline for a long time it might take many hours for them to catch up so that they can be converted back to primary nodes.

       kv-> ping

   Output:

       Pinging components of store mystore based upon topology sequence #117
       100 partitions and 6 storage nodes
       Time: 2024-04-05 07:39:18 UTC   Version: 24.1.11
       Shard Status: healthy: 1 writable-degraded: 0 read-only: 0 offline: 0 total: 1
       Admin Status: healthy
       Zone [name=Manhattan id=zn1 type=SECONDARY allowArbiters=false masterAffinity=false]
       RN Status: online: 3 read-only: 0 offline: 0 maxDelayMillis: 3 maxCatchupTimeSecs: 0
       Zone [name=JerseyCity id=zn2 type=PRIMARY allowArbiters=false masterAffinity=false]
       RN Status: online: 3 read-only: 0 offline: 0 maxDelayMillis: 4 maxCatchupTimeSecs: 0
       Storage Node [sn1] on nyc1: 5000    Zone: name=Manhattan id=zn1 type=SECONDARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:36:01 UTC
       Admin [admin1]  Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:38:14 UTC
       stateChangeTime: 2024-04-05 07:38:14 UTC  availableStorageSize: 2 GB
       Rep Node [rg1-rn1] Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5111
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:37:14 UTC
       stateChangeTime: 2024-04-05 07:37:20 UTC delayMillis: 0 catchupTimeSecs: 0
       Storage Node [sn2] on nyc1: 5100    Zone: [name=Manhattan id=zn1 type=SECONDARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise  isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:36:25 UTC
       Admin [admin2]  Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:38:34 UTC
       stateChangeTime: 2024-04-05 07:38:33 UTC availableStorageSize: 2 GB
       Rep Node [rg1-rn2]  Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5211
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:37:28 UTC
       stateChangeTime: 2024-04-05 07:37:33 UTC delayMillis: 0 catchupTimeSecs: 0
       Storage Node [sn3] on nyc1: 5200    Zone: [name=Manhattan id=zn1 type=SECONDARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:36:35 UTC
       Admin [admin3]  Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:38:56 UTC
       stateChangeTime: 2024-04-05 07:38:56 UTC  availableStorageSize: 2 GB
       Rep Node [rg1-rn3] Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5311
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:37:43 UTC
       stateChangeTime: 2024-04-05 07:37:49 UTC delayMillis: 3 catchupTimeSecs: 0
       Storage Node [sn4] on jersey1: 6000  Zone: [name=JerseyCity id=zn2 type=PRIMARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:05:44 UTC
       Admin [admin4]  Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:36:49 UTC
       stateChangeTime: 2024-04-05 07:36:47 UTC availableStorageSize: 2 GB
       Rep Node [rg1-rn4]  Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5411
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:36:36 UTC
       stateChangeTime: 2024-04-05 07:36:59 UTC delayMillis: 4 catchupTimeSecs: 0
       Storage Node [sn5] on jersey1: 6100    Zone: [name=JerseyCity id=zn2 type=PRIMARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise  isMasterBalanced: true
       serviceStartTime: 2024-04-059 07:05:54 UTC
       Admin [admin5] Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:36:49 UTC
       stateChangeTime: 2024-04-05 07:36:48 UTC  availableStorageSize: 2 GB
       Rep Node [rg1-rn5]  Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5511
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:36:36 UTC
       stateChangeTime: 2024-04-05 07:36:59 UTC delayMillis: 0 catchupTimeSecs: 0
       Storage Node [sn6] on jersey1: 6200    Zone: [name=JerseyCity id=zn2 type=PRIMARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise  isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:06:03 UTC
       Admin [admin6]  Status: RUNNING,MASTER  serviceStartTime: 2024-04-05 07:36:55 UTC
       stateChangeTime: 2024-04-05 07:36:46 UTC  availableStorageSize: 2 GB
       Rep Node [rg1-rn6] Status: RUNNING,MASTER sequenceNumber: 2,672 haPort: 5611
       availableStorageSize: 273 GB storageType: HD  serviceStartTime: 2024-04-05 07:36:36 UTC
       stateChangeTime: 2024-04-05 07:36:57 UTC

   In this case, 1800 seconds (30 minutes) is the value to be used.

5. Use the `await-consistency` command to specify the wait time (1800 seconds) used for the secondary zones to catch up with their masters.

   The system will only wait five minutes for nodes to catch up when attempting to change a zone's type. If the nodes do not catch up in that amount of time, the plan will fail.

   If the nodes will take more than five minutes to catch up, you should run the `await-consistency` command, specifying a longer wait time using the -timeout flag. In this case, the wait time (1800 seconds) is used:

       kv-> await-consistent -timeout 1800 -znname Manhattan
       The specified zone is consistent

   By default, nodes need to have a delay of no more than 1 second to be considered caught up. You can change this value by specifying the -replica-delay-threshold flag. You should do this if network delays prevent the nodes from catching up within 1 second of their masters.

   Note:

   If you do not want the switchover to wait for the nodes to catch up, you can use the -no-replica-delay threshold flag. In that case, nodes will be converted to primary nodes even if they are behind. You should evaluate whether this risk is worth taking.

6. Perform the switchover to convert the previously failed zone back to a primary zone , and the formerly secondary zone back to its earlier state.

       kv-> topology clone -current -name newTopo

       kv-> topology change-zone-type -name newTopo \
       -znname Manhattan -type primary

   Output:

       Changed zone type of zn1 to PRIMARY in newTopo

       kv-> topology change-zone-type -name newTopo \
       -znname JerseyCity -type secondary

   Output:

       Changed zone type of zn2 to SECONDARY in newTop

       kv-> plan deploy-topology -name newTopo -wait

   Output:

       Executed plan 11, waiting for completion...
       Plan 11 ended successfully

   Confirm the zone type change of the Manhattan zone to PRIMARY by running the ping command.

       kv-> ping

   Output:

       Pinging components of store mystore based upon topology sequence #117
       100 partitions and 6 storage nodes
       Time: 2024-04-05 07:39:18 UTC   Version: 24.1.11
       Shard Status: healthy: 1 writable-degraded: 0 read-only: 0 offline: 0 total: 1
       Admin Status: healthy
       Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]
       RN Status: online: 3 read-only: 0 offline: 0 maxDelayMillis: 3 maxCatchupTimeSecs: 0
       Zone [name=JerseyCity id=zn2 type=SECONDARY allowArbiters=false masterAffinity=false]
       RN Status: online: 3 read-only: 0 offline: 0 maxDelayMillis: 4 maxCatchupTimeSecs: 0
       Storage Node [sn1] on nyc1: 5000    Zone: name=Manhattan id=zn1 type=PRIMARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:36:01 UTC
       Admin [admin1]  Status: RUNNING,MASTER serviceStartTime: 2024-04-05 07:38:14 UTC
       stateChangeTime: 2024-04-05 07:38:14 UTC  availableStorageSize: 2 GB
       Rep Node [rg1-rn1]  Status: RUNNING,MASTER sequenceNumber: 2,672 haPort: 5111
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:37:14 UTC
       stateChangeTime: 2024-04-05 07:37:20 UTC delayMillis: 0 catchupTimeSecs: 0
       Storage Node [sn2] on nyc1: 5100  Zone: [name=Manhattan id=zn1 type=PRIMARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING  Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise  isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:36:25 UTC
       Admin [admin2]   Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:38:34 UTC
       stateChangeTime: 2024-04-05 07:38:33 UTC availableStorageSize: 2 GB
       Rep Node [rg1-rn2]  Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5211
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:37:28 UTC
       stateChangeTime: 2024-04-05 07:37:33 UTC delayMillis: 0 catchupTimeSecs: 0
       Storage Node [sn3] on nyc1: 5200    Zone: [name=Manhattan id=zn1 type=PRIMARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise    isMasterBalanced: true
        serviceStartTime: 2024-04-05 07:36:35 UTC
       Admin [admin3]        Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:38:56 UTC
       stateChangeTime: 2024-04-05 07:38:56 UTC  availableStorageSize: 2 GB
       Rep Node [rg1-rn3     Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5311
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:37:43 UTC
       stateChangeTime: 2024-04-05 07:37:49 UTC delayMillis: 3 catchupTimeSecs: 0
       Storage Node [sn4] on jersey1: 6000    Zone: [name=JerseyCity id=zn2 type=SECONDARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:05:44 UTC
       Admin [admin4]  Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:36:49 UTC
       stateChangeTime: 2024-04-05 07:36:47 UTC availableStorageSize: 2 GB
       Rep Node [rg1-rn4]  Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5411
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:36:36 UTC
       stateChangeTime: 2024-04-05 07:36:59 UTC delayMillis: 4 catchupTimeSecs: 0
       Storage Node [sn5] on jersey1: 6100 Zone: [name=JerseyCity id=zn2 type=SECONDARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:05:54 UTC
       Admin [admin5]    Status: RUNNING,REPLICA serviceStartTime: 2024-04-05 07:36:49 UTC
       stateChangeTime: 2024-04-05 07:36:48 UTC  availableStorageSize: 2 GB
       Rep Node [rg1-rn5]   Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5511
       availableStorageSize: 273 GB storageType: HD serviceStartTime: 2024-04-05 07:36:36 UTC
       stateChangeTime: 2024-04-05 07:36:59 UTC delayMillis: 0 catchupTimeSecs: 0
       Storage Node [sn6] on jersey1: 6200 Zone: [name=JerseyCity id=zn2 type=SECONDARY
       allowArbiters=false masterAffinity=false]
       Status: RUNNING   Ver: 24.1.11 2024-04-05 21:24:59 UTC
       Build id: 78bbc4cb976b Edition: Enterprise  isMasterBalanced: true
       serviceStartTime: 2024-04-05 07:06:03 UTC
       Admin [admin6]   Status: RUNNING,REPLICA  serviceStartTime: 2024-04-05 07:36:55 UTC
       stateChangeTime: 2024-04-05 07:36:46 UTC  availableStorageSize: 2 GB
       Rep Node [rg1-rn6]  Status: RUNNING,REPLICA sequenceNumber: 2,672 haPort: 5611
       availableStorageSize: 273 GB storageType: HD  serviceStartTime: 2024-04-05 07:36:36 UTC
       stateChangeTime: 2024-04-05 07:36:57 UTC