Verrazzano Enterprise Container Platform – Troubleshooting

Docs: Application Deployment

Mon, 01 Jan 0001 00:00:00 +0000

During application deployment, the oam-kubernetes-runtime and verrazzano-application-operator cooperate through the generation and update of Kubernetes resources. The oam-kubernetes-runtime processes the ApplicationConfiguration and Component resources provided by the user and generates workload and Trait resources. The verrazzano-application-operator processes Verrazzano specific workload and Trait resources. These are then used to generate additional child and related resources.

Troubleshooting application deployments should follow three general steps:

Review the status of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the logs of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the resources generated by the oam-kubernetes-runtime and the verrazzano-application-operator.

Review `oam-kubernetes-runtime` operator status

For application deployment to succeed, the oam-kubernetes-runtime pod must have a status of Running.

Use the following command to get the pod status.

$ kubectl get pods \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

If the pod status is not Running, then see the following instructions for reviewing the oam-kubernetes-runtime pod logs.

Review `verrazzano-application-operator` operator status

For application deployment to succeed, the verrazzano-application-operator pod must have a status of Running.

Use the following command to get the pod status.

$ kubectl get pods \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If the pod status is not Running, then see the following instructions for reviewing the verrazzano-application-operator logs.

Review `oam-kubernetes-runtime` operator logs

Review the oam-kubernetes-runtime pod logs for any indication that pod startup or the generation of workloads or traits has failed.

Use the following command to get the logs.

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

Review `verrazzano-application-operator` logs

Review the verrazzano-application-operator logs for any indication that pod startup or resource generation has failed.

Use the following command to get the logs.

$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

Review generated workload resources

The processing of a Component reference within an ApplicationConfiguration results in the generation of workloads. For example, a referenced Component might result in the generation of a VerrazzanoHelidonWorkload workload resource. In turn, the VerrazzanoHelidonWorkload workload resource will be processed and result in the generation of related Deployment and Service resources.

If the expected workload resource, for example VerrazzanoHelidonWorkload, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Deployment or Service, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to a VerrazzanoHelidonWorkload deployment.

$ kubectl get -n hello-helidon verrazzanohelidonworkload hello-helidon-workload
$ kubectl get -n hello-helidon deployment hello-helidon-deployment
$ kubectl get -n hello-helidon service hello-helidon-deployment

Review generated Trait resources

The processing of traits embedded with an ApplicationConfiguration results in the generation of Trait resources. For example, an IngressTrait embedded within an ApplicationConfiguration will result in the generation of an IngressTrait resource. In turn, the IngressTrait resource will be processed and result in the generation of related Certificate, Gateway, and VirtualService resources.

If the expected Trait resource, for example IngressTrait, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Certificate, Gateway, and VirtualService, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to an IngressTrait.

$ kubectl get -n hello-helidon ingresstrait hello-helidon-ingress
$ kubectl get -n istio-system Certificate hello-helidon-hello-helidon-appconf-cert
$ kubectl get -n hello-helidon gateway hello-helidon-hello-helidon-appconf-gw
$ kubectl get -n hello-helidon virtualservice hello-helidon-ingress-rule-0-vs

Check for RBAC privilege issues

The use of generic Kubernetes resources as workloads and traits can result in deployment failures if privileges are insufficient. In this case, the oam-kubernetes-runtime logs will contain errors containing the term forbidden.

The following command shows how to query for this type of failure message.

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime | grep forbidden

Check resource owners

Kubernetes maintains the child to parent relationship within metadata fields.

The following example returns the parent of the IngressTrait, named hello-helidon-ingress, in the hello-helidon namespace.

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .metadata.ownerReferences[*]}{.name}{"\n"}{end}'

The results of this command can help identify the lineage of a given resource.

Some resources also record the related resources affected during their processing. For example, when processed, an IngressTrait will create related Gateway, VirtualService, and Certificate resources.

The following command is an example of how to obtain the related resources of an IngressTraits.

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .status.resources[*]}{.kind}: {.name}{"\n"}{end}'

The results of this command can help identify which other resources, the given resource affected.

Docs: Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

This document describes some common problems you might encounter when using multicluster Verrazzano, and how to troubleshoot them.

If you created multicluster resources in the admin cluster, and specified a placement value in a managed cluster, then those resources will get created in that managed cluster. If they do not get created in the managed cluster, then use the following steps to troubleshoot:

Verify that the managed cluster is registered correctly and can connect to the admin cluster.
Verify that the VerrazzanoProject for the resource’s namespace, also has a placement in that managed cluster.
Check the multicluster resource’s status field on the admin cluster to know what the status of that resource is on each managed cluster to which it is targeted.

Verify managed cluster registration and connectivity

You can verify that a managed cluster was successfully registered with an admin cluster by viewing the corresponding VerrazzanoManagedCluster (VMC) resource on the admin cluster. For example, to verify that a managed cluster named managed1 was successfully registered:

# on the admin cluster
$ kubectl get verrazzanomanagedcluster managed1 \
    -n verrazzano-mc \
    -o yaml

Partial sample output from the previous command:

  status:
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:27Z"
      message: Ready
      status: "True"
      type: Ready
    lastAgentConnectTime: "2021-06-22T21:06:04Z"
    ... other fields ...

Check the lastAgentConnectTime in the status of the VMC resource. This is the last time at which the managed cluster connected to the admin cluster. If this value is not present, then the managed cluster named managed1 never successfully connected to the admin cluster. This could be due to several reasons:

The managed cluster registration process step of applying the registration YAML on the managed cluster, was not completed. For the complete setup instructions, see here.
The managed cluster does not have network connectivity to the admin cluster. The managed cluster will attempt to connect to the admin cluster at regular intervals, and any errors will be reported in the verrazzano-application-operator pod’s log on the managed cluster. View the logs using the following command.

# on the managed cluster
$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If these logs reveal that there is a connectivity issue, check the admin cluster Kubernetes server address that you provided during registration and ensure that it is correct, and that it is reachable from the managed cluster. If it is incorrect, then you will need to repeat the managed cluster registration process described in the setup instructions here.

Verify VerrazzanoProject placement

For Verrazzano to create an application namespace in a managed cluster, that namespace must be part of a VerrazzanoProject that:

Includes that namespace.
Has a placement value that includes that managed cluster.

View the details of the project that corresponds to your application’s namespace. In the example command that follows, the project name is assumed to be myproject. All projects are expected to be created in the verrazzano-mc namespace.

# on the admin cluster
$ kubectl get verrazzanoproject myproject \
    -n verrazzano-mc \
    -o yaml

The following partial sample output is for a project that will result in the namespace mynamespace being created on the managed cluster managed1.

spec:
  placement:
    clusters:
    - name: managed1
  template:
    namespaces:
    - metadata:
        name: mynamespace
....other fields....

Check the multicluster resource status

On the admin cluster, each multicluster resource’s status field is updated with the status of the underlying resource on each managed cluster in which it is placed.

The following example command shows how to view the status of a MultiClusterApplicationConfiguration named myapp, in the namespace mynamespace, that has a placement value that includes the managed cluster managed1

$ kubectl get multiclusterapplicationconfiguration myapp \
    -n mynamespace \
    -o yaml

The status of the underlying resource in each cluster specified in the placement is shown in the following partial sample output:

  status:
    clusters:
    - lastUpdateTime: "2021-06-22T21:05:04Z"
      message: OAM Application Configuration created
      name: managed1
      state: Succeeded
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:58Z"
      message: OAM Application Configuration created
      status: "True"
      type: DeployComplete
    state: Succeeded

The status message contains additional information on the operation’s success or failure.

Docs: OpenSearch Scaling and Resizing

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to recover an OpenSearch cluster’s health after it becomes unhealthy due to unassigned shards or disk pressure.

It also describes how to scale up the cluster’s data nodes and increase the size of the volumes. Because the volume size change in the Verrazzano operator also affects the master nodes volume size, you must take additional steps to address the volume resizing of a StatefulSet.

First:

# Edit the Verrazzano operator
$ kubectl -n verrazzano-system edit deploy verrazzano-operator

Then, change the following portion by increasing the number of ES_DATA_NODE_REPLICAS to 3, and the ES_DATA_STORAGE to 200Gi:

- name: ES_DATA_NODE_REPLICAS
  value: "3"

- name: ES_DATA_STORAGE
  value: "200"

Scaling OpenSearch data nodes

Follow this procedure to scale the OpenSearch data nodes.

Wait for the new data node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When you have a green state, replace the original data node -0 pods:

$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=0
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=0
$ kubectl -n verrazzano-system delete pod/vmi-system-es-data-0-xxxxxxxxx-xxxx pvc/vmi-system-es-data-0
$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=1
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=1

Wait for the new data node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When you have a green state, replace the original data node -1 pods:

$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=0
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=0
$ kubectl -n verrazzano-system delete pod/vmi-system-es-data-1-xxxxxxxxx-xxxx pvc/vmi-system-es-data-1
$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=1
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=1

Wait for the new data node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

You should now have three data nodes that are healthy and at 200GB volumes.

Address the master nodes’ StatefulSet

Now to address the master nodes. Because you cannot directly change the size of the volume associated with a volume template in a StatefulSet, you must follow this procedure:

First:

$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=0
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=0
$ kubectl -n verrazzano-system get sts vmi-system-es-master -o yaml > vmi-system-es-master.yaml

Edit the file created in the previous command, vmi-system-es-master.yaml

a. Remove the lines starting with:
```
creationTimestamp:
generation:
resourceVersion:
selfLink:
uid:
status:
```
b. Remove every line below status:

c. Edit the section to increase the storage to the same value that you used for the Verrazzano operator:
```
storage: 200Gi
```
d. Save that file.
The following command will delete the StatefulSet, but allow the associated pods to continue to run.
```
$ kubectl -n verrazzano-system delete sts vmi-system-es-master --cascade=orphan
```
Then run this command to recreate the StatefulSet with the new volume size defined:
```
$ kubectl -n verrazzano-system apply -f vmi-system-es-master.yaml
```
The next steps are to delete the existing master node pods, one at a time, allowing the cluster to become healthy before moving on to the next node:
```
$ kubectl -n verrazzano-system delete pod/vmi-system-es-master-0 pvc/elasticsearch-master-vmi-system-es-master-0
```

Wait for the new master node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When the cluster is healthy, continue to the next master node:

$ kubectl -n verrazzano-system delete pod/vmi-system-es-master-1 pvc/elasticsearch-master-vmi-system-es-master-1

Wait for the new master node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When the cluster is healthy, continue to the next master node:

$ kubectl -n verrazzano-system delete pod/vmi-system-es-master-2 pvc/elasticsearch-master-vmi-system-es-master-2

Wait for the new master node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When the cluster is healthy rescale the operators:

$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=1
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=1

Update the Verrazzano Custom Resource

Now you will edit the Verrazzano CR, so that when you upgrade, the above changes will not be overwritten. Follow this procedure:

Get the namespace and name of the Verrazzano CR:
```
$ kubectl get vz -A
```

Edit the Verrazzano CR:

$ kubectl -n <namespace> edit vz <name>

a. Alter to include the following:

spec:
  components:
    elasticsearch:
      installArgs:
      - name: nodes.data.replicas
        value: "3"
      - name: nodes.data.requests.storage
        value: 200Gi

b. Save the changes.

This completes the process.

Docs: Diagnostic Tools

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano Enterprise Container Platform – Troubleshooting

Docs: Application Deployment

Review oam-kubernetes-runtime operator status

Review verrazzano-application-operator operator status

Review oam-kubernetes-runtime operator logs

Review verrazzano-application-operator logs

Review generated workload resources

Review generated Trait resources

Check for RBAC privilege issues

Check resource owners

Check related resources

Docs: Multicluster Verrazzano

Verify managed cluster registration and connectivity

Verify VerrazzanoProject placement

Check the multicluster resource status

Docs: OpenSearch Scaling and Resizing

Scaling OpenSearch data nodes

Address the master nodes’ StatefulSet

Update the Verrazzano Custom Resource

Docs: Diagnostic Tools

Review `oam-kubernetes-runtime` operator status

Review `verrazzano-application-operator` operator status

Review `oam-kubernetes-runtime` operator logs

Review `verrazzano-application-operator` logs