Verrazzano Enterprise Container Platform – Troubleshoot and FAQ

Docs: Diagnostic Tools

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Fluentd Issues

Mon, 01 Jan 0001 00:00:00 +0000

Issue: Fluentd’s inability to push logs to OpenSearch due to a permission issue when SELinux is in enforcing mode.

If you are unable to see logs in OpenSearch, it may be due to a permission issue in Fluentd when SELinux is in enforcing mode, which prevents Fluentd from pushing the logs to OpenSearch.

To troubleshoot the Fluentd permission issue, follow these steps:

Check Fluentd pod logs.
- Identify the Fluentd pod related to the Verrazzano installation.
- View the logs of the Fluentd pod using the following command:
```
$ kubectl logs <fluentd-pod-name> -n verrazzano-system
```

Check for an error description.

Look for an error message in the Fluentd pod logs.
If you see an error with the following description, then follow the instructions in Step 3. Resolve permission issue.

unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/vz-fluentd-containers.log.pos", it indicates a permission issue. Fluentd doesn't have enough privilege to write `.pos` file.

Resolve permission issue.
- The issue occurs when SELinux is in enforcing mode on the worker nodes and Fluentd does not have the appropriate SELinux context to have read/write access to the logs (/var/log/) directory.
- Check if SELinux is in enforcing mode by running this command on the worker nodes:
```
$ sudo getenforce
```
  If SELinux is in enforcing mode, then follow the advice found here.
Verification.
- Verify that Fluentd is able to read and push the logs to OpenSearch by reviewing the Fluentd logs.
- Verify that the logs are visible in OpenSearch by accessing the OpenSearch dashboard or using relevant search queries.

Docs: MySQL Issues

Mon, 01 Jan 0001 00:00:00 +0000

There are known issues that can occur with MySQL. The Verrazzano platform operator will automatically detect each of the described issues and perform actions to repair them. The operator initiates a repair within a few minutes of detecting an issue.

The following sections are provided in the event that a manual repair of an issue is required.

MySQL pod stuck Terminating

A MySQL pod may get stuck in a terminating state. This may occur while upgrading the nodes of a Kubernetes cluster.

Here is an example of what this condition looks like. All the pod containers are terminated, but the pod never finishes terminating.

$ kubectl get pods -n keycloak -l component=mysqld
NAME      READY   STATUS        RESTARTS   AGE
mysql-0   0/3     Terminating   0          60m

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

MySQL pod waiting for readiness gates

The mysql StatefulSet may get stuck while waiting to reach the ready state. This will occur when one or more MySQL pods not meeting its set of ReadinessGates.

Here is an example of what this condition looks like.

$ kubectl describe pods -n keycloak -l component=mysqld

# Excerpt from the command output
Readiness Gates:
  Type                          Status
  mysql.oracle.com/configured   False
  mysql.oracle.com/ready        True

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

MySQL router pod in CrashLoopBackOff state

Here is an example of what this condition looks like.

$ kubectl get pods -n keycloak -l component=mysqlrouter
NAME                            READY   STATUS             RESTARTS   AGE
mysql-router-757595f6c5-pdgxj   1/2     CrashLoopBackOff   0          109m

You can repair this issue by deleting the pod that is in the CrashLoopBackOff state.

$ kubectl delete pod -n keycloak mysql-router-757595f6c5-pdgxj

InnoDBCluster object stuck Terminating

This condition has been observed to occur on an uninstallation of Verrazzano.

Here is an example of what this condition looks like.

$ kubectl get InnoDBCluster -n keycloak
NAME    STATUS    ONLINE   INSTANCES   ROUTERS   AGE
mysql   OFFLINE   0        1           1         7m51s

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

Docs: FAQ

Mon, 01 Jan 0001 00:00:00 +0000

Enable Google Chrome to accept self-signed Verrazzano certificates

There are some installation scenarios where Verrazzano generates SSL certificates that are not trusted by browsers:

The development (dev) profile installation, which uses its own self-signed CA to issue certificates.
Using the Let’s Encrypt Staging authority, which uses untrusted CAs to sign certificates.

These are typical development or testing scenarios, not recommended for production. When accessing Verrazzano sites using these certificates, some browsers like Firefox let you manually accept these certificates. However, Google Chrome now prevents users from being able to accept self-signed certificates by default. This will prevent you from accessing Verrazzano consoles that are using untrusted certificates.

When this occurs, while trying to access Verrazzano services, you will see an error message like the following:

opensearch.vmi.system.default.129.153.98.156.nip.io normally uses encryption to protect your information. When Chrome tried to connect to opensearch.vmi.system.default.129.153.98.156.nip.io this time, the website sent back unusual and incorrect credentials

You can choose to import the certificate into your local trust chain, but this will have to be done for each Verrazzano instance. From a security perspective, this is not recommended.

As an alternative, you can enter a secret passphrase in Chrome to enable it to prompt you to accept these certificates, by doing the following:

When you see an error such as the one shown previously, when the browser window has the keyboard focus, enter the phrase thisisunsafe.
Reload the site.
Chrome will prompt you to accept the certificate.

NOTE: This should be used only when accessing sites that are known to be safe, such as in this situation.

Verrazzano Enterprise Container Platform – Troubleshoot and FAQ

Docs: Diagnostic Tools

Docs: Fluentd Issues

Docs: MySQL Issues

MySQL pod stuck Terminating

MySQL pod waiting for readiness gates

MySQL router pod in CrashLoopBackOff state

InnoDBCluster object stuck Terminating

Docs: FAQ

Enable Google Chrome to accept self-signed Verrazzano certificates

Related articles