Verrazzano Enterprise Container Platform – Customize Verrazzano

Docs: AuthProxy

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano AuthProxy component enables authentication and authorization for Keycloak users accessing Verrazzano resources. You can customize the AuthProxy component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the AuthProxy component.

Path to Field Description

spec.components.authProxy.kubernetes.replicas The number of pods to replicate. The default is 2 for the prod profile and 1 for all other profiles.

Path to Field	Description
`spec.components.authProxy.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.authProxy.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes. spec: components: authProxy: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - verrazzano-authproxy topologyKey: kubernetes.io/hostname

spec.components.authProxy.kubernetes.affinity

The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes.

spec:
  components:
    authProxy:
      kubernetes:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    authProxy:
      overrides:
      - values:
          replicas: 3
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

Docs: Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano issues certificates to secure access from external clients to secure system endpoints.
A certificate from a certificate authority (CA) must be configured to issue the endpoint certificates in one of the following ways:

Let Verrazzano generate a self-signed CA (the default).
Configure a CA that you provide.
Configure LetsEncrypt as the certificate issuer (requires Oracle Cloud Infrastructure DNS).

In all cases, Verrazzano uses cert-manager to manage the creation of certificates.

NOTE

Self-signed certificate authorities generate certificates that are NOT signed by a trusted authority; typically, they are not used in production environments.

Use the Verrazzano self-signed CA

By default, Verrazzano creates its own self-signed CA. No configuration is required.

Use a custom CA

If you want to provide your own CA, you must:

(Optional) Create your own signing key pair and CA certificate.

For example, you can use the openssl CLI to create a key pair for the nip.io domain.
```
# Generate a CA private key
$ openssl genrsa -out tls.key 2048

# Create a self-signed certificate, valid for 10yrs with the 'signing' option set
$ openssl req -x509 -new -nodes -key tls.key -subj "/CN=*.nip.io" -days 3650 -reqexts v3_req -extensions v3_ca -out tls.crt
```
The output of these commands will be two files, tls.key and tls.crt, the key and certificate for your signing key pair. These files must be named in that manner for the next step.

If you already have generated your own key pair, you must name the private key and certificate, tls.key and tls.crt, respectively. If your issuer represents an intermediate, ensure that tls.crt contains the issuer’s full chain in the correct order.

You can find more details on providing your own CA, in the cert-manager CA documentation.

Save your signing key pair as a Kubernetes secret.

$ kubectl create ns mynamespace
$ kubectl create secret tls myca --namespace=mynamespace --cert=tls.crt --key=tls.key

Specify the secret name and namespace location in the Verrazzano custom resource.

The custom CA secret must be provided to cert-manager using the following fields in spec.components.certManager.certificate.ca in the Verrazzano custom resource:
- spec.components.certManager.certificate.ca.secretName
- spec.components.certManager.certificate.ca.clusterResourceNamespace

For example, if you created a CA secret named myca in the namespace mynamespace, you would configure it as shown:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-ca-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        ca:
          secretName: myca
          clusterResourceNamespace: mynamespace

Use LetsEncrypt certificates

You can configure Verrazzano to use certificates generated by LetsEncrypt. LetsEncrypt implements the ACME protocol, which provides a standard protocol for the automated issuance of certificates signed by a trusted authority. This is managed through the spec.components.certManager.certificate.acme field in the Verrazzano custom resource.

NOTE

Using LetsEncrypt for certificates also requires using Oracle Cloud Infrastructure DNS for DNS management. For details, see the Customize DNS page.

To configure cert-manager to use LetsEncrypt as the certificates provider, you must configure a cert-manager ACME provider with the following values in the Verrazzano custom resource:

Set the spec.components.certManager.certificate.acme.provider field to letsEncrypt.
Set the spec.components.certManager.certificate.acme.emailAddress field to a valid email address for the letsEncrypt account.
(Optional) Set the spec.components.certManager.certificate.acme.environment field to either staging or production (the default).

The following example configures Verrazzano to use the LetsEncrypt production environment by default, with Oracle Cloud Infrastructure DNS for DNS record management.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

The following example configures Verrazzano to use the LetsEncrypt staging environment with Oracle Cloud Infrastructure DNS.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
          environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

NOTE

Certificates issued by the LetsEncrypt staging environment are signed by untrusted authorities, similar to self-signed certificates. They are typically not used in production environments.

LetsEncrypt staging versus production

LetsEncrypt provides rate limits on generated certificates to ensure fair usage across all clients. The production environment limits can be exceeded more frequently in environments where Verrazzano may be installed or reinstalled frequently (like a test environment). This can result in failed installations due to rate limit exceptions on certificate generation.

In such environments, it is better to use the LetsEncrypt staging environment, which has much higher limits than the production environment. For test environments, the self-signed CA also may be more appropriate to completely avoid LetsEncrypt rate limits.

Docs: DNS

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports three DNS choices for Verrazzano services and applications:

Free wildcard DNS services (nip.io and sslip.io)
Oracle Cloud Infrastructure DNS managed by Verrazzano
Custom (user-managed) DNS

How Verrazzano constructs a DNS domain

Regardless of which DNS management you use, the value in the spec.environmentName field in your installation will be prepended to the configured domain in the spec.components.dns section of the custom resource, to form the full DNS domain name used to access Verrazzano endpoints.

For example, if spec.environmentName is set to sales and the domain is configured in spec.components.dns as us.example.com, Verrazzano will create sales.us.example.com as the DNS domain for the installation.

Verrazzano can be configured to use either the nip.io or sslip.io free wildcard DNS services. When queried with a host name with an embedded IP address, wildcard DNS services return that IP address.

For example, using the nip.io service, the following DNS names all map to the IP address 10.0.0.1:

10.0.0.1.nip.io
app.10.0.0.1.nip.io
customer1.app.10.0.0.1.nip.io

To configure Verrazzano to use one of these services, set the spec.wildcard.domain field in the Verrazzano custom resource to either nip.io or sslip.io; the default is nip.io.

For example, the following configuration uses sslip.io, instead of nip.io, for wildcard DNS with a dev installation profile.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    dns:
      wildcard:
        domain: sslip.io

Verrazzano can directly manage records in Oracle Oracle Cloud Infrastructure DNS when configured to use the spec.components.dns.oci field. This is achieved through the External DNS Service, which is a component that is conditionally installed when Oracle Cloud Infrastructure DNS is configured for DNS management in Verrazzano.

Prerequisites

The following prerequisites must be met before using Oracle Cloud Infrastructure DNS with Verrazzano:

You must have control of a DNS domain.
You must have an Oracle Cloud Infrastructure DNS Service Zone that is configured to manage records for that domain. Verrazzano also supports the use of both GLOBAL and PRIVATE Oracle Cloud Infrastructure DNS zones.

A DNS Service Zone is a distinct portion of a domain namespace. You must ensure that the zone is appropriately associated with a parent domain. For example, an appropriate zone name for parent domain example.com is us.example.com.

To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure CLI:
```
$ oci dns zone create \
    -c <compartment ocid> \
    --name <zone-name-prefix>.example.com \
    --zone-type PRIMARY
```
To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure Console, see Managing DNS Service Zones.
You must have a valid Oracle Cloud Infrastructure API signing key that can be used to communicate with Oracle Cloud Infrastructure DNS in your tenancy.

For example, you can create an API signing key using the Oracle Cloud Infrastructure CLI.
```
  $ oci setup keys --key-name myapikey
  Enter a passphrase for your private key (empty for no passphrase):
  Public key written to: /Users/jdoe/.oci/myapikey_public.pem
  Private key written to: /Users/jdoe/.oci/myapikey.pem
  Public key fingerprint: 39:08:44:69:9f:f5:73:86:7a:46:d8:ad:34:4f:95:29
```
If you haven’t already uploaded your API signing public key through the console, follow the instructions in this section, How to upload the public key.

After the key pair has been created, you must upload the public key to your account in your Oracle Cloud Infrastructure tenancy. For details, see the Oracle Cloud Infrastructure documentation, Required Keys and OCIDs.

Create an Oracle Cloud Infrastructure API secret in the target cluster

To communicate with Oracle Cloud Infrastructure DNS to manage DNS records, Verrazzano needs to be made aware of the necessary API credentials.
A generic Kubernetes secret must be created in the cluster’s verrazzano-install namespace with the required credentials. That secret must then be referenced by the custom resource that is used to install Verrazzano.

After you have an Oracle Cloud Infrastructure API key ready for use, create a YAML file, oci.yaml, with the API credentials in the form:

auth:
  region: <oci-region>
  tenancy: <oci-tenancy-ocid>
  user: <oci-user-ocid>
  key: |
    <oci-api-private-key-file-contents>
  fingerprint: <oci-api-private-key-fingerprint>

This information typically can be found in your Oracle Cloud Infrastructure CLI config file or in the Oracle Cloud Infrastructure Console. The <oci-api-private-key-file-contents> contents are the PEM-encoded contents of the key_file value within the Oracle Cloud Infrastructure CLI configuration profile.

For example, your oci.yaml file will look similar to the following:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1.....
  user: ocid1.user.oc1.....
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:d3:4c:gh:fd:9e:27:g8:b9:0d:9f:00:22:33:c3:gg

Verrazzano also supports the use of instance principals to communicate with Oracle Cloud Infrastructure in order to create or update Oracle Cloud Infrastructure DNS records. Instance principals require some prerequisites that can be found here.

When using instance principals, your oci.yaml file will look as follows:

auth:
  authtype: instance_principal

Then, you can create a generic Kubernetes secret in the cluster’s verrazzano-install namespace using kubectl.

$ kubectl create secret generic -n verrazzano-install <secret-name> --from-file=<path-to-oci-yaml-file>

For example, to create a secret named oci from a file oci.yaml, do the following:

$ kubectl create secret generic -n verrazzano-install oci --from-file=oci.yaml

This secret will later be referenced from the Verrazzano custom resource used during installation.

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Verrazzano also provides a helper script to create the necessary Kubernetes secret based on your Oracle Cloud Infrastructure CLI configuration file, assuming that you have the Oracle Cloud Infrastructure CLI installed and a valid Oracle Cloud Infrastructure CLI profile with the required API key information. The script create_oci_config_secret.sh reads your Oracle Cloud Infrastructure CLI configuration file to create the secret.

First, download the create_oci_config_secret.sh script.

$ curl \
    -o ./create_oci_config_secret.sh \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/platform-operator/scripts/install/create_oci_config_secret.sh

Next, set your KUBECONFIG environment variable to point to your cluster and run create_oci_config_secret.sh -h to display the script options.

$ chmod +x create_oci_config_secret.sh
$ export KUBECONFIG=<kubeconfig-file>
$ ./create_oci_config_secret.sh  -h
usage: ./create_oci_config_secret.sh [-o oci_config_file] [-s config_file_section]
  -o oci_config_file         The full path to the Oracle Cloud Infrastructure configuration file (default ~/.oci/config)
  -s config_file_section     The properties section within the Oracle Cloud Infrastructure configuration file.  Default is DEFAULT
  -k secret_name             The secret name containing the Oracle Cloud Infrastructure configuration.  Default is oci
  -c context_name            The kubectl context to use
  -a auth_type               The auth_type to be used to access Oracle Cloud Infrastructure. Valid values are user_principal/instance_principal. Default is user_principal.
  -h                         Help

For example, to have the script create the YAML file using your [DEFAULT] Oracle Cloud Infrastructure CLI profile and then create a Kubernetes secret named oci, you can run the script with no arguments, as follows:

$ ./create_oci_config_secret.sh
secret/oci created

The following example creates a secret myoci using an Oracle Cloud Infrastructure CLI profile named [dev].

$ ./create_oci_config_secret.sh -s dev -k myoci
secret/myoci created

When using instance principals, all other parameters will be ignored automatically. The following example creates a secret myoci using Oracle Cloud Infrastructure instance principal.

$ ./create_oci_config_secret.sh -a instance_principal
secret/myoci created

Installation

After the Oracle Cloud Infrastructure API secret is created, create a Verrazzano custom resource for the installation that is configured to use Oracle Cloud Infrastructure DNS and reference the secret you created.

As a starting point, download the sample Verrazzano custom resource install-oci.yaml file for Oracle Cloud Infrastructure DNS.

$ curl \
    -o ./install-oci.yaml \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/platform-operator/config/samples/install-oci.yaml

Edit the install-oci.yaml file to provide values for the following configuration settings in the custom resource spec:

spec.environmentName
spec.components.dns.oci.ociConfigSecret
spec.components.dns.oci.dnsZoneCompartmentOCID
spec.components.dns.oci.dnsZoneOCID
spec.components.dns.oci.dnsZoneName
spec.components.dns.oci.dnsScope

The field spec.components.dns.oci.ociConfigSecret should reference the secret created earlier. For details on the Oracle Cloud Infrastructure DNS configuration settings, see spec.components.dns.oci.

For example, a custom resource for a prod installation profile using Oracle Cloud Infrastructure DNS might look as follows, yielding a domain of myenv.example.com (Oracle Cloud Infrastructure identifiers redacted):

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com

If using a private DNS zone, then the same prod installation profile using Oracle Cloud Infrastructure DNS will look as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: my-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com
        dnsScope: PRIVATE

After the custom resource is ready, apply it using kubectl apply -f <path-to-custom-resource-file>.

You can specify your own externally managed, custom DNS domain. In this scenario, you manage your own DNS domain and all DNS records in that domain.

An externally managed DNS domain is specified in the spec.components.dns.external.suffix field of the Verrazzano custom resource.

When using an externally managed DNS domain, you are responsible for:

Configuring A records for Verrazzano ingress points (load balancers)
Configuring CNAME records for host names in the domain that point to the A records, as needed

The Verrazzano installer searches the DNS zone you provide for two specific A records.
These are used to configure the cluster and should refer to external addresses of the load balancers provisioned by the user.

The A records need to be created manually.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example, if spec.environmentName is set to myenv, and spec.components.dns.external.suffix is set to example.com, the A records would need to be set up as follows:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

This example assumes that load balancers exist for ingress-mgmt on 198.51.100.10 and for ingress-verrazzano on 203.0.113.10.

For a more complete example, see the documentation for setting up Verrazzano on the Oracle Cloud Native Environment Platform.

Docs: External Load Balancers

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires the following load balancers at installation:

Load balancer for NGINX ingress
Load balancer for Istio ingress

By default, Verrazzano automatically creates them as Kubernetes-managed load balancers, however, you have the option to use your own external load balancers. You can choose to replace either or both load balancers.

The following is an example of using external load balancers for both management and application ingress.

Prepare the external load balancers

External load balancer for management ingress:
- This load balancer must have a listener set up on port 443 with TCP protocol.
- The backend set for this listener needs to include the Kubernetes cluster node IP addresses on a port you pick, for example, 31443.
External load balancer for application ingress:
- This load balancer must have a listener set up on port 443 with TCP protocol.
- The backend set for this listener needs to include the Kubernetes cluster node IP addresses on a port you pick, for example, 32443.

Verrazzano installation options

External load balancer for management ingress:
- Set NodePort as the ingress type in the Ingress Component.
- Set controller.service.externalIPs with the IP address for the external management load balancer in the Ingress NGINX Overrides.
- Set ports in the Ingress Component with a PortConfig that has 443 as port, 31443 as nodePort, https as targetPort, and TCP as protocol.
External load balancer for application ingress using the Istio ingress gateway overrides:
- Set service Type to NodePort.
- Set service externalIPs to the external application load balancer IP address.
- Set service ports with a https named entry, 443 as port, 32443 as nodePort, 8443 as targetPort, and TCP as protocol.

Example Custom Resource with management and application external load balancers

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: myvz
spec:
  components:
    ingressNGINX:
      overrides:
      - values:
          controller:
            service:
              externalIPs:
              - 11.22.33.44
      type: NodePort
      ports:
      - name: https
        port: 443
        nodePort: 31443
        protocol: TCP
        targetPort: https
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              ingressGateways:
                - enabled: true
                  name: istio-ingressgateway
                  k8s:
                    service:
                      type: NodePort
                      ports:
                      - name: https
                        port: 443
                        nodePort: 32443
                        protocol: TCP
                        targetPort: 8443
                      externalIPs:
                      - 11.22.33.55

Docs: Network File System

Mon, 01 Jan 0001 00:00:00 +0000

Complete the following steps to configure NFS storage in an Oracle Cloud Native Environment.

Create an OLCNE cluster. See OLCNE cluster.

The cluster must have at least 3 worker nodes.
Create an NFS server. For an example that uses an NFS server on Oracle Linux, see Create an NFS server on Oracle Linux.

a. Install the NFS utility package on the server and client instances.
```
$ sudo dnf install -y nfs-utils
```
b. Create a directory for your shared files. Make sure that the server does not have root ownership.

c. Define the shared directory in /etc/exports with the correct permissions. Make sure to disable root squashing.
```
$ <path to directory> <ip-address/subnet-mask>(rw,sync,no_root_squash,no_subtree_check)
```
d. Set the firewall to allow NFS traffic.
```
$ sudo firewall-cmd --permanent --zone=public --add-service=nfs
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --list-all
```
e. Enable and start the NFS service.
```
$ sudo systemctl enable --now nfs-server
```

Deploy an NFS provisioner to your cluster.

a. Install an NFS client provisioner of your choice. For an example, see Kubernetes NFS Subdir External Provisioner.

b. Add the required Helm repo.

$ helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

c. Install the provisioner. Set your storage class as a default and create a service account.

$ helm install nfs-test \
   --set nfs.server=<server ip address> \
   --set nfs.path=<path> \
   --set storageClass.name=<name> \
   --set storageClass.defaultClass=true,rbac.create=true \
   --set storageClass.provisionerName=nfsclientprov/nfs \
   --set serviceAccount.create=true \
   --set serviceAccount.name=nfs-svc-acc-nfs nfs-subdir-external-provisioner/nfs-subdir-external-provisioner

d. Only one storage class should be listed as the default. If required, edit the other storage classes and delete the following annotation:

$ storageclass.kubernetes.io/is-default-class: "true"

Docs: Grafana

Mon, 01 Jan 0001 00:00:00 +0000

By default, Verrazzano automatically installs and configures a Grafana database. However, you can use your own external database.

If you prefer to use your own Grafana database, complete the following steps:

Create a secret named grafana-db in the verrazzano-install namespace which contains the login credentials. For example:

$ # Load the login credentials into variables
$ ROOT_SECRET=$(echo <database root user secret> | base64)
$ USER=$(echo <database user> | base64)
$ USER_SECRET=$(echo <database user secret> | base64)
$ #
$ # Create the secret
$ kubectl apply -f - <<-EOF
apiVersion: v1
kind: Secret
metadata:
  name: grafana-db
  namespace: verrazzano-install
type: Opaque
data:
  root-password: $ROOT_SECRET
  username: $USER
  password: $USER_SECRET
EOF

Configure the Grafana component of the Verrazzano custom resource. For example:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: grafana-db-example
spec:
  profile: dev
  components:
    grafana:
      database:
        host: mysql.verrazzano-install.svc.cluster.local
        name: grafana

For more information about the component definition, see Grafana component in the Verrazzano custom resource.

Docs: Ingress

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano uses NGINX for ingress to Verrazzano system components and Istio for application ingress. You can customize the NGINX and Istio ingress installation configurations using Helm overrides specified in the Verrazzano custom resource. For example, the following Verrazzano custom resource overrides the shape of an Oracle Cloud Infrastructure load balancer for both NGINX and Istio ingresses.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-lb-settings
spec:
  profile: prod
  components:
    ingress:
      type: LoadBalancer
      overrides:
      - values:
          controller:
            service:
              annotations:
                service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
                service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              ingressGateways:
                - enabled: true
                  name: istio-ingressgateway
                  k8s:
                    serviceAnnotations:
                      service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                      service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
                      service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"

For more information about setting component overrides, see Customizing the Chart Before Installing.

Docs: Istio

Mon, 01 Jan 0001 00:00:00 +0000

You can customize Verrazzano Istio component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the Istio component.

Path to Field	Description
`spec.components.istio.egress.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.istio.egress.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the Istio gateway pods across the available nodes. spec: components: istio: egress: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - istio-egressgateway topologyKey: kubernetes.io/hostname
`spec.components.istio.ingress.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.istio.ingress.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the Istio gateway pods across the available nodes. spec: components: istio: ingress: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - istio-ingressgateway topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3 for istio-ingressgateway and istio-egressgateway
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution for istio-ingressgateway and istio-egressgateway

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              egressGateways:
                - enabled: true
                  k8s:
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - podAffinityTerm:
                              labelSelector:
                                matchExpressions:
                                  - key: app
                                    operator: In
                                    values:
                                      - istio-egressgateway
                              topologyKey: kubernetes.io/hostname
                            weight: 100
                    replicaCount: 3
                  name: istio-egressgateway
              ingressGateways:
                - enabled: true
                  k8s:
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - podAffinityTerm:
                              labelSelector:
                                matchExpressions:
                                  - key: app
                                    operator: In
                                    values:
                                      - istio-ingressgateway
                              topologyKey: kubernetes.io/hostname
                            weight: 100
                    replicaCount: 3
                    service:
                      type: LoadBalancer
                  name: istio-ingressgateway

Docs: Load Balancers on OCI

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano sets up the following load balancers on Kubernetes at installation:

Load balancer for NGINX ingress
Load balancer for Istio ingress

Verrazzano allows customizing the load balancers allocated by Oracle Cloud Infrastructure (OCI) using annotations defined by the OCI Cloud Controller Manager (OCI-CCM). For a detailed description of different load balancer customization annotations, see the documentation here.

This document describes how to use these annotations to customize the following settings for Verrazzano load balancers:

Load balancer shape
Load balancer min/max bandwidth
Private IP address and subnet placement

Customize the load balancer shape

At installation, Verrazzano lets you customize the shape and size of the load balancers created. The shape of an OCI load balancer specifies its maximum total bandwidth. By default, load balancers are created with a shape of 100Mbps. Other shapes are available, including 400Mbps and 8000Mbps.

For more details on service limits and shape, see here.

For example, you can set up a flexible NGINX load balancer with a min and max bandwidth as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    ingress:
      type: LoadBalancer
      overrides:
        - values:
            controller:
              service:
                annotations:
                  service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                  service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
                  service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"

For example, you can set up a flexible Istio load balancer with a min and max bandwidth as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              components:
                ingressGateways:
                  - enabled: true
                    name: istio-ingressgateway
                    k8s:
                      serviceAnnotations:
                        service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                        service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
                        service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"

Use private IP addresses with a load balancer

At installation, Verrazzano lets you customize the IP address and subnet of the load balancers created. This is achieved using OCI-CCM annotations on the NGINX and Istio load balancer services, as documented here.

The following example configures the NGINX load balancer service to have a private load balancer IP address on the private subnet identified by the OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for Istio.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    ingress:
      type: LoadBalancer
      overrides:
        - values:
            controller:
              service:
                annotations:
                  service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                  service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures the Istio ingress gateway service to have a private load balancer IP address on the private subnet identified by the OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for NGINX.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:  
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              components:
                ingressGateways:
                  - enabled: true
                    name: istio-ingressgateway
                    k8s:
                      serviceAnnotations:
                        service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                        service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures both NGINX and Istio to have a private load balancer IP address on the private subnet identified by the OCID ocid1.subnet.oc1.phx.aaaa..sdjxa.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    ingress:
      type: LoadBalancer
      overrides:
        - values:
            controller:
              service:
                annotations:
                  service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                  service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1.phx.aaaa..sdjxa"
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              components:
                ingressGateways:
                  - enabled: true
                    name: istio-ingressgateway
                    k8s:
                      serviceAnnotations:
                        service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                        service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

Docs: OpenSearch

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports two cluster topologies for an OpenSearch cluster:

A single-node cluster (master, ingest, and data roles performed by a single node).
A multi-node cluster configuration with separate master, data, and ingest nodes.

Installation Profiles describes the default OpenSearch cluster configurations provided by Verrazzano.

Configure cluster topology

You can customize the node characteristics of your OpenSearch cluster by using the spec.components.opensearch.nodes field in the Verrazzano custom resource. When installing or upgrading Verrazzano, you can use this field to define an OpenSearch cluster using node groups.

The following example overrides the dev installation profile, OpenSearch configuration (a single-node cluster with 1Gi of memory and ephemeral storage) to use a multi-node cluster (three master nodes, and three combination data/ingest nodes) with persistent storage.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-opensearch-example
spec:
  profile: dev
  components:
    opensearch:
      nodes:
        - name: master
          replicas: 3
          roles:
            - master
          storage:
            size: 50Gi
          resources:
            requests:
              memory: 1.5Gi
        - name: data-ingest
          replicas: 3
          roles:
            - data
            - ingest
          storage:
            size: 100Gi
          resources:
            requests:
              memory: 1Gi
        # Override the default node groups because we are providing our own topology.
        - name: es-master
          replicas: 0
        - name: es-data
          replicas: 0
        - name: es-ingest
          replicas: 0

Listing the pods and persistent volumes in the verrazzano-system namespace for the previous configuration shows the expected nodes are running with the appropriate data volumes.

$ kubectl get pvc,pod -l verrazzano-component=opensearch -n verrazzano-system

# Sample output
NAME                                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/elasticsearch-master-vmi-system-master-0   Bound    pvc-9ace042a-dd68-4975-816d-f2ca0dc4d9d8   50Gi       RWO            standard       5m22s
persistentvolumeclaim/elasticsearch-master-vmi-system-master-1   Bound    pvc-8bf68c2c-235e-4bd5-8741-5a5cd3453934   50Gi       RWO            standard       5m21s
persistentvolumeclaim/elasticsearch-master-vmi-system-master-2   Bound    pvc-da8a48b1-5762-4669-98f0-8479f30043fc   50Gi       RWO            standard       5m21s
persistentvolumeclaim/vmi-system-data-ingest                     Bound    pvc-7ad9f275-632b-4aac-b7bf-c5115215937c   100Gi      RWO            standard       5m23s
persistentvolumeclaim/vmi-system-data-ingest-1                   Bound    pvc-8a293e51-2c20-4cae-916b-1ce46a780403   100Gi      RWO            standard       5m23s
persistentvolumeclaim/vmi-system-data-ingest-2                   Bound    pvc-0025fcef-1d8c-4307-977c-3921545c6730   100Gi      RWO            standard       5m22s

NAME                                                   READY   STATUS     RESTARTS   AGE
pod/coherence-operator-6ffb6bbd4d-bpssc                1/1     Running    1          8m2s
pod/fluentd-ndshl                                      2/2     Running    0          5m51s
pod/oam-kubernetes-runtime-85cfd899d8-z9gv6            1/1     Running    0          8m14s
pod/verrazzano-application-operator-5fbcdf6655-72tw9   1/1     Running    0          7m49s
pod/verrazzano-authproxy-5f9d479455-5bvvt              2/2     Running    0          7m43s
pod/verrazzano-console-5b857d7b47-djbrk                2/2     Running    0          5m51s
pod/verrazzano-monitoring-operator-b4b446567-pgnfw     2/2     Running    0          5m51s
pod/vmi-system-data-ingest-0-5485dcd95d-rkhvk          2/2     Running    0          5m21s
pod/vmi-system-data-ingest-1-8d7db6489-kdhbv           2/2     Running    1          5m21s
pod/vmi-system-data-ingest-2-699d6bdd9c-z7nzx          2/2     Running    0          5m21s
pod/vmi-system-grafana-7947cdd84b-b7mks                2/2     Running    0          5m21s
pod/vmi-system-kiali-6c7bd6658b-d2zq9                  2/2     Running    0          5m37s
pod/vmi-system-kibana-7d47f65dfc-zhjxp                 2/2     Running    0          5m21s
pod/vmi-system-master-0                                2/2     Running    0          5m21s
pod/vmi-system-master-1                                2/2     Running    0          5m21s
pod/vmi-system-master-2                                2/2     Running    0          5m21s
pod/weblogic-operator-666b548749-lj66t                 2/2     Running    0          7m48s

Running the command kubectl describe pod -n verrazzano-system vmi-system-data-ingest-0-5485dcd95d-rkhvk shows the requested amount of memory.

Containers:
  es-data:
    ...
    Requests:
      memory:   1Gi

Configure Index State Management policies

Index State Management policies configure OpenSearch to manage the data in your indices. Policies can be used to automatically rollover and prune old data, preventing your OpenSearch cluster from running out of disk space.

The following policy example configures OpenSearch to manage indices matching the pattern my-app-*. The data in these indices will be automatically pruned every 14 days, and will be rolled over if an index meets at least one of the following criteria:

Is three or more days old
Contains 1,000 documents or more
Is 10 GB in size or larger

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-opensearch-example
spec:
  profile: dev
  components:
    opensearch:
      policies:
        - policyName: my-app
          indexPattern: my-app-*
          minIndexAge: 14d
          rollover:
            minIndexAge: 3d
            minDocCount: 1000
            minSize: 10Gb

Install OpenSearch and OpenSearch Dashboards plug-ins

Verrazzano supports OpenSearch and OpenSearch Dashboard plug-in installation by providing plug-ins in the Verrazzano custom resource. To install plug-ins for OpenSearch, you define the field spec.components.opensearch.plugins in the Verrazzano custom resource.

The following Verrazzano custom resource example installs the analysis-stempel and opensearch-anomaly-detection plug-ins for OpenSearch:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-opensearch-example
spec:
  profile: dev
  components:
    opensearch:
      plugins:
        enabled: true
        installList:
          - analysis-stempel
          - https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-anomaly-detection/2.2.0.0/opensearch-anomaly-detection-2.2.0.0.zip

There are three ways to define a plug-in in the plugins.installList:

Define a plug-in by name:

There are some pre-built additional plug-ins that you can install by name.
```
installList:
        - analysis-icu
```

Define a plug-in from a remote ZIP file:

Provide the URL to a remote ZIP file that contains the required plug-in.

installList:
        - https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-anomaly-detection/2.2.0.0/opensearch-anomaly-detection-2.2.0.0.zip

Define a plug-in using Maven coordinates:

Provide the Maven coordinates for the available artifacts and versions hosted on Maven Central.
```
installList:
        - org.opensearch.plugin:opensearch-anomaly-detection:2.2.0.0
```

NOTE

Your environment must be able to connect to the Internet to access the provided plug-in URL or Maven Central to install the plug-in. If there is any error during plug-in installation, then the OS pods (one per deployment) will go into the CrashLoopBackOff state. Check the logs for the exact reason of the failure. In the case of an Internet issue, you might see SocketException or UnknownHostException exceptions in the logs. To resolve this issue, make sure that the pods are connected to the Internet.
Adding a new plug-in in the plugins.installList or removing a plug-in from the plugins.installList will result in restarting the OpenSearch related pods.
To be compatible, major, minor, and patch plug-in versions must match OpenSearch major, minor, and patch versions. For example, plug-ins versions 2.3.0.x are compatible only with OpenSearch version 2.3.0.

For OpenSearch Dashboard, you can provide the plug-ins by defining the field spec.components.opensearch-dashboards.plugins in the Verrazzano custom resource.

Here is a Verrazzano custom resource example to install plug-ins for the OpenSearch Dashboards:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-opensearch-example
spec:
  profile: dev
  components:
    opensearchDashboards:
      plugins:
        enabled: true
        installList:
          - <URL to OpenSearch Dashboard plugin ZIP file>

Docs: Persistent Storage

Mon, 01 Jan 0001 00:00:00 +0000

The following components can use persistent storage:

OpenSearch
OpenSearch Dashboards
Prometheus
Grafana
Keycloak/MySQL

By default, each Verrazzano installation profile has different storage characteristics. The dev profile uses ephemeral storage only, but in all other profiles, each of the listed components use persistent storage. For more information, see Profile Configurations.

NOTE

Ephemeral storage is not recommended for use in production; Kubernetes pods can be restarted at any time, leading to a loss of data and system instability if non-persistent storage is used. Persistent storage is recommended for all use cases beyond evaluation or development.

While each profile has its own default persistence settings, in each case you have the option to override the profile defaults to customize your persistence settings.

You can customize the persistence settings for these components through the VerrazzanoSpec, as follows:

Overriding the persistence settings for all components (Keycloak, Grafana, Prometheus, OpenSearch, and OpenSearch Dashboards) by using the defaultVolumeSource field.
Overriding the persistence settings for Keycloak by using the volumeSource field on that component’s configuration.

You can set the global defaultVolumeSource and component-level volumeSource fields to one of the following values:

Value	Storage
`emptyDir`	Ephemeral storage; should not be used for production scenarios.
`persistentVolumeClaim`	A `PersistentVolumeClaimVolumeSource` where the `claimSource` field references a named `volumeClaimSpecTemplate`.

When you want to use a persistentVolumeClaim to override the storage settings for components, you must do the following:

Create a volumeClaimSpecTemplate which identifies the desired persistence settings.
Configure a persistentVolumeClaim for the component where the claimName field references the template you created previously.

This lets you create named persistence settings that can be shared across multiple components within a Verrazzano configuration. Note that the existence of a persistence template in the volumeClaimSpecTemplates list does not directly result in the creation of a persistent volume, or affect any component storage settings until it is referenced by either defaultVolumeSource or a specific component’s volumeSource.

Examples

Review the following customizing persistent storage examples:

Customize persistence globally using `defaultVolumeSource`

If defaultVolumeSource is configured, then that setting will be used for all components that require storage.

For example, the following Verrazzano configuration uses the prod profile, but disables persistent storage for all components.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: no-storage-prod
spec:
  profile: prod
  defaultVolumeSource:
      emptyDir: {}

The following example uses persistentVolumeClaim to override persistence settings globally for a prod profile, to use 100Gi volumes for all components, instead of the default of 50Gi.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: prod-global-override
spec:
  profile: prod
  defaultVolumeSource:
    persistentVolumeClaim:
      claimName: globalOverride
  volumeClaimSpecTemplates:
    - metadata:
        name: globalOverride
      spec:
        resources:
          requests:
            storage: 100Gi

The following example uses a managed-cluster profile but overrides the persistence settings to use ephemeral storage.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: mgdcluster-empty-storage-example
spec:
  profile: managed-cluster
  defaultVolumeSource:
    emptyDir: {}  # Use emphemeral storage for all Components unless overridden

Customize PersistentVolumeClaim settings for Keycloak using `volumeSource`

The following example Verrazzano configuration enables a 100Gi PersistentVolumeClaim for the MySQL component in Keycloak in a dev profile configuration. This overrides the default of ephemeral storage for Keycloak in that profile, while retaining the default storage settings for other components.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: dev-mysql-storage-example
spec:
  profile: dev
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql  # Use the "mysql" PVC template for the MySQL volume configuration
  volumeClaimSpecTemplates:
  - metadata:
      name: mysql      
    spec:
      resources:
        requests:
          storage: 100Gi

Use global and local persistence settings together

The following example uses a dev installation profile, but overrides the profile persistence settings to:

Use 200Gi volumes for all components by default.
Use a 100Gi volume for the MySQL instance associated with Keycloak.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: dev-storage-example
spec:
  profile: dev
  defaultVolumeSource:
    persistentVolumeClaim:
      claimName: vmi     # Set storage globally for the metrics stack
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql  # Set storage separately for keycloak's MySql instance
  volumeClaimSpecTemplates:
    - metadata:
        name: mysql
      spec:
        resources:
          requests:
            storage: 100Gi
    - metadata:
        name: vmi
      spec:
        resources:
          requests:
            storage: 200Gi

Docs: Prometheus

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano installs Prometheus components, including Prometheus Operator and Prometheus, using the kube-prometheus-stack Helm chart. You can customize the installation configuration using Helm overrides specified in the Verrazzano custom resource. For example, the following Verrazzano custom resource overrides the number of Prometheus replicas.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            prometheus:
              prometheusSpec:
                replicas: 3

To enable Alertmanager, use the following Verrazzano custom resource:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            alertmanager:
              enabled: true
              alertmanagerSpec:
                podMetadata:
                  annotations:
                    sidecar.istio.io/inject: "false"

For more information about setting component overrides, see Customizing the Chart Before Installing.

Verrazzano Enterprise Container Platform – Customize Verrazzano

Docs: AuthProxy

Docs: Certificates

NOTE

Use the Verrazzano self-signed CA

Use a custom CA

Use LetsEncrypt certificates

NOTE

NOTE

LetsEncrypt staging versus production

Docs: DNS

How Verrazzano constructs a DNS domain

Prerequisites

Create an Oracle Cloud Infrastructure API secret in the target cluster

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Installation

Docs: External Load Balancers

Prepare the external load balancers

Verrazzano installation options

Example Custom Resource with management and application external load balancers

Docs: Network File System

Docs: Grafana

Docs: Ingress

Docs: Istio

Docs: Load Balancers on OCI

Customize the load balancer shape

Use private IP addresses with a load balancer

Docs: OpenSearch

Configure cluster topology

Configure Index State Management policies

Install OpenSearch and OpenSearch Dashboards plug-ins

NOTE

Docs: Persistent Storage

NOTE

Examples

Customize persistence globally using defaultVolumeSource

Customize PersistentVolumeClaim settings for Keycloak using volumeSource

Use global and local persistence settings together

Docs: Prometheus

Customize persistence globally using `defaultVolumeSource`

Customize PersistentVolumeClaim settings for Keycloak using `volumeSource`