Verrazzano Enterprise Container Platform – Customize Installations

Docs: Customize DNS

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports three DNS choices for Verrazzano services and applications:

Free wildcard DNS services (nip.io and sslip.io)
Oracle Cloud Infrastructure DNS managed by Verrazzano
Custom (user-managed) DNS

How Verrazzano constructs a DNS domain

Regardless of which DNS management you use, the value in the spec.environmentName field in your installation will be prepended to the configured domain in the spec.components.dns section of the custom resource, to form the full DNS domain name used to access Verrazzano endpoints.

For example, if spec.environmentName is set to sales and the domain is configured in spec.components.dns as us.example.com, Verrazzano will create sales.us.example.com as the DNS domain for the installation.

Verrazzano can be configured to use either the nip.io or sslip.io free wildcard DNS services. When queried with a hostname with an embedded IP address, wildcard DNS services return that IP address.

For example, using the nip.io service, the following DNS names all map to the IP address 10.0.0.1:

10.0.0.1.nip.io
app.10.0.0.1.nip.io
customer1.app.10.0.0.1.nip.io

To configure Verrazzano to use one of these services, set the spec.wildcard.domain field in the Verrazzano custom resource to either nip.io or sslip.io; the default is nip.io.

For example, the following configuration uses sslip.io, instead of nip.io, for wildcard DNS with a dev installation profile:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    dns:
      wildcard:
        domain: sslip.io

Verrazzano can directly manage records in Oracle Oracle Cloud Infrastructure DNS when configured to use the spec.components.dns.oci field. This is achieved through the External DNS Service, which is a component that is conditionally installed when Oracle Cloud Infrastructure DNS is configured for DNS management in Verrazzano.

Prerequisites

The following prerequisites must be met before using Oracle Cloud Infrastructure DNS with Verrazzano:

You must have control of a DNS domain.
You must have an Oracle Cloud Infrastructure DNS Service Zone that is configured to manage records for that domain. Verrazzano also supports the use of both GLOBAL and PRIVATE Oracle Cloud Infrastructure DNS zones.

A DNS Service Zone is a distinct portion of a domain namespace. You must ensure that the zone is appropriately associated with a parent domain. For example, an appropriate zone name for parent domain example.com is us.example.com.

To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure CLI:
```
$ oci dns zone create \
    -c <compartment ocid> \
    --name <zone-name-prefix>.example.com \
    --zone-type PRIMARY
```
To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure Console, see Managing DNS Service Zones.

You must have a valid Oracle Cloud Infrastructure API signing key that can be used to communicate with Oracle Cloud Infrastructure DNS in your tenancy.

For example, you can create an API signing key using the Oracle Cloud Infrastructure CLI:

  $ oci setup keys --key-name myapikey
  Enter a passphrase for your private key (empty for no passphrase):
  Public key written to: /Users/jdoe/.oci/myapikey_public.pem
  Private key written to: /Users/jdoe/.oci/myapikey.pem
  Public key fingerprint: 39:08:44:69:9f:f5:73:86:7a:46:d8:ad:34:4f:95:29


      If you haven't already uploaded your API signing public key through the
      console, follow the instructions on the page linked below in the section
      'How to upload the public key':

          https://docs.cloud.oracle.com/Content/API/Concepts/apisigningkey.htm#How2

After the key pair has been created, you must upload the public key to your account in your Oracle Cloud Infrastructure tenancy. For details, see the Oracle Cloud Infrastructure documentation, Required Keys and OCIDs.

Create an Oracle Cloud Infrastructure API secret in the target cluster

To communicate with Oracle Cloud Infrastructure DNS to manage DNS records, Verrazzano needs to be made aware of the necessary API credentials.
A generic Kubernetes secret must be created in the cluster’s verrazzano-install namespace with the required credentials. That secret must then be referenced by the custom resource that is used to install Verrazzano.

After you have an Oracle Cloud Infrastructure API key ready for use, create a YAML file, oci.yaml, with the API credentials in the form:

auth:
  region: <oci-region>
  tenancy: <oci-tenancy-ocid>
  user: <oci-user-ocid>
  key: |
    <oci-api-private-key-file-contents>
  fingerprint: <oci-api-private-key-fingerprint>

This information typically can be found in your Oracle Cloud Infrastructure CLI config file or in the Oracle Cloud Infrastructure Console. The <oci-api-private-key-file-contents> contents are the PEM-encoded contents of the key_file value within the Oracle Cloud Infrastructure CLI configuration profile.

For example, your oci.yaml file will look similar to the following:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1.....
  user: ocid1.user.oc1.....
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:d3:4c:gh:fd:9e:27:g8:b9:0d:9f:00:22:33:c3:gg

Verrazzano also supports the use of instance principals to communicate with Oracle Cloud Infrastructure in order to create or update Oracle Cloud Infrastructure DNS records. Instance principal requires some prerequisites that can be found here.

When using instance principals, your oci.yaml file will look as follows:

auth:
  authtype: instance_principal

Then, you can create a generic Kubernetes secret in the cluster’s verrazzano-install namespace using kubectl.

$ kubectl create secret generic -n verrazzano-install <secret-name> --from-file=<path-to-oci-yaml-file>

For example, to create a secret named oci from a file oci.yaml, do the following:

$ kubectl create secret generic -n verrazzano-install oci --from-file=oci.yaml

This secret will later be referenced from the Verrazzano custom resource used during installation.

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Verrazzano also provides a helper script to create the necessary Kubernetes secret based on your Oracle Cloud Infrastructure CLI config file, assuming that you have the Oracle Cloud Infrastructure CLI installed and a valid Oracle Cloud Infrastructure CLI profile with the required API key information. The script create_oci_config_secret.sh reads your Oracle Cloud Infrastructure CLI configuration file to create the secret.

First, download the create_oci_config_secret.sh script:

$ curl \
    -o ./create_oci_config_secret.sh \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.2.2/platform-operator/scripts/install/create_oci_config_secret.sh

Next, set your KUBECONFIG environment variable to point to your cluster and run create_oci_config_secret.sh -h to display the script options:

$ chmod +x create_oci_config_secret.sh
$ export KUBECONFIG=<kubeconfig-file>
$ ./create_oci_config_secret.sh  -h
usage: ./create_oci_config_secret.sh [-o oci_config_file] [-s config_file_section]
  -o oci_config_file         The full path to the Oracle Cloud Infrastructure configuration file (default ~/.oci/config)
  -s config_file_section     The properties section within the Oracle Cloud Infrastructure configuration file.  Default is DEFAULT
  -k secret_name             The secret name containing the Oracle Cloud Infrastructure configuration.  Default is oci
  -c context_name            The kubectl context to use
  -a auth_type               The auth_type to be used to access Oracle Cloud Infrastructure. Valid values are user_principal/instance_principal. Default is user_principal.
  -h                         Help

For example, to have the script create the YAML file using your [DEFAULT] Oracle Cloud Infrastructure CLI profile and then create a Kubernetes secret named oci, you can run the script with no arguments, as follows:

$ ./create_oci_config_secret.sh
secret/oci created

The following example creates a secret myoci using an Oracle Cloud Infrastructure CLI profile named [dev]:

$ ./create_oci_config_secret.sh -s dev -k myoci
secret/myoci created

When using instance principals all other parameters will be ignored automatically. The following example creates a secret myoci using Oracle Cloud Infrastructure instance principal:

$ ./create_oci_config_secret.sh -a instance_principal
secret/myoci created

Installation

After the Oracle Cloud Infrastructure API secret is created, create a Verrazzano custom resource for the installation that is configured to use Oracle Cloud Infrastructure DNS, and reference the secret you created.

As a starting point, download the sample Verrazzano custom resource install-oci.yaml file for Oracle Cloud Infrastructure DNS:

$ curl \
    -o ./install-oci.yaml \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.2.2/platform-operator/config/samples/install-oci.yaml

Edit the install-oci.yaml file to provide values for the following configuration settings in the custom resource spec:

spec.environmentName
spec.components.dns.oci.ociConfigSecret
spec.components.dns.oci.dnsZoneCompartmentOCID
spec.components.dns.oci.dnsZoneOCID
spec.components.dns.oci.dnsZoneName
spec.components.dns.oci.dnsScope

The field spec.components.dns.oci.ociConfigSecret should reference the secret created earlier. For details on the Oracle Cloud Infrastructure DNS configuration settings, see spec.components.dns.oci.

For example, a custom resource for a prod installation profile using Oracle Cloud Infrastructure DNS might look as follows, yielding a domain of myenv.example.com (Oracle Cloud Infrastructure identifiers redacted):

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com

If using a private DNS zone, then the same prod installation profile using Oracle Cloud Infrastructure DNS will look as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: my-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com
        dnsScope: PRIVATE

After the custom resource is ready, apply it using kubectl apply -f <path-to-custom-resource-file>.

You can specify your own externally managed, custom DNS domain. In this scenario, you manage your own DNS domain and all DNS records in that domain.

An externally managed DNS domain is specified in the spec.components.dns.external.suffix field of the Verrazzano custom resource.

When using an externally managed DNS domain, you are responsible for:

Configuring A records for Verrazzano ingress points (load balancers)
Configuring CNAME records for hostnames in the domain that point to the A records, as needed

The Verrazzano installer searches the DNS zone you provide for two specific A records.
These are used to configure the cluster and should refer to external addresses of the load balancers provisioned by the user.

The A records need to be created manually.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example, if spec.environmentName is set to myenv, and spec.components.dns.external.suffix is set to example.com, the A records would need to be set up as follows:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

This example assumes that load balancers exist for ingress-mgmt on 198.51.100.10 and for ingress-verrazzano on 203.0.113.10.

For a more complete example, see the documentation for setting up Verrazzano on the Oracle Cloud Native Environment Platform.

Docs: Customize Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano issues certificates to secure access from external clients to secure system endpoints.
A certificate from a certificate authority (CA) must be configured to issue the endpoint certificates in one of the following ways:

Let Verrazzano generate a self-signed CA (the default).
Configure a CA that you provide.
Configure LetsEncrypt as the certificate issuer (requires Oracle Cloud Infrastructure DNS).

In all cases, Verrazzano uses cert-manager to manage the creation of certificates.

NOTE

Self-signed certificate authorities generate certificates that are NOT signed by a trusted authority; typically, they are not used in production environments.

Use the Verrazzano self-signed CA

By default, Verrazzano creates its own self-signed CA. No configuration is required.

Use a custom CA

If you want to provide your own CA, you must:

(Optional) Create your own signing key pair and CA certificate.

For example, you can use the openssl CLI to create a key pair for the nip.io domain:
```
# Generate a CA private key
$ openssl genrsa -out tls.key 2048

# Create a self signed certificate, valid for 10yrs with the 'signing' option set
$ openssl req -x509 -new -nodes -key tls.key -subj "/CN=*.nip.io" -days 3650 -reqexts v3_req -extensions v3_ca -out tls.crt
```
The output of these commands will be two files, tls.key and tls.crt, the key and certificate for your signing key pair. These files must be named in that manner for the next step.

If you already have generated your own key pair, you must name the private key and certificate, tls.key and tls.crt, respectively. If your issuer represents an intermediate, ensure that tls.crt contains the issuer’s full chain in the correct order.

You can find more details on providing your own CA, in the cert-manager CA documentation.

Save your signing key pair as a Kubernetes secret.

$ kubectl create ns mynamespace
$ kubectl create secret tls myca --namespace=mynamespace --cert=tls.crt --key=tls.key

Specify the secret name and namespace location in the Verrazzano custom resource.

The custom CA secret must be provided to cert-manager using the following fields in spec.components.certManager.certificate.ca in the Verrazzano custom resource:
- spec.components.certManager.certificate.ca.secretName
- spec.components.certManager.certificate.ca.clusterResourceNamespace

For example, if you created a CA secret named myca in the namespace mynamespace, you would configure it as shown:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: custom-ca-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        ca:
          secretName: myca
          clusterResourceNamespace: mynamespace

Use LetsEncrypt certificates

You can configure Verrazzano to use certificates generated by LetsEncrypt. LetsEncrypt implements the ACME protocol, which provides a standard protocol for the automated issuance of certificates signed by a trusted authority. This is managed through the spec.components.certManager.certificate.acme field in the Verrazzano custom resource.

NOTE

Using LetsEncrypt for certificates also requires using Oracle Cloud Infrastructure DNS for DNS management. For details, see the Customize DNS page.

To configure cert-manager to use LetsEncrypt as the certificates provider, you must configure a cert-manager ACME provider with the following values in the Verrazzano custom resource:

Set the spec.components.certManager.certificate.acme.provider field to letsEncrypt.
Set the spec.components.certManager.certificate.acme.emailAddress field to a valid email address for the letsEncrypt account.
(Optional) Set the spec.components.certManager.certificate.acme.environment field to either staging or production (the default).

The following example configures Verrazzano to use the LetsEncrypt production environment by default, with Oracle Cloud Infrastructure DNS for DNS record management:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

The following example configures Verrazzano to use the LetsEncrypt staging environment with Oracle Cloud Infrastructure DNS:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
          environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

NOTE

Certificates issued by the LetsEncrypt staging environment are signed by untrusted authorities, similar to self-signed certificates. They are typically not used in production environments.

LetsEncrypt staging versus production

LetsEncrypt provides rate-limits on generated certificates to ensure fair usage across all clients. The production environment limits can be exceeded more frequently in environments where Verrazzano may be being installed or reinstalled frequently (like a test environment). This can result in failed installations due to rate limit exceptions on certificate generation.

In such environments, it is better to use the LetsEncrypt staging environment, which has much higher limits than the production environment. For test environments, the self-signed CA also may be more appropriate to completely avoid LetsEncrypt rate limits.

Docs: Customize Load Balancers on OKE

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano sets up the following load balancers on Kubernetes at installation:

Load balancer for NGINX ingress
Load balancer for Istio ingress

Verrazzano allows customizing the load balancers allocated by Oracle Container Engine (OKE) using annotations defined by OKE. For a detailed description of different load balancer customization annotations, see the OKE documentation here.

This document describes how to use these annotations to customize the following settings for Verrazzano load balancers:

Load balancer shape
Private IP address and subnet placement

Customize the load balancer shape

At installation, Verrazzano lets you customize the shape and size of the load balancers created. Oracle Cloud Infrastructure offers a flexible load balancer which uses Dynamic Shape:

10 Mbps
100 Mbps
400 Mbps
8,000 Mbps

For more details on service limits and shape, see here.

For example, you can set up an NGINX load balancer with 10Mbps as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
        value: "10Mbps"

For example, you can set up an Istio load balancer with 10Mbps as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      istioInstallArgs:
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
        value: "10Mbps"

Use private IP addresses with a load balancer

At installation, Verrazzano lets you customize the IP address and subnet of the load balancers created. This is achieved using OKE annotations on the NGINX and Istio load balancer services, as documented here.

The following example configures the NGINX load balancer service to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for Istio:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"    
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures the Istio ingress gateway service to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for NGINX:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer      
    istio:
      istioInstallArgs:
        - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
          value: "true"
        - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
          value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures both NGINX and Istio to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"
    istio:
      istioInstallArgs:
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

Docs: Customize AuthProxy

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano AuthProxy component enables authentication and authorization for Keycloak users accessing Verrazzano resources. You can customize the AuthProxy component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the AuthProxy component.

Path to Field Description

spec.components.authProxy.kubernetes.replicas The number of pods to replicate. The default is 2 for the prod profile and 1 for all other profiles.

Path to Field	Description
`spec.components.authProxy.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.authProxy.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes. spec: components: authProxy: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - verrazzano-authproxy topologyKey: kubernetes.io/hostname

spec.components.authProxy.kubernetes.affinity

The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes.

spec:
  components:
    authProxy:
      kubernetes:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    authproxy:
      kubernetes:
        replicas: 3
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - verrazzano-authproxy
                topologyKey: kubernetes.io/hostname

Docs: Customize Istio

Mon, 01 Jan 0001 00:00:00 +0000

You can customize Verrazzano Istio component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the Istio component.

Path to Field	Description
`spec.components.istio.egress.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.istio.egress.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the Istio gateway pods across the available nodes. spec: components: istio: egress: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - istio-egressgateway topologyKey: kubernetes.io/hostname
`spec.components.istio.ingress.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.istio.ingress.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the Istio gateway pods across the available nodes. spec: components: istio: ingress: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - istio-ingressgateway topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3 for istio-ingressgateway and istio-egressgateway
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution for istio-ingressgateway and istio-egressgateway

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    istio:
      ingress:
        kubernetes:
          replicas: 3
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - weight: 25
                    labelSelector:
                      matchExpressions:
                        - key: app
                          operator: In
                          values:
                            - istio-ingressgateway
                    topologyKey: kubernetes.io/hostname
      egress:
        kubernetes:
          replicas: 3
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - istio-egressgateway
                  topologyKey: kubernetes.io/hostname

Docs: Customize OpenSearch

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports two cluster topologies for an OpenSearch cluster:

A single-node cluster (master, ingest, and data roles performed by a single node).
A multi-node cluster configuration with separate master, data, and ingest nodes.

Installation Profiles describes the default OpenSearch cluster configurations provided by Verrazzano.

You can customize the node characteristics of your OpenSearch cluster by using the spec.components.elasticsearch.installArgs field in the Verrazzano custom resource. When installing Verrazzano, you can use this field to specify a list of Helm value overrides for the OpenSearch configuration.

These Helm overrides let you customize the following node characteristics:

Number of node replicas.
Memory request size per node.
Storage request size (data nodes only).

The following table lists the Helm values in the Verrazzano system chart related to OpenSearch nodes.

Name	Description
`nodes.master.replicas`	Number of master node replicas.
`nodes.master.requests.memory`	Memory request amount expressed as a Quantity.
`nodes.ingest.replicas`	Number of ingest node replicas.
`nodes.ingest.requests.memory`	Memory request amount expressed as a Quantity.
`nodes.data.replicas`	Number of data node replicas.
`nodes.data.requests.memory`	Memory request amount expressed as a Quantity.
`nodes.data.requests.storage`	Storage request amount expressed as a Quantity.

The following example overrides the dev installation profile, OpenSearch configuration (a single-node cluster with 1Gi of memory and ephemeral storage) to use a multi-node cluster with persistent storage. Note that the public API references Elasticsearch, the API will change to OpenSearch in an upcoming release.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: custom-es-example
spec:
  profile: dev
  components:
    elasticsearch:
      installArgs:
      - name: nodes.master.replicas
        value: "1"
      - name: nodes.master.requests.memory
        value: "1G"
      - name: nodes.ingest.replicas
        value: "1"
      - name: nodes.ingest.requests.memory
        value: "1G"
      - name: nodes.data.replicas
        value: "3"
      - name: nodes.data.requests.memory
        value: "1.5G"
      - name: nodes.data.requests.storage
        value: "10Gi"

Listing the pods and persistent volumes in the verrazzano-system namespace for the previous configuration shows the expected nodes are running with the appropriate data volumes:

$ kubectl  get pvc,pod -n verrazzano-system

# Sample output
NAME                                                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/elasticsearch-master-vmi-system-es-master-0   Bound    pvc-8ffff457-4d72-4a72-89ba-2cdcb8eade38   10Gi       RWO            standard       6m51s
persistentvolumeclaim/vmi-system-es-data                            Bound    pvc-e32c2182-46ba-4789-b577-195874b3dd69   10Gi       RWO            standard       6m53s
persistentvolumeclaim/vmi-system-es-data-1                          Bound    pvc-67789196-d688-4d06-b074-77655a913552   10Gi       RWO            standard       6m53s
persistentvolumeclaim/vmi-system-es-data-2                          Bound    pvc-43e07e3e-0713-4ab1-ac3f-812069c35cbb   10Gi       RWO            standard       6m53s

NAME                                                   READY   STATUS    RESTARTS   AGE
pod/coherence-operator-6986d6cf95-6b58p                1/1     Running   2          7m3s
pod/fluentd-fn28c                                      2/2     Running   2          7m12s
pod/oam-kubernetes-runtime-679c6f6775-79tvm            1/1     Running   0          5m11s
pod/verrazzano-api-58c5f65c8-6zbpc                     2/2     Running   0          7m12s
pod/verrazzano-application-operator-5766b899fd-9fjhb   1/1     Running   0          4m55s
pod/verrazzano-console-6599854544-pw56c                2/2     Running   0          7m12s
pod/verrazzano-monitoring-operator-55877766d4-9ktvh    1/1     Running   0          7m12s
pod/verrazzano-operator-75b5cd49fc-68cm4               1/1     Running   0          7m12s
pod/vmi-system-es-data-0-5884cfb84d-hn8xg              2/2     Running   0          6m52s
pod/vmi-system-es-data-1-679775494f-pdwzf              2/2     Running   0          6m52s
pod/vmi-system-es-data-2-5886d745c5-6pscm              2/2     Running   0          6m52s
pod/vmi-system-es-ingest-795749ddd8-cs4pc              3/3     Running   0          6m52s
pod/vmi-system-es-master-0                             2/2     Running   0          6m51s
pod/vmi-system-grafana-b94fcbb67-ktwf8                 3/3     Running   0          6m52s
pod/vmi-system-kibana-6594cfccc-j8gp5                  3/3     Running   0          6m51s
pod/vmi-system-prometheus-0-75864fc668-s5xv8           4/4     Running   0          44s
pod/weblogic-operator-5bd7bb6fb5-wz5cr                 2/2     Running   0          6m30s

Note that the master node uses the same amount of persistent storage as is configured for the data nodes.

Running the command kubectl describe pod -n verrazzano-system vmi-system-es-data-0-5884cfb84d-hn8xg shows the requested amount of memory:

Containers:
  es-data:
    Container ID:  containerd://cc01f24b107da0e1e90a05a49c7fd969761f59a81316fa01f7cc56a166684628
    Image:         ghcr.io/verrazzano/opensearch:1.2.3-20220207214930-833b159de83
    Image ID:      ghcr.io/verrazzano/elasticsearch@sha256:3d2cbb539f9ebba991c6f36db4fbaa9dc9c03e6192a28787869f7850cc2bd66c
    Ports:         9200/TCP, 9300/TCP
    Host Ports:    0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 29 Jul 2021 06:04:17 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      memory:   1500M

Docs: Customize Ingress

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano uses NGINX for ingress to Verrazzano system components and Istio for application ingress. You can customize the NGINX and Istio ingress installation configurations using Helm overrides specified in the Verrazzano custom resource. For example, the following Verrazzano custom resource overrides the shape of an Oracle Cloud Infrastructure load balancer for both NGINX and Istio ingresses:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: custom-lb-settings
spec:
  profile: prod
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
        value: "10Mbps"
    istio:
      istioInstallArgs:
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
        value: "10Mbps"

The previous entries use dot notation to represent YAML values.

For example:

    - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
      value: "10Mbps"

Is translated into:

controler:
   service:
     annotations:
       service.beta.kubernetes.io/oci-load-balancer-shape: 10Mbps

For more information about setting component overrides, see Customizing the Chart Before Installing.

Docs: Customize Persistent Storage

Mon, 01 Jan 0001 00:00:00 +0000

The following components allow persistent storage:

OpenSearch
Prometheus
Grafana
Keycloak/MySQL

By default, each Verrazzano installation profile has different storage characteristics. The dev profile uses ephemeral storage only, but in all other profiles, each of the listed components use persistent storage. For more information, see Profile Configurations.

NOTE

Ephemeral storage is not recommended for use in production; Kubernetes pods can be restarted at any time, leading to a loss of data and system instability if non-persistent storage is used. Persistent storage is recommended for all use cases beyond evaluation or development.

While each profile has its own default persistence settings, in each case you have the option to override the profile defaults to customize your persistence settings.

Customize persistent storage

The following components can use persistent storage:

OpenSearch
OpenSearch Dashboards
Prometheus
Grafana
Keycloak

You can customize the persistence settings for these components through the VerrazzanoSpec, as follows:

Overriding the persistence settings for all components (Keycloak, Grafana, Prometheus, OpenSearch, and OpenSearch Dashboards) by using the defaultVolumeSource field.
Overriding the persistence settings for Keycloak by using the volumeSource field on that component’s configuration.

You can set the global defaultVolumeSource and component-level volumeSource fields to one of the following values:

Value	Storage
`emptyDir`	Ephemeral storage; should not be used for production scenarios.
`persistentVolumeClaim`	A `PersistentVolumeClaimVolumeSource` where the `claimSource` field references a named `volumeClaimSpecTemplate`.

When you want to use a persistentVolumeClaim to override the storage settings for components, you must do the following:

Create a volumeClaimSpecTemplate which identifies the desired persistence settings.
Configure a persistentVolumeClaim for the component where the claimName field references the template you created previously.

This lets you create named persistence settings that can be shared across multiple components within a Verrazzano configuration. Note that the existence of a persistence template in the volumeClaimSpecTemplates list does not directly result in the creation of a persistent volume, or affect any component storage settings until it is referenced by either defaultVolumeSource or a specific component’s volumeSource.

Examples

Review the following customizing persistent storage examples:

Customize persistence globally using `defaultVolumeSource`

If defaultVolumeSource is configured, then that setting will be used for all components that require storage.

For example, the following Verrazzano configuration uses the prod profile, but disables persistent storage for all components:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: no-storage-prod
spec:
  profile: prod
  defaultVolumeSource:
      emptyDir: {}

The following example uses persistentVolumeClaim to override persistence settings globally for a prod profile, to use 100Gi volumes for all components, instead of the default of 50Gi:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: prod-global-override
spec:
  profile: prod
  defaultVolumeSource:
    persistentVolumeClaim:
      claimName: globalOverride
  volumeClaimSpecTemplates:
    - metadata:
        name: globalOverride
      spec:
        resources:
          requests:
            storage: 100Gi

The following example uses a managed-cluster profile but overrides the persistence settings to use ephemeral storage:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: mgdcluster-empty-storage-example
spec:
  profile: managed-cluster
  defaultVolumeSource:
    emptyDir: {}  # Use emphemeral storage for all Components unless overridden

Customize PersistentVolumeClaim settings for Keycloak using `volumeSource`

The following example Verrazzano configuration enables a 100Gi PersistentVolumeClaim for the MySQL component in Keycloak in a dev profile configuration. This overrides the default of ephemeral storage for Keycloak in that profile, while retaining the default storage settings for other components:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: dev-mysql-storage-example
spec:
  profile: dev
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql  # Use the "mysql" PVC template for the MySQL volume configuration
  volumeClaimSpecTemplates:
  - metadata:
      name: mysql      
    spec:
      resources:
        requests:
          storage: 100Gi

Use global and local persistence settings together

The following example uses a dev installation profile, but overrides the profile persistence settings to:

Use 200Gi volumes for all components by default.
Use a 100Gi volume for the MySQL instance associated with Keycloak.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: dev-storage-example
spec:
  profile: dev
  defaultVolumeSource:
    persistentVolumeClaim:
      claimName: vmi     # set storage globally for the metrics stack
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql  # set storage separately for keycloak's MySql instance
  volumeClaimSpecTemplates:
    - metadata:
        name: mysql
      spec:
        resources:
          requests:
            storage: 100Gi
    - metadata:
        name: vmi
      spec:
        resources:
          requests:
            storage: 200Gi

Verrazzano Enterprise Container Platform – Customize Installations

Docs: Customize DNS

How Verrazzano constructs a DNS domain

Prerequisites

Create an Oracle Cloud Infrastructure API secret in the target cluster

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Installation

Docs: Customize Certificates

NOTE

Use the Verrazzano self-signed CA

Use a custom CA

Use LetsEncrypt certificates

NOTE

NOTE

LetsEncrypt staging versus production

Docs: Customize Load Balancers on OKE

Customize the load balancer shape

Use private IP addresses with a load balancer

Docs: Customize AuthProxy

Docs: Customize Istio

Docs: Customize OpenSearch

Docs: Customize Ingress

Docs: Customize Persistent Storage

NOTE

Customize persistent storage

Examples

Customize persistence globally using defaultVolumeSource

Customize PersistentVolumeClaim settings for Keycloak using volumeSource

Use global and local persistence settings together

Customize persistence globally using `defaultVolumeSource`

Customize PersistentVolumeClaim settings for Keycloak using `volumeSource`