Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: Application Deployment Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Developing and deploying an application to Verrazzano consists of:

Packaging the application as a Docker image.
Publishing the application’s Docker image to a container registry.
Applying the application’s Verrazzano components to the cluster.
Applying the application’s Verrazzano applications to the cluster.

This guide does not provide the full details for the first two steps. An existing example application Docker image has been packaged and published for use.

Verrazzano supports application definition using Open Application Model (OAM). Verrrazzano applications are composed of components and application configurations. This document demonstrates creating OAM resources that define an application as well as the steps required to deploy those resources.

What you need

About 10 minutes.
Access to an existing Kubernetes cluster with Verrazzano installed.
Access to the application’s image in GitHub Container Registry.

Confirm access using this command to pull the example’s Docker image:
```
$ docker pull ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.12-1-20210218160249-d8db8f3
```

Application development

This guide uses an example application which was written with Java and Helidon. For the implementation details, see the Helidon MP tutorial. See the application source code in the Verrazzano examples repository.

The example application is a JAX-RS service and implements the following REST endpoints:

/greet - Returns a default greeting message that is stored in memory. This endpoint accepts the GET HTTP request method.
/greet/{name} - Returns a greeting message including the name provided in the path parameter. This endpoint accepts the GET HTTP request method.
/greet/greeting - Changes the greeting message to be used in future calls to the other endpoints. This endpoint accepts the PUT HTTP request method and a JSON payload.

The following code shows a portion of the application’s implementation. The Verrazzano examples repository contains the complete implementation. An important detail here is that the application contains a single resource exposed on path /greet.

package io.helidon.examples.quickstart.mp;
...
@Path("/greet")
@RequestScoped
public class GreetResource {

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getDefaultMessage() {
        ...
    }

    @Path("/{name}")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getMessage(@PathParam("name") String name) {
        ...
    }

    @Path("/greeting")
    @PUT
    @Consumes(MediaType.APPLICATION_JSON)
    ...
    public Response updateGreeting(JsonObject jsonObject) {
        ...
    }

}

A Dockerfile is used to package the completed application JAR file into a Docker image. The following code shows a portion of the Dockerfile. The Verrazzano examples repository contains the complete Dockerfile. Note that the Docker container exposes a single port 8080.

FROM ghcr.io/oracle/oraclelinux:7-slim
...
CMD java -cp /app/helidon-quickstart-mp.jar:/app/* io.helidon.examples.quickstart.mp.Main
EXPOSE 8080

Application deployment

When you deploy applications with Verrazzano, the platform sets up connections, network policies, and ingresses in the service mesh, and wires up a monitoring stack to capture the metrics, logs, and traces. Verrazzano employs OAM Components to define the functional units of a system that are then assembled and configured by defining associated application configurations.

Verrazzano components

A Verrazzano OAM Component is a Kubernetes Custom Resource describing an application’s general composition and environment requirements. The following code shows the component for the example application used in this guide. This resource describes a component which is implemented by a single Docker image containing a Helidon application exposing a single endpoint.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.10-3-20201016220428-56fb4d4"
              ports:
                - containerPort: 8080
                  name: http

A brief description of each field of the component:

apiVersion - Version of the component custom resource definition
kind - Standard name of the component custom resource definition
metadata.name - The name used to create the component’s custom resource
metadata.namespace - The namespace used to create this component’s custom resource
spec.workload.kind - VerrazzanoHelidonWorkload defines a stateless workload of Kubernetes
spec.workload.spec.deploymentTemplate.podSpec.metadata.name - The name used to create the stateless workload of Kubernetes
spec.workload.spec.deploymentTemplate.podSpec.containers - The implementation containers
spec.workload.spec.deploymentTemplate.podSpec.containers.ports - Ports exposed by the container

Verrazzano application configurations

A Verrazzano application configuration is a Kubernetes Custom Resource which provides environment specific customizations. The following code shows the application configuration for the example used in this guide. This resource specifies the deployment of the application to the hello-helidon namespace. Additional runtime features are specified using traits, or runtime overlays that augment the workload. For example, the ingress trait specifies the ingress host and path, while the metrics trait optionally provides the Prometheus scraper used to obtain the application related metrics. If no metrics trait is specified, the Verrazzano-supplied Prometheus component is used by default.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
              scraper: <optionally specify custom scraper>
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

A brief description of each field in the application configuration:

apiVersion - Version of the ApplicationConfiguration custom resource definition
kind - Standard name of the application configuration custom resource definition
metadata.name - The name used to create this application configuration resource
metadata.namespace - The namespace used for this application configuration custom resource
spec.components - Reference to the application’s components leveraged to specify runtime configuration
spec.components[].traits - The traits specified for the application’s components

To explore traits, we can examine the fields of an ingress trait:

apiVersion - Version of the OAM trait custom resource definition
kind - IngressTrait is the name of the OAM application ingress trait custom resource definition
spec.rules.paths - The context paths for accessing the application

Deploy the application

The following steps are required to deploy the example application. Steps similar to the apply steps would be used to deploy any application to Verrazzano.

Create a namespace for the example application and add labels identifying the namespace as managed by Verrazzano and enabled for Istio.
```
$ kubectl create namespace hello-helidon
$ kubectl label namespace hello-helidon verrazzano-managed=true istio-injection=enabled
```
Apply the application’s component.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/examples/hello-helidon/hello-helidon-comp.yaml -n hello-helidon
```
This step causes the validation and creation of the Component resource. No other resources or objects are created as a result. Application configurations applied in the future may reference this Component resource.
Apply the application configuration.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/examples/hello-helidon/hello-helidon-app.yaml -n hello-helidon
```
This step causes the validation and creation of the application configuration resource. This operation triggers the activation of a number of Verrazzano operators. These operators create other Kubernetes objects (for example, Deployments, ReplicaSets, Pods, Services, Ingresses) that collectively provide and support the application.
Configure the application’s DNS resolution.

After deploying the application, configure DNS to resolve the application’s ingress DNS name to the application’s load balancer IP address. The generated host name is obtained by querying Kubernetes for the gateway:
```
$ kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-gw \
    -n hello-helidon \
    -o jsonpath='{.spec.servers[0].hosts[0]}'
```
The load balancer IP is obtained by querying Kubernetes for the Istio ingress gateway status:
```
$ kubectl get service \
    -n istio-system istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```
DNS configuration steps are outside the scope of this guide. For DNS infrastructure that can be configured and used, see the Oracle Cloud Infrastructure DNS documentation. In some small non-production scenarios, DNS configuration using /etc/hosts or an equivalent may be sufficient.

Verify the deployment

Applying the application configuration initiates the creation of several Kubernetes objects. Actual creation and initialization of these objects occurs asynchronously. The following steps provide commands for determining when these objects are ready for use.

Note: Many other Kubernetes objects unrelated to the example application may also exist. Those have been omitted from the lists.

Verify the Helidon application pod is running.

$ kubectl get pods -n hello-helidon -l app=hello-helidon

# Sample output
NAME                                        READY   STATUS    RESTARTS   AGE
hello-helidon-deployment-8664954995-wcb9d   2/2     Running   0          5m5s

Verify that the Verrazzano application operator pod is running.
```
$ kubectl get pod -n verrazzano-system -l app=verrazzano-application-operator

# Sample output
NAME                                               READY   STATUS    RESTARTS   AGE
verrazzano-application-operator-79849b89ff-lr9w6   1/1     Running   0          13m
```
The namespace verrazzano-system is used by Verrazzano for non-application objects managed by Verrazzano. A single verrazzano-application-operator manages the life cycle of all OAM based applications within the cluster.

Verify the Verrazzano monitoring infrastructure is running.

$ kubectl get pods -n verrazzano-system | grep '^NAME\|vmi-system'

# Sample output
NAME                                               READY   STATUS    RESTARTS   AGE
vmi-system-es-master-0                             2/2     Running   0          47m
vmi-system-grafana-799d79648d-wsdp4                2/2     Running   0          47m
vmi-system-kiali-574c6dd94d-f49jv                  2/2     Running   0          51m
vmi-system-kibana-77f8d998f4-zzvqr                 2/2     Running   0          47m

$ kubectl get pods -n verrazzano-monitoring

# Sample output
NAME                                                   READY   STATUS    RESTARTS   AGE
prometheus-node-exporter-fstc7                         1/1     Running   0          14h
prometheus-operator-kube-p-operator-857fb66b74-szv4h   1/1     Running   0          14h
prometheus-prometheus-operator-kube-p-prometheus-0     3/3     Running   0          14h

These pods in the verrazzano-system and verrazzano-monitoring namespaces constitute a monitoring stack created by Verrazzano for the deployed applications.

The monitoring infrastructure comprises several components:

vmi-system-es - OpenSearch for log collection
vmi-system-grafana - Grafana for metric visualization
vms-system-kiali - Kiali for management console of istio service mesh
vmi-system-kibana - OpenSearch Dashboards for log visualization
prometheus-prometheus-operator-kube-p-prometheus - Prometheus for metric collection

Diagnose failures.

View the event logs of any pod not entering the Running state within a reasonable length of time, such as five minutes.
```
$ kubectl describe pod -n hello-helidon -l app=hello-helidon
```
Use the specific namespace and name for the pod being investigated.

Explore the application

Follow these steps to explore the application’s functionality. If DNS was not configured, then use the alternative commands.

Save the host name and IP address of the load balancer exposing the application’s REST service endpoints for later.

$ HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-gw \
      -n hello-helidon \
      -o jsonpath='{.spec.servers[0].hosts[0]}')
$ ADDRESS=$(kubectl get service \
      -n istio-system istio-ingressgateway \
      -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

NOTE:

The value of ADDRESS is used only if DNS has not been configured.
The following alternative commands may not work in conjunction with firewalls that validate HTTP Host headers.

Get the default message.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet"

# Expected response
{"message":"Hello World!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet" \
    --resolve ${HOST}:443:${ADDRESS}

Get a message for Robert.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert"

# Expected response
{"message":"Hello Robert!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET
    "https://${HOST}/greet/Robert" \
    --resolve ${HOST}:443:${ADDRESS}

Update the default greeting.

$ curl -sk \
    -X PUT \
    "https://${HOST}/greet/greeting" \
    -H 'Content-Type: application/json' \
    -d '{"greeting" : "Greetings"}'

If DNS has not been configured, then use this command.

$ curl -sk \
    -X PUT \
    "https://${HOST}/greet/greeting" \
    -H 'Content-Type: application/json' \
    -d '{"greeting" : "Greetings"}' \
    --resolve ${HOST}:443:${ADDRESS}

Get the new message for Robert.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert"

# Expected response
{"message":"Greetings Robert!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert" \
    --resolve ${HOST}:443:${ADDRESS}

Access the application’s logs

Deployed applications have log collection enabled. These logs are collected using OpenSearch and can be accessed using OpenSearch Dashboards. OpenSearch and OpenSearch Dashboards are examples of infrastructure Verrazzano creates in support of an application as a result of applying an application configuration. For more information on creating an index pattern and visualizing the log data collected in OpenSearch, see OpenSearch Dashboards.

Determine the URL to access OpenSearch Dashboards:

$ OSD_HOST=$(kubectl get ingress \
     -n verrazzano-system vmi-system-kibana \
     -o jsonpath='{.spec.rules[0].host}')
$ OSD_URL="https://${OSD_HOST}"
$ echo "${OSD_URL}"
$ open "${OSD_URL}"

The user name to access OpenSearch Dashboards defaults to verrazzano during the Verrazzano installation.

Determine the password to access OpenSearch Dashboards:

$ echo $(kubectl get secret \
      -n verrazzano-system verrazzano \
      -o jsonpath={.data.password} | base64 \
      --decode)

Access the application’s metrics

Deployed applications have metric collection enabled. Grafana can be used to access these metrics collected by Prometheus. Prometheus and Grafana are additional components Verrazzano creates as a result of applying an application configuration. For more information on visualizing Prometheus metrics data, see Grafana.

Determine the URL to access Grafana:

$ GRAFANA_HOST=$(kubectl get ingress \
      -n verrazzano-system vmi-system-grafana \
      -o jsonpath='{.spec.rules[0].host}')
$ GRAFANA_URL="https://${GRAFANA_HOST}"
$ echo "${GRAFANA_URL}"
$ open "${GRAFANA_URL}"

The user name to access Grafana is set to the default value verrazzano during the Verrazzano installation.

Determine the password to access Grafana:

$ echo $(kubectl get secret \
      -n verrazzano-system verrazzano \
      -o jsonpath={.data.password} | base64 \
      --decode)

Alternatively, metrics can be accessed directly using Prometheus. Determine the URL for this access:

$ PROMETHEUS_HOST=$(kubectl get ingress \
      -n verrazzano-system vmi-system-prometheus \
      -o jsonpath='{.spec.rules[0].host}')
$ PROMETHEUS_URL="https://${PROMETHEUS_HOST}"
$ echo "${PROMETHEUS_URL}"
$ open "${PROMETHEUS_URL}"

The user name and password for both Prometheus and Grafana are the same.

Suppress Kiali console warnings

For some applications, the Kiali console may show warnings for VirtualService and Gateway objects that replicate hostname/port configurations across multiple IngressTraits. These warnings do not impact functionality and can be suppressed with the following component override:

kiali:
  overrides:
    - values:
        kiali_feature_flags:
          validations:
            ignore: ["KIA1106", "KIA0301"]

Remove the application

Run the following commands to delete the application configuration, and optionally the component and namespace.

Delete the application configuration.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/examples/hello-helidon/hello-helidon-app.yaml
```
The deletion of the application configuration will result in the destruction of all application-specific Kubernetes objects.
(Optional) Delete the application’s component.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/examples/hello-helidon/hello-helidon-comp.yaml
```
Note: This step is not required if other application configurations for this component will be applied in the future.

(Optional) Delete the namespace.

$ kubectl delete namespace hello-helidon

Docs: Application Deployment

Mon, 01 Jan 0001 00:00:00 +0000

During application deployment, the oam-kubernetes-runtime and verrazzano-application-operator cooperate through the generation and update of Kubernetes resources. The oam-kubernetes-runtime processes the ApplicationConfiguration and Component resources provided by the user and generates workload and Trait resources. The verrazzano-application-operator processes Verrazzano specific workload and Trait resources. These are then used to generate additional child and related resources.

Troubleshooting application deployments should follow three general steps:

Review the status of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the logs of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the resources generated by the oam-kubernetes-runtime and the verrazzano-application-operator.

Review `oam-kubernetes-runtime` operator status

For application deployment to succeed, the oam-kubernetes-runtime pod must have a status of Running.

Use the following command to get the pod status:

$ kubectl get pods \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

If the pod status is not Running, then see the instructions for reviewing the oam-kubernetes-runtime pod logs.

Review `verrazzano-application-operator` operator status

For application deployment to succeed, the verrazzano-application-operator pod must have a status of Running.

Use the following command to get the pod status:

$ kubectl get pods \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If the pod status is not Running, then see the instructions for reviewing the verrazzano-application-operator logs.

Review `oam-kubernetes-runtime` operator logs

Review the oam-kubernetes-runtime pod logs for any indication that pod startup or the generation of workloads or traits has failed.

Use the following command to get the logs:

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

Review `verrazzano-application-operator` logs

Review the verrazzano-application-operator logs for any indication that pod startup or resource generation has failed.

Use the following command to get the logs:

$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

Review generated workload resources

The processing of a Component reference within an ApplicationConfiguration results in the generation of workloads. For example, a referenced Component might result in the generation of a VerrazzanoHelidonWorkload workload resource. In turn, the VerrazzanoHelidonWorkload workload resource will be processed and result in the generation of related Deployment and Service resources.

If the expected workload resource, for example VerrazzanoHelidonWorkload, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Deployment or Service, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to a VerrazzanoHelidonWorkload deployment:

$ kubectl get -n hello-helidon verrazzanohelidonworkload hello-helidon-workload
$ kubectl get -n hello-helidon deployment hello-helidon-deployment
$ kubectl get -n hello-helidon service hello-helidon-deployment

Review generated Trait resources

The processing of traits embedded with an ApplicationConfiguration results in the generation of Trait resources. For example, an IngressTrait embedded within an ApplicationConfiguration will result in the generation of an IngressTrait resource. In turn, the IngressTrait resource will be processed and result in the generation of related Certificate, Gateway, and VirtualService resources.

If the expected Trait resource, for example IngressTrait, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Certificate, Gateway, and VirtualService, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to an IngressTrait:

$ kubectl get -n hello-helidon ingresstrait hello-helidon-ingress
$ kubectl get -n istio-system Certificate hello-helidon-hello-helidon-appconf-cert
$ kubectl get -n hello-helidon gateway hello-helidon-hello-helidon-gw
$ kubectl get -n hello-helidon virtualservice hello-helidon-ingress-rule-0-vs

Check for RBAC privilege issues

The use of generic Kubernetes resources as workloads and traits can result in deployment failures if privileges are insufficient. In this case, the oam-kubernetes-runtime logs will contain errors containing the term forbidden.

The following command shows how to query for this type of failure message:

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime | grep forbidden

Check resource owners

Kubernetes maintains the child to parent relationship within metadata fields.

The following example returns the parent of the IngressTrait, named hello-helidon-ingress, in the hello-helidon namespace:

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .metadata.ownerReferences[*]}{.name}{"\n"}{end}'

The results of this command can help identify the lineage of a given resource.

Some resources also record the related resources affected during their processing. For example, when processed, an IngressTrait will create related Gateway, VirtualService, and Certificate resources.

The following command is an example of how to obtain the related resources of an IngressTraits:

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .status.resources[*]}{.kind}: {.name}{"\n"}{end}'

The results of this command can help identify which other resources, the given resource affected.

Docs: Application Security

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides the following support.

Keycloak

Applications can use the Verrazzano Keycloak server as an Identity Provider. Keycloak supports SAML 2.0 and OpenID Connect (OIDC) authentication and authorization flows. Verrazzano does not provide any explicit integrations for applications.

NOTE

If using Keycloak for application authentication and authorization, create a new realm to contain application users and clients. Do not use the verrazzano-system realm, or the default (Keycloak system) realm. The Keycloak root user account (keycloakadmin) has privileges to create realms.

Network security

Verrazzano uses Istio to authenticate and authorize incoming network connections for applications. Verrazzano also provides support for configuring Kubernetes NetworkPolicy on Verrazzano projects. NetworkPolicy rules control where network connections can be made.

NOTE

Enforcement of NetworkPolicy requires that a Kubernetes Container Network Interface (CNI) provider, such as Calico, be configured for the cluster.

For more information on how Verrazzano secures network traffic, see Network Security.

Pod security

By default, all containers within a pod run as root (UID 0) within the container. Most applications do not require this level of access and doing so is considered a security risk.

It is recommended that applications attempt to meet the requirements of the Kubernetes restricted Pod Security Standard. This essentially means running the container within a pod as a non-root user with minimal capabilities, and without the ability to escalate privileges. Each container image also should define a non-root user identity that the container process will run, as by default, for added security.

In the Kubernetes Pod specification, there is a Pod SecurityContext for defining security at the pod level and a Container SecurityContext used to define security for containers. Some fields are common between the two security contexts, and others are unique. For details, see the API specifications for each. Where there is overlap, settings defined at the container level override settings defined at the pod level.

The following sections describe implementing these standards in more detail.

Specify a non-root user in the container image

Unless otherwise specified, all containers run as the root user. It is recommended that each container image build explicitly creates an unprivileged, non-root user and group, and then uses that with the USER instruction in the Dockerfile for the container.

To achieve this, modify the container’s image build and use the USER <UID> instruction.

For example:

# Run as user 1000
USER 1000

This will make the process within the container run as UID 1000. Even if there is no entry in /etc/passwd matching the UID declared, the container will run as the specified UID with minimal privileges.

For example, this is illustrated by a running image using the kubectl run command with the defaults:

% kubectl run -it --rm myol --image=ghcr.io/oracle/oraclelinux:7-slim --restart=Never -- bash
If you don't see a command prompt, try pressing enter.
bash-4.2# whoami
root
bash-4.2# id
uid=0(root) gid=0(root) groups=0(root)
bash-4.2#

To run the same image as a non-root user, you can override the default user and group for the container process, as shown:

% kubectl run -it --rm myol --image=ghcr.io/oracle/oraclelinux:7-slim --restart=Never --overrides='{ "spec": { "securityContext": { "runAsUser": 1000, "runAsGroup": 1000, "runAsNonRoot": true } } }' -- bash
If you don't see a command prompt, try pressing enter.
bash-4.2$
bash-4.2$ whoami
whoami: cannot find name for user ID 1000
bash-4.2$ id
uid=1000 gid=1000 groups=1000

In the second example, the container is running as UID 1000 with a GID of 1000. Running whoami from within the container returns an error because USER 1000 is not defined in /etc/passwd, but running the id command from the shell shows that the container process is indeed running as the desired UID (1000).

Specify security settings for the Pod

By default, containers within Kubernetes pods run as the image default user, which in turn defaults to the root user (UID 0).

You can use the pod and container securityContext fields to force containers within a pod to run as non-root and prevent the container from acquiring escalated privileges. These will override any USER setting within the image.

apiVersion: apps/v1
kind: Deployment
spec:
  ...
  template:
    ...
    spec:
      # Define a security context for all containers in the pod
      securityContext:
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: some-container
        ...
        # Define a security context for the container; settings defined here have precedence over the pod securityContext
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
      ...

As mentioned previously, where there is overlap between the pod and container security settings, the settings defined at the container level override settings defined at the pod level.

Helidon pod security

The following YAML shows how to explicitly specify the pod security context for a Helidon application. With these settings, the Helidon application will meet the requirements of the Kubernetes restricted Pod Security Standard.

Note that the runAsUser 2000 UID does not exist in the container, as described previously.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
        version: v1
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          securityContext:
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: hello-helidon-container
...
              securityContext:
                runAsNonRoot: true
                runAsGroup: 2000
                runAsUser: 2000
                privileged: false
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL

Pod security for ContainerizedWorkload applications

The only means for controlling pod security for the ContainerizedWorkload type is to specify a non-root user, using the USER instruction in the container image build, as described in this section, Specify a non-root user in the container image.

Pod security for applications using standard Kubernetes resources

You can deploy applications using standard Kubernetes resources, as described in the Standard Kubernetes Resources example.

You configure security for these resources as you typically would for any Kubernetes Deployment resource.

For example:

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-deployment
spec:
  workload:
    kind: Deployment
    apiVersion: apps/v1
    name: oam-kube-dep
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: oam-kube-app
      template:
        metadata:
          labels:
            app: oam-kube-app
        spec:
          securityContext:
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: oam-kube-cnt
              image: hashicorp/http-echo
              args:
                - "-text=hello"
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
                privileged: false

Docs: AuthProxy

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano AuthProxy component enables authentication and authorization for Keycloak users accessing Verrazzano resources. You can customize the AuthProxy component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the AuthProxy component.

Path to Field Description

spec.components.authProxy.kubernetes.replicas The number of pods to replicate. The default is 2 for the prod profile and 1 for all other profiles.

Path to Field	Description
`spec.components.authProxy.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.authProxy.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes. spec: components: authProxy: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - verrazzano-authproxy topologyKey: kubernetes.io/hostname

spec.components.authProxy.kubernetes.affinity

The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes.

spec:
  components:
    authProxy:
      kubernetes:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    authProxy:
      overrides:
      - values:
          replicas: 3
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

Docs: Bob's Books

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Install Verrazzano by following the installation instructions.
To download the example image, you must first accept the license agreement.
- In a browser, navigate to https://container-registry.oracle.com/ and sign in.
- Search for example-bobbys-coherence, example-bobbys-front-end, example-bobs-books-order-manager, example-roberts-coherence, and weblogic.
- For each one:
  - Select the image name in the results.
  - From the drop-down menu, select your language and click Continue.
  - Then read and accept the license agreement.
NOTE: The Bob’s Books example application deployment files are contained in the Verrazzano project located at <VERRAZZANO_HOME>/examples/bobs-books, where <VERRAZZANO_HOME> is the root of the Verrazzano project.

Overview

Bob’s Books consists of three main parts:

A back-end “order processing” application, which is a Java EE application with REST services and a very simple JSP UI, which stores data in a MySQL database. This application runs on WebLogic Server.
A front-end web store “Robert’s Books”, which is a general book seller. This is implemented as a Helidon microservice, which gets book data from Coherence, uses a Coherence cache store to persist data for the order manager, and has a React web UI.
A front-end web store “Bobby’s Books”, which is a specialty children’s book store. This is implemented as a Helidon microservice, which gets book data from a (different) Coherence cache store, interfaces directly with the order manager, and has a JSF web UI running on WebLogic Server.

For more information and the source code of this application, see the Verrazzano Examples.

Deploy the application

NOTE

To run this application in the default namespace:

$ kubectl label namespace default verrazzano-managed=true istio-injection=enabled

If you chose the default namespace, you can skip Step 1 and ignore the -n option in the rest of the commands.

Create a namespace for the example and add a label identifying the namespace as managed by Verrazzano.

$ kubectl create namespace bobs-books
$ kubectl label namespace bobs-books verrazzano-managed=true istio-injection=enabled

Create a docker-registry secret to enable pulling the example image from the registry.

$ kubectl create secret docker-registry bobs-books-repo-credentials \
        --docker-server=container-registry.oracle.com \
        --docker-username=YOUR_REGISTRY_USERNAME \
        --docker-password=YOUR_REGISTRY_PASSWORD \
        --docker-email=YOUR_REGISTRY_EMAIL \
        -n bobs-books

Replace YOUR_REGISTRY_USERNAME, YOUR_REGISTRY_PASSWORD, and YOUR_REGISTRY_EMAIL with the values you use to access the registry.

Create secrets for the WebLogic domains:

# Replace the values of the WLS_USERNAME and WLS_PASSWORD environment variables as appropriate.
$ export WLS_USERNAME=<username>
$ export WLS_PASSWORD=<password>
$ kubectl create secret generic bobbys-front-end-weblogic-credentials \
    --from-literal=password=$WLS_PASSWORD \
    --from-literal=username=$WLS_USERNAME \
    -n bobs-books

$ kubectl create secret generic bobs-bookstore-weblogic-credentials \
    --from-literal=password=$WLS_PASSWORD \
    --from-literal=username=$WLS_USERNAME \
    -n bobs-books

$ kubectl create secret generic mysql-credentials \
    --from-literal=username=$WLS_USERNAME \
    --from-literal=password=$WLS_PASSWORD \
    --from-literal=url=jdbc:mysql://mysql.bobs-books.svc.cluster.local:3306/books \
    -n bobs-books

Note that the example application is preconfigured to use specific secret names. For the source code of this application, see the Verrazzano Examples. If you want to use secret names that are different from what is specified in the source code, you will need to update the corresponding YAML file and rebuild the Docker images for the example application.

To deploy the application, apply the example resources.

$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/examples/bobs-books/bobs-books-comp.yaml -n bobs-books
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/examples/bobs-books/bobs-books-app.yaml -n bobs-books

Wait for all of the pods in the Bob’s Books example application to be ready. You can monitor their progress by listing the pods and inspecting the output, or you can use the kubectl wait command.

You may need to repeat the kubectl wait command several times before it is successful. The WebLogic Server and Coherence pods may take a while to be created and Ready.
```
$ kubectl get pods -n bobs-books

# -or- #

$ kubectl wait \
    --for=condition=Ready pods \
    --all -n bobs-books \
    --timeout=600s
```

Get the EXTERNAL_IP address of the istio-ingressgateway service.

$ ADDRESS=$(kubectl get service \
    -n istio-system istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $ADDRESS

# Sample output
11.22.33.44

Get the generated host name for the application.

$ HOST=$(kubectl get gateways.networking.istio.io bobs-books-bobs-books-gw \
    -n bobs-books \
    -o jsonpath='{.spec.servers[0].hosts[0]}')
$ echo $HOST

# Sample output
bobs-books.bobs-books.11.22.33.44.nip.io

Access the application. To access the application in a browser, you will need to do one of the following:
- Option 1: If you are using nip.io, then you can access the application using the generated host name. For example:
  - Robert’s Books UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/.
  - Bobby’s Books UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/bobbys-front-end/.
  - Bob’s order manager UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/bobs-bookstore-order-manager/orders.
- Option 2: Temporarily, modify the /etc/hosts file (on Mac or Linux) or c:\Windows\System32\Drivers\etc\hosts file (on Windows 10), to add an entry mapping the host used by the application to the external IP address assigned to your gateway. For example:
```
11.22.33.44 bobs-books.example.com
```
  Then, you can use a browser to access the application, as shown:
  - Robert’s Books UI at https://bobs-books.example.com/.
  - Bobby’s Books UI at https://bobs-books.example.com/bobbys-front-end/.
  - Bob’s order manager UI at https://bobs-books.example.com/bobs-bookstore-order-manager/orders.
- Option 3: Alternatively, point your own DNS name to the load balancer’s external IP address. In this case, you would need to have edited the bobs-books-app.yaml file to use the appropriate values under the hosts section for the application (such as your-roberts-books-host.your.domain), before deploying the application. Then, you can use a browser to access the application, as shown:
  - Robert’s Books UI at https://<your-roberts-books-host.your.domain>/.
  - Bobby’s Books UI at https://<your-bobbys-books-host.your.domain>/bobbys-front-end/.
  - Bob’s order manager UI at https://<your-bobs-orders-host.your.domain>/.

Access the applications using the WebLogic Server Administration Console

Use the WebLogic Server Administration Console to access the applications as follows.

NOTE

It is recommended that the WebLogic Server Administration Console not be exposed publicly.

Access bobs-bookstore

Set up port forwarding.
```
$ kubectl port-forward pods/bobs-bookstore-adminserver 7001:7001 -n bobs-books
```
NOTE: If you are using the Oracle Cloud Infrastructure Cloud Shell to run kubectl, in order to access the WebLogic Server Administration Console using port forwarding, you will need to run kubectl on another machine.
Access the WebLogic Server Administration Console from your browser.
```
http://localhost:7001/console
```

Access bobbys-front-end

Set up port forwarding.
```
$ kubectl port-forward pods/bobbys-front-end-adminserver 7001:7001 -n bobs-books
```
NOTE: If you are using the Oracle Cloud Infrastructure Cloud Shell to run kubectl, in order to access the WebLogic Server Administration Console using port forwarding, you will need to run kubectl on another machine.
Access the WebLogic Server Administration Console from your browser.
```
http://localhost:7001/console
```

Verify the deployed application

Verify that the application configuration, domains, Coherence resources, and ingress trait all exist.

$ kubectl get ApplicationConfiguration -n bobs-books
$ kubectl get Domain -n bobs-books
$ kubectl get Coherence -n bobs-books
$ kubectl get IngressTrait -n bobs-books

Verify that the service pods are successfully created and transition to the READY state. Note that this may take a few minutes and that you may see some of the services terminate and restart.

$ kubectl get pods -n bobs-books

# Sample output
NAME                                                READY   STATUS    RESTARTS   AGE
bobbys-helidon-stock-application-868b5965c8-dk2xb   3/3     Running   0          19h
bobbys-coherence-0                                  2/2     Running   0          19h
bobbys-front-end-adminserver                        3/3     Running   0          19h
bobbys-front-end-managed-server1                    3/3     Running   0          19h
bobs-bookstore-adminserver                          3/3     Running   0          19h
bobs-bookstore-managed-server1                      3/3     Running   0          19h
mysql-669665fb54-9m8wq                              2/2     Running   0          19h
robert-helidon-96997fcd5-kzjkf                      3/3     Running   0          19h
robert-helidon-96997fcd5-nlswm                      3/3     Running   0          19h
roberts-coherence-0                                 2/2     Running   0          17h
roberts-coherence-1                                 2/2     Running   0          17h

Undeploy the application

To undeploy the application, delete the Bob’s Books OAM resources.

$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/examples/bobs-books/bobs-books-app.yaml -n bobs-books
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/examples/bobs-books/bobs-books-comp.yaml -n bobs-books

Delete the namespace bobs-books after the application pods are terminated. The secrets created for the WebLogic domain also will be deleted.
```
$ kubectl delete namespace bobs-books
```

Docs: Generic Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the generic install

Verrazzano requires that your Kubernetes cluster provides an implementation of network load balancers (Services of type LoadBalancer) for a production environment. If your generic Kubernetes implementation provides this feature, then you can use a default configuration of the Verrazzano custom resource with no customizations and follow the Installation Guide.

NOTE

Remember to not overlap network Classless Inter-Domain Routing (CIDR) blocks when designing and implementing your Kubernetes cluster; proper routing relies on that.

You can install a load balancer, such as MetalLB. This setup requires knowledge of networking both inside and outside your Kubernetes cluster. This would include specifics of your Container Network Interface (CNI) implementation, IP address allocation schemes, and routing that goes beyond the scope of this documentation. For a Kind implementation, see Install and configure MetalLB.

It is possible to use a Kubernetes Service of type NodePort to test aspects of Verrazzano. This requires a good working knowledge of networking and has limited use cases.

Customizations

Verrazzano is highly customizable. If your Kubernetes implementation requires a custom configuration, see Customize Verrazzano.

Next steps

To continue, see the Installation Guide.

Docs: Install Guide

Mon, 01 Jan 0001 00:00:00 +0000

The following instructions show you how to install Verrazzano in a single Kubernetes cluster.

Prerequisites

Find the Verrazzano prerequisite requirements here.
Review the list of the software versions supported and installed by Verrazzano.

NOTE

To avoid conflicts with Verrazzano system components, we recommend installing Verrazzano into an empty cluster.

Prepare for the installation

Before installing Verrazzano, see instructions on preparing Kubernetes platforms and installing the Verrazzano CLI (optional). Make sure that you have a valid kubeconfig file pointing to the Kubernetes cluster that you want to use for installing Verrazzano.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For instructions on how to install a CNI plug-in, see the documentation for your Kubernetes cluster.

You can install Verrazzano using the Verrazzano CLI or with kubectl. See the following respective sections.

Verrazzano provides a platform operator to manage the life cycle of Verrazzano installations. Using the Verrazzano custom resource, you can install, uninstall, and upgrade Verrazzano installations.

Perform the installation

Verrazzano supports the following installation profiles: development (dev), production (prod), and managed cluster (managed-cluster). For more information, see Installation Profiles.

This document shows how to create a basic Verrazzano installation using:

The development (dev) installation profile
Wildcard-DNS, where DNS is provided by nip.io (the default)

NOTE: Because the dev profile installs self-signed certificates, when installing Verrazzano on macOS, you might see: Your connection is not private. For a workaround, see this FAQ.

For a complete description of Verrazzano configuration options, see the Verrazzano Custom Resource Definition.

To use other DNS options, see Customizing DNS for more details.

Install Verrazzano

To create a Verrazzano installation as described in the previous section, run the following commands.

Install Verrazzano with its dev profile.

$ vz install -f - <<EOF
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  defaultVolumeSource:
    persistentVolumeClaim:
      claimName: verrazzano-storage
  volumeClaimSpecTemplates:
    - metadata:
        name: verrazzano-storage
      spec:
        resources:
          requests:
            storage: 2Gi
EOF

This command installs the Verrazzano platform operator and applies the Verrazzano custom resource.

Wait for the installation to complete. Installation logs will be streamed to the command window until the installation has completed or until the default timeout (30m) has been reached.

To use a different profile with the previous example, set the VZ_PROFILE environment variable to the name of the profile you want to install.

Install the Verrazzano platform operator

Verrazzano provides a platform operator to manage the life cycle of Verrazzano installations. Using the Verrazzano custom resource, you can install, uninstall, and upgrade Verrazzano installations.

To install the Verrazzano platform operator:

Deploy the Verrazzano platform operator.

$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/v1.4.8/verrazzano-platform-operator.yaml

Wait for the deployment to complete.

$ kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator

# Expected response
deployment "verrazzano-platform-operator" successfully rolled out

Confirm that the operator pod is correctly defined and running.

$ kubectl -n verrazzano-install get pods

# Sample output
NAME                                            READY   STATUS    RESTARTS   AGE
verrazzano-platform-operator-59d5c585fd-lwhsx   1/1     Running   0          114s

Perform the installation

Verrazzano supports the following installation profiles: development (dev), production (prod), and managed cluster (managed-cluster). For more information, see Installation Profiles.

This document shows how to create a basic Verrazzano installation using:

The development (dev) installation profile
Wildcard-DNS, where DNS is provided by nip.io (the default)

NOTE: Because the dev profile installs self-signed certificates, when installing Verrazzano on macOS, you might see: Your connection is not private. For a workaround, see this FAQ.

For a complete description of Verrazzano configuration options, see the Verrazzano Custom Resource Definition.

To use other DNS options, see Customzing DNS for more details.

Install Verrazzano

To create a Verrazzano installation as described in the previous section, run the following commands.

$ kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: ${VZ_PROFILE:-dev}
EOF
$ kubectl wait \
    --timeout=20m \
    --for=condition=InstallComplete verrazzano/example-verrazzano

To use a different profile with the previous example, set the VZ_PROFILE environment variable to the name of the profile you want to install.

If an error occurs, check the log output of the installation. You can view the logs with the following command.

$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
    | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'

Verify the installation

Verrazzano installs multiple objects in multiple namespaces. In the verrazzano-system namespaces, all the pods in the Running state, does not guarantee, but likely indicates that Verrazzano is up and running.

$ kubectl get pods -n verrazzano-system

# Sample output
coherence-operator-dcfb446df-24djp                 1/1     Running   1          49m
fluentd-h65xf                                      2/2     Running   1          45m
oam-kubernetes-runtime-6645df49cd-6q96c            1/1     Running   0          49m
verrazzano-application-operator-85ffd7f77b-rhwk7   1/1     Running   0          48m
verrazzano-authproxy-58db5b9484-nhnql              2/2     Running   0          45m
verrazzano-console-5dbdc579bd-hm4rh                2/2     Running   0          45m
verrazzano-monitoring-operator-599654889d-lbb4z    1/1     Running   0          45m
verrazzano-operator-7b6fd64dd5-8j9h8               1/1     Running   0          45m
vmi-system-es-master-0                             2/2     Running   0          45m
vmi-system-grafana-5558d65b46-pxg78                2/2     Running   0          45m
vmi-system-kiali-5949966fb8-465s8                  2/2     Running   0          48m
vmi-system-kibana-86b894d8f6-q4vb5                 2/2     Running   0          45m
weblogic-operator-646756c75c-hgz6j                 2/2     Running   0          49m

For installation troubleshooting help, see the Analysis Advice.

After the installation has completed, you can use the Verrazzano consoles. For information on how to get the consoles URLs and credentials, see Access Verrazzano.

Next steps

(Optional) Run the example applications located here.

Docs: Jaeger Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Jaeger is a distributed tracing system used for monitoring and troubleshooting microservices. For more information on Jaeger, see the Jaeger website.

Install Jaeger Operator

To install the Jaeger Operator, enable the jaegerOperator component in your Verrazzano custom resource. Here is an example YAML file that enables the Jaeger Operator. Verrazzano installs the Jaeger Operator in the verrazzano-monitoring namespace. If OpenSearch and Keycloak components are enabled in the Verrazzano custom resource, then a default Jaeger instance is also created by the Jaeger Operator in the verrazzano-monitoring namespace.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true

The Jaeger Operator will create Service custom resources for query and collection. After applying the Verrazzano custom resource, listing Jaeger resources will show output similar to the following.

$ kubectl get services,deployments -l app.kubernetes.io/instance=jaeger-operator-jaeger -n verrazzano-monitoring

NAME                                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                  AGE
service/jaeger-operator-jaeger-collector            ClusterIP   10.96.120.223   <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   79m
service/jaeger-operator-jaeger-collector-headless   ClusterIP   None            <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   79m
service/jaeger-operator-jaeger-query                ClusterIP   10.96.209.196   <none>        16686/TCP,16685/TCP                      79m

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger-operator-jaeger-collector   1/1     1            1           79m
deployment.apps/jaeger-operator-jaeger-query       1/1     1            1           79m

Customize Jaeger

Verrazzano installs the Jaeger Operator and Jaeger using the jaeger-operator Helm chart. Using Helm overrides specified in the Verrazzano custom resource, you can customize the installation configuration. For more information about setting component overrides, see Customizing the Chart Before Installing.

Customize a Jaeger instance to use an external OpenSearch or Elasticsearch for storage

You can use the default Jaeger instance with an external OpenSearch cluster. The following example shows you how to configure Jaeger Operator Helm overrides in the Verrazzano custom resource to use an external OpenSearch cluster with a TLS CA certificate mounted from a volume and the user/password stored in a secret. For more details, see the Jaeger documentation.

Prior to configuring the external OpenSearch for Jaeger in the Verrazzano custom resource, create a secret containing the OpenSearch credentials and certificates in the verrazzano-install namespace. Jaeger will use these credentials to connect to OpenSearch.
```
$ kubectl create secret generic jaeger-secret \
 --from-literal=ES_PASSWORD=<OPENSEARCH PASSWORD> \
 --from-literal=ES_USERNAME=<OPENSEARCH USERNAME> \
 --from-file=ca-bundle=<path to the file containing CA certs> \
 -n verrazzano-install
```
Use the Verrazzano custom resource to update the Jaeger resource:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger-external-opensearch
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              create: true
              spec:
                strategy: production
                storage:
                  type: elasticsearch
                  options:
                    es:
                      # Enter your OpenSearch cluster endpoint here.
                      server-urls: <External OpenSearch URL>
                      index-prefix: jaeger
                      tls:
                        ca: /verrazzano/certificates/ca-bundle
                  secretName: jaeger-secret
                volumeMounts:
                  - name: certificates
                    mountPath: /verrazzano/certificates/
                    readOnly: true
                volumes:
                  - name: certificates
                    secret:
                      secretName: jaeger-secret

Enable the Service Performance Monitoring experimental feature

To enable the Jaeger Service Performance Monitoring experimental feature in the default Jaeger instance created by Verrazzano, use the following Verrazzano custom resource. Verrazzano sets jaeger.spec.query.options.prometheus.server-url to the Prometheus server URL managed by Verrazzano, if it exists. To configure an external Prometheus server for your use case, override jaeger.spec.query.options.prometheus.server-url, jaeger.spec.query.options.prometheus.tls.enabled and jaeger.spec.query.options.prometheus.tls.ca appropriately in the Verrazzano custom resource. For more details, see the Jaeger documentation.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              spec:
                query:
                  metricsStorage:
                    type: prometheus

Disable default Jaeger instance creation

To disable the default Jaeger instance created by Verrazzano, use the following Verrazzano custom resource:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              create: false

Jaeger Operator Helm chart values that cannot be overridden

The following Jaeger Operator Helm values are not supported to be overridden in the Verrazzano custom resource:

nameOverride
fullnameOverride
serviceAccount.name
ingress.enabled
jaeger.spec.storage.dependencies.enabled

If you try to override the above Helm values in the Verrazzano custom resource, the request will be rejected and an error message returned.

Note: - Verrazzano does not support Jaeger Spark dependencies and hence the Helm chart value jaeger.spec.storage.dependencies.enabled, which is set to false for the Jaeger instance managed by Verrazzano, cannot be overridden.

Configure an application to export traces to Jaeger

The Jaeger agent sidecar is injected to application pods by the "sidecar.jaegertracing.io/inject": "true" annotation. You may apply this annotation to namespaces or pod controllers, such as Deployments. The subsequent snippet shows how to annotate an OAM Component for Jaeger agent injection.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-component
spec:
  workload:
    apiVersion: core.oam.dev/v1alpha2
    kind: ContainerizedWorkload
    metadata:
      name: example-workload
      annotations:
        # The component's Deployment will carry the Jaeger annotation.
        "sidecar.jaegertracing.io/inject": "true"

If you have multiple Jaeger instances in your cluster, specify the name of the Jaeger instance to which you intend to send the traces, as a value for the annotation sidecar.jaegertracing.io/inject. For more details, see the Jaeger documentation.

View traces on the Jaeger UI

After the installation has completed, you can use the Verrazzano Jaeger UI to view the traces. For information on how to get the Verrazzano Jaeger UI URL and credentials, see Access Verrazzano.

Configure the Istio mesh to use Jaeger tracing

You can view Istio mesh traffic by enabling Istio’s distributed tracing integration. Traces from the Istio mesh provide observability on application traffic that passes through Istio’s ingress and egress gateways.

Istio tracing is disabled by default. To turn on traces, customize your Istio component like the following example:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true

After enabling tracing, Istio will automatically configure itself with the the Jaeger instance managed by Verrazzano in your cluster, and Istio-injected pods will begin exporting traces to Jaeger.

To export traces to a different Jaeger instance than the one managed by Verrazzano, set meshConfig.defaultConfig.tracing.zipkin.address to the intended Jaeger Collector URL. Any new Istio-injected pods will begin exporting traces to the newly configured Jaeger instance. Existing pods require a restart to pull the new Istio configuration and start sending traces to the newly configured Jaeger instance.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true
                defaultConfig:
                  tracing:
                    zipkin:
                      address: <address:port of your Jaeger collector service>

Istio’s default sampling rate is 1%, meaning 1 in 100 requests will be traced in Jaeger. If you want a different sampling rate, configure your desired rate using the meshConfig.defaultConfig.tracing.sampling Istio installation argument.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true
                defaultConfig:
                  tracing:
                    sampling: 25.0

Management of Jaeger indices in OpenSearch

To clean old Jaeger data from OpenSearch, Verrazzano uses the index management provided by Jaeger. By default, a cron job with the following default values is created to clean old traces. To configure it to your use case, override the following Jaeger spec values in the Verrazzano custom resource with your desired values.

storage:
  type: elasticsearch
  esIndexCleaner:
    enabled: true                                 // turn the cron job deployment on and off
    numberOfDays: 7                               // number of days to wait before deleting a record
    schedule: "55 23 * * *"                       // cron expression for it to run

Jaeger tracing in a multicluster Verrazzano environment

If the Jaeger Operator component is enabled in the managed cluster, after successful registration with the admin cluster, a Jaeger collector service runs in the managed cluster, which exports the traces to the OpenSearch or Elasticsearch storage configured in the admin cluster.

NOTE: Traces are exported to the admin cluster only when the Jaeger instance in the admin cluster is configured with the OpenSearch or Elasticsearch storage.

Listing Jaeger resources in the managed cluster shows output similar to the following.

$ kubectl get jaegers -n verrazzano-monitoring
NAME                                STATUS    VERSION   STRATEGY     STORAGE         AGE
jaeger-verrazzano-managed-cluster   Running   1.34.1    production   elasticsearch   11m

Configure the Istio mesh in a managed cluster to export Jaeger traces to the admin cluster

To export the Istio mesh traces in the managed cluster to the admin cluster, set meshConfig.defaultConfig.tracing.zipkin.address to the Jaeger Collector URL created in the managed cluster that exports the traces to the OpenSearch or Elasticsearch storage configured in the admin cluster.

Configure the Istio mesh on the managed cluster at the time of the Verrazzano installation, as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: managed-cluster
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            meshConfig:
              enableTracing: true
              defaultConfig:
                tracing:
                  zipkin:
                    address: jaeger-verrazzano-managed-cluster-collector.verrazzano-monitoring.svc.cluster.local.:9411

View the managed cluster traces

You can see the managed cluster traces from the Jaeger UI in the admin cluster only. To find the Jaeger UI URL for your admin cluster, follow the instructions for Accessing Verrazzano.

The spans include the Process tag verrazzano_cluster, which has the name of the managed cluster. To see the traces for the managed cluster only, search based on the tag verrazzano_cluster=<managed cluster name>.

Sample output of Jager UI screens

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires the following:

A Kubernetes cluster and a compatible kubectl.
dev profile - At least two CPUs, 100 GB disk storage, and 16 GB RAM available on the Kubernetes worker nodes. Depending on the resource requirements of the applications you deploy, this may or may not be sufficient.
prod profile - At least four CPUs, 100 GB disk storage, and 32 GB RAM available on the Kubernetes worker nodes. Depending on the resource requirements of the applications you deploy, this may or may not be sufficient.

NOTE

To avoid conflicts with Verrazzano system components, we recommend installing Verrazzano into an empty cluster.

Supported hardware

Verrazzano requires x86-64; other architectures are not supported.

Supported software versions

Verrazzano supports the following software versions.

Kubernetes

You can install Verrazzano on the following Kubernetes versions.

Verrazzano	Kubernetes Versions
1.4	1.21, 1.22, 1.23, 1.24
1.3	1.21, 1.22, 1.23
1.2	1.19, 1.20, 1.21
1.1	1.19, 1.20, 1.21
1.0	1.18, 1.19, 1.20

For more information, see Kubernetes Release Documentation. For platform specific details, see Verrazzano platform setup.

WebLogic Server

The supported versions of WebLogic Server are dependent on the WebLogic Kubernetes Operator version. See the WebLogic Server versions supported here.

Coherence

The supported versions of Coherence are dependent on the Coherence Operator version. See the Coherence versions supported here.

Helidon

Verrazzano supports all versions of Helidon. For more information, see Helidon and Helidon Commercial Offerings.

Installed components

Verrazzano installs a curated set of open source components. The following table lists each component, its version, and a brief description.

Component	Version	Description
alert-manager	0.24.0	Handles alerts sent by client applications, such as the Prometheus server.
cert-manager	1.7.1	Automates the management and issuance of TLS certificates.
Coherence Operator	3.2.9	Assists with deploying and managing Coherence clusters.
ExternalDNS	0.10.2	Synchronizes exposed Kubernetes Services and ingresses with DNS providers.
Fluentd	1.14.5	Collects logs and sends them to OpenSearch.
Grafana	7.5.17	Tool to help you examine, analyze, and monitor metrics.
Istio	1.14.3	Service mesh that layers transparently onto existing distributed applications.
Jaeger	1.34.1	Distributed tracing system for monitoring and troubleshooting distributed systems.
Jaeger Operator	1.34.1	Provides management for Jaeger tools.
Keycloak	15.0.2	Provides single sign-on with Identity and Access Management.
Kiali	1.42.0	Management console for the Istio service mesh.
kube-state-metrics	2.4.2	Provides metrics about the state of Kubernetes API objects.
MySQL	8.0.29	Open source relational database management system used by Keycloak.
NGINX Ingress Controller	1.1.1	Traffic management solution for cloud‑native applications in Kubernetes.
Node Exporter	1.3.1	Prometheus exporter for hardware and OS metrics.
OAM Kubernetes Runtime	0.3.0	Plug-in for implementing the Open Application Model (OAM) control plane with Kubernetes.
OpenSearch	1.2.3	Provides a distributed, multitenant-capable full-text search engine.
OpenSearch Dashboards	1.2.0	Provides search and data visualization capabilities for data indexed in OpenSearch.
Prometheus	2.34.0	Provides event monitoring and alerting.
Prometheus Adapter	0.9.1	Provides metrics in support of pod autoscaling.
Prometheus Operator	0.55.1	Provides management for Prometheus monitoring tools.
Prometheus Pushgateway	1.4.2	Allows ephemeral and batch jobs to expose their metrics to Prometheus.
Rancher	2.6.8	Manages multiple Kubernetes clusters.
Rancher Backup Operator	2.1.3	Manages backup and restore of Rancher configurations and data.
Velero	1.9.1	Manages backup and restore of Kubernetes configurations and data.
WebLogic Kubernetes Operator	3.4.10	Assists with deploying and managing WebLogic domains.
WebLogic Monitoring Exporter	2.1.5	Exports Prometheus-compatible metrics from WebLogic instances.

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides Velero and rancher-backup for backup and recovery at the component and platform level. Use the following instructions to enable and configure these components in your environment.

NOTE: The backup functionality for OpenSearch can be used only if the components are enabled explicitly in the Verrazzano CR.

Enable backup components

To back up and restore persistent data, first you must enable the velero and rancherBackup components. The following configuration shows how to enable the backup components with a prod installation profile.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:    
    velero:
      enabled: true
    rancherBackup:
      enabled: true

NOTE: rancherBackup will be enabled only in cases when rancher is also enabled.

After they’re enabled, check for Velero pods running in the verrazzano-backup namespace.

# Sample of pods running after enabling the velero component

$ kubectl get pod -n verrazzano-backup
NAME                      READY   STATUS    RESTARTS   AGE
restic-ndxfk              1/1     Running   0          21h
velero-5ff8766fd4-xbn4z   1/1     Running   0          21h

For rancher-backup, the pods will be created in the cattle-resources-system namespace.

# Sample of pods running after enabling the rancherBackup component

$ kubectl get pod -n cattle-resources-system
NAME                              READY   STATUS    RESTARTS   AGE
rancher-backup-5c4b985697-xw7md   1/1     Running   0          2d4h

Configure backup components

Next, meet the following prerequisite requirements for both velero and rancherBackup components:

Object store bucket name.
- Both components require an object store that is Amazon S3 compatible, therefore, you need to have an object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - Make a note of the bucket name and tenancy name for reference.
  - For more information about creating a bucket with Object Storage, refer to this page.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region information.
A signing key, which is required to authenticate with the Amazon S3 compatible object store. Follow these steps to create a Customer Secret Key.

Velero CLI (optional)

The Velero CLI helps you access Velero objects in a more descriptive manner; you can also manage them using kubectl.

If desired, install the Velero CLI on Oracle Linux as follows:

$ rpm -ivh https://yum.oracle.com/repo/OracleLinux/OL7/developer/olcne/x86_64/getPackage/velero-1.8.1-1.el7.x86_64.rpm

Component-specific prerequisites

Meet the following component-specific prerequisites:

Velero operator prerequisites

Now, create the following objects:

Create a backup-secret.txt file, which has the object store credentials.

[default]
aws_access_key_id=<object store access key>
aws_secret_access_key=<object store secret key>

In the namespace verrazzano-backup, create a Kubernetes secret verrazzano-backup-creds.

$ kubectl create secret generic -n <backup-namespace> <secret-name> --from-file=<key>=<full_path_to_creds_file>

Example

$ kubectl create secret generic -n verrazzano-backup verrazzano-backup-creds --from-file=cloud=backup-secret.txt

NOTE: To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.

Create BackupStorageLocation, which the backup component will reference for subsequent backups. See the following BackupStorageLocation example. For more information, see here.

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: verrazzano-backup-location
  namespace: verrazzano-backup
spec:
  provider: aws
  objectStorage:
    bucket: example-verrazzano
    prefix: backup-demo
  credential:
    name: verrazzano-backup-creds
    key: cloud
  config:
    region: us-phoenix-1
    s3ForcePathStyle: "true"
    s3Url: https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com

rancher-backup operator prerequisites

Now, in the namespace verrazzano-backup, create a Kubernetes secret rancher-backup-creds.

$ kubectl create secret generic -n <backup-namespace> <secret-name> --from-literal=accessKey=<accesskey> --from-literal=secretKey=<secretKey>

Example

$ kubectl create secret generic -n verrazzano-backup rancher-backup-creds --from-literal=accessKey="s5VLpXwa0xNZQds4UTVV" --from-literal=secretKey="nFFpvyxpQvb0dIQovsl0"

Docs: Verrazzano Analysis Tools

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides the vz analyze command-line tool, which assists in troubleshooting issues in your environment. You can use it to analyze a cluster as well as, to analyze a cluster snapshot captured by the vz bug-report tool. For detailed information about vz bug-report, see here.

The vz analyze command-line tool analyzes the cluster or a cluster snapshot, reports the issues found, and prescribes related actions to take. Users, developers, and Continuous Integration (CI) can use this tooling to quickly identify the root cause of encountered problems, determine mitigation actions, and provide a sharable report with other users or tooling.

Set up the CLI tool

To set up the vz command-line tool, follow the steps here.

Analyze clusters

To analyze a Kubernetes cluster:

$ vz analyze

Analyze cluster snapshots

Use the vz bug-report tool to capture a cluster snapshot.

To create a bug report in a TAR file named my-bug-report.tar.gz and extract it to a directory my-cluster-snapshot:
```
$ vz bug-report my-bug-report.tar.gz
  mkdir my-cluster-snapshot
  tar -xvf my-bug-report.tar.gz -C my-cluster-snapshot
```
Use the vz analyze tool to analyze the cluster snapshot.

To perform an analysis of the cluster snapshot under my-cluster-snapshot:
```
$ vz analyze --capture-dir my-cluster-snapshot
```

Use the `vz analyze` tool to analyze multiple snapshots

The vz analyze tool will find and analyze all cluster snapshot directories found under a specified root directory. This lets you create a directory to hold the cluster snapshots of related clusters in subdirectories, which the tool can then analyze.

For example:

my-cluster-snapshots
    CAPTURE_DIR-1
        cluster-snapshot
            ...
    CAPTURE_DIR-2
        cluster-snapshot
            ...

To perform an analysis of the clusters under my-cluster-snapshots:

$ vz analyze --capture-dir my-cluster-snapshots

Usage information

Use the following syntax to run vz commands from your terminal window.

$ vz analyze [flags]

Available options

Command	Definition
`--capture-dir string`	Directory holding the captured data.
`-h, --help`	Help for the `vz analyze` command.
`--report-file string`	Name of the report output file. (Default `stdout`)
`--report-format string`	The format of the report output. Valid report formats are “summary” and “detailed”. (Default “summary”)
`-v, --verbose`	Enable verbose output.

Available flags

These flags apply to all the commands.

Flag	Definition
`--context string`	The name of the kubeconfig file context to use.
`--kubeconfig string`	Path to the kubeconfig file to use.

Docs: Verrazzano and the Open Application Model

Mon, 01 Jan 0001 00:00:00 +0000

Open Application Model (OAM) is a runtime-agnostic specification for defining cloud native applications; it allows developers to focus on the application instead of the complexities of a particular runtime infrastructure. OAM provides the specification for several file formats and rules for a runtime to interpret. Verrazzano uses OAM to enable the definition of a composite application abstraction and makes OAM constructs available within a VerrazzanoApplication YAML file. Verrazzano provides the flexibility to combine what you want into a multicloud enablement. It uses the VerrazzanoApplication as a means to encapsulate a set of components, scopes, and traits, and deploy them on a selected cluster.

OAM’s workload concept makes it easy to use many different workload types. Verrazzano includes specific workload types with special handling to deploy and manage those types, such as WebLogic, Coherence, and Helidon. OAM’s flexibility lets you create a grouping that is managed as a unit, although each component can be scaled or updated independently.

How does OAM work?

OAM has five core concepts:

Workloads - Declarations of the kinds of resources supported by the platform and the OpenAPI schema for that resource. Most Kubernetes CRDs can be exposed as workloads. Standard Kubernetes resource types can also be used (for example, Deployment, Service, Pod, ConfigMap).
Components - Wrap a workload resource’s specification data within OAM specific metadata.
Application Configurations - Describe a collection of components that comprise an application. This is also where customization (such as, environmental) of each component is done. Customization is achieved using scopes and traits.
Scopes - Apply customization to several components.
Traits - Apply customization to a single component.

Docs: Verrazzano in a Multicluster Environment

Mon, 01 Jan 0001 00:00:00 +0000

Review the following key concepts to understand multicluster Verrazzano.

Admin cluster - A Kubernetes cluster that serves as the central management point for deploying and monitoring applications in managed clusters.
Managed clusters - A Kubernetes cluster that has the following characteristics:
- It is registered with an admin cluster with a unique name.
- Verrazzano multicluster applications may be deployed to the managed cluster from the admin cluster.
- Logs and metrics for Verrazzano system components and Verrazzano multicluster applications deployed on the managed cluster are viewable from the admin cluster.
Verrazzano multicluster resources - Custom Kubernetes resources defined by Verrazzano.
- Each multicluster resource serves as a wrapper for an underlying resource type.
- A multicluster resource allows the placement of the underlying resource to be specified as a list of names of the clusters in which the resource must be placed.

For more details, see here.

Docs: Verrazzano Projects

Mon, 01 Jan 0001 00:00:00 +0000

A project provides a way to group application namespaces that are owned or administered by the same user or group of users. When creating a project, you can specify the subjects: users, groups, or service accounts, that are to be granted access to the namespaces governed by the project. Two types of subjects may be specified:

Project admins, who have both read and write access to the project’s namespaces.
Project monitors, who have read-only access to the project’s namespaces.

For more information, see Projects.

Docs: Backup

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides Velero and rancher-backup for backup and recovery at the component and platform level. First, ensure that the backup component prerequisites are met, as indicated here.

The following sections provide detailed configuration information for:

The rancher-backup operator, to back up persistent data and Rancher-related configuration. See Rancher backup.
Velero hooks, to ensure a consistent backup experience for these components:
- OpenSearch. See OpenSearch backup.
- For all other components, refer to the Velero Backup Reference.

Rancher backup

To initiate a Rancher backup, create the following example custom resource YAML file that uses an Amazon S3 compatible object store as a backend. The operator uses the credentialSecretNamespace value to determine where to look for the Amazon S3 backup secret. Note that in the prerequisites example, you previously created the secret in the verrazzano-backup namespace.

apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
  name: rancher-backup-test
spec:
  storageLocation:
    s3:
      credentialSecretName: rancher-backup-creds
      credentialSecretNamespace: verrazzano-backup
      bucketName: myvz-bucket
      folder: rancher-backup
      region: us-phoenix-1
      endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
  resourceSetName: rancher-resource-set

The operator creates the backup file, in the *.tar.gz format, and stores it in the location configured in the storageLocation field.

Scheduled backups

Similar to Velero, rancher-backup allows scheduled backups.

OpenSearch backup

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero while invoking a backup. Due to the nature of transient data handled by OpenSearch, the hook invokes OpenSearch snapshot APIs to back up and restore data streams appropriately, thereby ensuring that there is no loss of data and avoids data corruption as well.

The following example shows a sample Velero Backup API object that you can invoke to make an OpenSearch backup.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: verrazzano-opensearch-backup
  namespace: verrazzano-backup
spec:
  includedNamespaces:
    - verrazzano-system
  labelSelector:
    matchLabels:
      verrazzano-component: opensearch
  defaultVolumesToRestic: false
  storageLocation:  verrazzano-backup-location
  hooks:
    resources:
      - name: opensearch-backup-test
        includedNamespaces:
          - verrazzano-system
        labelSelector:
          matchLabels:
            statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
        post:                           
          - exec:
              container: es-master
              command:
                - /usr/share/opensearch/bin/verrazzano-backup-hook
                - -operation
                - backup
                - -velero-backup-name
                - verrazzano-opensearch-backup
              onError: Fail
              timeout: 10m

The preceding example backs up the OpenSearch components:

In this case, you are not backing up the PersistentVolumes directly, rather executing a hook that invokes the OpenSearch APIs to take a snapshot of the data.
So that Velero ignores the associated PVC’s, defaultVolumesToRestic needs to be false.
In this case, the hook can be pre or post.
The command used in the hook requires an operation flag and the Velero backup name as an input.
The container on which the hook needs to be run is identified by the pod label selectors, followed by the container name. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.

After the backup is processed, you can see the hook logs using the velero backup logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch backup logs

# To display the logs from the backup, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-backup-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

Scheduled backups

Velero supports a Schedule API that is a repeatable request that is sent to the Velero server to perform a backup for a given cron notation. After the Schedule object is created, the Velero server will start the backup process. Then, it will wait for the next valid point in the given cron expression and run the backup process on a repeating basis.

Docs: Bug Reports

Mon, 01 Jan 0001 00:00:00 +0000

Use the vz bug-report tool to selectively capture cluster information and create an archive (*.tar.gz) file to help diagnose problems. The archive file helps development and support teams analyze issues and provide recommendations.

CLI setup

To set up the vz command-line tool, follow the steps here.

Use the `vz bug-report` tool

To create a bug report in a TAR file named my-bug-report.tar.gz:

$ vz bug-report --report-file my-bug-report.tar.gz

We suggest that you review the contents of the bug report before sharing it with support and development teams.

Usage information

Use the following syntax to run vz commands from your terminal window.

$ vz bug-report [flags]

Available options

Command	Definition
`-h, --help`	Help for the `vz bug-report` command.
`-i, --include-namespaces strings`	A comma-separated list of namespaces, in addition to the ones collected by default (system namespaces), for collecting cluster information. This flag can be specified multiple times, such as `--include-namespaces ns1 --include-namespaces ns...`
`-r, --report-file string`	The report file created by the `vz bug-report` command, as a `*.tar.gz` file. Defaults to `bug-report.tar.gz` in the current directory.
`-l --include-logs`	Include logs from the pods in one or more namespaces; this is specified along with the `--include-namespaces` flag.
`-d --duration duration`	The time period during which the logs are collected in seconds, minutes, and hours.
`-v, --verbose`	Enable verbose output.

Available flags

These flags apply to all the commands.

Flag	Definition
`--context string`	The name of the kubeconfig file context to use.
`--kubeconfig string`	Path to the kubeconfig file to use.

Examples

Create a bug report file, bugreport.tar.gz, by collecting data from the cluster:
```
$ vz bug-report --report-file bugreport.tar.gz
```
When --report-file is not provided, the command creates bug-report.tar.gz in the current directory.
Create a bug report file, bugreport.tar.gz, including the additional namespace ns1 from the cluster:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1
```
The flag --include-namespaces accepts comma-separated values and can be specified multiple times. For example, the following commands create a bug report by including the additional namespaces ns1, ns2, and ns3:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2,ns3
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-namespaces ns3
```
Use the --include-logs flag to collect the logs from the pods in one or more namespaces, by specifying the --include-namespaces flag. For example, the following command creates a bug report by including the logs from the additional namespaces ns1 and ns2:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs
```
The --duration flag collects logs for the specified time period. The default value is zero (0), which collects the complete pod log. You can specify seconds, minutes, and hours. For example, the following commands create bug reports by including the logs from the additional namespaces ns1 and ns2 during the specified periods of time:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 5m
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 2h
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 300s
```
The values specified for the flag --include-namespaces are case-sensitive.

Docs: Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano issues certificates to secure access from external clients to secure system endpoints.
A certificate from a certificate authority (CA) must be configured to issue the endpoint certificates in one of the following ways:

Let Verrazzano generate a self-signed CA (the default).
Configure a CA that you provide.
Configure LetsEncrypt as the certificate issuer (requires Oracle Cloud Infrastructure DNS).

In all cases, Verrazzano uses cert-manager to manage the creation of certificates.

NOTE

Self-signed certificate authorities generate certificates that are NOT signed by a trusted authority; typically, they are not used in production environments.

Use the Verrazzano self-signed CA

By default, Verrazzano creates its own self-signed CA. No configuration is required.

Use a custom CA

If you want to provide your own CA, you must:

(Optional) Create your own signing key pair and CA certificate.

For example, you can use the openssl CLI to create a key pair for the nip.io domain.
```
# Generate a CA private key
$ openssl genrsa -out tls.key 2048

# Create a self-signed certificate, valid for 10yrs with the 'signing' option set
$ openssl req -x509 -new -nodes -key tls.key -subj "/CN=*.nip.io" -days 3650 -reqexts v3_req -extensions v3_ca -out tls.crt
```
The output of these commands will be two files, tls.key and tls.crt, the key and certificate for your signing key pair. These files must be named in that manner for the next step.

If you already have generated your own key pair, you must name the private key and certificate, tls.key and tls.crt, respectively. If your issuer represents an intermediate, ensure that tls.crt contains the issuer’s full chain in the correct order.

You can find more details on providing your own CA, in the cert-manager CA documentation.

Save your signing key pair as a Kubernetes secret.

$ kubectl create ns mynamespace
$ kubectl create secret tls myca --namespace=mynamespace --cert=tls.crt --key=tls.key

Specify the secret name and namespace location in the Verrazzano custom resource.

The custom CA secret must be provided to cert-manager using the following fields in spec.components.certManager.certificate.ca in the Verrazzano custom resource:
- spec.components.certManager.certificate.ca.secretName
- spec.components.certManager.certificate.ca.clusterResourceNamespace

For example, if you created a CA secret named myca in the namespace mynamespace, you would configure it as shown:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-ca-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        ca:
          secretName: myca
          clusterResourceNamespace: mynamespace

Use LetsEncrypt certificates

You can configure Verrazzano to use certificates generated by LetsEncrypt. LetsEncrypt implements the ACME protocol, which provides a standard protocol for the automated issuance of certificates signed by a trusted authority. This is managed through the spec.components.certManager.certificate.acme field in the Verrazzano custom resource.

NOTE

Using LetsEncrypt for certificates also requires using Oracle Cloud Infrastructure DNS for DNS management. For details, see the Customize DNS page.

To configure cert-manager to use LetsEncrypt as the certificates provider, you must configure a cert-manager ACME provider with the following values in the Verrazzano custom resource:

Set the spec.components.certManager.certificate.acme.provider field to letsEncrypt.
Set the spec.components.certManager.certificate.acme.emailAddress field to a valid email address for the letsEncrypt account.
(Optional) Set the spec.components.certManager.certificate.acme.environment field to either staging or production (the default).

The following example configures Verrazzano to use the LetsEncrypt production environment by default, with Oracle Cloud Infrastructure DNS for DNS record management.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

The following example configures Verrazzano to use the LetsEncrypt staging environment with Oracle Cloud Infrastructure DNS.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
          environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

NOTE

Certificates issued by the LetsEncrypt staging environment are signed by untrusted authorities, similar to self-signed certificates. They are typically not used in production environments.

LetsEncrypt staging versus production

LetsEncrypt provides rate limits on generated certificates to ensure fair usage across all clients. The production environment limits can be exceeded more frequently in environments where Verrazzano may be installed or reinstalled frequently (like a test environment). This can result in failed installations due to rate limit exceptions on certificate generation.

In such environments, it is better to use the LetsEncrypt staging environment, which has much higher limits than the production environment. For test environments, the self-signed CA also may be more appropriate to completely avoid LetsEncrypt rate limits.

Docs: Default User Accounts

Mon, 01 Jan 0001 00:00:00 +0000

During installation, Verrazzano generates several default accounts.

System	Account	Secret	Secret Namespace	Description
Keycloak	keycloakadmin	`keycloak-http`	`keycloak`	Keycloak root user: full administrative privileges for Keycloak.
Keycloak	verrazzano	`verrazzano`	`verrazzano-system`	Verrazzano root user: can manage the verrazzano-system realm in Keycloak, including managing users in that realm. This user is a member of the verrazzano-admins group, and, if default role bindings are used, has the verrazzano-admin role.
Rancher	admin	`rancher-admin-secret`	`cattle-system`	Rancher root user: full administrative privileges for Rancher.

Docs: Hello World Helidon

Mon, 01 Jan 0001 00:00:00 +0000

The Hello World Helidon example is a Helidon-based service that returns a “Hello World” response when invoked. The example application is specified using Open Application Model (OAM) component and application configuration YAML files, and then deployed by applying those files.

The example application has two endpoints, which differ in configuration source:

/greet- uses a microprofile properties file. Deploy this application by using the instructions here.
/config- uses a Kubernetes ConfigMap. Deploy this application by using the instructions here.

For more information and the code of this application, see the Verrazzano examples.

Docs: IngressTrait

Mon, 01 Jan 0001 00:00:00 +0000

The IngressTrait custom resource contains the configuration of host and path rules for traffic routing to an application. Here is a sample ApplicationConfiguration that specifies an IngressTrait. To deploy an example application that demonstrates this IngressTrait, see Hello World Helidon.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

In the sample configuration, the IngressTrait hello-helidon-ingress is set on the hello-helidon-component application component and defines an ingress rule that configures a path and path type. This exposes a route for external access to the application. Note that because no hosts list is given for the IngressRule, a DNS host name is automatically generated.

For example, with the sample application configuration successfully deployed, the application will be accessible with the path specified in the IngressTrait and the generated host name.

$ HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-gw -n hello-helidon -o jsonpath={.spec.servers[0].hosts[0]})
$ echo $HOST
hello-helidon-appconf.hello-helidon.11.22.33.44.nip.io

$ curl -sk -X GET https://${HOST}/greet

Load balancer session affinity is configured using an HTTP cookie in a destination rule. Here is an updated sample ApplicationConfiguration that includes a destination rule with an HTTP cookie.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix
                - destination:
                    httpCookie:
                      name: sessioncookie
                      path: "/"
                      ttl: 600

Additionally, an authorization policy limiting access to specific request principals and optionally predicated on additional conditions, can be specified for a path. Request for the path will be limited to matching request principals that meet the defined conditions, otherwise the request will be denied.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix
                      authorizationPolicy:
                        rules:
                          - from:
                              requestPrincipals:
                                - "*"
                              when:
                                - key: request.auth.claims[realm_access][roles]
                                  values:
                                    - "customer"

Use the following rules related to the host name:

If you provide a host name, then you have an option to provide a certificate. If you do not provide a certificate, then Verrazzano generates one for you.
If you provide a certificate, then you must provide a host name.
If you do not provide either a host name or a certificate, then Verrazzano generates them for you.

IngressTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	IngressTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	IngressTraitSpec	The desired state of an ingress trait.	Yes

IngressTraitSpec

IngressTraitSpec specifies the desired state of an ingress trait.

Field	Type	Description	Required
`rules`	IngressRule array	A list of ingress rules to for an ingress trait.	Yes
`tls`	IngressSecurity	The security parameters for an ingress trait. This is required only if specific hosts are given in an IngressRule.	No

IngressRule

IngressRule specifies a rule for an ingress trait.

Field	Type	Description	Required
`hosts`	string array	One or more hosts exposed by the ingress trait. Wildcard hosts or hosts that are empty are filtered out. If there are no valid hosts provided, then a DNS host name is automatically generated and used.	No
`paths`	IngressPath array	The paths to be exposed for an ingress trait.	Yes
`destination`	IngressDestination	The destination host and port for the ingress paths.	No

IngressPath

IngressPath specifies a specific path to be exposed for an ingress trait.

Field	Type	Description	Required
`path`	string	If no path is provided, it defaults to forward slash (`/`).	No
`pathType`	string	Path type values are case-sensitive and formatted as follows: `exact`: exact string match `prefix`: prefix-based match `regex`: regex-based match If the provided ingress path doesn’t contain a `pathType`, it defaults to `prefix` if the path is `/` and `exact` otherwise.	No
`authorizationPolicy`	AuthorizationPolicy	Defines the set of rules for authorizing a request.	No

IngressDestination

IngressDestination specifies a specific destination host and port for the ingress paths.

Field	Type	Description	Required
`host`	string	Destination host.	No
`port`	uint32	Destination port.	No
`httpCookie`	HttpCookie	Session affinity cookie.	No

NOTE

If there are multiple ports defined for a service, then the destination port must be specified OR the service port name must have the prefix http.

HttpCookie

HttpCookie specifies a session affinity cookie for an ingress trait.

Field	Type	Description	Required
`name`	string	The name of the HTTP cookie.	No
`path`	string	The path of the HTTP cookie.	No
`ttl`	uint32	The lifetime of the HTTP cookie (in seconds).	No

IngressSecurity

IngressSecurity specifies the secret containing the certificate securing the transport for an ingress trait.

Field	Type	Description	Required
`secretName`	string	The name of a secret containing the certificate securing the transport. The specification of a secret here implies that a certificate was created for specific hosts, as specified in an IngressRule.	Yes

AuthorizationPolicy

AuthorizationPolicy defines the set of rules for authorizing a request.

Field	Type	Description	Required
`rules`	string array	Rules are used to match requests from request principals to specific paths given an optional list of conditions.	Yes

AuthorizationRule

AuthorizationRule matches requests from a list of request principals that access a specific path subject to a list of conditions.

Field	Type	Description	Required
`from`	AuthorizationRuleFrom	Specifies the request principals for access to a request. An asterisk (`*`) will match when the value is not empty, for example, if any request principal is found in the request.	Yes
`when`	AuthorizationRuleCondition	Specifies a list of additional conditions for access to a request.	No

AuthorizationRuleFrom

Provides a list of request principals.

Field	Type	Description	Required
`requestPrincipals`	string array	Specifies the request principals for access to a request.	Yes

AuthorizationRuleCondition

Provides additional required attributes for authorization.

Field	Type	Description	Required
`key`	string	The name of a request attribute.	Yes
`values`	string array	A list of allowed values for the attribute.	Yes

Docs: Installation Profiles

Mon, 01 Jan 0001 00:00:00 +0000

This document describes built-in configuration profiles that you can use to simplify a Verrazzano installation. An installation profile is a well-known configuration of Verrazzano settings that can be referenced by name, which then can be customized as needed.

The following table describes the Verrazzano installation profiles.

Profile	Description	Characteristics
`prod`	Full installation, production configuration.	Default profile: - Full installation. - Persistent storage. - Production OpenSearch cluster topology.
`dev`	Development or evaluation configuration.	Lightweight installation: - For evaluation purposes. - No persistence. - Single-node OpenSearch cluster topology.
`managed-cluster`	A specialized installation for managed clusters in a multicluster topology.	Minimal installation for a managed cluster: - Cluster must be registered with an admin cluster to use multicluster features.

Use an installation profile

To specify an installation profile when installing Verrazzano, set the profile name in the profile field of your Verrazzano custom resource.

For example, to use the dev profile:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev

To use a different profile, replace dev with prod or managed-cluster.

Customize an installation profile

You can override the profile settings for any component regardless of the profile. The following example uses a customized dev profile to configure a small 8 Gi persistent volume for the MySQL instance used by Keycloak to provide more stability for the Keycloak service.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-dev-example
spec:
  profile: dev
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql
  volumeClaimSpecTemplates:
  - metadata:
      name: mysql      
    spec:
      resources:
        requests:
          storage: 8Gi

For details on how to customize Verrazzano components, see Customize an Installation.

Profile configurations

The following table lists the Verrazzano components that are installed with each profile. Note that you can customize any Verrazzano installation, regardless of the profile.

Component	dev	prod	managed-cluster
Istio	✔️	✔️	✔️
NGINX	✔️	✔️	✔️
cert-manager	✔️	✔️	✔️
External-DNS	️	️
Prometheus	✔️	✔️	✔️
OpenSearch	✔️	✔️
Console	✔️	✔️
OpenSearch Dashboards	✔️	✔️
Grafana	✔️	✔️
Rancher	✔️	✔️
Keycloak	✔️	✔️

Prometheus and Grafana configurations

The following table describes the Prometheus and Grafana configurations in each profile.

Profile	Prometheus	Grafana
`prod`	One replica (128 MB memory, 50 Gi storage)	One replica (48 MB memory, 50 Gi storage)
`dev`	One replica (128 MB memory, ephemeral storage)	One replica (48 MB memory, ephemeral storage)
`managed-cluster`	One replica (128 MB memory, 50 Gi storage)	Not installed

OpenSearch Dashboards and OpenSearch configurations

The following table describes the OpenSearch Dashboards and OpenSearch cluster topology in each profile.

Profile	OpenSearch	OpenSearch Dashboards
`prod`	Three master replicas (1.4 Gi memory, 50 Gi storage each) One ingest replica (2.5 Gi memory, no storage) Three data replicas (4.8 Gi memory, 50 Gi storage each)	One replica (192 MB memory, ephemeral storage)
`dev`	One master/data/ingest replica (1 Gi memory, ephemeral storage)	One replica (192 MB memory, ephemeral storage)
`managed-cluster`	Not installed	Not installed

NOTE

OpenSearch containers are configured to use 75% of the configured request memory for the Java min/max heap settings.

Profile-independent defaults

The following table shows the settings for components that are profile-independent (consistent across all profiles unless overridden).

Component	Default
DNS	Wildcard DNS provider nip.io.
Certificates	Uses the cert-manager self-signed ClusterIssuer for certificates.
Ingress-type	Defaults to `LoadBalancer` service type for the ingress.

For details on how to customize Verrazzano components, see Customize an Installation.

Docs: Kind

Mon, 01 Jan 0001 00:00:00 +0000

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. Follow these instructions to prepare a Kind cluster for running Verrazzano.

NOTE

Kind is not recommended for use on macOS and Windows because the Docker network is not directly exposed to the host.

Prerequisites

Install Docker
Install Kind

Prepare the Kind cluster

To prepare the Kind cluster for use with Verrazzano, you must create the cluster and then install and configure MetalLB in that cluster.

You can create the Kind cluster in two ways: with or without image caching; image caching can speed up your installation time.

Create a Kind cluster

Kind images are prebuilt for each release. To find images suitable for a given release, check the release notes for your Kind version (check with kind version). There you’ll find a complete listing of images created for a Kind release.

The following example references a Kubernetes v1.21.1-based image built for Kind v0.11.1. Replace that image with one suitable for the Kind release you are using.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
EOF

Create a Kind cluster with image caching

While developing or experimenting with Verrazzano, you might destroy and re-create your Kind cluster multiple times. To speed up Verrazzano installation, follow these steps to ensure that the image cache used by containerd inside a Kind cluster, is preserved across clusters. Subsequent installations will be faster because they will not need to pull the images again.

1. Create a named Docker volume that will be used for the image cache and note its mountPoint path. In this example, the volume is named containerd.

$ docker volume create containerd

$ docker volume inspect containerd

#Sample output
{
    "CreatedAt": "2021-01-11T16:27:47Z",
    "Driver": "local",
    "Labels": {},
    "Mountpoint": "/var/lib/docker/volumes/containerd/_data",
    "Name": "containerd",
    "Options": {},
    "Scope": "local"
}

2. Specify the mountPoint path obtained, as the hostPath under extraMounts in your Kind configuration file, with a containerPath of /var/lib/containerd, which is the default containerd image caching location inside the Kind container. An example of the modified Kind configuration is shown in the following create cluster command.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
    extraMounts:
      - hostPath: /var/lib/docker/volumes/containerd/_data
        containerPath: /var/lib/containerd #This is the location of the image cache inside the Kind container
EOF

Install and configure MetalLB

By default, Kind does not provide an implementation of network load balancers (Services of type LoadBalancer). MetalLB offers a network load balancer implementation.

To install MetalLB:

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
$ kubectl create secret generic \
    -n metallb-system memberlist \
    --from-literal=secretkey="$(openssl rand -base64 128)"
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

For further details, see the MetalLB installation guide.

MetalLB is idle until configured. Configure MetalLB in Layer 2 mode and give it control over a range of IP addresses in the kind Docker network. In versions v0.7.0 and earlier, Kind uses Docker’s default bridge network; in versions v0.8.0 and later, it creates its own bridge network in Kind.

To determine the subnet of the kind Docker network in Kind v0.8.0 and later:

$ docker inspect kind | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.18.0.0/16

To determine the subnet of the kind Docker network in Kind v0.7.0 and earlier:

$ docker inspect bridge | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.17.0.0/16

For use by MetalLB, assign a range of IP addresses at the end of the kind network’s subnet CIDR range.

$ kubectl apply -f - <<-EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: my-ip-space
      protocol: layer2
      addresses:
      - 172.18.0.230-172.18.0.250
EOF

Next steps

To continue, see the Installation Guide.

Docs: LoggingTrait

Mon, 01 Jan 0001 00:00:00 +0000

The LoggingTrait custom resource contains the configuration for an additional logging sidecar with a custom image and Fluentd configuration file. Here is a sample ApplicationConfiguration that includes a LoggingTrait. To deploy an example application with this LoggingTrait, replace the ApplicationConfiguration of the ToDo-List example application with the following sample.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: todo-appconf
  namespace: todo-list
  annotations:
    version: v1.0.0
    description: "ToDo List example application"
spec:
  components:
    - componentName: todo-domain
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: LoggingTrait
            metadata:
              name: logging-trait-example
              namespace: todo-list
            spec:
              loggingImage: fluent/fleuntd-example-image # Replace with custom Fluentd Image
              loggingConfig: |-
                # Replace with Fluentd config file
                <match **>
                @type stdout
                </match>                
    - componentName: todo-jdbc-configmap
    - componentName: todo-mysql-configmap
    - componentName: todo-mysql-service
    - componentName: todo-mysql-deployment

In this sample configuration, the LoggingTrait logging-trait-example is set on the todo-domain application component and defines a logging sidecar with the given Fluentd image and configuration file. This sidecar will be attached to the component’s pod and will gather logs according to the given Fluentd configuration file. In order for the Fluentd DaemonSet to collect the custom logs, the Fluentd configuration file needs to direct the logs to STDOUT, as demonstrated in the previous example.

For example, when the ToDo-List example ApplicationConfiguration is successfully deployed with a LoggingTrait, the tododomain-adminserver pod will have a container named logging-stdout.

$ kubectl get pods tododomain-adminserver -n todo-list -o jsonpath='{.spec.containers[*].name}'
  ... logging-stdout ...

In this example, the logging-stdout container will run the image given in the LoggingTrait and a ConfigMap named logging-stdout-todo-domain-domain will be created with the custom Fluentd configuration file.

LoggingTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	LoggingTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	LoggingTraitSpec	The desired state of a logging trait.	Yes

LoggingTraitSpec

LoggingTraitSpec specifies the desired state of a logging trait.

Field	Type	Description	Required
`loggingConfig`	string	A string representation of the Fluentd configuration.	Yes
`loggingImage`	string	The name of the custom Fluentd image.	Yes

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano metrics stack automates metrics aggregation and consists of Prometheus and Grafana components. Metrics sources expose system and application metrics. The Prometheus components retrieve and store the metrics and Grafana provides dashboards to visualize them.

Metrics sources

The following sections describe metrics sources that Verrazzano provides for OAM and standard Kubernetes applications.

OAM

Metrics sources produce metrics and expose them to the Kubernetes Prometheus system using annotations in the pods. The metrics annotations may differ slightly depending on the resource type. The following is an example of the WebLogic Prometheus-related configuration specified in the todo-list application pod:

$ kubectl describe pod tododomain-adminserver -n todo-list

Annotations:  prometheus.io/path: /wls-exporter/metrics
              prometheus.io/port: 7001
              prometheus.io/scrape: true

For other resource types, such as Coherence or Helidon, the annotations would look similar to this:

Annotations:  verrazzano.io/metricsEnabled: true
              verrazzano.io/metricsPath: /metrics
              verrazzano.io/metricsPort: 8080

To look directly at the metrics that are being made available by the metric source, map the port and then access the path.

For example, for the previous metric source:

Map the port being used to expose the metrics.

$ kubectl port-forward tododomain-adminserver 7001:7001 -n todo-list

Get the user name and password used to access the metrics source from the corresponding secret.

$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.username} | base64 \
    --decode; echo
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

Access the metrics at the exported path, using the user name and password retrieved in the previous step.
```
$ curl -u USERNAME:PASSWORD localhost:7001/wls-exporter/metrics
```

Standard Kubernetes workloads

Verrazzano supports enabling metric sources for Kubernetes workloads deployed without OAM Components. To enable metrics for Kubernetes workloads, you must create a Service Monitor or Pod Monitor, as applicable. For details on Service Monitor and Pod Monitor, refer to the Prometheus Operator documentation.

When creating the Service Monitor or Pod Monitor for your workload, include the label release, with the value prometheus-operator on the monitor resource.

Verify metrics collection

To verify that the metrics are being collected for your workload, follow these steps.

Access the Prometheus console.
From the console, use the navigation bar to access Status/Targets.
On this page, you will see a target name with this formatting: <monitor-type>/<workload-namespace>_<workload-name>_<workload-type>, where monitor-type may be serviceMonitor or podMonitor.
Copy this job name from the target labels for use in future queries.
Verify that the State of this target is UP.
Next, use the navigation bar to access the Graph.
Here, use the job name you copied to construct this expression: {job="<job_name>"}
Use the graph to run this expression and verify that you see application metrics appear.

Metrics Traits use Service Monitors which require Services for metrics collection. If you are unable to verify metrics collection, you might need to manually create a Service for the workload.

For more information on Prometheus solutions, see Troubleshooting Prometheus.

Legacy workloads

Standard Kubernetes workloads that were metrics sources in earlier versions of Verrazzano (1.3.x or earlier), will continue to be supported when upgrading to later versions of Verrazzano.

For workloads that used the legacy default metrics template, Verrazzano will create a Service Monitor in the workload’s namespace to ensure that metrics continue to be scraped. You can make any ongoing changes to the metrics source configuration by editing the Service Monitor.

For workloads that used a legacy custom metrics template, Verrazzano will configure the Prometheus Operator to ensure that metrics continue to be scraped.

Metrics server

Verrazzano installs the Prometheus Operator in the verrazzano-monitoring namespace.
A single Prometheus pod is created by Prometheus Operator in the same namespace.
Discovers exposed metrics source endpoints.
Scrapes metrics from metrics sources.
Responsible for exposing all metrics.

Grafana

Grafana provides visualization for your Prometheus metric data.

Single pod per cluster.
Named vmi-system-grafana-* in the verrazzano-system namespace.
Provides dashboards for metrics visualization.

To access Grafana:

Get the host name from the Grafana ingress.

$ kubectl get ingress vmi-system-grafana -n verrazzano-system

# Sample output
NAME                 CLASS    HOSTS                                              ADDRESS          PORTS     AGE
vmi-system-grafana   <none>   grafana.vmi.system.default.123.456.789.10.nip.io   123.456.789.10   80, 443   26h

Get the password for the user verrazzano.

$ kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

Access Grafana in a browser using the host name.
Log in using the verrazzano user and the password.

From here, you can select an existing dashboard or create a new dashboard. To select an existing dashboard, use the drop-down list in the top left corner. The initial value of this list is Home.

To view host level metrics, select Host Metrics. This will provide system metrics for all of the nodes in your cluster.

To view the application metrics for the todo-list example application, select WebLogic Server Dashboard because the todo-list application is a WebLogic application.

Docs: Metrics Template

Mon, 01 Jan 0001 00:00:00 +0000

Due to the integration of the Prometheus Operator, the Metrics Template will no longer be used to provide metrics from default Kubernetes workloads. Instead, we recommend using Service Monitors and Pod Monitors.

For more information on setting up metrics for Kubernetes workloads, see Verrazzano metrics.

MetricsTemplate

Field	Type	Description	Required
`apiVersion`	string	`app.verrazzano.io/v1alpha1`	Yes
`kind`	string	MetricsTemplate	Yes
`metadata`	ObjectMeta	Refer to the Kubernetes API documentation for fields of metadata.	No
`spec`	MetricsTemplateSpec	The desired state of a metrics trait.	Yes

MetricsTemplateSpec

Field	Type	Description	Required
`workloadSelector`	WorkloadSelector	Selector for target workloads.	No
`prometheusConfig`	PrometheusConfig	Prometheus configuration details.	No

WorkloadSelector

Field	Type	Description	Required
`namespaceSelector`	LabelSelector	Scopes the template to a namespace.	No
`objectSelector`	LabelSelector	Scopes the template to a specific workload object.	No
`apiGroups`	[]string	Scopes the template to given API Groups.	No
`apiVersions`	[]string	Scopes the template to given API Versions.	No
`resources`	[]string	Scopes the template to given API Resources.	No

PrometheusConfig

Field	Type	Description	Required
`targetConfigMap`	TargetConfigMap	Identity of the ConfigMap to be updated with the scrape configuration specified in `scrapeConfigTemplate`.	Yes
`scrapeConfigTemplate`	string	Scrape configuration template to be added to the Prometheus configuration.	Yes

TargetConfigMap

Field	Type	Description	Required
`namespace`	string	Namespace of the ConfigMap to be updated with the scrape target configuration.	Yes
`name`	string	Name of the ConfigMap to be updated with the scrape target configuration.	Yes

Docs: MetricsTrait

Mon, 01 Jan 0001 00:00:00 +0000

The MetricsTrait custom resource contains the configuration information needed to enable metrics for an application component. Component workloads configured with a MetricsTrait are set up to emit metrics through an endpoint that are scraped by a given Prometheus deployment. Here is a sample ApplicationConfiguration that specifies a MetricsTrait. To deploy an example application that demonstrates a MetricsTrait, see Hello World Helidon.

Note that if an ApplicationConfiguration does not specify a MetricsTrait, then a default MetricsTrait will be generated with values appropriate for the workload type.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

In the sample configuration, a MetricsTrait is specified for the hello-helidon-component application component.

With the sample application configuration successfully deployed, you can query for metrics from the application component.

$ HOST=$(kubectl get ingress \
     -n verrazzano-system vmi-system-prometheus \
     -o jsonpath={.spec.rules[0].host})
$ echo $HOST

prometheus.vmi.system.default.<ip>.nip.io

$ VZPASS=$(kubectl get secret \
     --namespace verrazzano-system verrazzano \
     -o jsonpath={.data.password} | base64 \
     --decode; echo)
$ curl -sk \
    --user verrazzano:${VZPASS} \
    -X GET https://${HOST}/api/v1/query?query=vendor_requests_count_total

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"vendor_requests_count_total","app":"hello-helidon","app_oam_dev_component":"hello-helidon-component","app_oam_dev_name":"hello-helidon-appconf","app_oam_dev_resourceType":"WORKLOAD","app_oam_dev_revision":"hello-helidon-component-v1","containerizedworkload_oam_crossplane_io":"496df78f-ef8b-4753-97fd-d9218d2f38f1","job":"hello-helidon-appconf_default_helidon-logging_hello-helidon-component","namespace":"helidon-logging","pod_name":"hello-helidon-workload-b7d9d95d8-ht7gb","pod_template_hash":"b7d9d95d8"},"value":[1616535232.487,"4800"]}]}}

MetricsTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	MetricsTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	MetricsTraitSpec	The desired state of a metrics trait.	Yes

MetricsTraitSpec

MetricsTraitSpec specifies the desired state of a metrics trait.

Field	Type	Description	Required
`port`	integer	The HTTP port for the related metrics endpoint. Defaults to `8080`.	No
`ports`	[]PortSpec	The HTTP endpoints for the related metrics.	No
`path`	string	The HTTP path for the related metrics endpoint. Defaults to `/metrics`.	No
`secret`	string	The name of an opaque secret (for example, user name and password) within the workload’s namespace for metrics endpoint access.	No
`scraper`	string	The Prometheus deployment used to scrape the related metrics endpoints. By default, the Verrazzano-supplied Prometheus component is used to scrape the endpoint.	No

PortSpec

PortSpec defines an HTTP port and path combination.

Field	Type	Description	Required
`port`	integer	The HTTP port for the related metrics endpoint. Defaults to `8080`.	No
`path`	string	The HTTP path for the related metrics endpoint. Defaults to `/metrics`.	No

Docs: MultiClusterApplicationConfiguration

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterApplicationConfiguration custom resource is an envelope used to distribute core.oam.dev/v1alpha2/ApplicationConfiguration resources in a multicluster environment.

Here is a sample MultiClusterApplicationConfiguration that specifies an ApplicationConfiguration resource to create on the cluster named managed1. To deploy an example application that demonstrates a MultiClusterApplicationConfiguration, see Multicluster ToDo List.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterApplicationConfiguration
metadata:
  name: todo-appconf
  namespace: mc-todo-list
spec:
  template:
    metadata:
      annotations:
        version: v1.0.0
        description: "ToDo List example application"
    spec:
      components:
        - componentName: todo-domain
          traits:
            - trait:
                apiVersion: oam.verrazzano.io/v1alpha1
                kind: MetricsTrait
            - trait:
                apiVersion: oam.verrazzano.io/v1alpha1
                kind: IngressTrait
                spec:
                  rules:
                    - paths:
                        - path: "/todo"
                          pathType: Prefix
        - componentName: todo-jdbc-config
        - componentName: mysql-initdb-config
        - componentName: todo-mysql-service
        - componentName: todo-mysql-deployment
  placement:
    clusters:
      - name: managed1
  secrets:
    - tododomain-repo-credentials
    - tododomain-jdbc-tododb
    - tododomain-weblogic-credentials

MultiClusterApplicationConfiguration

A MultiClusterApplicationConfiguration is an envelope to create core.oam.dev/v1alpha2/ApplicationConfiguration resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterApplicationConfiguration	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterApplicationConfigurationSpec	The desired state of a `core.oam.dev/v1alpha2/ApplicationConfiguration` resource.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterApplicationConfigurationSpec

MultiClusterApplicationConfigurationSpec specifies the desired state of a core.oam.dev/v1alpha2/ApplicationConfiguration resource.

Field	Type	Description	Required
`template`	ApplicationConfigurationTemplate	The embedded `core.oam.dev/v1alpha2/ApplicationConfiguration` resource.	Yes
`placement`	Placement	Clusters in which the resource is to be placed.	Yes
`secrets`	string array	List of secrets used by the application. These secrets must be created in the application’s namespace before deploying a MultiClusterApplicationConfiguration resource.	No

ApplicationConfigurationTemplate

ApplicationConfigurationTemplate has the metadata and spec of the core.oam.dev/v1alpha2/ApplicationConfiguration resource.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	ApplicationConfigurationSpec	An instance of the `struct` ApplicationConfigurationSpec defined in core_types.go.	No

Docs: MultiClusterComponent

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterComponent custom resource is an envelope used to distribute core.oam.dev/v1alpha2/Component resources in a multicluster environment.

NOTE

Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterComponent custom resource not be used; instead directly use core.oam.dev/v1alpha2/Component resources in your application. See the example application, Multicluster ToDo List, which directly uses core.oam.dev/v1alpha2/Component resources.

Here is a sample MultiClusterComponent that specifies a OAM Component resource to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterComponent
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  template:
    spec:
      workload:
        apiVersion: oam.verrazzano.io/v1alpha1
        kind: VerrazzanoHelidonWorkload
        metadata:
          name: hello-helidon-workload
          namespace: hello-helidon
          labels:
            app: hello-helidon
        spec:
          deploymentTemplate:
            metadata:
              name: hello-helidon-deployment
            podSpec:
              containers:
                - name: hello-helidon-container
                  image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.12-1-20210409130027-707ecc4"
                  ports:
                    - containerPort: 8080
                      name: http
  placement:
    clusters:
      - name: managed1

MultiClusterComponent

A MultiClusterComponent is an envelope to create core.oam.dev/v1alpha2/Component resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterComponent	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterComponentSpec	The desired state of a `core.oam.dev/v1alpha2/Component` resource.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterComponentSpec

MultiClusterComponentSpec specifies the desired state of a core.oam.dev/v1alpha2/Component resource.

Field	Type	Description	Required
`template`	ComponentTemplate	The embedded `core.oam.dev/v1alpha2/Component` resource.	Yes
`placement`	Placement	Clusters in which the resource is to be placed.	Yes

ComponentTemplate

ComponentTemplate has the metadata and spec of the core.oam.dev/v1alpha2/Component resource.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	ComponentSpec	An instance of the `struct` ComponentSpec defined in core_types.go.	No

Docs: MultiClusterConfigMap

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterConfigMap custom resource is an envelope used to distribute Kubernetes ConfigMap resources in a multicluster environment.

NOTE

Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterConfigMap custom resource not be used; instead directly use core.oam.dev/v1alpha2/Component to define ConfigMap resources in your application. See the example application, Multicluster ToDo List, which uses core.oam.dev/v1alpha2/Component resources to define ConfigMaps.

Here is a sample MultiClusterConfigMap that specifies a Kubernetes ConfigMap to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterConfigMap
metadata:
  name: mymcconfigmap
  namespace: multiclustertest
spec:
  template:
    metadata:
      name: myconfigmap
      namespace: myns
    data:
      simple.key: "simplevalue"
  placement:
    clusters:
      - name: managed1

MultiClusterConfigMap

A MultiClusterConfigMap is an envelope to create Kubernetes ConfigMap resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterConfigMap	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterConfigMapSpec	The desired state of a Kubernetes ConfigMap.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterConfigMapSpec

MultiClusterConfigMapSpec specifies the desired state of a Kubernetes ConfigMap.

Field	Type	Description	Required
`template`	ConfigMapTemplate	The embedded Kubernetes ConfigMap.	Yes
`placement`	Placement	Clusters in which the ConfigMap is to be placed.	Yes

ConfigMapTemplate

ConfigMapTemplate has the metadata and spec of the Kubernetes ConfigMap.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`immutable`	*bool	Corresponds to the `immutable` field of the `struct` ConfigMap defined in types.go.	No
`data`	map[string]string	Corresponds to the `data` field of the `struct` ConfigMap defined in types.go.	No
`binaryData`	map[string][]byte	Corresponds to the `binaryData` field of the `struct` ConfigMap defined in types.go.	No

Docs: MultiClusterResourceStatus Subresource

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterResourceStatus subresource is shared by multicluster custom resources.

MultiClusterResourceStatus

MultiClusterResourceStatus specifies the status portion of a multicluster resource.

Field	Type	Description	Required
`conditions`	Condition array	The current state of a multicluster resource.	No
`state`	string	The state of the multicluster resource. State values are case-sensitive and formatted as follows: `Pending`: deployment to cluster is in progress `Succeeded`: deployment to cluster successfully completed `Failed`: deployment to cluster failed	No
`clusters`	ClusterLevelStatus array	Array of status information for each cluster.	No

Condition

Condition describes current state of a multicluster resource across all clusters.

Field	Type	Description	Required
`type`	string	The condition of the multicluster resource which can be checked with a `kubectl wait` command. Condition values are case-sensitive and formatted as follows: `DeployComplete`: deployment to all clusters completed successfully `DeployFailed`: deployment to all clusters failed	Yes
`status`	ConditionStatus	An instance of the type ConditionStatus that is defined in types.go.	Yes
`lastTransitionTime`	string	The last time the condition transitioned from one status to another.	No
`message`	string	A message with details about the last transition.	No

ClusterLevelStatus

ClusterLevelStatus describes the status of the multicluster resource on an individual cluster.

Field	Type	Description	Required
`name`	string	Name of the cluster.	Yes
`state`	string	The state of the multicluster resource. State values are case-sensitive and formatted as follows: `Pending`: deployment is in progress `Succeeded`: deployment successfully completed `Failed`: deployment failed	No
`message`	string	Message with details about the status in this cluster.	No
`lastUpdateTime`	string	The last time the resource state was updated.	Yes

Docs: MultiClusterSecret

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterSecret custom resource is an envelope used to distribute Kubernetes Secret resources in a multicluster environment.

NOTE

Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterSecret custom resource not be used; instead specify secrets in the MultiClusterApplicationConfiguration resource. See the example application, Multicluster ToDo List where secrets are specified in a MultiClusterApplicationConfiguration resource.

Here is a sample MultiClusterSecret that specifies a Kubernetes secret to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterSecret
metadata:
  name: mymcsecret
  namespace: multiclustertest
spec:
  template:
    data:
      username: <base64-encoded value>
  spec:
  placement:
    clusters:
      - name: managed1

MultiClusterSecret

A MultiClusterSecret is an envelope to create Kubernetes Secret resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterSecret	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterSecretSpec	The desired state of a Kubernetes Secret.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterSecretSpec

MultiClusterSecretSpec specifies the desired state of a Kubernetes Secret.

Field	Type	Description	Required
`template`	SecretTemplate	The embedded Kubernetes Secret.	Yes
`placement`	Placement	Clusters in which the Secret is to be placed.	Yes

SecretTemplate

SecretTemplate has the metadata and spec of the Kubernetes Secret.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`data`	map[string][]byte	Corresponds to the `data` field of the `struct` Secret defined in types.go.	No
`stringData`	map[string]string	Corresponds to the `stringData` field of the `struct` Secret defined in types.go.	No
`type`	string	Corresponds to the `type` field of the `struct` Secret defined in types.go.	No

Docs: Network Security

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano manages and secures network traffic between Verrazzano system components and deployed applications. Verrazzano does not manage or secure traffic for the Kubernetes cluster itself, or for non-Verrazzano services or applications running in the cluster. Traffic is secured at two levels in the network stack:

ISO Layer 3/4: Using NetworkPolicies to control IP access to Pods.
ISO Layer 6: Using TLS and mutual TLS authentication (mTLS) to provide authentication, confidentiality, and integrity for connections within the cluster and for external connections.

NetworkPolicies

By default, all Pods in a Kubernetes cluster have network access to all other Pods in the cluster. Kubernetes has a NetworkPolicy resource that provides network level 3 and 4 security for Pods, restricting both ingress and egress IP traffic for a set of Pods in a namespace. Verrazzano configures all system components with NetworkPolicies to control ingress. Egress is not restricted.

NOTE: A NetworkPolicy resource needs a NetworkPolicy controller to implement the policy, otherwise the policy has no effect. You must install a Kubernetes Container Network Interface (CNI) plug-in that provides a NetworkPolicy controller, such as Calico, before installing Verrazzano, or else the policies are ignored.

NetworkPolicies for system components

Verrazzano installs a set of NetworkPolicies for system components to control ingress into the Pods. A policy is scoped to a namespace and uses selectors to specify the Pods that the policy applies to, along with the ingress and egress rules. For example, the following policy applies to the Verrazzano API Pod in the verrazzano-system namespace. This policy allows network traffic from NGINX Ingress Controller on port 8775 and from Prometheus on port 15090. No other Pods can reach those ports or any other ports of the Verrazzano API Pod. Notice that namespace selectors need to be used; the NetworkPolicy resource does not support specifying the namespace name.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
...
spec:
  PodSelector:
    matchLabels:
      app: verrazzano-api
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: ingress-nginx
      PodSelector:
        matchLabels:
          app.kubernetes.io/instance: ingress-controller
    ports:
    - port: 8775
      protocol: TCP
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      PodSelector:
        matchLabels:
          app: system-prometheus
    ports:
    - port: 15090
      protocol: TCP

The following table shows all of the ingresses that allow network traffic into system components. The ports shown are Pod ports, which is what NetworkPolicies require.

Component	Pod Port	From	Description
Verrazzano Application Operator	9443	Kubernetes API Server	Webhook entrypoint
Verrazzano Platform Operator	9443	Kubernetes API Server	Webhook entrypoint
Verrazzano Console	8000	NGINX Ingress	Access from external client
Verrazzano Console	15090	Prometheus	Prometheus scraping
Verrazzano Authentication Proxy	8775	NGINX Ingress	Access from external client
Verrazzano Authentication Proxy	15090	Prometheus	Prometheus scraping
cert-manager	9402	Prometheus	Prometheus scraping
Coherence Operator	9443	Prometheus	Webhook entrypoint
OpenSearch	8775	NGINX Ingress	Access from external client
OpenSearch	8775	Fluentd	Access from Fluentd
OpenSearch	9200	OpenSearch Dashboards, Internal	OpenSearch data port
OpenSearch	9300	Internal	OpenSearch cluster port
OpenSearch	15090	Prometheus	Envoy metrics scraping
Istio control plane	15012	Envoy	Envoy access to `istiod`
Istio control plane	15014	Prometheus	Prometheus scraping.
Istio control plane	15017	Kubernetes API Server	Webhook entrypoint
Istio ingress gateway	8443	External	Application ingress
Istio ingress gateway	15090	Prometheus	Prometheus scraping
Istio egress gateway	8443	Mesh services	Application egress
Istio egress gateway	15090	Prometheus	Prometheus scraping
Keycloak	8080	NGINX Ingress	Access from external client
Keycloak	15090	Prometheus	Prometheus scraping
MySql	15090	Prometheus	Prometheus scraping
MySql	3306	Keycloak	Keycloak datastore
Node exporter	9100	Prometheus	Prometheus scraping
Rancher	80	NGINX Ingress	Access from external client
Rancher	9443	Kubernetes API Server	Webhook entrypoint
Prometheus	8775	NGINX Ingress	Access from external client
Prometheus	9090	Grafana	Access for Grafana UI

NetworkPolicies for applications

By default, applications do not have NetworkPolicies that restrict ingress into the application or egress from it. You can configure them for the application namespaces using the NetworkPolicy section of a Verrazzano project.

NOTE

Verrazzano requires specific ingress to and egress from application pods. If you add a NetworkPolicy for your application namespace or pods, you must add an additional policy to ensure that Verrazzano still has the required access it needs. The ingress policy is needed only if you restrict ingress. Likewise, the egress policy is needed only if you restrict egress. The following are the ingress and egress NetworkPolicies:

Ingress NetworkPolicies

  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-ingressgateway
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: system-prometheus
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: weblogic-operator

Egress NetworkPolicies

  egress:
  - ports:
    - port: 15012
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-egressgateway
  - ports:
    - port: 53
      protocol: TCP
    - port: 53
      protocol: UDP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: kube-system
  - ports:
    - port: 8000
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator

NetworkPolicies for Envoy sidecar proxies

As mentioned, Envoy sidecar proxies run in both system component pods and application pods. Each proxy sends requests to the Istio control plane pod, istiod, for a variety of reasons. During installation, Verrazzano creates a NetworkPolicy named istiod-access in the istio-system namespace to give ingress to system component and application sidecar proxies.

Mutual TLS authentication (mTLS)

Istio can be enabled to use mTLS between services in the mesh, and also between the Istio gateways and Envoy sidecar proxies. There are various options to customize mTLS usage, for example it can be disabled on a per-port level. The Istio control plane, Istiod, is a CA and provides key and certificate rotation for the Envoy proxies, both gateways and sidecars.

Verrazzano configures Istio to have strict mTLS for the mesh. All components and applications put into the mesh will use mTLS, with the exception of Coherence clusters, which are not in the mesh. Also, all traffic between the Istio ingress gateway and mesh sidecars use mTLS, and the same is true between the proxy sidecars and the egress gateway.

Verrazzano sets up mTLS during installation with the PeerAuthentication resource as follows:

apiVersion: v1
items:
- apiVersion: security.istio.io/v1beta1
  kind: PeerAuthentication
  ...
  spec:
    mtls:
      mode: STRICT

TLS

TLS is used by external clients to access the cluster, both through the NGINX Ingress Controller and the Istio ingress gateway. The certificate used by these TLS connections vary; see Verrazzano security for details. All TLS connections are terminated at the ingress proxy. Traffic between the two proxies and the internal cluster Pods always uses mTLS, because those Pods are all in the Istio mesh.

Istio mesh

Istio provides extensive security protection for both authentication and authorization, as described in Istio Security. Access control and mTLS are two security features that Verrazzano configures. These security features are available in the context of a service mesh.

A service mesh is an infrastructure layer that provides certain capabilities like security, observability, load balancing, and such, for services. Istio defines a service mesh here. In the context of Istio on Kubernetes, a service in the mesh is a Kubernetes Service. Consider the Bob’s Books example application, which has several OAM Components defined. At runtime, there is a Kubernetes Service for each component, and each Service is in the mesh, with one or more Pods associated with the service. All services in the mesh have an Envoy proxy in front of their Pods, intercepting network traffic to and from the Pod. In Kubernetes, that proxy happens to be a sidecar running in each Pod.

There are various ways to put a service in the mesh. Verrazzano uses the namespace label, istio-injection: enabled, to designate that all Pods in a given namespace are in the mesh. When a Pod is created in that namespace, the Istio control plane mutating webhook, changes the Pod spec to add the Envoy proxy sidecar container, causing the Pod to be in the mesh.

Disabling sidecar injection

In certain cases, Verrazzano needs to disable sidecar injection for specific Pods in a namespace. This is done in two ways: first, during installation, Verrazzano modifies the istio-sidecar-injector ConfigMap using a Helm override file for the Istio chart. This excludes several components from the mesh, such as the Verrazzano application operator. Second, certain Pods, such as Coherence Pods, are labeled at runtime with sidecar.istio.io/inject="false" to exclude them from the mesh.

Components in the mesh

The following Verrazzano components are in the mesh and use mTLS for all service to service communication.

OpenSearch
Fluentd
Grafana
Kiali
OpenSearch Dashboards
Keycloak
MySQL
NGINX Ingress Controller
Prometheus
Verrazzano Authentication Proxy
Verrazzano Console
WebLogic Kubernetes Operator

Some of these components, have mesh-related details that are worth noting, as described in the following sections.

NGINX

The NGINX Ingress Controller listens for HTTPS traffic, and provides ingress into the cluster. NGINX is configured to do TLS termination of client connections. All traffic from NGINX to the mesh services use mTLS, which means that traffic is fully encrypted from the client to the target back-end services.

Keycloak and MySQL

Keycloak and MySQL are also in the mesh and use mTLS for network traffic. Because all of the components that use Keycloak are in the mesh, there is end to end mTLS security for all identity management handled by Keycloak. The following components access Keycloak:

Verrazzano Authentication Proxy
Verrazzano Console
OpenSearch
Prometheus
Grafana
Kiali
OpenSearch Dashboards

Prometheus

Although Prometheus is in the mesh, it is configured to use the Envoy sidecar and mTLS only when communicating with Keycloak. All the traffic related to scraping metrics, bypasses the sidecar proxy, doesn’t use the service IP address, but rather connects to the scrape target using the Pod IP address. If the scrape target is in the mesh, then HTTPS is used; otherwise, HTTP is used. For Verrazzano multicluster, Prometheus also connects from the admin cluster to the Prometheus server in the managed cluster by using the managed cluster NGINX Ingress, using HTTPS. Prometheus is in the managed cluster and never establishes connections to targets outside the cluster.

Because Prometheus is in the mesh, additional configuration is done to allow the Envoy sidecar to be bypassed when scraping Pods. This is done with the Prometheus Pod annotation traffic.sidecar.istio.io/includeOutboundIPRanges: <keycloak-service-ip>. This causes traffic bound for Keycloak to go through the Envoy sidecar, and all other traffic to bypass the sidecar.

WebLogic Kubernetes Operator

Applications in the mesh

Before you create a Verrazzano application, you should decide if it should be in the mesh. You control sidecar injection, for example, mesh inclusion, by labeling the application namespace with istio-injection=enabled or istio-injection=disabled. By default, applications will not be put in the mesh if that label is missing. If your application uses a Verrazzano project, then Verrazzano will label the namespaces in the project to enable injection. If the application is in the mesh, then mTLS will be used. You can change the PeerAuthentication mTLS mode as desired if you don’t want strict mTLS. Also, if you need to add mTLS port exceptions, you can do this with DestinationRules or by creating another PeerAuthentication resource in the application namespace. Consult the Istio documentation for more information.

WebLogic

When the WebLogic Kubernetes Operator creates a domain, it needs to communicate with the Pods in the domain. Verrazzano puts the operator in the mesh so that it can communicate with the domain Pods using mTLS. Because of that, the WebLogic domain must be created in the mesh. Also, because mTLS is used, do not configure WebLogic to use TLS. If you want to use a custom certificate for your application, you can specify that in the ApplicationConfiguration, but that TLS connection will be terminated at the Istio ingress gateway, which you configure using a Verrazzano IngressTrait.

Coherence

Coherence clusters are represented by the Coherence resource, and are not in the mesh. When Verrazzano creates a Coherence cluster in a namespace that is annotated to do sidecar injection, it disables injection of the Coherence resource using the sidecar.istio.io/inject="false" label shown previously. Furthermore, Verrazzano will create a DestinationRule in the application namespace to disable mTLS for the Coherence extend port 9000. This allows a service in the mesh to call the Coherence extend proxy. For an example, see Bobs Books.

Here is an example of a DestinationRule created for the Bob’s Books application which includes a Coherence cluster.

API Version:  networking.istio.io/v1beta1
Kind:         DestinationRule
...
Spec:
  Host:  *.bobs-books.svc.cluster.local
  Traffic Policy:
    Port Level Settings:
      Port:
        Number:  9000
      Tls:
    Tls:
      Mode:  ISTIO_MUTUAL

Istio access control

Istio lets you control access to your workload in the mesh using the AuthorizationPolicy resource. This lets you control which services or Pods can access your workloads. Some of these options require mTLS; for more information, see Authorization Policy.

Verrazzano always creates AuthorizationPolicies for applications but never for system components. During application deployment, Verrazzano creates the policy in the application namespace and configures it to allow access from the following:

Other Pods in the application
Istio ingress gateway
Prometheus scraper

This prevents other Pods in the cluster from gaining network access to the application Pods. Istio uses a service identity to determine the identity of the request’s origin; for Kubernetes this identity is a service account. Verrazzano creates a per-application AuthorizationPolicy as follows:

AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
...
spec:
  rules:
    - from:
    - source:
  principals:
    - cluster.local/ns/sales/sa/greeter
    - cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
    - cluster.local/ns/verrazzano-system/sa/verrazzano-monitoring-operator

WebLogic domain access

For WebLogic applications, the WebLogic Kubernetes Operator must have access to the domain Pods for two reasons. First, it must access the domain servers to get health status; second, it must inject configuration into the Monitoring Exporter sidecar running in the domain server Pods. When a WebLogic domain is created, Verrazzano adds an additional source, cluster.local/ns/verrazzano-system/sa/weblogic-operator-sa to the principals section to permit that access.

Docs: Placement Subresource

Mon, 01 Jan 0001 00:00:00 +0000

The Placement subresource is shared by multicluster custom resources.

Placement

Placement contains the name of each cluster where this resource will be located.

Field	Type	Description	Required
`clusters`	Cluster array	An array of cluster locations.	Yes

Cluster

Cluster contains the name of a single cluster.

Field	Type	Description	Required
`cluster`	string	The name of a cluster.	Yes

Docs: Verrazzano v1alpha1

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano custom resource contains the configuration information for an installation. Here is a sample Verrazzano custom resource file that uses Oracle Cloud Infrastructure DNS. See other examples here.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  environmentName: env
  profile: prod
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: emailAddress@example.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: dnsZoneCompartmentOcid
        dnsZoneOCID: dnsZoneOcid
        dnsZoneName: my.dns.zone.name
    ingress:
      type: LoadBalancer

VerrazzanoSpec

Field	Type	Description	Required
`environmentName`	string	Name of the installation. This name is part of the endpoint access URLs that are generated. The default value is `default`.	No
`profile`	string	The installation profile to select. Valid values are `prod` (production), `dev` (development), and `managed-cluster`. The default is `prod`.	No
`version`	string	The version to install. Valid versions can be found here. Defaults to the current version supported by the Verrazzano platform operator.	No
`components`	Components	The Verrazzano components.	No
`defaultVolumeSource`	VolumeSource	Defines the type of volume to be used for persistence for all components unless overridden, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of an existing `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.	No
`volumeClaimSpecTemplates`	VolumeClaimSpecTemplate	Defines a named set of PVC configurations that can be referenced from components to configure persistent volumes.	No

VolumeClaimSpecTemplate

Field	Type	Description	Required
`metadata`	ObjectMeta	Metadata about the PersistentVolumeClaimSpec template.	No
`spec`	PersistentVolumeClaimSpec	A `PersistentVolumeClaimSpec` template that can be referenced by a Component to override its default storage settings for a profile. At present, only a subset of the `resources.requests` object are honored depending on the component.	No

Components

Field	Type	Description	Required
`authProxy`	AuthProxyComponent	The AuthProxy component configuration.	No
`certManager`	CertManagerComponent	The cert-manager component configuration.	No
`dns`	DNSComponent	The DNS component configuration.	No
`ingress`	IngressComponent	The ingress component configuration.	No
`istio`	IstioComponent	The Istio component configuration.	No
`fluentd`	FluentdComponent	The Fluentd component configuration.	No
`jaegerOperator`	JaegerOperatorComponent	The Jaeger Operator component configuration.	No
`keycloak`	KeycloakComponent	The Keycloak component configuration.	No
`elasticsearch`	OpenSearchComponent	The OpenSearch component configuration.	No
`prometheus`	PrometheusComponent	The Prometheus component configuration.	No
`kibana`	OpenSearchDashboardsComponent	The OpenSearch Dashboards component configuration.	No
`grafana`	GrafanaComponent	The Grafana component configuration.	No
`kiali`	KialiComponent	The Kiali component configuration.	No
`prometheusOperator`	PrometheusOperatorComponent	The Prometheus Operator component configuration.	No
`prometheusAdapter`	PrometheusAdapterComponent	The Prometheus Adapter component configuration.	No
`kubeStateMetrics`	KubeStateMetricsComponent	The kube-state-metrics component configuration.	No
`velero`	VeleroComponent	The Velero component configuration.	No
`rancherBackup`	RancherBackupComponent	The rancherBackup component configuration.	No

AuthProxy Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then AuthProxy will be installed.	No
`kubernetes`	AuthProxyKubernetes	The Kubernetes resources than can be configured for AuthProxy.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

AuthProxy Kubernetes Configuration

Field	Type	Description	Required
`replicas`	uint32	The number of pods to replicate.	No
`affinity`	Affinity	A Kubernetes affinity definition.	No

CertManager Component

Field	Type	Description	Required
`certificate`	Certificate	The certificate configuration.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Certificate

Field	Type	Description	Required
`acme`	Acme	The ACME configuration. Either `acme` or `ca` must be specified.	No
`ca`	CertificateAuthority	The certificate authority configuration. Either `acme` or `ca` must be specified.	No

Acme

Field	Type	Description	Required
`provider`	string	Name of the Acme provider.	Yes
`emailAddress`	string	Email address of the user.	Yes

CertificateAuthority

Field	Type	Description	Required
`secretName`	string	The secret name.	Yes
`clusterResourceNamespace`	string	The secrete namespace.	Yes

DNS Component

Field	Type	Description	Required
`wildcard`	DNS-Wilcard	Wildcard DNS configuration. This is the default with a domain of `nip.io`.	No
`oci`	DNS-OCI	Oracle Cloud Infrastructure DNS configuration.	No
`external`	DNS-External	External DNS configuration.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

DNS Wildcard

Field	Type	Description	Required
`domain`	string	The type of wildcard DNS domain. For example, `nip.io`, `sslip.io`, and such.	Yes

DNS Oracle Cloud Infrastructure

Field	Type	Description	Required
`ociConfigSecret`	string	Name of the Oracle Cloud Infrastructure configuration secret. Generate a secret based on the Oracle Cloud Infrastructure configuration profile you want to use. You can specify a profile other than DEFAULT and specify the secret name. See instructions by running `./install/create_oci_config_secret.sh`.	Yes
`dnsZoneCompartmentOCID`	string	The Oracle Cloud Infrastructure DNS compartment OCID.	Yes
`dnsZoneOCID`	string	The Oracle Cloud Infrastructure DNS zone OCID.	Yes
`dnsZoneName`	string	Name of Oracle Cloud Infrastructure DNS zone.	Yes
`dnsScope`	string	Scope of the Oracle Cloud Infrastructure DNS zone (`PRIVATE`, `GLOBAL`). If not specified, then defaults to `GLOBAL`.	No

DNS External

Field	Type	Description	Required
`suffix`	string	The suffix for DNS names.	Yes

Ingress Component

Field	Type	Description	Required
`type`	string	The ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`. If the ingress type is `NodePort`, a valid and accessible IP address must be specified using the `controller.service.externalIPs` key in NGINXInstallArgs. For sample usage, see External Load Balancers.	No
`nginxInstallArgs`	NGINXInstallArgs list	A list of values to use during NGINX installation.	No
`ports`	PortConfig list	The list port configurations used by the ingress.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

NGINX Install Args

Name	Type	ValueType	Description	Required
`controller.service.externalIPs`	NameValue	string list	The external IP address used by the NGINX Ingress Controller.	No
`controller.service.externalTrafficPolicy`	NameValue	string	Preserves the client source IP address. See Bare-metal considerations.	No
`controller.service.annotations.*`	NameValue	string	Annotations used for NGINX Ingress Controller. For sample usage, see Customize Ingress.	No
`controller.autoscaling.enabled`	NameValue	Boolean	If true, then enable horizonal pod autoscaler. Default `false`.	No
`controller.autoscaling.minReplicas`	NameValue	string	Minimum replicas used for autoscaling. Default `1`.	No

Port Config

Field	Type	Description	Required
`name`	string	The port name.	No
`port`	string	The port value.	Yes
`targetPort`	string	The target port value. The default is same as the port value.	Yes
`protocol`	string	The protocol used by the port. `TCP` is the default.	No
`nodePort`	string	The `nodePort` value.	No

Name Value

Field	Type	Description	Required
`name`	string	The name of a Helm override for a Verrazzano component chart, specified with a `set` flag on the Helm command line, for example, `helm install --set name=value`. For more information about chart overrides, see Customize Ingress.	Yes
`value`	string	The value of a Helm override for a Verrazzano component chart, specified with a `set` flag on the Helm command line, for example, `helm install --set name=value`. Either `value` or `valueList` must be specified. For more information about chart overrides, see Customize Ingress.	No
`valueList`	string list	The list of Helm override values for a Verrazzano component, each specified with a `set` flag on the Helm command line, for example, `helm install --set name[0]=<first element of valueList> —set name[1]=<second element of valueList>`. Either `value` or `valueList` must be specified. For more information about chart overrides, see Customize Ingress.	No
`setString`	Boolean	Specifies if the argument requires the Helm `--set-string` command-line flag to override a chart value, for example, `helm install --set-string name=value`.	No

Istio Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Istio will be installed.	No
`istioIngress`	IstioIngress	The Istio ingress gateway configuration.	No
`istioEgress`	IstioEgress	The Istio egress gateway configuration.	No
`istioInstallArgs`	IstioInstallArgs list	A list of values to use during Istio installation. Each argument is specified as either a `name/value` or `name/valueList` pair.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for default IstioOperator. Lower Overrides have precedence over the ones above them. You can find all possible values here. Passing through an invalid IstioOperator resource will result in an error.	No

Istio Ingress Configuration

Field	Type	Description	Required
`type`	string	The Istio ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`. If the Istio ingress type is `NodePort`, a valid and accessible IP address must be specified using the `gateways.istio-ingressgateway.externalIPs` key in IstioInstallArgs. For sample usage, see External Load Balancers.	No
`ports`	PortConfig list	The list port configurations used by the Istio ingress.	No
`kubernetes`	IstioKubernetes	The Kubernetes resources than can be configured for an Istio ingress gateway.	No

Istio Egress Configuration

Field	Type	Description	Required
`kubernetes`	IstioKubernetes	The Kubernetes resources than can be configured for an Istio egress gateway.	No

Istio Kubernetes Configuration

Field	Type	Description	Required
`replicas`	uint32	The number of pods to replicate.	No
`affinity`	Affinity	A Kubernetes affinity definition.	No

Istio Install Args

Name	Type	ValueType	Description	Required
`gateways.istio-ingressgateway.externalIPs`	NameValue	string list	The external IP address used by the Istio ingress gateway.	No
`gateways.istio-ingressgateway.serviceAnnotations.*`	NameValue	string	Annotations used for the Istio ingress gateway. For sample usage, see Customize Ingress.	No
`meshConfig.enableTracing`	NameValue	string	If `"true"`, Istio will export tracing when Jaeger is installed. Defaults to `"false"`.	No
`meshConfig.defaultConfig.tracing.sampling`	NameValue	string	Sampling rate for Istio tracing. Defaults to `"1"`, meaning a 1% sampling rate.	No

Fluentd Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Fluentd will be installed.	No
`extraVolumeMounts`	ExtraVolumeMount list	A list of host path volume mounts in addition to `/var/log` into the Fluentd DaemonSet. The Fluentd component collects log files in the `/var/log/containers` directory of Kubernetes worker nodes. The `/var/log/containers` directory may contain symbolic links to files located outside the `/var/log` directory. If the host path directory containing the log files is located outside of `/var/log`, the Fluentd DaemonSet must have the volume mount of that directory to collect the logs.	No
`elasticsearchURL`	string	The target OpenSearch URLs. Specify this option in this format. The default `http://vmi-system-es-ingest-oidc:8775` is the VMI OpenSearch URL.	No
`elasticsearchSecret`	string	The secret containing the credentials for connecting to OpenSearch. This secret needs to be created in the `verrazzano-install` namespace prior to creating the Verrazzano custom resource. Specify the OpenSearch login credentials in the `username` and `password` fields in this secret. Specify the CA for verifying the OpenSearch certificate in the `ca-bundle` field, if applicable. The default `verrazzano` is the secret for connecting to the VMI OpenSearch.	No
`oci`	OCILoggingConfiguration	The Oracle Cloud Infrastructure Logging configuration.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Jaeger Operator Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Jaeger Operator will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Extra Volume Mount

Field	Type	Description	Required
`source`	string	The source host path.	Yes
`destination`	string	The destination path on the Fluentd Container, defaults to the `source` host path.	No
`readOnly`	Boolean	Specifies if the volume mount is read-only, defaults to `true`.	No

Oracle Cloud Infrastructure Logging Configuration

Field	Type	Description	Required
`systemLogId`	string	The OCID of the Oracle Cloud Infrastructure Log that will collect system logs.	Yes
`defaultAppLogId`	string	The OCID of the Oracle Cloud Infrastructure Log that will collect application logs.	Yes
`apiSecret`	string	The name of the secret containing the Oracle Cloud Infrastructure API configuration and private key.	No

Keycloak Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Keycloak will be installed.	No
`mysql`	MySQLComponent	Contains the MySQL component configuration needed for Keycloak.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

MySQL Component

Field	Type	Description	Required
`volumeSource`	VolumeSource	Defines the type of volume to be used for persistence for Keycloak/MySQL, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of a `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

OpenSearch Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then OpenSearch will be installed.	No
`installArgs`	OpenSearchInstallArgs list	A list of values to use during OpenSearch installation. Each argument is specified as either a `name/value` or `name/valueList` pair. For sample usage, see Customize OpenSearch.	No
`policies`	Policy list	A list of Index State Management policies to enable on OpenSearch.	No
`plugins`	OpenSearchPlugins	OpenSearch plug-ins to be installed in OpenSearch.	No
`nodes`	Node list	A list of OpenSearch node groups.	No

OpenSearch Node Groups

Field	Type	Description	Required
`name`	string	Name of the node group.	Yes
`replicas`	integer	Node group replica count.	No
`roles`	list	Role(s) that nodes in the group will assume. May be `master`, `data`, and/or `ingest`.	Yes
`storage`	Storage	Storage settings for the node group.	No
`resources`	Resources	Kubernetes container resources for nodes in the node group.	No

OpenSearch Node Group Storage

Field	Type	Description	Required
`size`	string	Node group storage size expressed as a Quantity.	Yes

OpenSearch Index Management Policies

Field	Type	Description	Required
`policyName`	string	Name of the Index State Management policy.	Yes
`indexPattern`	string	An Index Pattern is an index name or pattern like `my-index-*`. If an index matches the pattern, the associated policy will attach to the index.	Yes
`minIndexAge`	Time	Amount of time until a managed index is deleted. Default is seven days (`7d`).	No
`rollover`	Rollover	Index rollover settings.	No

OpenSearch Plugins

Field	Type	Description	Required
`enabled`	bool	To enable or disable the OpenSearch plug-ins.	Yes
`installList`	string list	List of OpenSearch plug-ins to be installed in OpenSearch.	No

OpenSearch Dashboards Plugins

Field	Type	Description	Required
`enabled`	bool	To enable or disable the OpenSearch Dashboards plug-ins.	Yes
`installList`	string list	List of OpenSearch Dashboards plug-ins to be installed in the OpenSearch Dashboards.	No

OpenSearch Install Args

To configure OpenSearch, instead of using install args, Oracle recommends that you use OpenSearch Node Groups.

Name	Type	ValueType	Description	Required
`nodes.master.replicas`	NameValue	string	The number of master node replicas.	No
`nodes.master.requests.memory`	NameValue	string	The master node memory request amount expressed as a Quantity.	No
`nodes.master.requests.storage`	NameValue	string	The master storage request amount expressed as a Quantity.	No
`nodes.ingest.replicas`	NameValue	string	The number of ingest node replicas.	No
`nodes.ingest.requests.memory`	NameValue	string	The ingest node memory request amount expressed as a Quantity.	No
`nodes.data.replicas`	NameValue	string	The number of data node replicas.	No
`nodes.data.requests.memory`	NameValue	string	The data node memory request amount expressed as a Quantity.	No
`nodes.data.requests.storage`	NameValue	string	The data storage request amount expressed as a Quantity.	No

OpenSearch Index Management Rollover

Field	Type	Description	Required
`minIndexAge`	Time	Amount of time until a managed index is rolled over. Default is 1 day (`1d`).	No
`minSize`	Bytes	The size at which a managed index is rolled over.	No
`minDocCount`	uint32	Amount of documents in a managed index that triggers a rollover.	No

OpenSearch Dashboards Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then OpenSearch Dashboards will be installed.	No
`plugins`	OpenSearchDashboardsPlugins	OpenSearch Dashboards plug-ins to be installed in the OpenSearch Dashboards.	No

Prometheus Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Prometheus will be installed. Defaults to `true`. This is a legacy setting; the preferred way to configure Prometheus is using the prometheusOperator component.	No

Grafana Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Grafana will be installed.	No
`replicas`	integer	The number of pods to replicate. The default is `1`.	No
`database`	DatabaseInfo	The information to configure a connection to an external Grafana database.	No

Grafana Database Info

Field	Type	Description	Required
`host`	string	The host of the database.	No
`name`	string	The name of the database.	No

Kiali Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Kiali will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Prometheus Operator Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then the Prometheus Operator will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Prometheus Adapter Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then the Prometheus Adapter will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Kube State Metrics Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then kube-state-metrics will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Overrides

Field	Type	Description	Required
`configMapRef`	ConfigMapKeySelector	Selector for ConfigMap containing override data.	No
`secretRef`	SecretKeySelector	Selector for Secret containing override data.	No
`values`	JSON	Configure overrides using inline YAML.	No

Velero Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Velero will be installed.	No

Rancher Backup Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then rancherBackup will be installed. rancherBackup is dependant on Rancher being installed.	No

Docs: Verrazzano v1beta1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  environmentName: env
  profile: prod
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: emailAddress@example.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: dnsZoneCompartmentOcid
        dnsZoneOCID: dnsZoneOcid
        dnsZoneName: my.dns.zone.name
    ingress:
      type: LoadBalancer

VerrazzanoSpec

Field	Type	Description	Required
`environmentName`	string	Name of the installation. This name is part of the endpoint access URLs that are generated. The default value is `default`.	No
`profile`	string	The installation profile to select. Valid values are `prod` (production), `dev` (development), and `managed-cluster`. The default is `prod`.	No
`version`	string	The version to install. Valid versions can be found here. Defaults to the current version supported by the Verrazzano platform operator.	No
`components`	Components	The Verrazzano components.	No
`defaultVolumeSource`	VolumeSource	Defines the type of volume to be used for persistence for all components unless overridden, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of an existing `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.	No
`volumeClaimSpecTemplates`	VolumeClaimSpecTemplate	Defines a named set of PVC configurations that can be referenced from components to configure persistent volumes.	No

VolumeClaimSpecTemplate

Field	Type	Description	Required
`metadata`	ObjectMeta	Metadata about the PersistentVolumeClaimSpec template.	No
`spec`	PersistentVolumeClaimSpec	A `PersistentVolumeClaimSpec` template that can be referenced by a Component to override its default storage settings for a profile. At present, only a subset of the `resources.requests` object are honored depending on the component.	No

Components

Field	Type	Description	Required
`authProxy`	AuthProxyComponent	The AuthProxy component configuration.	No
`certManager`	CertManagerComponent	The cert-manager component configuration.	No
`dns`	DNSComponent	The DNS component configuration.	No
`ingressNGINX`	IngressComponent	The ingress component configuration.	No
`istio`	IstioComponent	The Istio component configuration.	No
`fluentd`	FluentdComponent	The Fluentd component configuration.	No
`jaegerOperator`	JaegerOperatorComponent	The Jaeger Operator component configuration.	No
`keycloak`	KeycloakComponent	The Keycloak component configuration.	No
`opensearch`	OpenSearchComponent	The OpenSearch component configuration.	No
`prometheus`	PrometheusComponent	The Prometheus component configuration.	No
`opensearchDashboards`	OpenSearchDashboardsComponent	The OpenSearch Dashboards component configuration.	No
`grafana`	GrafanaComponent	The Grafana component configuration.	No
`kiali`	KialiComponent	The Kiali component configuration.	No
`prometheusOperator`	PrometheusOperatorComponent	The Prometheus Operator component configuration.	No
`prometheusAdapter`	PrometheusAdapterComponent	The Prometheus Adapter component configuration.	No
`kubeStateMetrics`	KubeStateMetricsComponent	The kube-state-metrics component configuration.	No
`velero`	VeleroComponent	The Velero component configuration.	No
`rancherBackup`	RancherBackupComponent	The rancherBackup component configuration.	No

AuthProxy Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then AuthProxy will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

CertManager Component

Field	Type	Description	Required
`certificate`	Certificate	The certificate configuration.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Certificate

Field	Type	Description	Required
`acme`	Acme	The ACME configuration. Either `acme` or `ca` must be specified.	No
`ca`	CertificateAuthority	The certificate authority configuration. Either `acme` or `ca` must be specified.	No

Acme

Field	Type	Description	Required
`provider`	string	Name of the Acme provider.	Yes
`emailAddress`	string	Email address of the user.	Yes

CertificateAuthority

Field	Type	Description	Required
`secretName`	string	The secret name.	Yes
`clusterResourceNamespace`	string	The secrete namespace.	Yes

DNS Component

Field	Type	Description	Required
`wildcard`	DNS-Wilcard	Wildcard DNS configuration. This is the default with a domain of `nip.io`.	No
`oci`	DNS-OCI	Oracle Cloud Infrastructure DNS configuration.	No
`external`	DNS-External	External DNS configuration.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

DNS Wildcard

Field	Type	Description	Required
`domain`	string	The type of wildcard DNS domain. For example, `nip.io`, `sslip.io`, and such.	Yes

DNS Oracle Cloud Infrastructure

Field	Type	Description	Required
`ociConfigSecret`	string	Name of the Oracle Cloud Infrastructure configuration secret. Generate a secret based on the Oracle Cloud Infrastructure configuration profile you want to use. You can specify a profile other than DEFAULT and specify the secret name. See instructions by running `./install/create_oci_config_secret.sh`.	Yes
`dnsZoneCompartmentOCID`	string	The Oracle Cloud Infrastructure DNS compartment OCID.	Yes
`dnsZoneOCID`	string	The Oracle Cloud Infrastructure DNS zone OCID.	Yes
`dnsZoneName`	string	Name of Oracle Cloud Infrastructure DNS zone.	Yes
`dnsScope`	string	Scope of the Oracle Cloud Infrastructure DNS zone (`PRIVATE`, `GLOBAL`). If not specified, then defaults to `GLOBAL`.	No

DNS External

Field	Type	Description	Required
`suffix`	string	The suffix for DNS names.	Yes

Ingress NGINX Component

Field	Type	Description	Required
`type`	string	The ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`. If the ingress type is `NodePort`, a valid and accessible IP address must be specified using the `controller.service.externalIPs` key in ingressNGINX.overrides. For sample usage, see External Load Balancers.	No
`ports`	PortConfig list	The list port configurations used by the ingress.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Port Config

Field	Type	Description	Required
`name`	string	The port name.	No
`port`	string	The port value.	Yes
`targetPort`	string	The target port value. The default is same as the port value.	Yes
`protocol`	string	The protocol used by the port. `TCP` is the default.	No
`nodePort`	string	The `nodePort` value.	No

Name Value

Field	Type	Description	Required
`name`	string	The name of a Helm override for a Verrazzano component chart, specified with a `set` flag on the Helm command line, for example, `helm install --set name=value`. For more information about chart overrides, see Customize Ingress.	Yes
`value`	string	The value of a Helm override for a Verrazzano component chart, specified with a `set` flag on the Helm command line, for example, `helm install --set name=value`. Either `value` or `valueList` must be specified. For more information about chart overrides, see Customize Ingress.	No
`valueList`	string list	The list of Helm override values for a Verrazzano component, each specified with a `set` flag on the Helm command line, for example, `helm install --set name[0]=<first element of valueList> —set name[1]=<second element of valueList>`. Either `value` or `valueList` must be specified. For more information about chart overrides, see Customize Ingress.	No
`setString`	Boolean	Specifies if the argument requires the Helm `--set-string` command-line flag to override a chart value, for example, `helm install --set-string name=value`.	No

Istio Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Istio will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for default IstioOperator. Lower Overrides have precedence over the ones above them. You can find all possible values here. Passing through an invalid IstioOperator resource will result in an error.	No

Fluentd Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Fluentd will be installed.	No
`extraVolumeMounts`	ExtraVolumeMount list	A list of host path volume mounts in addition to `/var/log` into the Fluentd DaemonSet. The Fluentd component collects log files in the `/var/log/containers` directory of Kubernetes worker nodes. The `/var/log/containers` directory may contain symbolic links to files located outside the `/var/log` directory. If the host path directory containing the log files is located outside of `/var/log`, the Fluentd DaemonSet must have the volume mount of that directory to collect the logs.	No
`opensearchURL`	string	The target OpenSearch URLs. Specify this option in this format. The default `http://vmi-system-es-ingest-oidc:8775` is the VMI OpenSearch URL.	No
`opensearchSecret`	string	The secret containing the credentials for connecting to OpenSearch. This secret needs to be created in the `verrazzano-install` namespace prior to creating the Verrazzano custom resource. Specify the OpenSearch login credentials in the `username` and `password` fields in this secret. Specify the CA for verifying the OpenSearch certificate in the `ca-bundle` field, if applicable. The default `verrazzano` is the secret for connecting to the VMI OpenSearch.	No
`oci`	OCILoggingConfiguration	The Oracle Cloud Infrastructure Logging configuration.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Jaeger Operator Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Jaeger Operator will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Extra Volume Mount

Field	Type	Description	Required
`source`	string	The source host path.	Yes
`destination`	string	The destination path on the Fluentd Container, defaults to the `source` host path.	No
`readOnly`	Boolean	Specifies if the volume mount is read-only, defaults to `true`.	No

Oracle Cloud Infrastructure Logging Configuration

Field	Type	Description	Required
`systemLogId`	string	The OCID of the Oracle Cloud Infrastructure Log that will collect system logs.	Yes
`defaultAppLogId`	string	The OCID of the Oracle Cloud Infrastructure Log that will collect application logs.	Yes
`apiSecret`	string	The name of the secret containing the Oracle Cloud Infrastructure API configuration and private key.	No

Keycloak Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Keycloak will be installed.	No
`mysql`	MySQLComponent	Contains the MySQL component configuration needed for Keycloak.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

MySQL Component

Field	Type	Description	Required
`volumeSource`	VolumeSource	Defines the type of volume to be used for persistence for Keycloak/MySQL, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of a `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

OpenSearch Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then OpenSearch will be installed.	No
`policies`	Policy list	A list of Index State Management policies to enable on OpenSearch.	No
`plugins`	OpenSearchPlugins	OpenSearch plug-ins to be installed in OpenSearch.	No
`nodes`	Node list	A list of OpenSearch node groups. For sample usage, see Customize OpenSearch.	No

OpenSearch Node Groups

Field	Type	Description	Required
`name`	string	Name of the node group.	Yes
`replicas`	integer	Node group replica count.	No
`roles`	list	Role(s) that nodes in the group will assume. May be `master`, `data`, and/or `ingest`.	Yes
`storage`	Storage	Storage settings for the node group.	No
`resources`	Resources	Kubernetes container resources for nodes in the node group.	No

OpenSearch Node Group Storage

Field	Type	Description	Required
`size`	string	Node group storage size expressed as a Quantity.	Yes

OpenSearch Index Management Policies

Field	Type	Description	Required
`policyName`	string	Name of the Index State Management policy.	Yes
`indexPattern`	string	An Index Pattern is an index name or pattern like `my-index-*`. If an index matches the pattern, the associated policy will attach to the index.	Yes
`minIndexAge`	Time	Amount of time until a managed index is deleted. Default is seven days (`7d`).	No
`rollover`	Rollover	Index rollover settings.	No

OpenSearch Index Management Rollover

Field	Type	Description	Required
`minIndexAge`	Time	Amount of time until a managed index is rolled over. Default is 1 day (`1d`).	No
`minSize`	Bytes	The size at which a managed index is rolled over.	No
`minDocCount`	uint32	Amount of documents in a managed index that triggers a rollover.	No

OpenSearch Dashboards Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then OpenSearch Dashboards will be installed.	No
`plugins`	OpenSearchDashboardsPlugins	OpenSearch Dashboards plug-ins to be installed in OpenSearch Dashboards.	No

OpenSearch Plugins

Field	Type	Description	Required
`enabled`	bool	To enable or disable the OpenSearch plug-ins.	Yes
`installList`	string list	List of OpenSearch plug-ins to be installed in OpenSearch.	No

OpenSearch Dashboards Plugins

Field	Type	Description	Required
`enabled`	bool	To enable or disable the OpenSearch Dashboards plug-ins.	Yes
`installList`	string list	List of OpenSearch Dashboards plug-ins to be installed in the OpenSearch Dashboards.	No

Prometheus Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Prometheus will be installed. Defaults to `true`. This is a legacy setting; the preferred way to configure Prometheus is using the prometheusOperator component.	No

Grafana Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Grafana will be installed.	No
`replicas`	integer	The number of pods to replicate. The default is `1`.	No
`database`	DatabaseInfo	The information to configure a connection to an external Grafana database.	No

Grafana Database Info

Field	Type	Description	Required
`host`	string	The host of the database.	No
`name`	string	The name of the database.	No

Kiali Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Kiali will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Prometheus Operator Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then the Prometheus Operator will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Prometheus Adapter Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then the Prometheus Adapter will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Kube State Metrics Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then kube-state-metrics will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Overrides

Field	Type	Description	Required
`configMapRef`	ConfigMapKeySelector	Selector for ConfigMap containing override data.	No
`secretRef`	SecretKeySelector	Selector for Secret containing override data.	No
`values`	JSON	Configure overrides using inline YAML.	No

Velero Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Velero will be installed.	No

Rancher Backup Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then rancherBackup will be installed. rancherBackup is dependant on Rancher being installed.	No

Docs: Verrazzano Workloads

Mon, 01 Jan 0001 00:00:00 +0000

VerrazzanoCoherenceWorkload

The VerrazzanoCoherenceWorkload custom resource contains the configuration information for a Coherence workload within Verrazzano. Here is a sample component that specifies a VerrazzanoCoherenceWorkload. To deploy an example application that demonstrates this workload type, see Sock Shop.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: carts
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: carts-coh
        spec:
          cluster: SockShop
          role: Carts
          replicas: 1
          image: ghcr.io/helidon-sockshop/carts-coherence:2.2.0
          imagePullPolicy: Always
          application:
            type: helidon
          jvm:
            args:
              - "-Dcoherence.k8s.operator.health.wait.dcs=false"
              - "-Dcoherence.metrics.legacy.names=false"
            memory:
              heapSize: 2g
          coherence:
            logLevel: 9
          ports:
            - name: http
              port: 7001
              service:
                name: carts
                port: 80
              serviceMonitor:
                enabled: true
            - name: metrics
              port: 7001
              serviceMonitor:
                enabled: true

VerrazzanoCoherenceWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoCoherenceWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoCoherenceWorkloadSpec	The desired state of a Verrazzano Coherence workload.	Yes

VerrazzanoCoherenceWorkloadSpec

VerrazzanoCoherenceWorkloadSpec specifies the desired state of a Verrazzano Coherence workload.

Field	Type	Description	Required
`template`	RawExtension	The metadata and spec for the underlying Coherence resource.	Yes

VerrazzanoHelidonWorkload

The VerrazzanoHelidonWorkload custom resource contains the configuration information for a Helidon workload within Verrazzano. Here is a sample component that specifies a VerrazzanoHelidonWorkload. To deploy an example application that demonstrates this workload type, see Hello World Helidon.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.10-3-20201016220428-56fb4d4"
              ports:
                - containerPort: 8080
                  name: http

VerrazzanoHelidonWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoHelidonWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoHelidonWorkloadSpec	The desired state of a Verrazzano Helidon workload.	Yes

VerrazzanoHelidonWorkloadSpec

VerrazzanoHelidonWorkloadSpec specifies the desired state of a Verrazzano Helidon workload.

Field	Type	Description	Required
`deploymentTemplate`	DeploymentTemplate	The embedded deployment.	Yes

DeploymentTemplate

DeploymentTemplate specifies the metadata and pod spec of the underlying deployment.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`strategy`	DeploymentStrategy	The replacement strategy of the underlying deployment.	No
`podSpec`	PodSpec	The pod spec of the underlying deployment.	Yes

VerrazzanoWebLogicWorkload

The VerrazzanoWebLogicWorkload custom resource contains the configuration information for a WebLogic Domain workload within Verrazzano. Here is a sample component that specifies a VerrazzanoWebLogicWorkload. To deploy an example application that demonstrates this workload type, see the ToDo List Lift-and-Shift application.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: todo-domain
  namespace: todo-list
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoWebLogicWorkload
    spec:
      template:
        metadata:
          name: todo-domain
          namespace: todo-list
        spec:
          domainUID: tododomain
          domainHome: /u01/domains/tododomain
          image: container-registry.oracle.com/verrazzano/example-todo:0.8.0
          imagePullSecrets:
            - name: tododomain-repo-credentials
          domainHomeSourceType: "FromModel"
          includeServerOutInPodLog: true
          replicas: 1
          webLogicCredentialsSecret:
            name: tododomain-weblogic-credentials
          configuration:
            introspectorJobActiveDeadlineSeconds: 900
            model:
              configMap: tododomain-jdbc-config
              domainType: WLS
              modelHome: /u01/wdt/models
              runtimeEncryptionSecret: tododomain-runtime-encrypt-secret
            secrets:
              - tododomain-jdbc-tododb
          serverPod:
            env:
              - name: JAVA_OPTIONS
                value: "-Dweblogic.StdoutDebugEnabled=false"
              - name: USER_MEM_ARGS
                value: "-Djava.security.egd=file:/dev/./urandom -Xms64m -Xmx256m "
              - name: WL_HOME
                value: /u01/oracle/wlserver
              - name: MW_HOME
                value: /u01/oracle

VerrazzanoWebLogicWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoWebLogicWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoWebLogicWorkloadSpec	The desired state of a Verrazzano WebLogic workload.	Yes

VerrazzanoWebLogicWorkloadSpec

VerrazzanoWebLogicWorkloadSpec specifies the desired state of a Verrazzano WebLogic workload.

Field	Type	Description	Required
`template`	RawExtension	The metadata and spec for the underlying WebLogic Domain resource.	Yes

Docs: VerrazzanoManagedCluster

Mon, 01 Jan 0001 00:00:00 +0000

The VerrazzanoManagedCluster custom resource is used to register a managed cluster with an admin cluster. Here is a sample VerrazzanoManagedCluster that registers the cluster named managed1. To deploy an example application that demonstrates a VerrazzanoManagedCluster, see Multicluster Hello World Helidon.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Managed Cluster 1"
  caSecret: ca-secret-managed1

VerrazzanoManagedCluster

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoManagedCluster	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	VerrazzanoManagedClusterSpec	The managed cluster specification.	Yes
`status`	VerrazzanoManagedClusterStatus	The runtime status this resource.	No

VerrazzanoManagedClusterSpec

VerrazzanoManagedClusterSpec specifies a managed cluster to associate with an admin cluster.

Field	Type	Description	Required
`description`	string	The description of the managed cluster.	No
`caSecret`	string	The name of a Secret that contains the CA certificate of the managed cluster. This is used to configure the admin cluster to scrape metrics from the Prometheus endpoint on the managed cluster. See the pre-registration instructions for how to create this Secret.	Yes
`serviceAccount`	string	The name of the ServiceAccount that was generated for the managed cluster. This field is managed by a Verrazzano Kubernetes operator.	No
`managedClusterManifestSecret`	string	The name of the Secret containing generated YAML manifest file to be applied by the user to the managed cluster. This field is managed by a Verrazzano Kubernetes operator.	No

VerrazzanoManagedClusterStatus

Field	Type	Description	Required
`conditions`	Condition array	The current state of this resource.	No
`lastAgentConnectTime`	string	The last time the agent from this managed cluster connected to the admin cluster.	No
`apiUrl`	string	The Verrazzano API server URL for the managed cluster.	No

Condition

Condition describes current state of this resource.

Field	Type	Description	Required
`type`	string	The condition of the multicluster resource which can be checked with a `kubectl wait` command. Condition values are case-sensitive and formatted as follows: `Ready`: the VerrazzanoManagedCluster is ready to be used and all resources needed have been generated.	Yes
`status`	ConditionStatus	An instance of the type ConditionStatus that is defined in types.go.	Yes
`lastTransitionTime`	string	The last time the condition transitioned from one status to another.	No
`message`	string	A message with details about the last transition.	No

Docs: VerrazzanoProject

Mon, 01 Jan 0001 00:00:00 +0000

The VerrazzanoProject custom resource is used to create the application namespaces and their associated security settings on one or more clusters. The namespaces are always created on the admin cluster. Here is a sample VerrazzanoProject that specifies a namespace to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoProject
metadata:
  name: hello-helidon
  namespace: verrazzano-mc
spec:
  template:
    namespaces:
      - metadata:
          name: hello-helidon
  placement:
    clusters:
      - name: managed1

VerrazzanoProject

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoProject	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	VerrazzanoProjectSpec	The project specification.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

VerrazzanoProjectSpec

VerrazzanoProjectSpec specifies the namespaces to create and on which clusters to create them.

Field	Type	Description	Required
`template`	ProjectTemplate	The project template.	Yes
`placement`	Placement	Clusters on which the namespaces are to be created.	Yes

ProjectTemplate

ProjectTemplate contains the list of namespaces to create and the optional security configuration for each namespace.

Field	Type	Description	Required
`namespaces`	NamespaceTemplate array	The list of application namespaces to create for this project.	Yes
`security`	SecuritySpec	The project security configuration.	No
`networkPolicies`	NetworkPolicyTemplate array	The network policies applied to namespaces in the project.	No

NamespaceTemplate

NamespaceTemplate contains the metadata and specification of a Kubernetes namespace.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	NamespaceSpec	An instance of the `struct` NamespaceSpec defined in types.go.	No

SecuritySpec

SecuritySpec defines the security configuration for a project.

Field	Type	Description	Required
`projectAdminSubjects`	Subject	The subject to bind to the `verrazzano-project-admin` role. Encoded as an instance of the `struct` Subject defined in types.go.	No
`projectMonitorSubjects`	Subject	The subject to bind to the `verrazzano-project-monitoring` role. Encoded as an instance of the `struct` Subject defined in types.go.	No

NetworkPolicyTemplate

NetworkPolicyTemplate contains the metadata and specification of the underlying NetworkPolicy.

NOTE

To add an application NetworkPolicy, see NetworkPolicies for applications.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	NetworkPolicySpec	An instance of the `struct` NetworkPolicySpec defined in types.go.	No

Docs: DNS

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports three DNS choices for Verrazzano services and applications:

Free wildcard DNS services (nip.io and sslip.io)
Oracle Cloud Infrastructure DNS managed by Verrazzano
Custom (user-managed) DNS

How Verrazzano constructs a DNS domain

Regardless of which DNS management you use, the value in the spec.environmentName field in your installation will be prepended to the configured domain in the spec.components.dns section of the custom resource, to form the full DNS domain name used to access Verrazzano endpoints.

For example, if spec.environmentName is set to sales and the domain is configured in spec.components.dns as us.example.com, Verrazzano will create sales.us.example.com as the DNS domain for the installation.

Verrazzano can be configured to use either the nip.io or sslip.io free wildcard DNS services. When queried with a host name with an embedded IP address, wildcard DNS services return that IP address.

For example, using the nip.io service, the following DNS names all map to the IP address 10.0.0.1:

10.0.0.1.nip.io
app.10.0.0.1.nip.io
customer1.app.10.0.0.1.nip.io

To configure Verrazzano to use one of these services, set the spec.wildcard.domain field in the Verrazzano custom resource to either nip.io or sslip.io; the default is nip.io.

For example, the following configuration uses sslip.io, instead of nip.io, for wildcard DNS with a dev installation profile.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    dns:
      wildcard:
        domain: sslip.io

Verrazzano can directly manage records in Oracle Oracle Cloud Infrastructure DNS when configured to use the spec.components.dns.oci field. This is achieved through the External DNS Service, which is a component that is conditionally installed when Oracle Cloud Infrastructure DNS is configured for DNS management in Verrazzano.

Prerequisites

The following prerequisites must be met before using Oracle Cloud Infrastructure DNS with Verrazzano:

You must have control of a DNS domain.
You must have an Oracle Cloud Infrastructure DNS Service Zone that is configured to manage records for that domain. Verrazzano also supports the use of both GLOBAL and PRIVATE Oracle Cloud Infrastructure DNS zones.

A DNS Service Zone is a distinct portion of a domain namespace. You must ensure that the zone is appropriately associated with a parent domain. For example, an appropriate zone name for parent domain example.com is us.example.com.

To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure CLI:
```
$ oci dns zone create \
    -c <compartment ocid> \
    --name <zone-name-prefix>.example.com \
    --zone-type PRIMARY
```
To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure Console, see Managing DNS Service Zones.
You must have a valid Oracle Cloud Infrastructure API signing key that can be used to communicate with Oracle Cloud Infrastructure DNS in your tenancy.

For example, you can create an API signing key using the Oracle Cloud Infrastructure CLI.
```
  $ oci setup keys --key-name myapikey
  Enter a passphrase for your private key (empty for no passphrase):
  Public key written to: /Users/jdoe/.oci/myapikey_public.pem
  Private key written to: /Users/jdoe/.oci/myapikey.pem
  Public key fingerprint: 39:08:44:69:9f:f5:73:86:7a:46:d8:ad:34:4f:95:29
```
If you haven’t already uploaded your API signing public key through the console, follow the instructions in this section, How to upload the public key.

After the key pair has been created, you must upload the public key to your account in your Oracle Cloud Infrastructure tenancy. For details, see the Oracle Cloud Infrastructure documentation, Required Keys and OCIDs.

Create an Oracle Cloud Infrastructure API secret in the target cluster

To communicate with Oracle Cloud Infrastructure DNS to manage DNS records, Verrazzano needs to be made aware of the necessary API credentials.
A generic Kubernetes secret must be created in the cluster’s verrazzano-install namespace with the required credentials. That secret must then be referenced by the custom resource that is used to install Verrazzano.

After you have an Oracle Cloud Infrastructure API key ready for use, create a YAML file, oci.yaml, with the API credentials in the form:

auth:
  region: <oci-region>
  tenancy: <oci-tenancy-ocid>
  user: <oci-user-ocid>
  key: |
    <oci-api-private-key-file-contents>
  fingerprint: <oci-api-private-key-fingerprint>

This information typically can be found in your Oracle Cloud Infrastructure CLI config file or in the Oracle Cloud Infrastructure Console. The <oci-api-private-key-file-contents> contents are the PEM-encoded contents of the key_file value within the Oracle Cloud Infrastructure CLI configuration profile.

For example, your oci.yaml file will look similar to the following:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1.....
  user: ocid1.user.oc1.....
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:d3:4c:gh:fd:9e:27:g8:b9:0d:9f:00:22:33:c3:gg

Verrazzano also supports the use of instance principals to communicate with Oracle Cloud Infrastructure in order to create or update Oracle Cloud Infrastructure DNS records. Instance principals require some prerequisites that can be found here.

When using instance principals, your oci.yaml file will look as follows:

auth:
  authtype: instance_principal

Then, you can create a generic Kubernetes secret in the cluster’s verrazzano-install namespace using kubectl.

$ kubectl create secret generic -n verrazzano-install <secret-name> --from-file=<path-to-oci-yaml-file>

For example, to create a secret named oci from a file oci.yaml, do the following:

$ kubectl create secret generic -n verrazzano-install oci --from-file=oci.yaml

This secret will later be referenced from the Verrazzano custom resource used during installation.

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Verrazzano also provides a helper script to create the necessary Kubernetes secret based on your Oracle Cloud Infrastructure CLI configuration file, assuming that you have the Oracle Cloud Infrastructure CLI installed and a valid Oracle Cloud Infrastructure CLI profile with the required API key information. The script create_oci_config_secret.sh reads your Oracle Cloud Infrastructure CLI configuration file to create the secret.

First, download the create_oci_config_secret.sh script.

$ curl \
    -o ./create_oci_config_secret.sh \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/platform-operator/scripts/install/create_oci_config_secret.sh

Next, set your KUBECONFIG environment variable to point to your cluster and run create_oci_config_secret.sh -h to display the script options.

$ chmod +x create_oci_config_secret.sh
$ export KUBECONFIG=<kubeconfig-file>
$ ./create_oci_config_secret.sh  -h
usage: ./create_oci_config_secret.sh [-o oci_config_file] [-s config_file_section]
  -o oci_config_file         The full path to the Oracle Cloud Infrastructure configuration file (default ~/.oci/config)
  -s config_file_section     The properties section within the Oracle Cloud Infrastructure configuration file.  Default is DEFAULT
  -k secret_name             The secret name containing the Oracle Cloud Infrastructure configuration.  Default is oci
  -c context_name            The kubectl context to use
  -a auth_type               The auth_type to be used to access Oracle Cloud Infrastructure. Valid values are user_principal/instance_principal. Default is user_principal.
  -h                         Help

For example, to have the script create the YAML file using your [DEFAULT] Oracle Cloud Infrastructure CLI profile and then create a Kubernetes secret named oci, you can run the script with no arguments, as follows:

$ ./create_oci_config_secret.sh
secret/oci created

The following example creates a secret myoci using an Oracle Cloud Infrastructure CLI profile named [dev].

$ ./create_oci_config_secret.sh -s dev -k myoci
secret/myoci created

When using instance principals, all other parameters will be ignored automatically. The following example creates a secret myoci using Oracle Cloud Infrastructure instance principal.

$ ./create_oci_config_secret.sh -a instance_principal
secret/myoci created

Installation

After the Oracle Cloud Infrastructure API secret is created, create a Verrazzano custom resource for the installation that is configured to use Oracle Cloud Infrastructure DNS and reference the secret you created.

As a starting point, download the sample Verrazzano custom resource install-oci.yaml file for Oracle Cloud Infrastructure DNS.

$ curl \
    -o ./install-oci.yaml \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.4.8/platform-operator/config/samples/install-oci.yaml

Edit the install-oci.yaml file to provide values for the following configuration settings in the custom resource spec:

spec.environmentName
spec.components.dns.oci.ociConfigSecret
spec.components.dns.oci.dnsZoneCompartmentOCID
spec.components.dns.oci.dnsZoneOCID
spec.components.dns.oci.dnsZoneName
spec.components.dns.oci.dnsScope

The field spec.components.dns.oci.ociConfigSecret should reference the secret created earlier. For details on the Oracle Cloud Infrastructure DNS configuration settings, see spec.components.dns.oci.

For example, a custom resource for a prod installation profile using Oracle Cloud Infrastructure DNS might look as follows, yielding a domain of myenv.example.com (Oracle Cloud Infrastructure identifiers redacted):

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com

If using a private DNS zone, then the same prod installation profile using Oracle Cloud Infrastructure DNS will look as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: my-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com
        dnsScope: PRIVATE

After the custom resource is ready, apply it using kubectl apply -f <path-to-custom-resource-file>.

You can specify your own externally managed, custom DNS domain. In this scenario, you manage your own DNS domain and all DNS records in that domain.

An externally managed DNS domain is specified in the spec.components.dns.external.suffix field of the Verrazzano custom resource.

When using an externally managed DNS domain, you are responsible for:

Configuring A records for Verrazzano ingress points (load balancers)
Configuring CNAME records for host names in the domain that point to the A records, as needed

The Verrazzano installer searches the DNS zone you provide for two specific A records.
These are used to configure the cluster and should refer to external addresses of the load balancers provisioned by the user.

The A records need to be created manually.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example, if spec.environmentName is set to myenv, and spec.components.dns.external.suffix is set to example.com, the A records would need to be set up as follows:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

This example assumes that load balancers exist for ingress-mgmt on 198.51.100.10 and for ingress-verrazzano on 203.0.113.10.

For a more complete example, see the documentation for setting up Verrazzano on the Oracle Cloud Native Environment Platform.

Docs: Install Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Before you begin, read this document, Verrazzano in a multicluster environment.

Overview

To set up a multicluster Verrazzano environment, you will need two or more Kubernetes clusters. One of these clusters will the admin cluster; the others will be managed clusters.

The instructions assume an admin cluster and a single managed cluster. For each additional managed cluster, simply repeat the managed cluster instructions.

Install Verrazzano

Install Verrazzano on each Kubernetes cluster.

On one cluster, install Verrazzano using the dev or prod profile; this will be the admin cluster.
On the other cluster, install Verrazzano using the managed-cluster profile; this will be a managed cluster. The managed-cluster profile contains only the components that are required for a managed cluster.

Create the environment variables, KUBECONFIG_ADMIN, KUBECONTEXT_ADMIN, KUBECONFIG_MANAGED1, and KUBECONTEXT_MANAGED1, and point them to the kubeconfig files and contexts for the admin and managed cluster, respectively. You will use these environment variables in subsequent steps when registering the managed cluster. The following shows an example of how to set these environment variables.

$ export KUBECONFIG_ADMIN=/path/to/your/adminclusterkubeconfig
$ export KUBECONFIG_MANAGED1=/path/to/your/managedclusterkubeconfig

# Lists the contexts in each kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN config get-contexts -o=name
my-admin-cluster-context
some-other-cluster-context

$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 config get-contexts -o=name
my-managed-cluster-context
some-other-cluster2-context

# Choose the right context name for your admin and managed clusters from the output shown and set the KUBECONTEXT
# environment variables
$ export KUBECONTEXT_ADMIN=<admin-cluster-context-name>
$ export KUBECONTEXT_MANAGED1=<managed-cluster-context-name>

For detailed instructions on how to install and customize Verrazzano on a Kubernetes cluster using a specific profile, see the Installation Guide and Installation Profiles.

Register the managed cluster with the admin cluster

The following sections show you how to register the managed cluster with the admin cluster. As indicated, some of these steps are performed on the admin cluster and some on the managed cluster. The commands provided use the environment variables set previously to connect to the appropriate cluster.

Preregistration setup

Before registering the managed cluster, first you’ll need to set up the following items.

A Secret containing the managed cluster’s CA certificate. Note that the cacrt field in this secret can be empty only if the managed cluster uses a well-known CA. This CA certificate is used by the admin cluster to scrape metrics from the managed cluster, for both applications and Verrazzano components.
A ConfigMap containing the externally reachable address of the admin cluster. This will be provided to the managed cluster during registration so that it can connect to the admin cluster.

Follow these preregistration setup steps.

If needed for the admin cluster, obtain the managed cluster’s CA certificate. The admin cluster scrapes metrics from the managed cluster’s Prometheus endpoint. If the managed cluster Verrazzano installation uses self-signed certificates or LetsEncrypt staging certificates, then the admin cluster will need the managed cluster’s CA certificate to make an https connection.
- Depending on whether the Verrazzano installation on the managed cluster uses self-signed certificates, LetsEncrypt staging certificates, or certificates signed by a well-known certificate authority, choose the appropriate instructions:
- If you are unsure what type of certificates are used, use the following instructions.
  - To check if the verrazzano resource is configured to use LetsEncrypt staging certificates:
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     describe verrazzano
```
    If the output contains the following information, then LetsEncrypt staging certificates are being used.
```
Cert Manager:
  Certificate:
    Acme:
      Environment:    staging
      Provider:       letsEncrypt
```
  - To check the ca.crt field of the verrazzano-tls secret in the verrazzano-system namespace on the managed cluster:
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     -n verrazzano-system get secret verrazzano-tls -o jsonpath='{.data.ca\.crt}'
```
    If this value is empty, then your managed cluster is using certificates signed by a well-known certificate authority. Otherwise, your managed cluster is using self-signed certificates.
    
    Well-known CA
    
    In this case, no additional configuration is necessary.
    
    Self-signed certificates
    
    If the managed cluster certificates are self-signed, create a file called managed1.yaml containing the CA certificate of the managed cluster as the value of the cacrt field. In the following commands, the managed cluster’s CA certificate is saved in an environment variable called MGD_CA_CERT. Then use the --dry-run option of the kubectl command to generate the managed1.yaml file.
```
# On the managed cluster
$ export MGD_CA_CERT=$(kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     get secret verrazzano-tls \
     -n verrazzano-system \
     -o jsonpath="{.data.ca\.crt}" | base64 --decode)
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
  create secret generic "ca-secret-managed1" \
  -n verrazzano-mc \
  --from-literal=cacrt="$MGD_CA_CERT" \
  --dry-run=client \
  -o yaml > managed1.yaml
```
    Create a Secret on the admin cluster that contains the CA certificate for the managed cluster. This secret will be used for scraping metrics from the managed cluster. The managed1.yaml file that was created in the previous step provides input to this step.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
     apply -f managed1.yaml

# After the command succeeds, you may delete the managed1.yaml file
$ rm managed1.yaml
```
    LetsEncrypt staging certificates
    
    If the managed cluster certificates are LetsEncrypt staging, then create a file called managed1.yaml containing the CA certificate of the managed cluster as the value of the cacrt field. In the following commands, the managed cluster’s CA certificate is saved in an environment variable called MGD_CA_CERT. Then use the --dry-run option of the kubectl command to generate the managed1.yaml file.
```
# On the managed cluster
$ export MGD_CA_CERT=$(kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     get secret tls-ca-additional \
     -n cattle-system \
     -o jsonpath="{.data.ca-additional\.pem}" | base64 --decode)
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
  create secret generic "ca-secret-managed1" \
  -n verrazzano-mc \
  --from-literal=cacrt="$MGD_CA_CERT" \
  --dry-run=client \
  -o yaml > managed1.yaml
```
    Create a Secret on the admin cluster that contains the CA certificate for the managed cluster. This secret will be used for scraping metrics from the managed cluster. The managed1.yaml file that was created in the previous step provides input to this step.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
     apply -f managed1.yaml

# After the command succeeds, you may delete the managed1.yaml file
$ rm managed1.yaml
```
Use the following instructions to obtain the Kubernetes API server address for the admin cluster. This address must be accessible from the managed cluster.
- Most Kubernetes Clusters
- Kind Clusters
  
  Most Kubernetes Clusters
  
  For most types of Kubernetes clusters, except for Kind clusters, you can find the externally accessible API server address of the admin cluster from its kubeconfig file.
```
# View the information for the admin cluster in your kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN config view --minify

# Sample output
apiVersion: v1
kind: Config
clusters:
- cluster:
  certificate-authority-data: DATA+OMITTED
  server: https://11.22.33.44:6443
  name: my-admin-cluster
contexts:
....
....
```
  In the output of this command, you can find the URL of the admin cluster API server from the server entry. Set the value of the ADMIN_K8S_SERVER_ADDRESS variable to this URL.
```
$ export ADMIN_K8S_SERVER_ADDRESS=<the server address from the config output>
```
  Kind Clusters
  
  Kind clusters run within a Docker container. If your admin and managed clusters are Kind clusters, the API server address of the admin cluster in its kubeconfig file is usually a local address on the host machine, which will not be accessible from the managed cluster. Use the kind command to obtain the “internal” kubeconfig of the admin cluster, which will contain a server address accessible from other Kind clusters on the same machine, and therefore in the same Docker network.
```
$ kind get kubeconfig --internal --name <your-admin-cluster-name> | grep server
```
  In the output of this command, you can find the URL of the admin cluster API server from the server entry. Set the value of the ADMIN_K8S_SERVER_ADDRESS variable to this URL.
```
$ export ADMIN_K8S_SERVER_ADDRESS=<the server address from the config output>
```

On the admin cluster, create a ConfigMap that contains the externally accessible admin cluster Kubernetes server address found in the previous step.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: v1
kind: ConfigMap
metadata:
  name: verrazzano-admin-cluster
  namespace: verrazzano-mc
data:
  server: "${ADMIN_K8S_SERVER_ADDRESS}"
EOF

Registration steps

Perform the first three registration steps on the admin cluster, and the last step, on the managed cluster. The cluster against which to run the command is indicated in each code block.

On the admin cluster

To begin the registration process for a managed cluster named managed1, apply the VerrazzanoManagedCluster object on the admin cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Test VerrazzanoManagedCluster object"
  caSecret: ca-secret-managed1
EOF

Wait for the VerrazzanoManagedCluster resource to reach the Ready status. At that point, it will have generated a YAML file that must be applied on the managed cluster to complete the registration process.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    wait --for=condition=Ready \
    vmc managed1 -n verrazzano-mc
```

Export the YAML file created to register the managed cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register.yaml

On the managed cluster

Apply the registration file exported in the previous step, on the managed cluster.

# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register.yaml

# After the command succeeds, you may delete the register.yaml file
$ rm register.yaml

After this step, the managed cluster will begin connecting to the admin cluster periodically. When the managed cluster connects to the admin cluster, it will update the Status field of the VerrazzanoManagedCluster resource for this managed cluster, with the following information:

The timestamp of the most recent connection made from the managed cluster, in the lastAgentConnectTime status field.
The host address of the Prometheus instance running on the managed cluster, in the prometheusHost status field. This is then used by the admin cluster to scrape metrics from the managed cluster.
The API address of the managed cluster, in the apiUrl status field. This is used by the admin cluster’s authentication proxy to route incoming requests for managed cluster information, to the managed cluster’s authentication proxy.

Verify that managed cluster registration has completed

You can perform all the verification steps on the admin cluster.

Verify that the managed cluster can connect to the admin cluster. View the status of the VerrazzanoManagedCluster resource on the admin cluster, and check whether the lastAgentConnectTime, prometheusUrl, and apiUrl fields are populated. This may take up to two minutes after completing the registration steps.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get vmc managed1 -n verrazzano-mc -o yaml

# Sample output showing the status field
spec:
  ....
  ....
status:
  apiUrl: https://verrazzano.default.172.18.0.211.nip.io
  conditions:
  - lastTransitionTime: "2021-07-07T15:49:43Z"
    message: Ready
    status: "True"
    type: Ready
  lastAgentConnectTime: "2021-07-16T14:47:25Z"
  prometheusHost: prometheus.vmi.system.default.172.18.0.211.nip.io

Verify that the managed cluster is successfully registered with Rancher. When you perform the registration steps, Verrazzano also registers the managed cluster with Rancher. View the Rancher UI on the admin cluster. If the registration with Rancher was successful, then your cluster will be listed in Rancher’s list of clusters, and will be in Active state. You can find the Rancher UI URL for your cluster by following the instructions for Accessing Verrazzano.

Verify that managed cluster metrics are being collected

Verify that the admin cluster is collecting metrics from the managed cluster. The Prometheus output will include records that contain the name of the Verrazzano cluster (labeled as verrazzano_cluster).

You can find the Prometheus UI URL for your cluster by following the instructions for Accessing Verrazzano. Run a query for a metric (for example, node_disk_io_time_seconds_total).

Sample output of a Prometheus query

An alternative approach to using the Prometheus UI is to query metrics from the command line. Here is an example of how to obtain Prometheus metrics from the command line. Search the output of the query for responses that have the verrazzano_cluster field set to the name of the managed cluster.

# On the admin cluster
$ prometheusUrl=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.prometheusUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} "${prometheusUrl}/api/v1/query?query=node_disk_io_time_seconds_total"

Verify that managed cluster logs are being collected

Verify that the admin cluster is collecting logs from the managed cluster. The output will include records which have the name of the managed cluster in the cluster_name field.

You can find the OpenSearch Dashboards URL for your cluster by following the instructions for Accessing Verrazzano. Searching the verrazzano-system data stream for log records with the cluster_name set to the managed cluster name yields logs for the managed cluster.

Sample output of a OpenSearch Dashboards screen

An alternative approach to using the OpenSearch Dashboards is to query OpenSearch from the command line. Here is an example of how to obtain log records from the command line. Search the output of the query for responses that have the cluster_name field set to the name of the managed cluster.

# On the admin cluster
$ OS_URL=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.openSearchUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} -X POST -H 'kbn-xsrf: true' "${OS_URL}/verrazzano-system/_search?size=25"

Run applications in multicluster Verrazzano

The Verrazzano multicluster setup is now complete and you can deploy applications by following the Multicluster Hello World Helidon example application.

Use the admin cluster UI

The admin cluster serves as a central point from which to register and deploy applications to managed clusters.

In the Verrazzano UI on the admin cluster, you can view the following:

The managed clusters registered with this admin cluster.
VerrazzanoProjects located on this admin cluster or any of its registered managed clusters, or both.
Applications located on this admin cluster or any of its registered managed clusters, or both.

Docs: Keycloak and SSO

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano can be deployed to a number of different hosted and on-premises Kubernetes environments. Particularly in hosted environments, it may not be possible to choose the authentication providers configured for the Kubernetes API server, and Verrazzano may have no ability to view, manage, or authenticate users.

Verrazzano installs Keycloak to provide a common user store across all Kubernetes environments. The Verrazzano admin user can create and manage user accounts in Keycloak, and Verrazzano can authenticate and authorize Keycloak users.

Also, you can configure Keycloak to delegate authentication to an external user store, such as Active Directory or an LDAP server.

Because Keycloak is not configured as an authentication provider for the Kubernetes API, authenticating Keycloak users to Kubernetes requires the use of a proxy that impersonates Keycloak users when making Kubernetes API requests. For more information about the Verrazzano authentication proxy, see Verrazzano Proxies.

Keycloak is also used when authenticating to the Verrazzano Console and the various Verrazzano Monitoring Instance (VMI) logging and metrics consoles. The Verrazzano Console uses the OpenID Connect (OIDC) PKCE flow to authenticate users against Keycloak and obtain ID and access tokens. Authentication for VMI consoles is provided by the Verrazzano authentication proxy, which also uses PKCE to authenticate users, validates the resulting tokens, and authorizes incoming requests. For more information about the Verrazzano authentication proxy, see Verrazzano Proxies.

Docs: Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

This document describes some common problems you might encounter when using multicluster Verrazzano, and how to troubleshoot them.

If you created multicluster resources in the admin cluster, and specified a placement value in a managed cluster, then those resources will get created in that managed cluster. If they do not get created in the managed cluster, then use the following steps to troubleshoot:

Verify that the managed cluster is registered correctly and can connect to the admin cluster.
Verify that the VerrazzanoProject for the resource’s namespace, also has a placement in that managed cluster.
Check the multicluster resource’s status field on the admin cluster to know what the status of that resource is on each managed cluster to which it is targeted.

If you update the DNS of the admin cluster and notice that the managed cluster status is unavailable in the Rancher console, along with the error x509: certificate is valid for <rancher new url>, not <rancher old url> seen in the cattle-cluster-agent (Rancher Agent) logs on the managed cluster, then re-register the managed cluster, as described here.

Verify managed cluster registration and connectivity

You can verify that a managed cluster was successfully registered with an admin cluster by viewing the corresponding VerrazzanoManagedCluster (VMC) resource on the admin cluster. For example, to verify that a managed cluster named managed1 was successfully registered:

# on the admin cluster
$ kubectl get verrazzanomanagedcluster managed1 \
    -n verrazzano-mc \
    -o yaml

Partial sample output from the previous command.

  status:
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:27Z"
      message: Ready
      status: "True"
      type: Ready
    lastAgentConnectTime: "2021-06-22T21:06:04Z"
    ... other fields ...

Check the lastAgentConnectTime in the status of the VMC resource. This is the last time at which the managed cluster connected to the admin cluster. If this value is not present, then the managed cluster named managed1 never successfully connected to the admin cluster. This could be due to several reasons:

The managed cluster registration process step of applying the registration YAML on the managed cluster, was not completed. For the complete setup instructions, see here.
The managed cluster does not have network connectivity to the admin cluster. The managed cluster will attempt to connect to the admin cluster at regular intervals, and any errors will be reported in the verrazzano-application-operator pod’s log on the managed cluster. View the logs using the following command:

# on the managed cluster
$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If these logs reveal that there is a connectivity issue, check the admin cluster Kubernetes server address that you provided during registration and ensure that it is correct, and that it is reachable from the managed cluster. If it is incorrect, then you will need to repeat the managed cluster registration process described in the setup instructions here.

Verify VerrazzanoProject placement

For Verrazzano to create an application namespace in a managed cluster, that namespace must be part of a VerrazzanoProject that:

Includes that namespace.
Has a placement value that includes that managed cluster.

View the details of the project that corresponds to your application’s namespace. In the example command that follows, the project name is assumed to be myproject. All projects are expected to be created in the verrazzano-mc namespace.

# on the admin cluster
$ kubectl get verrazzanoproject myproject \
    -n verrazzano-mc \
    -o yaml

The following partial sample output is for a project that will result in the namespace mynamespace being created on the managed cluster managed1.

spec:
  placement:
    clusters:
    - name: managed1
  template:
    namespaces:
    - metadata:
        name: mynamespace
....other fields....

Check the multicluster resource status

On the admin cluster, each multicluster resource’s status field is updated with the status of the underlying resource on each managed cluster in which it is placed.

The following example command shows how to view the status of a MultiClusterApplicationConfiguration named myapp, in the namespace mynamespace, that has a placement value that includes the managed cluster managed1.

$ kubectl get multiclusterapplicationconfiguration myapp \
    -n mynamespace \
    -o yaml

The status of the underlying resource in each cluster specified in the placement is shown in the following partial sample output.

  status:
    clusters:
    - lastUpdateTime: "2021-06-22T21:05:04Z"
      message: OAM Application Configuration created
      name: managed1
      state: Succeeded
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:58Z"
      message: OAM Application Configuration created
      status: "True"
      type: DeployComplete
    state: Succeeded

The status message contains additional information on the operation’s success or failure.

Re-register the managed cluster

Perform the following steps to re-register the managed cluster with the admin cluster. The cluster against which to run the command is indicated in each code block.

On the admin cluster, export the register YAML file newly created on the admin cluster to re-register the managed cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register_new.yaml

On the managed cluster, apply the registration file exported in the previous step.

# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register_new.yaml

# After the command succeeds, you may delete the register_new.yaml file
$ rm register_new.yaml

On the admin cluster, run kubectl patch clusters.management.cattle.io to trigger redeployment of the Rancher agent on the managed cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get clusters.management.cattle.io

# Sample output
NAME      AGE
c-mzb2h   4h48m
local     4h56m

$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    patch clusters.management.cattle.io <the managed cluster name from the above output> \
    -p '{"status":{"agentImage":"dummy"}}' --type merge

# Sample output
cluster.management.cattle.io/c-mzb2h patched

Docs: Network Traffic

Mon, 01 Jan 0001 00:00:00 +0000

Network traffic refers to the data flowing across the network. In the context of this document, it is useful to think of network traffic from two perspectives: traffic based on direction and traffic related to component types, system, or applications. Traffic direction is either north-south traffic, which enters and leaves the cluster, or east-west traffic, which stays within the cluster.

First is a description of getting traffic into the cluster, then how traffic flows after it is in the cluster.

Ingress

Ingress is an overloaded term, so it needs to be understood in context. Sometimes the term means external access into the cluster, as in “ingress to the cluster.” The term also refers to the Kubernetes Ingress resource. In addition, it might be used to mean network ingress to a container in a Pod. Here, it’s used to refer to both general ingress into the cluster and the Kubernetes Ingress resource.

During installation, Verrazzano creates the necessary network resources to access both system components and applications. The following ingress and load balancers descriptions are in the context of a Verrazzano installation.

LoadBalancer Services

To reach Pods from outside a cluster, an external IP address must be exposed using a LoadBalancer or NodePort service. Verrazzano creates two LoadBalancer services, one for system component traffic and another for application traffic. The specifics of how the service gets traffic into the cluster depends on the underlying Kubernetes platform. With Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE), creating a LoadBalancer type service will result in an Oracle Cloud Infrastructure load balancer being created and configured to load balance to a set of Pods.

Ingress for system components

To provide ingress to system components, Verrazzano installs a NGINX Ingress Controller, which includes a NGINX load balancer. Verrazzano also creates Kubernetes Ingress resources to configure ingress for each system component that requires ingress. An Ingress resource is used is to specify HTTP/HTTPS routes to Kubernetes services, along with an endpoint host name and a TLS certificate. An Ingress by itself doesn’t do anything; it is just a resource. An ingress controller is needed to watch Ingress resources and reconcile them, configuring the underlying Kubernetes load balancer to handle the service routing. The NGINX Ingress Controller processes Ingress resources and configures NGINX with the ingress route information, and such.

The NGINX Ingress Controller is a LoadBalancer service, as seen here:

$ kubectl get service -n ingress-nginx

# Sample output
ingress-controller-ingress-nginx-controller           LoadBalancer

Using the OKE example, traffic entering the Oracle Cloud Infrastructure load balancer is routed to the NGINX load balancer, then routed from there to the Pods belonging to the services described in the Ingress.

Ingress for applications

Verrazzano also provides ingress into applications, but uses an Istio ingress gateway, which is an Envoy proxy, instead of NGINX. Istio has a Gateway resource that provides load balancer information, such as hosts, ports, and certificates for traffic coming into the mesh. For more information, see Istio Gateway. Just as an Ingress needs a corresponding Ingress controller, the same is true for the Gateway resource, where there is a corresponding Istio ingress gateway controller. However, unlike the Ingress, the Gateway resource doesn’t have service routing information. That is handled by the Istio VirtualService resource. The combination of Gateway and VirtualService is basically a superset of Ingress, because the combination provides more features than Ingress. In summary, the Istio ingress gateway provides ingress to the cluster using information from both the Gateway and VirtualService resources.

Because Verrazzano doesn’t create any applications during installations, there is no need to create a Gateway and VirtualService at that time. However, during installation, Verrazzano does create the Istio ingress gateway, which is a LoadBalancer service, along with the Istio egress gateway, which is a ClusterIP service.

$ kubectl get service -n istio-system

# Sample output
istio-ingressgateway   LoadBalancer

Again, referring to the OKE use case, this means that there will another Oracle Cloud Infrastructure load balancer created, routing traffic to the Istio ingress gateway Pod, for example, the Envoy proxy.

External DNS

When you install Verrazzano, you can optionally specify an external DNS for your domain. If you do that, Verrazzano will not only create the DNS records, using ExternalDNS, but also it will configure your host name in the Ingress resources. You can then use that host name to access the system components through the NGINX Ingress Controller.

System traffic

System traffic includes all traffic that enters and leaves system Pods.

North-south system traffic

North-south traffic includes all system traffic that enters or leaves a Kubernetes cluster.

Ingress

The following lists the Verrazzano system components which are accessed through the NGINX Ingress Controller from a client external to the cluster:

OpenSearch
Keycloak
OpenSearch Dashboards
Grafana
Prometheus
Rancher
Verrazzano Console
Verrazzano API

Egress

The following table shows Verrazzano system components that initiate requests to a destination outside the cluster.

Component	Destination	Description
cert-manager	Let’s Encrypt	Gets signed certificate.
ExternalDNS	External DNS	Creates and deletes DNS entries in an external DNS.
Fluentd	OpenSearch	Fluentd on the managed cluster calls OpenSearch on the admin cluster.
Prometheus	Prometheus	Prometheus on the admin cluster scrapes metrics from Prometheus on the managed cluster.
Rancher Agent	Rancher	Rancher agent on the managed cluster sends requests to Rancher on the admin cluster.
Verrazzano Authentication Proxy	Keycloak	Calls Keycloak for authentication, which includes redirects.
Verrazzano Platform Operator	Kubernetes API server	Multicluster agent on the managed cluster calls API server on the admin cluster.

East-west system traffic

The following tables show Verrazzano system components that send traffic to a destination inside the cluster, with the following exceptions:

Usage of CoreDNS: It can be assumed that any Pod in the cluster can access CoreDNS for name resolution.
Envoy to Istiod: The Envoy proxies all make requests to the Istio control plane to get dynamic configuration, and such. This includes both the gateways and the mesh sidecar proxies. That traffic is not shown.
Traffic within a component is not shown, for example, traffic between OpenSearch Pods.
Prometheus scraping traffic is shown in the second table.

Component	Destination	Description
cert-manager	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Fluentd	OpenSearch	Fluentd sends data to OpenSearch.
Grafana	Prometheus	UI for Prometheus data.
OpenSearch Dashboards	OpenSearch	UI for OpenSearch.
NGINX Ingress Controller	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Istio	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Rancher	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Authentication Proxy	Keycloak	Calls Keycloak for token authentication.
Verrazzano Authentication Proxy	VMI components	Access UIs for OpenSearch Dashboards, Grafana, and such.
Verrazzano Authentication Proxy	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Application Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Monitoring Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Platform Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Platform Operator	Rancher	Registers the managed cluster with Rancher.

Prometheus scraping traffic

This table shows Prometheus traffic for each system component scrape target.

Target	Description
cadvisor	Kubernetes metrics
OpenSearch	Envoy metrics
Grafana	Envoy metrics
Istiod	Istio control plane metrics
Istiod	Envoy metrics
Istio egress gateway	Envoy metrics
Istio ingress gateway	Envoy metrics
Keycloak	Envoy metrics
OpenSearch Dashboards	Envoy metrics
MySQL	Envoy metrics
NGINX Ingress Controller	Envoy metrics
NGINX Ingress Controller	NGINX metrics
NGINX default back end	Envoy metrics
Node exporter	Node metrics
Prometheus	Envoy metrics
Prometheus	Prometheus metrics
Verrazzano Console	Envoy metrics
Verrazzano API	Envoy metrics
WebLogic operator	Envoy metrics

Webhooks

Several of the system components are controllers and some of those have webhooks. Webhooks are called by the Kubernetes API server on a component HTTPS port to validate or mutate API payloads before they reach the API server.

The following components use webhooks:

cert-manager
Coherence Operator
Istio
Rancher
Verrazzano Application Operator
Verrazzano Platform Operator

Application traffic

Application traffic includes all traffic to and from Verrazzano applications.

North-south application traffic

After Verrazzano is installed, you can deploy applications into the Istio mesh. When doing so, you will likely need ingress into the application. As previously mentioned, this can be done with Istio using the Gateway and VirtualService resources. Verrazzano will create those resources for you when you use an IngressTrait in your ApplicationConfiguration. The Istio ingress gateway created during installation will be shared by all applications in the mesh, and the Gateway resource is bound to the Istio ingress gateway that was created during installation. This is done by the selector field in the Gateway.

   selector:
     istio: ingressgateway

Verrazzano creates a Gateway/VirtualService pair for each IngressTrait. Following is an example of those two resources created by Verrazzano.

Here is the Gateway; in this case both the host name and certificate were generated by Verrazzano.

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: Gateway
  metadata:
   ...
    name: hello-helidon-hello-helidon-gw
    namespace: hello-helidon
  ...
  spec:
    selector:
      istio: ingressgateway
    servers:
    - hosts:
      - hello-helidon-appconf.hello-helidon.1.2.3.4.nip.io
      port:
        name: HTTPS
        number: 443
        protocol: HTTPS
      tls:
        credentialName: hello-helidon-hello-helidon-appconf-cert-secret
        mode: SIMPLE

Here is the VirtualService; notice that it refers back to the Gateway and that it contains the service routing information.

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
  ...
    name: hello-helidon-ingress-rule-0-vs
    namespace: hello-helidon
  spec:
    gateways:
    - hello-helidon-hello-helidon-gw
    hosts:
    - hello-helidon-appconf.hello-helidon.1.2.3.4.nip.io
    HTTP:
    - match:
      - uri:
          prefix: /greet
      route:
      - destination:
          host: hello-helidon
          port:
            number: 8080

East-west application traffic

To manage east-west traffic, each service in the mesh should be routed using a VirtualService and an optional DestinationRule. You can still send east-west traffic without either of these resources, but you won’t get any custom routing or load balancing. Verrazzano doesn’t configure east-west traffic. Consider bobbys-front-end in the Bob’s Books example at bobs-books-comp.yaml. When deploying Bob’s Books, a VirtualService is created for bobbys-front-end, because of the IngressTrait, but there are no VirtualServices for the other services in the application. When bobbys-front-end sends requests to bobbys-helidon-stock-application, this east-west traffic still goes to bobbys-helidon-stock-application through the Envoy sidecar proxies in the source and destination Pods, but there is no VirtualService representing bobbys-helidon-stock-application where you could specify a canary deployment or custom load balancing. This is something you could configure manually, but it is not configured by Verrazzano.

Proxies

Verrazzano uses network proxies in multiple places. The two proxy products are Envoy and NGINX. The following table shows which proxies are used and in which Pod they run.

Usage	Proxy	Pod	Namespace	Description
System ingress	NGINX	`ingress-controller-ingress-nginx-controller-*`	`ingress-nginx`	Provides external access to Verrazzano system components.
Verrazzano authentication proxy	NGINX	`verrazzano-authproxy-*`	`verrazzano-system`	Verrazzano authentication proxy server for Kubernetes API and Single Sign-On (SSO).
Application ingress	Envoy	`istio-ingressgateway-*`	`istio-system`	Provides external access to Verrazzano applications.
Application egress	Envoy	`istio-egressgateway-*`	`istio-system`	Provides control of application egress traffic.
Istio mesh sidecar	Envoy	`ingress-controller-ingress-nginx-controller-*`	`ingress-nginx`	NGINX Ingress Controller in the Istio mesh.
Istio mesh sidecar	Envoy	`ingress-controller-ingress-nginx-defaultbackend-*`	`ingress-nginx`	NGINX default backend in the Istio mesh.
Istio mesh sidecar	Envoy	`fluentd-*`	`verrazzano-system`	Fluentd in the Istio mesh.
Istio mesh sidecar	Envoy	`keycloak-*`	`keycloak`	Keycloak in the Istio mesh.
Istio mesh sidecar	Envoy	`mysql-*`	`keycloak`	MySQL used by Keycloak in the Istio mesh.
Istio mesh sidecar	Envoy	`verrazzano-api-*`	`verrazzano-system`	Verrazzano API in the Istio mesh.
Istio mesh sidecar	Envoy	`verrazzano-console-*`	`verrazzano-system`	Verrazzano Console in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-master-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-data-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-ingest-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-kibana-*`	`verrazzano-system`	OpenSearch Dashboards in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-grafana-*`	`verrazzano-system`	Grafana in the Istio mesh.
Istio mesh sidecar	Envoy	`weblogic-operator-*`	`verrazzano-system`	WebLogic Kubernetes Operator in the Istio mesh.
Istio mesh sidecar	Envoy	`prometheus-prometheus-operator-kube-p-prometheus-*`	`verrazzano-monitoring`	Prometheus in the Istio mesh.

Multicluster

Some Verrazzano components send traffic between Kubernetes clusters. Those components are the Verrazzano agent, Verrazzano authentication proxy, and Prometheus.

Multicluster egress

The following table shows Verrazzano system components that initiate requests between the admin and managed clusters. All of these requests go through the NGINX Ingress Controller on the respective destination cluster.

Source Cluster	Source Component	Destination Cluster	Destination Component	Description
Admin	Prometheus	Managed	Prometheus	Scapes metrics on managed clusters.
Admin	Verrazzano Console	Managed	Verrazzano Authentication Proxy	Admin cluster proxy sends Kubernetes API requests to managed cluster proxy.
Managed	Fluentd	Admin	OpenSearch	Fluentd sends logs to OpenSearch.
Managed	Rancher Agent	Admin	Rancher	Rancher Agent sends requests Rancher.
Managed	Verrazzano Authentication Proxy	Admin	Keycloak	Proxy sends requests to Keycloak.
Managed	Verrazzano Agent	Admin	Kubernetes API server	Agent, in the platform operator, sends requests Kubernetes API server.

Verrazzano agent

In the multicluster topology, the Verrazzano platform operator has an agent thread running on the managed cluster that sends requests to the Kubernetes API server on the admin cluster. The URL for the admin cluster Kubernetes API server is registered on the managed cluster by the user.

Verrazzano authentication proxy

In a multicluster topology, the Verrazzano authentication proxy runs on both the admin and managed clusters. On the admin cluster, the authentication proxy connects to in-cluster Keycloak, using the Keycloak Service. On the managed cluster, the authentication proxy connects to Keycloak on the admin cluster through the NGINX Ingress Controller running on the admin cluster.

For Single Sign-On (SSO), the authentication proxy also needs to send requests to Keycloak, either in-cluster or through the cluster ingress. When a request comes into the authentication proxy without an authentication header, the proxy sends a request to Keycloak through the NGINX Ingress Controller, so the request exits the cluster. Otherwise, if the authentication proxy is on the admin cluster, then the request is sent directly to Keycloak within the cluster. If the authentication proxy is on the managed cluster, then it must send requests to Keycloak on the admin cluster.

Prometheus

A single Prometheus service in the cluster, scrapes metrics from Pods in system components and applications. It also scrapes Pods in the Istio mesh using HTTPS, and outside the mesh using HTTP. In the multicluster case, Prometheus on the admin cluster, scrapes metrics from Prometheus on the managed cluster, through the NGINX Ingress Controller on the managed cluster.

Docs: Oracle Cloud Infrastructure Logging Service

Mon, 01 Jan 0001 00:00:00 +0000

The Oracle Cloud Infrastructure Logging service is a highly scalable and fully managed single view for all the logs in your tenancy. You can configure Verrazzano to send logs to Oracle Cloud Infrastructure Logging instead of OpenSearch. For general information, see Oracle Cloud Infrastructure Logging Overview.

Set up custom logs

Verrazzano can send its logs to Oracle Cloud Infrastructure custom logs. You will need to provide two Oracle Cloud Infrastructure Log identifiers in your Verrazzano installation resource: one for Verrazzano system logs and one for application logs. Follow the steps in Creating Custom Logs to create two custom logs. Do not create an agent configuration when creating a custom log, otherwise the log records will be duplicated.

Configure credentials

The Fluentd plug-in included with Verrazzano will use Oracle Cloud Infrastructure instance principal authentication by default. Optionally, you can configure Verrazzano with a user API signing key. API signing key authentication is required to send logs to Oracle Cloud Infrastructure Logging if the cluster is running outside of Oracle Cloud Infrastructure.

Instance principal authentication

Create a dynamic group that includes the compute instances in your cluster’s node pools and assign the appropriate policy, so that the dynamic group is allowed to send log entries to the custom logs you created earlier. Pay close attention to the required permissions.

If the dynamic group and policy are configured incorrectly, then Fluentd will fail to send logs to Oracle Cloud Infrastructure Logging.

User API signing key

If you do not already have an API signing key, then see Required Keys and OCIDs in the Oracle Cloud Infrastructure documentation. You need to create an Oracle Cloud Infrastructure configuration file with the credential details and then use that configuration file to create a secret.

The following requirements must be met for Fluentd Oracle Cloud Infrastructure Logging to work:

The profile name in the Oracle Cloud Infrastructure configuration file must be DEFAULT.
The key_file path in the Oracle Cloud Infrastructure configuration file must be /root/.oci/key. The actual key file does not need to be in that location, because you will be providing the actual key file location in a secret.
The user associated with the API key must have the appropriate Oracle Cloud Infrastructure Identity and Access Management (IAM) policy in place to allow the Fluentd plug-in to send logs to Oracle Cloud Infrastructure. See Details for Logging in the Oracle Cloud Infrastructure documentation for the IAM policies used by the Oracle Cloud Infrastructure Logging service.

After the Verrazzano platform operator has been installed, create an opaque secret in the verrazzano-install namespace from the Oracle Cloud Infrastructure configuration and private key files. The key for the configuration file must be config and the key for the private key file data must be key.

Here is an example kubectl command that will create the secret.

$ kubectl create secret generic oci-fluentd -n verrazzano-install \
      --from-file=config=/home/myuser/oci_config --from-file=key=/home/myuser/keys/oci_api.pem

The secret should look something like this.

apiVersion: v1
data:
  config: W0RFRkFVTFRdCnVzZXI9b2NpZDEudXN...
  key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS...
kind: Secret
metadata:
  name: oci-fluentd
  namespace: verrazzano-install
type: Opaque

For convenience, there is a helper script available here that you can point at an existing Oracle Cloud Infrastructure configuration file and it will create the secret for you. The script allows you to override the default configuration file location, profile name, and the name of the secret.

Install Verrazzano

Oracle Cloud Infrastructure Logging is enabled in your cluster when installing Verrazzano. The Verrazzano installation custom resource has fields for specifying two custom logs: one for system logs and one for application logs. Here is an example Verrazzano installation YAML file for each type of credential.

Instance principal credentials

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: vz-oci-logging
spec:
  profile: dev
  components:
    fluentd:
      enabled: true
      oci:
        systemLogId: ocid1.log.oc1.iad.system.example
        defaultAppLogId: ocid1.log.oc1.iad.app.example
    opensearch:
      enabled: false
    opensearchDashboards:
      enabled: false

User API credentials

When using user API credentials, you need to configure the name of the secret in the Verrazzano custom resource, under the Oracle Cloud Infrastructure section of the Fluentd component settings. Your YAML file should look something like this.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: vz-oci-logging
spec:
  profile: dev
  components:
    fluentd:
      enabled: true
      oci:
        systemLogId: ocid1.log.oc1.iad.system.example
        defaultAppLogId: ocid1.log.oc1.iad.app.example
        apiSecret: oci-fluentd
    opensearch:
      enabled: false
    opensearchDashboards:
      enabled: false

The apiSecret value must match the secret you created earlier when configuring the user API credentials.

Override the default log objects

You can override the Oracle Cloud Infrastructure Log object on an individual namespace. To specify a log identifier on a namespace, add an annotation named verrazzano.io/oci-log-id to the namespace. The value of the annotation is the Oracle Cloud Infrastructure Log object identifier.

Here is an example namespace.

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    verrazzano.io/oci-log-id: ocid1.log.oc1.iad.ns.app.example
  creationTimestamp: "2022-01-14T15:09:19Z"
  labels:
    istio-injection: enabled
    verrazzano-managed: "true"
  name: example
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Note that if you add and subsequently remove the annotation, then the logs will revert to the default Oracle Cloud Infrastructure Log object specified in the Verrazzano custom resource.

Search logs

To search Verrazzano logs, you can use the Oracle Cloud Infrastructure Console, Oracle Cloud Infrastructure CLI, or Oracle Cloud Infrastructure SDK.

For example, use the Oracle Cloud Infrastructure CLI to search the system logs for records emitted by the verrazzano-application-operator container:

$ oci logging-search search-logs --search-query=\
     "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \
     where \"data\".\"kubernetes.container_name\" = 'verrazzano-application-operator' | sort by datetime desc" \
     --time-start 2021-12-07 --time-end 2021-12-17

Search for all application log records in the springboot namespace:

$ oci logging-search search-logs --search-query=\
     "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \
     where \"data\".\"kubernetes.namespace_name\" = 'springboot' | sort by datetime desc" \
     --time-start 2021-12-07 --time-end 2021-12-17

For more information on searching logs, see the Logging Query Language Specification.

Troubleshooting

If you are not able to view Verrazzano logs in Oracle Cloud Infrastructure Logging, then check the Fluentd container logs in the cluster to see if there are errors.

$ kubectl logs -n verrazzano-system -l app=fluentd --tail=-1

If you see not authorized error messages, then there is likely a problem with the Oracle Cloud Infrastructure Dynamic Group or IAM policy that is preventing the Fluentd plug-in from communicating with the Oracle Cloud Infrastructure API.

To ensure the appropriate permissions are in place, review the Oracle Cloud Infrastructure Logging required permissions documentation.

Docs: Restore

Mon, 01 Jan 0001 00:00:00 +0000

Before proceeding, ensure that the backup component prerequisites are met, as indicated here. This document also assumes that a successful backup was previously made using either Velero or rancher-backup, as shown here.

Use the following component-specific instructions to restore application data:

Rancher restore

To initiate a Rancher restore, create the following example custom resource YAML file. When a Restore custom resource is created, the operator accesses the backup *.tar.gz file specified and restores the application data from that file.

apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
  name: s3-restore
spec:
  backupFilename: rancher-backup-test-1111111-2222-3333-2022-07-26T02-44-21Z.tar.gz
  storageLocation:
    s3:
      credentialSecretName: rancher-backup-creds
      credentialSecretNamespace: verrazzano-backup
      bucketName: myvz-bucket
      folder: rancher-backup
      region: us-phoenix-1
      endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com

The rancher-backup operator scales down the Rancher deployment during the restore operation and scales it back up after the restoration completes.

Resources are restored in this order:

Custom Resource Definitions (CRDs)
Cluster-scoped resources
Namespace resources

OpenSearch restore

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero, to perform a restore operation. Due to the nature of transient data handled by OpenSearch, the hook invokes OpenSearch snapshot APIs to back up and restore data streams appropriately, thereby ensuring there is no loss of data and avoids data corruption as well.

To initiate an OpenSearch restore, first delete the existing OpenSearch cluster running on the system and all related data.

Scale down Verrazzano Monitoring Operator.

$ kubectl scale deploy -n verrazzano-system verrazzano-monitoring-operator --replicas=0

Then, clean up the OpenSearch components.

# These are sample commands to demonstrate the OpenSearch restore process

$ kubectl delete sts -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete deploy -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete pvc -n verrazzano-system  -l verrazzano-component=opensearch

To perform an OpenSearch restore, you can invoke the following example Velero Restore API object.

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: verrazzano-opensearch-restore
  namespace: verrazzano-backup
spec:
  backupName: verrazzano-opensearch-backup
  includedNamespaces:
    - verrazzano-system
  labelSelector:
    matchLabels:
      verrazzano-component: opensearch
  restorePVs: false
  hooks:
    resources:
      - name: opensearch-test
        includedNamespaces:
          - verrazzano-system       
        labelSelector:
          matchLabels:            
            statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
        postHooks:
          - exec:
              container: es-master
              command:
                - /usr/share/opensearch/bin/verrazzano-backup-hook
                - -operation
                - restore
                - -velero-backup-name
                - verrazzano-opensearch-backup
              waitTimeout: 30m
              execTimeout: 30m
              onError: Fail

The preceding example will restore an OpenSearch cluster from an existing backup.

It will recreate a new OpenSearch cluster (with new indexes).
The postHook will invoke the OpenSearch APIs that restores the snapshot data.
The container on which the hook needs to be run is identified by the pod label selectors, followed by the container name. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.

NOTE: The hook needs to be a postHook because it must be applied after the Kubernetes objects are restored.

After the restore operation is processed, you can see the hook logs using the velero restore logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch restore logs

# To display the logs from the restore, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-restore-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

Docs: Uninstall

Mon, 01 Jan 0001 00:00:00 +0000

Uninstall considerations

Before uninstalling Verrazzano, you should delete your Verrazzano applications because they may not function properly after the uninstall is done.

When you uninstall Verrazzano:

All of the Verrazzano components are uninstalled
The CRDs installed by Verrazzano are not deleted
Any applications that were deployed will still exist, but they may not be functional

Perform the uninstall

You can uninstall Verrazzano using the Verrazzano CLI or with kubectl. See the following respective sections.

Uninstall Verrazzano.
```
$ vz uninstall
```

Wait for the uninstall to complete. The uninstall logs from the Verrazzano platform operator will be streamed to the command window until the uninstall has completed or until the default timeout (20m) has been reached.

The following is an example of the output:

Uninstalling Verrazzano
2022-11-22T16:31:20.377Z info Reconciling Verrazzano resource default/verrazzano, generation 2, version 1.4.2
2022-11-22T16:31:20.377Z info Deleting Verrazzano installation
2022-11-22T16:31:20.418Z info Uninstalling components
2022-11-22T16:31:20.418Z info Uninstalling Verrazzano default/verrazzano
...

To delete a Verrazzano installation, delete the Verrazzano custom resource you used to install it into your cluster.

The following example starts a deletion of a Verrazzano installation in the background and then uses the kubectl logs -f command to tail the output of the pod performing the uninstall.

Get the name of the Verrazzano custom resource.

$ MYVZ=$(kubectl  get vz -o jsonpath="{.items[0].metadata.name}")

Delete the Verrazzano custom resource. Once the delete is done, the Verrazzano uninstall will be complete.
```
$ kubectl delete verrazzano $MYVZ
```

If you want to see the uninstall logs during the deletion, you can view them from the Verrazzano platform operator with the following command:

$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
    | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'

Docs: Coherence Workload

Mon, 01 Jan 0001 00:00:00 +0000

A Verrazzano application can contain any number of Coherence component workloads, where each workload is a standalone Coherence cluster, independent from other Coherence clusters in the application.

Verrazzano uses the standard Coherence Operator to provision and manage clusters, as documented at Coherence Operator. The Coherence Operator uses a CRD, coherence.oracle.com (Coherence resource), to represent a Coherence cluster. When a Verrazzano application with Coherence is provisioned, Verrazzano configures the default logging and metrics for the Coherence cluster. Logs are sent to OpenSearch and metrics to Prometheus. You can view this telemetry data using the OpenSearch Dashboards and Grafana consoles.

OAM Component

The custom resource YAML file for the Coherence cluster is specified as a VerrazzanoCoherenceWorkload custom resource. In the following example, everything under the spec: section is standard Coherence resource YAML that you would typically use to provision a Coherence cluster. Including this Component reference in your ApplicationConfiguration will result in a new Coherence cluster being provisioned. You can have multiple clusters in the same application with no conflict.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: orders
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: orders-coh
        spec:
          cluster: SockShop
          ...

Life cycle

With Verrazzano, you manage the life cycle of applications using Component and ApplicationConfiguration resources. Typically, you would modify the Coherence cluster resource to make changes or to do lifecycle operations, like scale in and scale out. However, in the Verrazzano environment, the cluster resource is owned by the Verrazzano application operator and will be reconciled to match the Component workload resource. Therefore, you need to manage the cluster configuration by modifying the resource, either by kubectl edit or applying a new YAML file. Verrazzano will notice that the Component resource changed and will update the Coherence resource as needed.

Provisioning

When you apply the Component YAML file shown previously, Kubernetes will create a component.oam.verrazzano.io resource, but the Coherence cluster will not be created until you create the ApplicationConfiguration resource, which references the Coherence component. When the application is created, Verrazzano creates a Coherence custom resource for each cluster, which is subsequently processed by the Coherence Operator, resulting in a new cluster. After a cluster is created, the Coherence Operator will monitor the Coherence resource to reconcile the state of the cluster. You can add a new Coherence workload to a running application, or remove an existing workload, by modifying the ApplicationConfiguration resource, and adding or removing the Coherence component.

Scaling

Scaling a Coherence cluster is done by modifying the replicas field in the Component resource. Verrazzano will modify the Coherence resource replicas field and the cluster will be scaled accordingly. The following example configuration shows the replicas field that specifies the number of pods in the cluster.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: orders
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: orders-coh
        spec:
          cluster: SockShop
          replicas: 3
          ...

NOTE: A Coherence cluster provisioned with Verrazzano does not support autoscaling with a Horizontal Pod Autoscaler.

Termination

You can terminate the Coherence cluster by removing the Component from the ApplicationConfiguration or by deleting the ApplicationConfiguration resource entirely.

NOTE

Do not delete the Coherence component if the application is still using it.

Logging

When a Coherence cluster is provisioned, Verrazzano configures it to send logs to OpenSearch. This is done by injecting a Fluentd sidecar configuration into the Coherence resource. The Coherence Operator will create the pod with the Fluentd sidecar. This sidecar periodically copies the Coherence logs from /logs to stdout, enabling the Fluentd DaemonSet in the verrazzano-system namespace to send the logs to OpenSearch. Note that the Fluend sidecar running in the Coherence pod never communicates with OpenSearch or any other network endpoint.

The logs are placed in a per-namespace OpenSearch data stream named verrazzano-application-<namespace>, for example: verrazzano-application-sockshop. All logs from Coherence pods in the same namespace will go into the same data stream, even for different applications. This is standard behavior and there is no way to disable or change it.

Each log record has some Coherence and application fields, along with the log message itself. For example:

 kubernetes.labels.coherenceCluster        SockShop
 kubernetes.labels.app_oam_dev/name        sockshop-appconf
 kubernetes.labels.app_oam_dev/component   orders
 ...

Metrics

Verrazzano uses Prometheus to scrape metrics from Coherence cluster pods. Like logging, metrics scraping is also enabled during provisioning, however, the Coherence resource YAML file must have proper metrics configuration. For details, see Coherence Metrics. In summary, there are two ways to configure the Coherence metrics endpoint. Coherence has a default metrics endpoint that you can enable. If your application serves metrics from its own endpoint, such as a Helidon application, then do not use the native Coherence metrics endpoint. To see the difference, examine the socks-shop and bobs-books examples.

Bobs Books

The bobs-books example uses the default Coherence metrics endpoint, so the configuration must enable this feature, shown in the following metrics section of the roberts-coherence component in the YAML file, bobs-books-comp.yaml.

          coherence:
            metrics:
              enabled: true

Sock Shop

The sock-shop example, which is a Helidon application with embedded Coherence, explicitly specifies the metrics port 7001 and doesn’t enable Coherence metrics. Coherence metrics still will be scraped, but not at the default endpoint.

          ports:
            ...
            - name: metrics
              port: 7001
              serviceMonitor:
                enabled: true

Because sock-shop components are not using the default Coherence metrics port, you must add a MetricsTrait section to the ApplicationConfiguration for each component, specifying the metrics port as follows:

        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            metadata:
              name: carts-metrics
            spec:
              port: 7001

Prometheus configuration

Prometheus is configured using the Prometheus Operator to scrape application targets. During application deployment, Verrazzano creates or updates Service Monitors based on the MetricsTrait specified in the ApplicationConfiguration. When the application is deleted, Verrazzano removes the Service Monitors so that metrics are no longer collected for it.

Here is an example of the sock-shop Prometheus Service Monitor resource for catalog-coh in the application namespace.
Notice that services with certain labels are targeted. Prometheus Operator will find the Service Monitor and generate the scrape configuration to be used by Prometheus.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  ....
  name: catalog-coh-metrics
  namespace: sockshop
  ....
spec:
  endpoints:
  - bearerTokenSecret:
      key: ""
    port: metrics
    relabelings:
    - action: labeldrop
      regex: (endpoint|instance|job|service)
  namespaceSelector: {}
  selector:
    matchLabels:
      coherenceCluster: SockShop
      coherenceComponent: coherence-service
      coherenceDeployment: catalog-coh
      coherencePort: metrics
      coherenceRole: Catalog

Here are the labels on the corresponding catalog-coh-metrics service.

kind: Service
metadata:
  labels:
    coherenceCluster: SockShop
    coherenceComponent: coherence-service
    coherenceDeployment: catalog-coh
    coherencePort: metrics
    coherenceRole: Catalog
spec:
  ports:
  - name: metrics
    port: 9612
    protocol: TCP
    targetPort: 9612
  ....

Istio integration

Verrazzano ensures that Coherence clusters are not included in an Istio mesh, even if the namespace has the istio-injection: enabled label. This is done by adding the sidecar.istio.io/inject: "false" annotation to the Coherence resource, resulting in Coherence pods being created with that label. However, other application components in the mesh using mutual TLS authentication (mTLS) may need to communicate with Coherence. To handle this case, Verrazzano automatically creates an Istio DestinationRule to disable TLS for the Coherence port. This policy disables mTLS for port 9000, which happens to be used as a Coherence extend port for Bob’s Books.

  trafficPolicy:
    portLevelSettings:
    - port:
        number: 9000
      tls: {}
   ...

Currently, port 9000 is the only port where TLS is disabled, so you need to use this as the Coherence extend port if other components in the mesh access Coherence over the extend protocol.

Docs: External Load Balancers

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires the following load balancers at installation:

Load balancer for NGINX ingress
Load balancer for Istio ingress

By default, Verrazzano automatically creates them as Kubernetes-managed load balancers, however, you have the option to use your own external load balancers. You can choose to replace either or both load balancers.

The following is an example of using external load balancers for both management and application ingress.

Prepare the external load balancers

External load balancer for management ingress:
- This load balancer must have a listener set up on port 443 with TCP protocol.
- The backend set for this listener needs to include the Kubernetes cluster node IP addresses on a port you pick, for example, 31443.
External load balancer for application ingress:
- This load balancer must have a listener set up on port 443 with TCP protocol.
- The backend set for this listener needs to include the Kubernetes cluster node IP addresses on a port you pick, for example, 32443.

Verrazzano installation options

External load balancer for management ingress:
- Set NodePort as the ingress type in the Ingress Component.
- Set controller.service.externalIPs with the IP address for the external management load balancer in the Ingress NGINX Overrides.
- Set ports in the Ingress Component with a PortConfig that has 443 as port, 31443 as nodePort, https as targetPort, and TCP as protocol.
External load balancer for application ingress using the Istio ingress gateway overrides:
- Set service Type to NodePort.
- Set service externalIPs to the external application load balancer IP address.
- Set service ports with a https named entry, 443 as port, 32443 as nodePort, 8443 as targetPort, and TCP as protocol.

Example Custom Resource with management and application external load balancers

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: myvz
spec:
  components:
    ingressNGINX:
      overrides:
      - values:
          controller:
            service:
              externalIPs:
              - 11.22.33.44
      type: NodePort
      ports:
      - name: https
        port: 443
        nodePort: 31443
        protocol: TCP
        targetPort: https
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              ingressGateways:
                - enabled: true
                  name: istio-ingressgateway
                  k8s:
                    service:
                      type: NodePort
                      ports:
                      - name: https
                        port: 443
                        nodePort: 32443
                        protocol: TCP
                        targetPort: 8443
                      externalIPs:
                      - 11.22.33.55

Docs: Helidon Workload

Mon, 01 Jan 0001 00:00:00 +0000

Helidon is a collection of Java libraries for writing microservices. Helidon provides an open source, lightweight, fast, reactive, cloud native framework for developing Java microservices. It is available as two frameworks:

Helidon SE is a compact toolkit that embraces the latest Java SE features: reactive streams, asynchronous and functional programming, and fluent-style APIs.
Helidon MP implements and supports Eclipse MicroProfile, a baseline platform definition that leverages Java EE and Jakarta EE technologies for microservices and delivers application portability across multiple runtimes.

Helidon is designed and built with container-first philosophy.

Small footprint, low memory usage and faster startup times.
All 3rd party dependencies are stored separately to enable Docker layering.
Provides readiness, liveness and customizable health information for container schedulers like Kubernetes.

Containerized Helidon applications are generally deployed as Deployment in Kubernetes.

Verrazzano integration

Verrazzano supports application definition using Open Application Model (OAM). Verrrazzano applications are composed of components and application configurations.

Helidon applications are first class citizens in Verrazzano with specialized Helidon workload support, for example, VerrazzanoHelidonWorkload. VerrazzanoHelidonWorkload is supported as part of verrazzano-application-operator in the Verrazzano installation and no additional operator setup or installation is required. VerrazzanoHelidonWorkload also supports all the Traits and Scopes defined by Verrazzano along with core ones defined by the OAM specification.

VerrazzanoHelidonWorkload is modeled after ContainerizedWorkload, for example, it is used for long-running workloads in containers. However, VerrazzanoHelidonWorkload closely resembles and directly refers to Kubernetes Deployment schema. This enables an easy lift and shift of existing containerized Helidon applications.

The complete VerrazzanoHelidonWorkload API definition and description is available at VerrazzanoHelidonWorkload.

Verrazzano Helidon application development

With Verrazzano, you manage the life cycle of applications using Component and ApplicationConfiguration resources. A Verrazzano application can contain any number of VerrazzanoHelidonWorkload components, where each workload is a standalone containerized Helidon application, independent of any other in the application.

In the following example, everything under the spec: section is the custom resource YAML file for the containerized Helidon application, as defined by the VerrazzanoHelidonWorkload custom resource. Including this Component reference in your ApplicationConfiguration will result in a new containerized Helidon application being provisioned.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              ...
              ...

The Application Development Guide provides end-to-end instructions for developing and deploying the Verrazzano Helidon application.

For more Verrazzano Helidon application examples, see Examples.

Customizing Helidon Workload service

By default, deploying a Helidon Workload will create a service to access the Helidon pod. That service can be customized by added a serviceTemplate to the VerrazzanoHelidonWorkload spec. This supports customizing the service metadata and serviceSpec sections to do things, like add labels and annotations, customize the service ports, modify the pod selector, and such. It is not necessary to provide complete metadata and serviceSpec sections. Just add the fields you would like to customize and Verrazzano will create the rest of the fields, based the information provided in the deploymentTemplate. When customizing service ports, refer to Protocol Selection in the Istio documentation.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              ...
              ...
      serviceTemplate:
        metadata:
          name: hello-helidon-service
        serviceSpec:
          ports:
            - name: http-hello-helidon
              ...
              ...

Provisioning

When you apply the previous Component YAML file, Kubernetes will create a component.oam.verrazzano.io resource, but the containerized Helidon application will not be created until you create the ApplicationConfiguration resource, which references the VerrazzanoHelidonWorkload component. When the application is created, Verrazzano creates a Deployment and Service resource for each containerized Helidon application.

Typically, you would modify the Deployment and Service resource to make changes or to do lifecycle operations, like scale in and scale out. However, in the Verrazzano environment, the containerized Helidon application resource is owned by the verrazzano-application-operator and will be reconciled to match the component workload resource. Therefore, you need to manage the application configuration by modifying the VerrazzanoHelidonWorkload or ApplicationConfiguration resource, either by kubectl edit or applying a new YAML file. Verrazzano will notice the Component resource change and will update the Deployment and Service resource as needed.

You can add a new VerrazzanoHelidonWorkload to a running application, or remove an existing workload, by modifying the ApplicationConfiguration resource and adding or removing the VerrazzanoHelidonWorkload component.

Scaling

The recommended way to scale containerized Helidon application replicas is to specify a ManualScalerTrait with the VerrazzanoHelidonWorkload in the ApplicationConfiguration. The following example configuration shows the replicaCount field that specifies the number of replicas for the application.

...
    spec:
      components:
      - componentName: hello-helidon-component
        traits:
        - trait:                      
            apiVersion: core.oam.dev/v1alpha2
            kind: ManualScalerTrait
            spec:
              replicaCount: 2
...

Verrazzano will modify the Deployment resource replicas field and the containerized Helidon application replicas will be scaled accordingly.

NOTE

Make sure the replicas defined on the VerrazzanoHelidonWorkload component and that the replicaCount defined on the ManualScalerTrait for that component match, or else the DeploymentController in Kubernetes and OAM runtime in verrazzano-application-operator will compete to create a different number of Pods for same containerized Helidon application. To avoid confusion, we recommend that you specify replicaCount defined on the ManualScalerTrait and leave replicas undefined on VerrazzanoHelidonWorkload (as it is optional).

Logging

When a containerized Helidon application is provisioned on Verrazzano, Verrazzano will configure the default logging and send logs to OpenSearch. You can view the logs using the OpenSearch Dashboards.

The logs are placed in a per-namespace OpenSearch data stream named verrazzano-application-<namespace>, for example: verrazzano-application-hello-helidon. All logs from containerized Helidon application pods in the same namespace will go into the same data stream, even for different applications. This is standard behavior and there is no way to disable or change it.

Metrics

Verrazzano uses Prometheus to scrape metrics from containerized Helidon application pods. Like logging, metrics scraping is also enabled during provisioning. You can view metrics using the Grafana console.

Using the MetricsTrait custom resource, you can customize configuration information needed to enable metrics for an application component.

Ingress

Using the IngressTrait custom resource, you can configure traffic routing to a containerized Helidon application for an application component.

Troubleshooting

Whenever you have a problem with your Verrazzano Helidon application, there are some basic techniques you can use to troubleshoot. Troubleshooting shows you some simple things to try, as well as how to solve common problems you may encounter.

Docs: Kubernetes RBAC

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano uses Kubernetes Role-Based Access Control (RBAC) to protect Verrazzano resources.

Verrazzano includes a set of roles that can be granted to users, enabling access to Verrazzano resources managed by Kubernetes. In addition, Verrazzano creates a number of roles that grant permissions needed by various Verrazzano system components (operators and third-party components).

Verrazzano creates default role bindings during installation and for projects, at project creation or update.

NOTE

Kubernetes RBAC must be enabled in every cluster to which Verrazzano is deployed or access control will not work. RBAC is enabled by default in most Kubernetes environments.

Verrazzano user roles

The following table lists the defined Verrazzano user roles. Each is a ClusterRole intended to be granted directly to users or groups. (In some scenarios, it may be appropriate to grant a user role to a service account.)

Verrazzano Role	Binding Scope	Description
verrazzano-admin	Cluster	Manage Verrazzano system components, clusters, and projects. Install and update Verrazzano.
verrazzano-monitor	Cluster	View and monitor Verrazzano system components, clusters, and projects.
verrazzano-project-admin	Namespace	Deploy and manage applications.
verrazzano-project-monitor	Namespace	View and monitor applications.

Kubernetes user roles

Verrazzano roles do not include permissions for Kubernetes itself. Instead, it relies on the default user roles provided by Kubernetes. This allows Verrazzano to easily grant the Kubernetes access appropriate to a Verrazzano role, without having to maintain a long list of fine-grained Kubernetes permissions in the Verrazzano roles.

The following table shows the default Kubernetes roles that are granted by default for each Verrazzano role.

Verrazzano Role	Kubernetes Role	Binding Scope
verrazzano-admin	admin	Cluster
verrazzano-monitor	view	Cluster
verrazzano-project-admin	admin	Namespace
verrazzano-project-monitor	view	Namespace

Default role bindings

Verrazzano creates role bindings for the system and for projects, binding Verrazzano ClusterRoles to one or more Kubernetes Subjects. By default, each role is bound to a Keycloak group, so all Keycloak users who are members of that group will be granted the role.

Also, Verrazzano creates role bindings for the corresponding Kubernetes user roles. The Kubernetes role appropriate for a given Verrazzano role is bound to the same set of Subjects as the corresponding Verrazzano role.

The default bindings can be overridden by specifying one or more Kubernetes Subjects to which the role should be bound. Any valid Subject can be specified (user, group, or service account), but two caveats should be kept in mind:

It’s generally better to grant a role to a group, rather than a specific user, so that roles can be granted (or withdrawn) by editing a user’s group memberships, rather than deleting a role binding and creating a new one.
If you do want to grant a role directly to a specific user, then the user must be specified using its unique ID, not its user name. This is because the authentication proxy impersonates the sub (subject) field from the user’s token, which contains the ID. Keycloak user IDs are guaranteed to be unique, unlike user names.

Default system role bindings

Verrazzano creates role bindings for system users during installation. The default role bindings are listed as follows:

Role	Default Binding Subject
verrazzano-admin	group: verrazzano-admins
verrazzano-monitor	group: verrazzano-monitors

Default project role bindings

Verrazzano creates role bindings for project users at project creation or update. The default role bindings are listed as follows:

Role	Default Binding Subject
verrazzano-project-admin	group: verrazzano-project-<proj_name>-admins
verrazzano-project-monitor	group: verrazzano-project-<proj_name>-monitors

NOTE

The role bindings for project roles are created automatically, but the project-specific groups that they refer to are not automatically created. You must create those groups using the Keycloak console or API, or specify different binding subjects for the project.

Override default role bindings

You can override the default role bindings that are created for system and project roles.

Override system role bindings

To override the set of subjects that are bound to Verrazzano (and Kubernetes) roles during installation, add the Subjects to the Verrazzano CR you use to install Verrazzano, as shown in the following example:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  ...
  security:
    adminSubjects:
    - name: admin-group
      kind: Group
    monitorSubjects:
    - name: view-group
      kind: Group
  ...

You can specify multiple subjects for both admin and monitor roles. You can also specify a subject or subjects for one role, but not the other. If no subjects are specified for a role, then the default binding subjects will be used.

Override project role bindings

To override the set of subjects that are bound to Verrazzano (and Kubernetes) roles for a project, add the Subjects to the VerrazzanoProject CR for the project, as shown in the following example.

Note that the generated role bindings will be updated if you update the VerrazzanoProject CR and change the subjects specified for either role.

apiVersion: clusters.verrazzano.io/v1beta1
kind: VerrazzanoProject
metadata:
  name: my-project
spec:
  ...
  security:
    projectAdminSubjects:
    - name: my-project-admin-group
      kind: Group
    projectMonitorSubjects:
    - name: my-project-view-group
      kind: Group
  ...

As with the system role bindings, you can specify multiple subjects for both project-admin and project-monitor roles. You can also specify a subject or subjects for one role, but not the other. If no subjects are specified for a role, then the default binding subjects will be used.

Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: Application Deployment Guide

Overview

What you need

Application development

Application deployment

Verrazzano components

Verrazzano application configurations

Deploy the application

Verify the deployment

Explore the application

Access the application’s logs

Access the application’s metrics

Suppress Kiali console warnings

Remove the application

Docs: Application Deployment

Review oam-kubernetes-runtime operator status

Review verrazzano-application-operator operator status

Review oam-kubernetes-runtime operator logs

Review verrazzano-application-operator logs

Review generated workload resources

Review generated Trait resources

Check for RBAC privilege issues

Check resource owners

Check related resources

Docs: Application Security

Keycloak

NOTE

Network security

NOTE

Pod security

Specify a non-root user in the container image

Specify security settings for the Pod

Helidon pod security

Pod security for ContainerizedWorkload applications

Pod security for applications using standard Kubernetes resources

Docs: AuthProxy

Docs: Bob's Books

Before you begin

Overview

Deploy the application

NOTE

Access the applications using the WebLogic Server Administration Console

NOTE

Access bobs-bookstore

Access bobbys-front-end

Verify the deployed application

Undeploy the application

Docs: Generic Kubernetes

Prepare for the generic install

NOTE

Customizations

Next steps

Docs: Install Guide

Prerequisites

NOTE

Prepare for the installation

Perform the installation

Install Verrazzano

Install the Verrazzano platform operator

Perform the installation

Install Verrazzano

Verify the installation

Next steps

Docs: Jaeger Tracing

Install Jaeger Operator

Customize Jaeger

Customize a Jaeger instance to use an external OpenSearch or Elasticsearch for storage

Enable the Service Performance Monitoring experimental feature

Disable default Jaeger instance creation

Jaeger Operator Helm chart values that cannot be overridden

Configure an application to export traces to Jaeger

View traces on the Jaeger UI

Configure the Istio mesh to use Jaeger tracing

Management of Jaeger indices in OpenSearch

Jaeger tracing in a multicluster Verrazzano environment

Configure the Istio mesh in a managed cluster to export Jaeger traces to the admin cluster

View the managed cluster traces

Docs: Prerequisites

NOTE

Review `oam-kubernetes-runtime` operator status

Review `verrazzano-application-operator` operator status

Review `oam-kubernetes-runtime` operator logs

Review `verrazzano-application-operator` logs

Use the `vz analyze` tool to analyze multiple snapshots

Use the `vz bug-report` tool