Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: Application Deployment Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Developing and deploying an application to Verrazzano consists of:

Packaging the application as a Docker image.
Publishing the application’s Docker image to a container registry.
Applying the application’s Verrazzano components to the cluster.
Applying the application’s Verrazzano applications to the cluster.

This guide does not provide the full details for the first two steps. An existing example application Docker image has been packaged and published for use.

Verrazzano supports application definition using Open Application Model (OAM). Verrrazzano applications are composed of components and application configurations. This document demonstrates creating OAM resources that define an application as well as the steps required to deploy those resources.

What you need

About 10 minutes.
Access to an existing Kubernetes cluster with Verrazzano installed.
Access to the application’s image in GitHub Container Registry.

Confirm access using this command to pull the example’s Docker image:
```
$ docker pull ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.12-1-20210218160249-d8db8f3
```

Application development

This guide uses an example application which was written with Java and Helidon. For the implementation details, see the Helidon MP tutorial. See the application source code in the Verrazzano examples repository.

The example application is a JAX-RS service and implements the following REST endpoints:

/greet - Returns a default greeting message that is stored in memory. This endpoint accepts the GET HTTP request method.
/greet/{name} - Returns a greeting message including the name provided in the path parameter. This endpoint accepts the GET HTTP request method.
/greet/greeting - Changes the greeting message to be used in future calls to the other endpoints. This endpoint accepts the PUT HTTP request method and a JSON payload.

The following code shows a portion of the application’s implementation. The Verrazzano examples repository contains the complete implementation. An important detail here is that the application contains a single resource exposed on path /greet.

package io.helidon.examples.quickstart.mp;
...
@Path("/greet")
@RequestScoped
public class GreetResource {

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getDefaultMessage() {
        ...
    }

    @Path("/{name}")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getMessage(@PathParam("name") String name) {
        ...
    }

    @Path("/greeting")
    @PUT
    @Consumes(MediaType.APPLICATION_JSON)
    ...
    public Response updateGreeting(JsonObject jsonObject) {
        ...
    }

}

A Dockerfile is used to package the completed application JAR file into a Docker image. The following code shows a portion of the Dockerfile. The Verrazzano examples repository contains the complete Dockerfile. Note that the Docker container exposes a single port 8080.

FROM ghcr.io/oracle/oraclelinux:7-slim
...
CMD java -cp /app/helidon-quickstart-mp.jar:/app/* io.helidon.examples.quickstart.mp.Main
EXPOSE 8080

Application deployment

When you deploy applications with Verrazzano, the platform sets up connections, network policies, and ingresses in the service mesh, and wires up a monitoring stack to capture the metrics, logs, and traces. Verrazzano employs OAM Components to define the functional units of a system that are then assembled and configured by defining associated application configurations.

Verrazzano components

A Verrazzano OAM Component is a Kubernetes Custom Resource describing an application’s general composition and environment requirements. The following code shows the component for the example application used in this guide. This resource describes a component which is implemented by a single Docker image containing a Helidon application exposing a single endpoint.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.10-3-20201016220428-56fb4d4"
              ports:
                - containerPort: 8080
                  name: http

A brief description of each field of the component:

apiVersion - Version of the component custom resource definition
kind - Standard name of the component custom resource definition
metadata.name - The name used to create the component’s custom resource
metadata.namespace - The namespace used to create this component’s custom resource
spec.workload.kind - VerrazzanoHelidonWorkload defines a stateless workload of Kubernetes
spec.workload.spec.deploymentTemplate.podSpec.metadata.name - The name used to create the stateless workload of Kubernetes
spec.workload.spec.deploymentTemplate.podSpec.containers - The implementation containers
spec.workload.spec.deploymentTemplate.podSpec.containers.ports - Ports exposed by the container

Verrazzano application configurations

A Verrazzano application configuration is a Kubernetes Custom Resource which provides environment specific customizations. The following code shows the application configuration for the example used in this guide. This resource specifies the deployment of the application to the hello-helidon namespace. Additional runtime features are specified using traits, or runtime overlays that augment the workload. For example, the ingress trait specifies the ingress host and path, while the metrics trait provides the Prometheus scraper used to obtain the application related metrics.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
                scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

A brief description of each field in the application configuration:

apiVersion - Version of the ApplicationConfiguration custom resource definition
kind - Standard name of the application configuration custom resource definition
metadata.name - The name used to create this application configuration resource
metadata.namespace - The namespace used for this application configuration custom resource
spec.components - Reference to the application’s components leveraged to specify runtime configuration
spec.components[].traits - The traits specified for the application’s components

To explore traits, we can examine the fields of an ingress trait:

apiVersion - Version of the OAM trait custom resource definition
kind - IngressTrait is the name of the OAM application ingress trait custom resource definition
spec.rules.paths - The context paths for accessing the application

Deploy the application

The following steps are required to deploy the example application. Steps similar to the apply steps would be used to deploy any application to Verrazzano.

Create a namespace for the example application and add labels identifying the namespace as managed by Verrazzano and enabled for Istio.
```
$ kubectl create namespace hello-helidon
$ kubectl label namespace hello-helidon verrazzano-managed=true istio-injection=enabled
```
Apply the application’s component.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/hello-helidon/hello-helidon-comp.yaml -n hello-helidon
```
This step causes the validation and creation of the Component resource. No other resources or objects are created as a result. Application configurations applied in the future may reference this Component resource.
Apply the application configuration.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/hello-helidon/hello-helidon-app.yaml -n hello-helidon
```
This step causes the validation and creation of the application configuration resource. This operation triggers the activation of a number of Verrazzano operators. These operators create other Kubernetes objects (for example, Deployments, ReplicaSets, Pods, Services, Ingresses) that collectively provide and support the application.
Configure the application’s DNS resolution.

After deploying the application, configure DNS to resolve the application’s ingress DNS name to the application’s load balancer IP address. The generated host name is obtained by querying Kubernetes for the gateway:
```
$ kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-appconf-gw \
    -n hello-helidon \
    -o jsonpath='{.spec.servers[0].hosts[0]}'
```
The load balancer IP is obtained by querying Kubernetes for the Istio ingress gateway status:
```
$ kubectl get service \
    -n istio-system istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```
DNS configuration steps are outside the scope of this guide. For DNS infrastructure that can be configured and used, see the Oracle Cloud Infrastructure DNS documentation. In some small non-production scenarios, DNS configuration using /etc/hosts or an equivalent may be sufficient.

Verify the deployment

Applying the application configuration initiates the creation of several Kubernetes objects. Actual creation and initialization of these objects occurs asynchronously. The following steps provide commands for determining when these objects are ready for use.

Note: Many other Kubernetes objects unrelated to the example application may also exist. Those have been omitted from the lists.

Verify the Helidon application pod is running.

$ kubectl get pods -n hello-helidon -l app=hello-helidon

# Sample output
NAME                                        READY   STATUS    RESTARTS   AGE
hello-helidon-deployment-8664954995-wcb9d   2/2     Running   0          5m5s

Verify that the Verrazzano application operator pod is running.
```
$ kubectl get pod -n verrazzano-system -l app=verrazzano-application-operator

# Sample output
NAME                                               READY   STATUS    RESTARTS   AGE
verrazzano-application-operator-79849b89ff-lr9w6   1/1     Running   0          13m
```
The namespace verrazzano-system is used by Verrazzano for non-application objects managed by Verrazzano. A single verrazzano-application-operator manages the life cycle of all OAM based applications within the cluster.

Verify the Verrazzano monitoring infrastructure is running.

$ kubectl get pods -n verrazzano-system | grep '^NAME\|vmi-system'

# Sample output
NAME                                               READY   STATUS    RESTARTS   AGE
vmi-system-es-master-0                             2/2     Running   0          47m
vmi-system-grafana-799d79648d-wsdp4                2/2     Running   0          47m
vmi-system-kiali-574c6dd94d-f49jv                  2/2     Running   0          51m
vmi-system-kibana-77f8d998f4-zzvqr                 2/2     Running   0          47m
vmi-system-prometheus-0-7f89d54fbf-brg6x           3/3     Running   0          45m

These pods in the verrazzano-system namespace constitute a monitoring stack created by Verrazzano for the deployed applications.

The monitoring infrastructure comprises several components:

vmi-system-es - OpenSearch for log collection
vmi-system-grafana - Grafana for metric visualization
vms-system-kiali - Kiali for management console of istio service mesh
vmi-system-kibana - OpenSearch Dashboards for log visualization
vmi-system-prometheus - Prometheus for metric collection

Diagnose failures.

View the event logs of any pod not entering the Running state within a reasonable length of time, such as five minutes.
```
$ kubectl describe pod -n hello-helidon -l app=hello-helidon
```
Use the specific namespace and name for the pod being investigated.

Explore the application

Follow these steps to explore the application’s functionality. If DNS was not configured, then use the alternative commands.

Save the host name and IP address of the load balancer exposing the application’s REST service endpoints for later.

$ HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-appconf-gw \
      -n hello-helidon \
      -o jsonpath='{.spec.servers[0].hosts[0]}')
$ ADDRESS=$(kubectl get service \
      -n istio-system istio-ingressgateway \
      -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

NOTE:

The value of ADDRESS is used only if DNS has not been configured.
The following alternative commands may not work in conjunction with firewalls that validate HTTP Host headers.

Get the default message.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet"

# Expected response
{"message":"Hello World!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet" \
    --resolve ${HOST}:443:${ADDRESS}

Get a message for Robert.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert"

# Expected response
{"message":"Hello Robert!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET
    "https://${HOST}/greet/Robert" \
    --resolve ${HOST}:443:${ADDRESS}

Update the default greeting.

$ curl -sk \
    -X PUT \
    "https://${HOST}/greet/greeting" \
    -H 'Content-Type: application/json' \
    -d '{"greeting" : "Greetings"}'

If DNS has not been configured, then use this command.

$ curl -sk \
    -X PUT \
    "https://${HOST}/greet/greeting" \
    -H 'Content-Type: application/json' \
    -d '{"greeting" : "Greetings"}' \
    --resolve ${HOST}:443:${ADDRESS}

Get the new message for Robert.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert"

# Expected response
{"message":"Greetings Robert!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert" \
    --resolve ${HOST}:443:${ADDRESS}

Access the application’s logs

Deployed applications have log collection enabled. These logs are collected using OpenSearch and can be accessed using OpenSearch Dashboards. OpenSearch and OpenSearch Dashboards are examples of infrastructure Verrazzano creates in support of an application as a result of applying an application configuration. For more information on creating an index pattern and visualizing the log data collected in OpenSearch, see OpenSearch Dashboards.

Determine the URL to access OpenSearch Dashboards:

$ OSD_HOST=$(kubectl get ingress \
     -n verrazzano-system vmi-system-kibana \
     -o jsonpath='{.spec.rules[0].host}')
$ OSD_URL="https://${OSD_HOST}"
$ echo "${OSD_URL}"
$ open "${OSD_URL}"

The user name to access OpenSearch Dashboards defaults to verrazzano during the Verrazzano installation.

Determine the password to access OpenSearch Dashboards:

$ echo $(kubectl get secret \
      -n verrazzano-system verrazzano \
      -o jsonpath={.data.password} | base64 \
      --decode)

Access the application’s metrics

Deployed applications have metric collection enabled. Grafana can be used to access these metrics collected by Prometheus. Prometheus and Grafana are additional components Verrazzano creates as a result of applying an application configuration. For more information on visualizing Prometheus metrics data, see Grafana.

Determine the URL to access Grafana:

$ GRAFANA_HOST=$(kubectl get ingress \
      -n verrazzano-system vmi-system-grafana \
      -o jsonpath='{.spec.rules[0].host}')
$ GRAFANA_URL="https://${GRAFANA_HOST}"
$ echo "${GRAFANA_URL}"
$ open "${GRAFANA_URL}"

The user name to access Grafana is set to the default value verrazzano during the Verrazzano installation.

Determine the password to access Grafana:

$ echo $(kubectl get secret \
      -n verrazzano-system verrazzano \
      -o jsonpath={.data.password} | base64 \
      --decode)

Alternatively, metrics can be accessed directly using Prometheus. Determine the URL for this access:

$ PROMETHEUS_HOST=$(kubectl get ingress \
      -n verrazzano-system vmi-system-prometheus \
      -o jsonpath='{.spec.rules[0].host}')
$ PROMETHEUS_URL="https://${PROMETHEUS_HOST}"
$ echo "${PROMETHEUS_URL}"
$ open "${PROMETHEUS_URL}"

The user name and password for both Prometheus and Grafana are the same.

Suppress Kiali console warnings

For some applications, the Kiali console may show warnings for VirtualService and Gateway objects that replicate hostname/port configurations across multiple IngressTraits. These warnings do not impact functionality and can be suppressed with the following component override:

kiali:
  overrides:
    - values:
        kiali_feature_flags:
          validations:
            ignore: ["KIA1106", "KIA0301"]

Remove the application

Run the following commands to delete the application configuration, and optionally the component and namespace.

Delete the application configuration.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/hello-helidon/hello-helidon-app.yaml
```
The deletion of the application configuration will result in the destruction of all application-specific Kubernetes objects.
(Optional) Delete the application’s component.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/hello-helidon/hello-helidon-comp.yaml
```
Note: This step is not required if other application configurations for this component will be applied in the future.

(Optional) Delete the namespace.

$ kubectl delete namespace hello-helidon

Docs: Application Deployment

Mon, 01 Jan 0001 00:00:00 +0000

During application deployment, the oam-kubernetes-runtime and verrazzano-application-operator cooperate through the generation and update of Kubernetes resources. The oam-kubernetes-runtime processes the ApplicationConfiguration and Component resources provided by the user and generates workload and Trait resources. The verrazzano-application-operator processes Verrazzano specific workload and Trait resources. These are then used to generate additional child and related resources.

Troubleshooting application deployments should follow three general steps:

Review the status of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the logs of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the resources generated by the oam-kubernetes-runtime and the verrazzano-application-operator.

Review `oam-kubernetes-runtime` operator status

For application deployment to succeed, the oam-kubernetes-runtime pod must have a status of Running.

Use the following command to get the pod status:

$ kubectl get pods \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

If the pod status is not Running, then see the instructions for reviewing the oam-kubernetes-runtime pod logs.

Review `verrazzano-application-operator` operator status

For application deployment to succeed, the verrazzano-application-operator pod must have a status of Running.

Use the following command to get the pod status:

$ kubectl get pods \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If the pod status is not Running, then see the instructions for reviewing the verrazzano-application-operator logs.

Review `oam-kubernetes-runtime` operator logs

Review the oam-kubernetes-runtime pod logs for any indication that pod startup or the generation of workloads or traits has failed.

Use the following command to get the logs:

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

Review `verrazzano-application-operator` logs

Review the verrazzano-application-operator logs for any indication that pod startup or resource generation has failed.

Use the following command to get the logs:

$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

Review generated workload resources

The processing of a Component reference within an ApplicationConfiguration results in the generation of workloads. For example, a referenced Component might result in the generation of a VerrazzanoHelidonWorkload workload resource. In turn, the VerrazzanoHelidonWorkload workload resource will be processed and result in the generation of related Deployment and Service resources.

If the expected workload resource, for example VerrazzanoHelidonWorkload, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Deployment or Service, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to a VerrazzanoHelidonWorkload deployment:

$ kubectl get -n hello-helidon verrazzanohelidonworkload hello-helidon-workload
$ kubectl get -n hello-helidon deployment hello-helidon-deployment
$ kubectl get -n hello-helidon service hello-helidon-deployment

Review generated Trait resources

The processing of traits embedded with an ApplicationConfiguration results in the generation of Trait resources. For example, an IngressTrait embedded within an ApplicationConfiguration will result in the generation of an IngressTrait resource. In turn, the IngressTrait resource will be processed and result in the generation of related Certificate, Gateway, and VirtualService resources.

If the expected Trait resource, for example IngressTrait, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Certificate, Gateway, and VirtualService, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to an IngressTrait:

$ kubectl get -n hello-helidon ingresstrait hello-helidon-ingress
$ kubectl get -n istio-system Certificate hello-helidon-hello-helidon-appconf-cert
$ kubectl get -n hello-helidon gateway hello-helidon-hello-helidon-appconf-gw
$ kubectl get -n hello-helidon virtualservice hello-helidon-ingress-rule-0-vs

Check for RBAC privilege issues

The use of generic Kubernetes resources as workloads and traits can result in deployment failures if privileges are insufficient. In this case, the oam-kubernetes-runtime logs will contain errors containing the term forbidden.

The following command shows how to query for this type of failure message:

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime | grep forbidden

Check resource owners

Kubernetes maintains the child to parent relationship within metadata fields.

The following example returns the parent of the IngressTrait, named hello-helidon-ingress, in the hello-helidon namespace:

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .metadata.ownerReferences[*]}{.name}{"\n"}{end}'

The results of this command can help identify the lineage of a given resource.

Some resources also record the related resources affected during their processing. For example, when processed, an IngressTrait will create related Gateway, VirtualService, and Certificate resources.

The following command is an example of how to obtain the related resources of an IngressTraits:

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .status.resources[*]}{.kind}: {.name}{"\n"}{end}'

The results of this command can help identify which other resources, the given resource affected.

Docs: DNS

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports three DNS choices for Verrazzano services and applications:

Free wildcard DNS services (nip.io and sslip.io)
Oracle Cloud Infrastructure DNS managed by Verrazzano
Custom (user-managed) DNS

How Verrazzano constructs a DNS domain

Regardless of which DNS management you use, the value in the spec.environmentName field in your installation will be prepended to the configured domain in the spec.components.dns section of the custom resource, to form the full DNS domain name used to access Verrazzano endpoints.

For example, if spec.environmentName is set to sales and the domain is configured in spec.components.dns as us.example.com, Verrazzano will create sales.us.example.com as the DNS domain for the installation.

Verrazzano can be configured to use either the nip.io or sslip.io free wildcard DNS services. When queried with a hostname with an embedded IP address, wildcard DNS services return that IP address.

For example, using the nip.io service, the following DNS names all map to the IP address 10.0.0.1:

10.0.0.1.nip.io
app.10.0.0.1.nip.io
customer1.app.10.0.0.1.nip.io

To configure Verrazzano to use one of these services, set the spec.wildcard.domain field in the Verrazzano custom resource to either nip.io or sslip.io; the default is nip.io.

For example, the following configuration uses sslip.io, instead of nip.io, for wildcard DNS with a dev installation profile:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    dns:
      wildcard:
        domain: sslip.io

Verrazzano can directly manage records in Oracle Oracle Cloud Infrastructure DNS when configured to use the spec.components.dns.oci field. This is achieved through the External DNS Service, which is a component that is conditionally installed when Oracle Cloud Infrastructure DNS is configured for DNS management in Verrazzano.

Prerequisites

The following prerequisites must be met before using Oracle Cloud Infrastructure DNS with Verrazzano:

You must have control of a DNS domain.
You must have an Oracle Cloud Infrastructure DNS Service Zone that is configured to manage records for that domain. Verrazzano also supports the use of both GLOBAL and PRIVATE Oracle Cloud Infrastructure DNS zones.

A DNS Service Zone is a distinct portion of a domain namespace. You must ensure that the zone is appropriately associated with a parent domain. For example, an appropriate zone name for parent domain example.com is us.example.com.

To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure CLI:
```
$ oci dns zone create \
    -c <compartment ocid> \
    --name <zone-name-prefix>.example.com \
    --zone-type PRIMARY
```
To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure Console, see Managing DNS Service Zones.
You must have a valid Oracle Cloud Infrastructure API signing key that can be used to communicate with Oracle Cloud Infrastructure DNS in your tenancy.

For example, you can create an API signing key using the Oracle Cloud Infrastructure CLI:
```
  $ oci setup keys --key-name myapikey
  Enter a passphrase for your private key (empty for no passphrase):
  Public key written to: /Users/jdoe/.oci/myapikey_public.pem
  Private key written to: /Users/jdoe/.oci/myapikey.pem
  Public key fingerprint: 39:08:44:69:9f:f5:73:86:7a:46:d8:ad:34:4f:95:29
```
If you haven’t already uploaded your API signing public key through the console, follow the instructions in this section, How to upload the public key.

After the key pair has been created, you must upload the public key to your account in your Oracle Cloud Infrastructure tenancy. For details, see the Oracle Cloud Infrastructure documentation, Required Keys and OCIDs.

Create an Oracle Cloud Infrastructure API secret in the target cluster

To communicate with Oracle Cloud Infrastructure DNS to manage DNS records, Verrazzano needs to be made aware of the necessary API credentials.
A generic Kubernetes secret must be created in the cluster’s verrazzano-install namespace with the required credentials. That secret must then be referenced by the custom resource that is used to install Verrazzano.

After you have an Oracle Cloud Infrastructure API key ready for use, create a YAML file, oci.yaml, with the API credentials in the form:

auth:
  region: <oci-region>
  tenancy: <oci-tenancy-ocid>
  user: <oci-user-ocid>
  key: |
    <oci-api-private-key-file-contents>
  fingerprint: <oci-api-private-key-fingerprint>

This information typically can be found in your Oracle Cloud Infrastructure CLI config file or in the Oracle Cloud Infrastructure Console. The <oci-api-private-key-file-contents> contents are the PEM-encoded contents of the key_file value within the Oracle Cloud Infrastructure CLI configuration profile.

For example, your oci.yaml file will look similar to the following:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1.....
  user: ocid1.user.oc1.....
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:d3:4c:gh:fd:9e:27:g8:b9:0d:9f:00:22:33:c3:gg

Verrazzano also supports the use of instance principals to communicate with Oracle Cloud Infrastructure in order to create or update Oracle Cloud Infrastructure DNS records. Instance principal requires some prerequisites that can be found here.

When using instance principals, your oci.yaml file will look as follows:

auth:
  authtype: instance_principal

Then, you can create a generic Kubernetes secret in the cluster’s verrazzano-install namespace using kubectl.

$ kubectl create secret generic -n verrazzano-install <secret-name> --from-file=<path-to-oci-yaml-file>

For example, to create a secret named oci from a file oci.yaml, do the following:

$ kubectl create secret generic -n verrazzano-install oci --from-file=oci.yaml

This secret will later be referenced from the Verrazzano custom resource used during installation.

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Verrazzano also provides a helper script to create the necessary Kubernetes secret based on your Oracle Cloud Infrastructure CLI config file, assuming that you have the Oracle Cloud Infrastructure CLI installed and a valid Oracle Cloud Infrastructure CLI profile with the required API key information. The script create_oci_config_secret.sh reads your Oracle Cloud Infrastructure CLI configuration file to create the secret.

First, download the create_oci_config_secret.sh script:

$ curl \
    -o ./create_oci_config_secret.sh \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/platform-operator/scripts/install/create_oci_config_secret.sh

Next, set your KUBECONFIG environment variable to point to your cluster and run create_oci_config_secret.sh -h to display the script options:

$ chmod +x create_oci_config_secret.sh
$ export KUBECONFIG=<kubeconfig-file>
$ ./create_oci_config_secret.sh  -h
usage: ./create_oci_config_secret.sh [-o oci_config_file] [-s config_file_section]
  -o oci_config_file         The full path to the Oracle Cloud Infrastructure configuration file (default ~/.oci/config)
  -s config_file_section     The properties section within the Oracle Cloud Infrastructure configuration file.  Default is DEFAULT
  -k secret_name             The secret name containing the Oracle Cloud Infrastructure configuration.  Default is oci
  -c context_name            The kubectl context to use
  -a auth_type               The auth_type to be used to access Oracle Cloud Infrastructure. Valid values are user_principal/instance_principal. Default is user_principal.
  -h                         Help

For example, to have the script create the YAML file using your [DEFAULT] Oracle Cloud Infrastructure CLI profile and then create a Kubernetes secret named oci, you can run the script with no arguments, as follows:

$ ./create_oci_config_secret.sh
secret/oci created

The following example creates a secret myoci using an Oracle Cloud Infrastructure CLI profile named [dev]:

$ ./create_oci_config_secret.sh -s dev -k myoci
secret/myoci created

When using instance principals all other parameters will be ignored automatically. The following example creates a secret myoci using Oracle Cloud Infrastructure instance principal:

$ ./create_oci_config_secret.sh -a instance_principal
secret/myoci created

Installation

After the Oracle Cloud Infrastructure API secret is created, create a Verrazzano custom resource for the installation that is configured to use Oracle Cloud Infrastructure DNS, and reference the secret you created.

As a starting point, download the sample Verrazzano custom resource install-oci.yaml file for Oracle Cloud Infrastructure DNS:

$ curl \
    -o ./install-oci.yaml \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/platform-operator/config/samples/install-oci.yaml

Edit the install-oci.yaml file to provide values for the following configuration settings in the custom resource spec:

spec.environmentName
spec.components.dns.oci.ociConfigSecret
spec.components.dns.oci.dnsZoneCompartmentOCID
spec.components.dns.oci.dnsZoneOCID
spec.components.dns.oci.dnsZoneName
spec.components.dns.oci.dnsScope

The field spec.components.dns.oci.ociConfigSecret should reference the secret created earlier. For details on the Oracle Cloud Infrastructure DNS configuration settings, see spec.components.dns.oci.

For example, a custom resource for a prod installation profile using Oracle Cloud Infrastructure DNS might look as follows, yielding a domain of myenv.example.com (Oracle Cloud Infrastructure identifiers redacted):

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com

If using a private DNS zone, then the same prod installation profile using Oracle Cloud Infrastructure DNS will look as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: my-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com
        dnsScope: PRIVATE

After the custom resource is ready, apply it using kubectl apply -f <path-to-custom-resource-file>.

You can specify your own externally managed, custom DNS domain. In this scenario, you manage your own DNS domain and all DNS records in that domain.

An externally managed DNS domain is specified in the spec.components.dns.external.suffix field of the Verrazzano custom resource.

When using an externally managed DNS domain, you are responsible for:

Configuring A records for Verrazzano ingress points (load balancers)
Configuring CNAME records for hostnames in the domain that point to the A records, as needed

The Verrazzano installer searches the DNS zone you provide for two specific A records.
These are used to configure the cluster and should refer to external addresses of the load balancers provisioned by the user.

The A records need to be created manually.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example, if spec.environmentName is set to myenv, and spec.components.dns.external.suffix is set to example.com, the A records would need to be set up as follows:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

This example assumes that load balancers exist for ingress-mgmt on 198.51.100.10 and for ingress-verrazzano on 203.0.113.10.

For a more complete example, see the documentation for setting up Verrazzano on the Oracle Cloud Native Environment Platform.

Docs: Install Guide

Mon, 01 Jan 0001 00:00:00 +0000

The following instructions show you how to install Verrazzano in a single Kubernetes cluster.

Prerequisites

Find the Verrazzano prerequisite requirements here.
Review the list of the software versions supported and installed by Verrazzano.

Prepare for the install

Before installing Verrazzano, see instructions on preparing Kubernetes platforms.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For instructions on how to install a CNI plug-in, see the documentation for your Kubernetes cluster.

Install the Verrazzano platform operator

Verrazzano provides a platform operator to manage the life cycle of Verrazzano installations. Using the Verrazzano custom resource, you can install, uninstall, and upgrade Verrazzano installations.

To install the Verrazzano platform operator:

Deploy the Verrazzano platform operator.

$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/operator.yaml

Wait for the deployment to complete.

$ kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator

# Expected response
deployment "verrazzano-platform-operator" successfully rolled out

Confirm that the operator pod is correctly defined and running.

$ kubectl -n verrazzano-install get pods

# Sample output
NAME                                            READY   STATUS    RESTARTS   AGE
verrazzano-platform-operator-59d5c585fd-lwhsx   1/1     Running   0          114s

Perform the install

Verrazzano supports the following installation profiles: development (dev), production (prod), and managed cluster (managed-cluster). For more information on profiles, see Installation Profiles.

This page shows how to create a basic Verrazzano installation using:

The development (dev) installation profile
Wildcard-DNS, where DNS is provided by nip.io (the default)

NOTE

Because the dev profile installs self-signed certificates, when installing Verrazzano on macOS, you might see: Your connection is not private. For a workaround, see this FAQ.

For a complete description of Verrazzano configuration options, see the Verrazzano Custom Resource Definition.

To use other DNS options, see the Customzing DNS page for more details.

Install Verrazzano

To create a Verrazzano installation as described in the previous section, run the following commands:

$ kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: ${VZ_PROFILE:-dev}
EOF
$ kubectl wait \
    --timeout=20m \
    --for=condition=InstallComplete verrazzano/example-verrazzano

To use a different profile with the previous example, set the VZ_PROFILE environment variable to the name of the profile you want to install.

If an error occurs, check the log output of the installation. You can view the logs with the following command:

$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
    | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'

For more help troubleshooting the installation, see Analysis Advice.

After the installation is complete, you can use the console URLs. For more information on how to access the Verrazzano consoles, see Access Verrazzano.

Verify the install

Verrazzano installs multiple objects in multiple namespaces. In the verrazzano-system namespaces, all the pods in the Running state, does not guarantee, but likely indicates that Verrazzano is up and running.

$ kubectl get pods -n verrazzano-system

# Sample output
coherence-operator-dcfb446df-24djp                 1/1     Running   1          49m
fluentd-h65xf                                      2/2     Running   1          45m
oam-kubernetes-runtime-6645df49cd-6q96c            1/1     Running   0          49m
verrazzano-application-operator-85ffd7f77b-rhwk7   1/1     Running   0          48m
verrazzano-authproxy-58db5b9484-nhnql              2/2     Running   0          45m
verrazzano-console-5dbdc579bd-hm4rh                2/2     Running   0          45m
verrazzano-monitoring-operator-599654889d-lbb4z    1/1     Running   0          45m
verrazzano-operator-7b6fd64dd5-8j9h8               1/1     Running   0          45m
vmi-system-es-master-0                             2/2     Running   0          45m
vmi-system-grafana-5558d65b46-pxg78                2/2     Running   0          45m
vmi-system-kiali-5949966fb8-465s8                  2/2     Running   0          48m
vmi-system-kibana-86b894d8f6-q4vb5                 2/2     Running   0          45m
vmi-system-prometheus-0-859fcd87dc-m5ws9           3/3     Running   0          44m
weblogic-operator-646756c75c-hgz6j                 2/2     Running   0          49m

(Optional) Run the example applications

Example applications are located here.

To get the consoles URLs and credentials, see Access Verrazzano.

Docs: Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

This document describes some common problems you might encounter when using multicluster Verrazzano, and how to troubleshoot them.

If you created multicluster resources in the admin cluster, and specified a placement value in a managed cluster, then those resources will get created in that managed cluster. If they do not get created in the managed cluster, then use the following steps to troubleshoot:

Verify that the managed cluster is registered correctly and can connect to the admin cluster.
Verify that the VerrazzanoProject for the resource’s namespace, also has a placement in that managed cluster.
Check the multicluster resource’s status field on the admin cluster to know what the status of that resource is on each managed cluster to which it is targeted.

If you update the DNS of the admin cluster and notice that the managed cluster status is unavailable in the Rancher console, along with the error x509: certificate is valid for <rancher new url>, not <rancher old url> seen in the cattle-cluster-agent (Rancher Agent) logs on the managed cluster, then re-register the managed cluster, as described here.

Verify managed cluster registration and connectivity

You can verify that a managed cluster was successfully registered with an admin cluster by viewing the corresponding VerrazzanoManagedCluster (VMC) resource on the admin cluster. For example, to verify that a managed cluster named managed1 was successfully registered:

# on the admin cluster
$ kubectl get verrazzanomanagedcluster managed1 \
    -n verrazzano-mc \
    -o yaml

Partial sample output from the previous command:

  status:
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:27Z"
      message: Ready
      status: "True"
      type: Ready
    lastAgentConnectTime: "2021-06-22T21:06:04Z"
    ... other fields ...

Check the lastAgentConnectTime in the status of the VMC resource. This is the last time at which the managed cluster connected to the admin cluster. If this value is not present, then the managed cluster named managed1 never successfully connected to the admin cluster. This could be due to several reasons:

The managed cluster registration process step of applying the registration YAML on the managed cluster, was not completed. For the complete setup instructions, see here.
The managed cluster does not have network connectivity to the admin cluster. The managed cluster will attempt to connect to the admin cluster at regular intervals, and any errors will be reported in the verrazzano-application-operator pod’s log on the managed cluster. View the logs using the following command:

# on the managed cluster
$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If these logs reveal that there is a connectivity issue, check the admin cluster Kubernetes server address that you provided during registration and ensure that it is correct, and that it is reachable from the managed cluster. If it is incorrect, then you will need to repeat the managed cluster registration process described in the setup instructions here.

Verify VerrazzanoProject placement

For Verrazzano to create an application namespace in a managed cluster, that namespace must be part of a VerrazzanoProject that:

Includes that namespace.
Has a placement value that includes that managed cluster.

View the details of the project that corresponds to your application’s namespace. In the example command that follows, the project name is assumed to be myproject. All projects are expected to be created in the verrazzano-mc namespace.

# on the admin cluster
$ kubectl get verrazzanoproject myproject \
    -n verrazzano-mc \
    -o yaml

The following partial sample output is for a project that will result in the namespace mynamespace being created on the managed cluster managed1.

spec:
  placement:
    clusters:
    - name: managed1
  template:
    namespaces:
    - metadata:
        name: mynamespace
....other fields....

Check the multicluster resource status

On the admin cluster, each multicluster resource’s status field is updated with the status of the underlying resource on each managed cluster in which it is placed.

The following example command shows how to view the status of a MultiClusterApplicationConfiguration named myapp, in the namespace mynamespace, that has a placement value that includes the managed cluster managed1.

$ kubectl get multiclusterapplicationconfiguration myapp \
    -n mynamespace \
    -o yaml

The status of the underlying resource in each cluster specified in the placement is shown in the following partial sample output:

  status:
    clusters:
    - lastUpdateTime: "2021-06-22T21:05:04Z"
      message: OAM Application Configuration created
      name: managed1
      state: Succeeded
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:58Z"
      message: OAM Application Configuration created
      status: "True"
      type: DeployComplete
    state: Succeeded

The status message contains additional information on the operation’s success or failure.

Re-register the managed cluster

Perform the following steps to re-register the managed cluster with the admin cluster. The cluster against which to run the command is indicated in each code block.

On the admin cluster, export the register YAML file newly created on the admin cluster to re-register the managed cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register_new.yaml

On the managed cluster, apply the registration file exported in the previous step.

# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register_new.yaml

# Once the command succeeds, you may delete the register_new.yaml file
$ rm register_new.yaml

On the admin cluster, run kubectl patch clusters.management.cattle.io to trigger redeployment of the Rancher agent on the managed cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get clusters.management.cattle.io

# Sample output
NAME      AGE
c-mzb2h   4h48m
local     4h56m

$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \ 
    patch clusters.management.cattle.io <the managed cluster name from the above output> \
    -p '{"status":{"agentImage":"dummy"}}' --type merge

# Sample output
cluster.management.cattle.io/c-mzb2h patched

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires the following:

A Kubernetes cluster and a compatible kubectl.
At least two CPUs, 100GB disk storage, and 16GB RAM available on the Kubernetes worker nodes. This is sufficient to install the development profile of Verrazzano. Depending on the resource requirements of the applications you deploy, this may or may not be sufficient for deploying your applications.

Supported hardware

Verrazzano requires x86-64; other architectures are not supported.

Supported software versions

Verrazzano supports the following software versions.

Kubernetes

You can install Verrazzano on the following Kubernetes versions.

Verrazzano	Kubernetes Versions
1.3	1.21, 1.22, 1.23
1.2	1.19, 1.20, 1.21
1.1	1.19, 1.20, 1.21
1.0	1.18, 1.19, 1.20

For more information, see Kubernetes Release Documentation. For platform specific details, see Verrazzano platform setup.

WebLogic Server

The supported versions of WebLogic Server are dependent on the WebLogic Kubernetes Operator version. See the WebLogic Server versions supported here.

Coherence

The supported versions of Coherence are dependent on the Coherence Operator version. See the Coherence versions supported here.

Helidon

Verrazzano supports all versions of Helidon. For more information, see Helidon and Helidon Commercial Offerings.

Installed components

Verrazzano installs a curated set of open source components. The following table lists each component, its version, and a brief description.

Component	Version	Description
alert-manager	0.24.0	Handles alerts sent by client applications, such as the Prometheus server.
cert-manager	1.7.1	Automates the management and issuance of TLS certificates.
Coherence Operator	3.2.5	Assists with deploying and managing Coherence clusters.
ExternalDNS	0.10.2	Synchronizes exposed Kubernetes Services and ingresses with DNS providers.
Fluentd	1.12.3	Collects logs and sends them to OpenSearch.
Grafana	7.5.11	Tool to help you examine, analyze, and monitor metrics.
Istio	1.13.5	Service mesh that layers transparently onto existing distributed applications.
Jaeger Operator	1.32.0	Distributed tracing system for monitoring and troubleshooting distributed systems.
Keycloak	15.0.2	Provides single sign-on with Identity and Access Management.
Kiali	1.42.0	Management console for the Istio service mesh.
kube-state-metrics	2.4.2	Provides metrics about the state of Kubernetes API objects.
MySQL	8.0.28	Open source relational database management system used by Keycloak.
NGINX Ingress Controller	1.1.1	Traffic management solution for cloud‑native applications in Kubernetes.
Node Exporter	1.3.1	Prometheus exporter for hardware and OS metrics.
OAM Kubernetes Runtime	0.3.0	Plug-in for implementing Open Application Model (OAM) control plane with Kubernetes.
OpenSearch	1.2.3	Provides a distributed, multitenant-capable full-text search engine.
OpenSearch Dashboards	1.2.0	Provides search and data visualization capabilities for data indexed in OpenSearch.
Prometheus	2.34.0	Provides event monitoring and alerting.
Prometheus Adapter	0.9.1	Provides metrics in support of pod autoscaling.
Prometheus Operator	0.55.1	Provides management for Prometheus monitoring tools.
Prometheus Pushgateway	1.4.2	Allows ephemeral and batch jobs to expose their metrics to Prometheus.
Rancher	2.6.6	Manages multiple Kubernetes clusters.
WebLogic Kubernetes Operator	3.4.3	Assists with deploying and managing WebLogic domains.

Docs: Verrazzano Analysis Tools

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides tooling which assists in troubleshooting issues in your environment:

k8s-dump-cluster.sh
verrazzano-analysis

Tools Setup

These tools are available for Linux and Mac: https://github.com/verrazzano/verrazzano/releases/.

Linux Instructions

Use these instructions to obtain the analysis tools on Linux machines.

Download the tooling

 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/k8s-dump-cluster.sh
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/k8s-dump-cluster.sh.sha256
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/verrazzano-analysis-linux-amd64.tar.gz
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/verrazzano-analysis-linux-amd64.tar.gz.sha256

Verify the downloaded files

 $ sha256sum -c k8s-dump-cluster.sh.sha256
 $ sha256sum -c verrazzano-analysis-linux-amd64.tar.gz.sha256

Unpack the `verrazzano-analysis` binary

 $ tar xvf verrazzano-analysis-linux-amd64.tar.gz

Mac Instructions

Use these instructions to obtain the analysis tools on Mac machines.

Download the tooling

 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/k8s-dump-cluster.sh
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/k8s-dump-cluster.sh.sha256
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/verrazzano-analysis-darwin-amd64.tar.gz
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.3.8/verrazzano-analysis-darwin-amd64.tar.gz.sha256

Verify the downloaded files

 $ shasum -a 256 -c k8s-dump-cluster.sh.sha256
 $ shasum -a 256 -c verrazzano-analysis-darwin-amd64.tar.gz.sha256

Unpack the `verrazzano-analysis` binary

 $ tar xvf verrazzano-analysis-darwin-amd64.tar.gz

Use the `k8s-dump-cluster.sh` tool

The k8s-dump-cluster.sh tool is a shell script which runs various kubectl and helm commands against a cluster.

Note that the data captured by this script might include sensitive information. This data is under your control; you can choose whether to share it.

The directory structure created by the k8s-dump-cluster.sh tool, for a specific cluster dump, appears as follows:

$ CAPTURE_DIR
  cluster-dump
    directory per namespace (a directory at this level is assumed to represent a namespace)
      acme-orders.json
      application-configurations.json
      certificate-requests.json
      cluster-role-bindings.json
      cluster-roles.json
      cluster-roles.json
      coherence.json
      components.json
      {CONFIGNAME}.configmap (a file at this level for each configmap in the namespace)
      daemonsets.json
      deployments.json
      events.json
      gateways.json
      ingress-traits.json
      jobs.json
      multicluster-application-configurations.json
      multicluster-components.json
      multicluster-config-maps.json
      multicluster-logging-scopes.json
      multicluster-secrets.json
      namespace.json
      persistent-volume-claims.json
      persistent-volumes.json
      pods.json
      replicasets.json
      replication-controllers.json
      role-bindings.json
      services.json
      verrazzano-managed-clusters.json
      verrazzano-projects.json
      verrazzano_resources.json
      virtualservices.json
      weblogic-domains.json
      directory per pod (a directory at this level is assumed to represent a specific pod)
        logs.txt (includes logs for all containers and initContainers)
    api-resources.out
    application-configurations.json
    cluster-issuers.txt
    coherence.json
    configmap_list.out
    crd.json
    es_indexes.out
    gateways.json
    helm-ls.json
    helm-version.out
    images-on-nodes.csv
    ingress.json
    ingress-traits.json
    kubectl-version.json
    namespace_list.out
    network-policies.json
    network-policies.txt
    nodes.json
    pv.json
    verrazzano_resources.out
    virtualservices.json

The script shows the kubectl and helm commands which are run. The basic structure, shown previously, is formed by running the command, $ kubectl cluster-info dump --all-namespaces, with additional data captured into that directory structure.

To perform a dump of a cluster into a directory named my-cluster-dump:

$ sh k8s-dump-cluster.sh -d my-cluster-dump

Use the `verrazzano-analysis` tool

The verrazzano-analysis tool analyzes data from a cluster dump captured using k8s-dump-cluster.sh, reports the issues found, and prescribes related actions to take. These tools are continually evolving with regard to what may be captured, the knowledge base of issues and actions, and the types of analysis that can be performed.

Users, developers, and Continuous Integration (CI) can use this tooling to quickly identify the root cause of encountered problems, determine mitigation actions, and provide a sharable report with other users or tooling.

The data that the analysis examines follows the structure created by the corresponding capture tooling. For example, k8s-dump-cluster.sh dumps a cluster into a specific structure, which might contain data that you do not want to share. The tooling analyzes the data and provides you with a report, which identifies issues and provides you with actions to take.

The verrazzano-analysis tool will find and analyze all cluster dump directories found under a specified root directory. This lets you create a directory to hold the cluster dumps of related clusters into sub-directories which the tool can analyze.

For example:

my-cluster-dumps
    CAPTURE_DIR-1
        cluster-dump
            ...
    CAPTURE_DIR-2
        cluster-dump
            ...

The tool analyzes each cluster dump directory found; you need to provide only the single root directory.

To perform an analysis of the clusters:

$ verrazzano-analysis my-cluster-dumps

Usage information

Usage: verrazzano-analysis [options] captured-data-directory

Parameter	Definition	Default
`-actions`	Include actions in the report.	`true`
`-help`	Display usage help.
`-info`	Include informational messages.	`true`
`-minConfidence`	Minimum confidence threshold to report for issues, 0-10.	`0`
`-minImpact`	Minimum impact threshold to report for issues, 0-10.	`0`
`-reportFile`	Name of report output file.	Output to stdout.
`-support`	Include support data in the report.	`true`
`-version`	Display tool version.

Docs: Verrazzano and the Open Application Model

Mon, 01 Jan 0001 00:00:00 +0000

Open Application Model (OAM) is a runtime-agnostic specification for defining cloud native applications; it allows developers to focus on the application instead of the complexities of a particular runtime infrastructure. OAM provides the specification for several file formats and rules for a runtime to interpret. Verrazzano uses OAM to enable the definition of a composite application abstraction and makes OAM constructs available within a VerrazzanoApplication YAML file. Verrazzano provides the flexibility to combine what you want into a multicloud enablement. It uses the VerrazzanoApplication as a means to encapsulate a set of components, scopes, and traits, and deploy them on a selected cluster.

OAM’s workload concept makes it easy to use many different workload types. Verrazzano includes specific workload types with special handling to deploy and manage those types, such as WebLogic, Coherence, and Helidon. OAM’s flexibility lets you create a grouping that is managed as a unit, although each component can be scaled or updated independently.

How does OAM work?

OAM has five core concepts:

Workloads - Declarations of the kinds of resources supported by the platform and the OpenAPI schema for that resource. Most Kubernetes CRDs can be exposed as workloads. Standard Kubernetes resource types can also be used (for example, Deployment, Service, Pod, ConfigMap).
Components - Wrap a workload resource’s specification data within OAM specific metadata.
Application Configurations - Describe a collection of components that comprise an application. This is also where customization (such as, environmental) of each component is done. Customization is achieved using scopes and traits.
Scopes - Apply customization to several components.
Traits - Apply customization to a single component.

Docs: Verrazzano in a Multicluster Environment

Mon, 01 Jan 0001 00:00:00 +0000

Review the following key concepts to understand multicluster Verrazzano:

Admin cluster - A Kubernetes cluster that serves as the central management point for deploying and monitoring applications in managed clusters.
Managed clusters - A Kubernetes cluster that has the following characteristics:
- It is registered with an admin cluster with a unique name.
- Verrazzano multicluster applications may be deployed to the managed cluster from the admin cluster.
- Logs and metrics for Verrazzano system components and Verrazzano multicluster applications deployed on the managed cluster are viewable from the admin cluster.
Verrazzano multicluster resources - Custom Kubernetes resources defined by Verrazzano.
- Each multicluster resource serves as a wrapper for an underlying resource type.
- A multicluster resource allows the placement of the underlying resource to be specified as a list of names of the clusters in which the resource must be placed.

For more details, see here.

Docs: Verrazzano Projects

Mon, 01 Jan 0001 00:00:00 +0000

A project provides a way to group application namespaces that are owned or administered by the same user or group of users. When creating a project, you can specify the subjects: users, groups and/or service accounts, that are to be granted access to the namespaces governed by the project. Two types of subjects may be specified:

Project admins, who have both read and write access to the project’s namespaces.
Project monitors, who have read-only access to the project’s namespaces.

For more information, see Projects.

Docs: Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano issues certificates to secure access from external clients to secure system endpoints.
A certificate from a certificate authority (CA) must be configured to issue the endpoint certificates in one of the following ways:

Let Verrazzano generate a self-signed CA (the default).
Configure a CA that you provide.
Configure LetsEncrypt as the certificate issuer (requires Oracle Cloud Infrastructure DNS).

In all cases, Verrazzano uses cert-manager to manage the creation of certificates.

NOTE

Self-signed certificate authorities generate certificates that are NOT signed by a trusted authority; typically, they are not used in production environments.

Use the Verrazzano self-signed CA

By default, Verrazzano creates its own self-signed CA. No configuration is required.

Use a custom CA

If you want to provide your own CA, you must:

(Optional) Create your own signing key pair and CA certificate.

For example, you can use the openssl CLI to create a key pair for the nip.io domain:
```
# Generate a CA private key
$ openssl genrsa -out tls.key 2048

# Create a self signed certificate, valid for 10yrs with the 'signing' option set
$ openssl req -x509 -new -nodes -key tls.key -subj "/CN=*.nip.io" -days 3650 -reqexts v3_req -extensions v3_ca -out tls.crt
```
The output of these commands will be two files, tls.key and tls.crt, the key and certificate for your signing key pair. These files must be named in that manner for the next step.

If you already have generated your own key pair, you must name the private key and certificate, tls.key and tls.crt, respectively. If your issuer represents an intermediate, ensure that tls.crt contains the issuer’s full chain in the correct order.

You can find more details on providing your own CA, in the cert-manager CA documentation.

Save your signing key pair as a Kubernetes secret.

$ kubectl create ns mynamespace
$ kubectl create secret tls myca --namespace=mynamespace --cert=tls.crt --key=tls.key

Specify the secret name and namespace location in the Verrazzano custom resource.

The custom CA secret must be provided to cert-manager using the following fields in spec.components.certManager.certificate.ca in the Verrazzano custom resource:
- spec.components.certManager.certificate.ca.secretName
- spec.components.certManager.certificate.ca.clusterResourceNamespace

For example, if you created a CA secret named myca in the namespace mynamespace, you would configure it as shown:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: custom-ca-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        ca:
          secretName: myca
          clusterResourceNamespace: mynamespace

Use LetsEncrypt certificates

You can configure Verrazzano to use certificates generated by LetsEncrypt. LetsEncrypt implements the ACME protocol, which provides a standard protocol for the automated issuance of certificates signed by a trusted authority. This is managed through the spec.components.certManager.certificate.acme field in the Verrazzano custom resource.

NOTE

Using LetsEncrypt for certificates also requires using Oracle Cloud Infrastructure DNS for DNS management. For details, see the Customize DNS page.

To configure cert-manager to use LetsEncrypt as the certificates provider, you must configure a cert-manager ACME provider with the following values in the Verrazzano custom resource:

Set the spec.components.certManager.certificate.acme.provider field to letsEncrypt.
Set the spec.components.certManager.certificate.acme.emailAddress field to a valid email address for the letsEncrypt account.
(Optional) Set the spec.components.certManager.certificate.acme.environment field to either staging or production (the default).

The following example configures Verrazzano to use the LetsEncrypt production environment by default, with Oracle Cloud Infrastructure DNS for DNS record management:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

The following example configures Verrazzano to use the LetsEncrypt staging environment with Oracle Cloud Infrastructure DNS:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
          environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

NOTE

Certificates issued by the LetsEncrypt staging environment are signed by untrusted authorities, similar to self-signed certificates. They are typically not used in production environments.

LetsEncrypt staging versus production

LetsEncrypt provides rate limits on generated certificates to ensure fair usage across all clients. The production environment limits can be exceeded more frequently in environments where Verrazzano may be being installed or reinstalled frequently (like a test environment). This can result in failed installations due to rate limit exceptions on certificate generation.

In such environments, it is better to use the LetsEncrypt staging environment, which has much higher limits than the production environment. For test environments, the self-signed CA also may be more appropriate to completely avoid LetsEncrypt rate limits.

Docs: IngressTrait Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The IngressTrait custom resource contains the configuration of host and path rules for traffic routing to an application. Here is a sample ApplicationConfiguration that specifies an IngressTrait. To deploy an example application that demonstrates this IngressTrait, see Hello World Helidon.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
                scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

In the sample configuration, the IngressTrait hello-helidon-ingress is set on the hello-helidon-component application component and defines an ingress rule that configures a path and path type. This exposes a route for external access to the application. Note that because no hosts list is given for the IngressRule, a DNS host name is automatically generated.

For example, with the sample application configuration successfully deployed, the application will be accessible with the path specified in the IngressTrait and the generated host name.

$ HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-appconf-gw -n hello-helidon -o jsonpath={.spec.servers[0].hosts[0]})
$ echo $HOST
hello-helidon-appconf.hello-helidon.11.22.33.44.nip.io

$ curl -sk -X GET https://${HOST}/greet

Alternatively, specific host names can be given in an IngressRule. Doing this implies that a secret and certificate have been created for the specific hosts and the secret name has been specified in the associated IngressSecurity secretName field.

Load balancer session affinity is configured using an HTTP cookie in a destination rule. Here is an updated sample ApplicationConfiguration that includes a destination rule with an HTTP cookie.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
                scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix
                - destination:
                    httpCookie:
                      name: sessioncookie
                      path: "/"
                      ttl: 600

IngressTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	IngressTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	IngressTraitSpec	The desired state of an ingress trait.	Yes

IngressTraitSpec

IngressTraitSpec specifies the desired state of an ingress trait.

Field	Type	Description	Required
`rules`	IngressRule array	A list of ingress rules to for an ingress trait.	Yes
`tls`	IngressSecurity	The security parameters for an ingress trait. This is required only if specific hosts are given in an IngressRule.	No

IngressRule

IngressRule specifies a rule for an ingress trait.

Field	Type	Description	Required
`hosts`	string array	One or more hosts exposed by the ingress trait. Wildcard hosts or hosts that are empty are filtered out. If there are no valid hosts provided, then a DNS host name is automatically generated and used.	No
`paths`	IngressPath array	The paths to be exposed for an ingress trait.	Yes
`destination`	IngressDestination	The destination host and port for the ingress paths.	No

IngressPath

IngressPath specifies a specific path to be exposed for an ingress trait.

Field	Type	Description	Required
`path`	string	If no path is provided, it defaults to `/`.	No
`pathType`	string	Path type values are case-sensitive and formatted as follows: `exact`: exact string match `prefix`: prefix-based match `regex`: regex-based match If the provided ingress path doesn’t contain a `pathType`, it defaults to `prefix` if the path is `/` and `exact` otherwise.	No

IngressDestination

IngressDestination specifies a specific destination host and port for the ingress paths.

Field	Type	Description	Required
`host`	string	Destination host.	No
`port`	uint32	Destination port.	No
`httpCookie`	HttpCookie	Session affinity cookie.	No

NOTE

If there are multiple ports defined for a service, then the destination port must be specified OR the service port name must have the prefix “http”.

HttpCookie

HttpCookie specifies a session affinity cookie for an ingress trait.

Field	Type	Description	Required
`name`	string	The name of the HTTP cookie.	No
`path`	string	The path of the HTTP cookie.	No
`ttl`	uint32	The lifetime of the HTTP cookie (in seconds).	No

IngressSecurity

IngressSecurity specifies the secret containing the certificate securing the transport for an ingress trait.

Field	Type	Description	Required
`secretName`	string	The name of a secret containing the certificate securing the transport. The specification of a secret here implies that a certificate was created for specific hosts, as specified in an IngressRule.	Yes

Docs: Installation Profiles

Mon, 01 Jan 0001 00:00:00 +0000

This document describes built-in configuration profiles that you can use to simplify a Verrazzano installation. An installation profile is a well-known configuration of Verrazzano settings that can be referenced by name, which then can be customized as needed.

The following table describes the Verrazzano installation profiles.

Profile	Description	Characteristics
`prod`	Full install, production configuration.	Default profile: - Full installation. - Persistent storage. - Production OpenSearch cluster topology.
`dev`	Development or evaluation configuration.	Lightweight installation: - For evaluation purposes. - No persistence. - Single-node OpenSearch cluster topology.
`managed-cluster`	A specialized installation for managed clusters in a multicluster topology.	Minimal installation for a managed cluster: - Cluster must be registered with an admin cluster to use multicluster features.

Use an installation profile

To use a profile to install Verrazzano, set the profile name in the profile field of your Verrazzano custom resource.

For example, to use the dev profile:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev

To use a different profile, simply replace dev with prod or managed-cluster.

Customize an installation profile

You can override the profile settings for any component regardless of the profile. The following example uses a customized dev profile to configure a small 8Gi persistent volume for the MySQL instance used by Keycloak to provide more stability for the Keycloak service:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: custom-dev-example
spec:
  profile: dev
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql
  volumeClaimSpecTemplates:
  - metadata:
      name: mysql      
    spec:
      resources:
        requests:
          storage: 8Gi

For details on how to customize Verrazzano components, see Customize an Installation.

Profile configurations

The following table lists the Verrazzano components that are installed with each profile. Note that you can customize any Verrazzano installation, regardless of the profile.

Component	dev	prod	managed-cluster
Istio	✔️	✔️	✔️
NGINX	✔️	✔️	✔️
cert-manager	✔️	✔️	✔️
External-DNS	️	️
Prometheus	✔️	✔️	✔️
OpenSearch	✔️	✔️
Console	✔️	✔️
OpenSearch Dashboards	✔️	✔️
Grafana	✔️	✔️
Rancher	✔️	✔️
Keycloak	✔️	✔️

Prometheus and Grafana configurations

The following table describes the Prometheus and Grafana configurations in each profile.

Profile	Prometheus	Grafana
`prod`	1 replica (128M memory, 50Gi storage)	1 replica (48M memory, 50Gi storage)
`dev`	1 replica (128M memory, ephemeral storage)	1 replica (48M memory, ephemeral storage)
`managed-cluster`	1 replica (128M memory, 50Gi storage)	Not installed

OpenSearch Dashboards and OpenSearch configurations

The following table describes the OpenSearch Dashboards and OpenSearch cluster topology in each profile.

Profile	OpenSearch	OpenSearch Dashboards
`prod`	3 master replicas (1.4Gi memory, 50Gi storage each) 1 ingest replica (2.5Gi memory, no storage) 3 data replicas (4.8Gi memory, 50Gi storage each)	1 replica (192M memory, ephemeral storage)
`dev`	1 master/data/ingest replica (1Gi memory, ephemeral storage)	1 replica (192M memory, ephemeral storage)
`managed-cluster`	Not installed	Not installed

NOTE

OpenSearch containers are configured to use 75% of the configured request memory for the Java min/max heap settings.

Profile-independent defaults

The following table shows the settings for components that are profile-independent (consistent across all profiles unless overridden).

Component	Default
DNS	Wildcard DNS provider nip.io.
Certificates	Uses the cert-manager self-signed ClusterIssuer for certificates.
Ingress-type	Defaults to `LoadBalancer` service type for the ingress.

For details on how to customize Verrazzano components, see Customize an Installation.

Docs: Kubernetes RBAC

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano uses Kubernetes Role-Based Access Control (RBAC) to protect Verrazzano resources.

Verrazzano includes a set of roles that can be granted to users, enabling access to Verrazzano resources managed by Kubernetes. In addition, Verrazzano creates a number of roles that grant permissions needed by various Verrazzano system components (operators and third-party components).

Verrazzano creates default role bindings during installation and for projects, at project creation or update.

NOTE

Kubernetes RBAC must be enabled in every cluster to which Verrazzano is deployed or access control will not work. RBAC is enabled by default in most Kubernetes environments.

Verrazzano user roles

The following table lists the defined Verrazzano user roles. Each is a ClusterRole intended to be granted directly to users or groups. (In some scenarios, it may be appropriate to grant a user role to a service account.)

Verrazzano Role	Binding Scope	Description
verrazzano-admin	Cluster	Manage Verrazzano system components, clusters, and projects. Install/update Verrazzano.
verrazzano-monitor	Cluster	View/monitor Verrazzano system components, clusters, and projects.
verrazzano-project-admin	Namespace	Deploy/manage applications.
verrazzano-project-monitor	Namespace	View/monitor applications.

Kubernetes user roles

Verrazzano roles do not include permissions for Kubernetes itself. Instead, it relies on the default user roles provided by Kubernetes. This allows Verrazzano to easily grant the Kubernetes access appropriate to a Verrazzano role, without having to maintain a long list of fine-grained Kubernetes permissions in the Verrazzano roles.

The following table shows the default Kubernetes roles that are granted by default for each Verrazzano role.

Verrazzano Role	Kubernetes Role	Binding Scope
verrazzano-admin	admin	Cluster
verrazzano-monitor	view	Cluster
verrazzano-project-admin	admin	Namespace
verrazzano-project-monitor	view	Namespace

Default role bindings

Verrazzano creates role bindings for the system and for projects, binding Verrazzano ClusterRoles to one or more Kubernetes Subjects. By default, each role is bound to a Keycloak group, so all Keycloak users who are members of that group will be granted the role.

Also, Verrazzano creates role bindings for the corresponding Kubernetes user roles. The Kubernetes role appropriate for a given Verrazzano role is bound to the same set of Subjects as the corresponding Verrazzano role.

The default bindings can be overridden by specifying one or more Kubernetes Subjects to which the role should be bound. Any valid Subject can be specified (user, group, or service account), but two caveats should be kept in mind:

It’s generally better to grant a role to a group, rather than a specific user, so that roles can be granted (or withdrawn) by editing a user’s group memberships, rather than deleting a role binding and creating a new one.
If you do want to grant a role directly to a specific user, the user must be specified using its unique ID, not its user name. This is because the authentication proxy impersonates the sub (subject) field from the user’s token, which contains the ID. Keycloak user IDs are guaranteed to be unique, unlike user names.

Default system role bindings

Verrazzano creates role bindings for system users during installation. The default role bindings are listed as follows:

Role	Default Binding Subject
verrazzano-admin	group: verrazzano-admins
verrazzano-monitor	group: verrazzano-monitors

Default project role bindings

Verrazzano creates role bindings for project users at project creation or update. The default role bindings are listed as follows:

Role	Default Binding Subject
verrazzano-project-admin	group: verrazzano-project-<proj_name>-admins
verrazzano-project-monitor	group: verrazzano-project-<proj_name>-monitors

NOTE

The role bindings for project roles are created automatically, but the project-specific groups that they refer to are not automatically created. You must create those groups using the Keycloak console or API, or specify different binding subjects for the project.

Override default role bindings

You can override the default role bindings that are created for system and project roles.

Override system role bindings

To override the set of subjects that are bound to Verrazzano (and Kubernetes) roles during installation, add the Subjects to the Verrazzano CR you use to install Verrazzano, as shown in the following example:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  ...
  security:
    adminSubjects:
    - name: admin-group
      kind: Group
    monitorSubjects:
    - name: view-group
      kind: Group
  ...

You can specify multiple subjects for both admin and monitor roles. You can also specify a subject or subjects for one role, but not the other. If no subjects are specified for a role, then the default binding subjects will be used.

Override project role bindings

To override the set of subjects that are bound to Verrazzano (and Kubernetes) roles for a project, add the Subjects to the VerrazzanoProject CR for the project, as shown in the following example.

Note that the generated role bindings will be updated if you update the VerrazzanoProject CR and change the subjects specified for either role.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoProject
metadata:
  name: my-project
spec:
  ...
  security:
    projectAdminSubjects:
    - name: my-project-admin-group
      kind: Group
    projectMonitorSubjects:
    - name: my-project-view-group
      kind: Group
  ...

As with the system role bindings, you can specify multiple subjects for both project-admin and project-monitor roles. You can also specify a subject or subjects for one role, but not the other. If no subjects are specified for a role, then the default binding subjects will be used.

Docs: Lift-and-Shift Guide

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to move (“Lift-and-Shift”) an on-premises WebLogic Server domain to a cloud environment running Kubernetes using Verrazzano.

Overview

The Initial steps create a very simple on-premises domain that you will move to Kubernetes. The sample domain is the starting point for the lift and shift process; it contains one application (ToDo List) and one data source. First, you’ll configure the database and the WebLogic Server domain. Then, in Lift and Shift, you will move the domain to Kubernetes with Verrazzano. This guide does not include the setup of the networking that would be needed to access an on-premises database, nor does it document how to migrate a database to the cloud.

What you need

The Git command-line tool and access to GitHub
MySQL Database 8.x - a database server
WebLogic Server 12.2.1.4.0 - an application server; Note that all WebLogic Server installers are supported except the Quick Installer.
Maven - to build the application
WebLogic Deploy Tooling (WDT) - v1.9.15 or later, to convert the WebLogic Server domain to and from metadata
WebLogic Image Tool (WIT) - v1.9.13 or later, to build the Docker image

Initial steps

In the initial steps, you create a sample domain that represents your on-premises WebLogic Server domain.

Create a database using MySQL called `tododb`

Download the MySQL image from Docker Hub.
```
$ docker pull mysql:latest
```

Start the container database (and optionally mount a volume for data).

$ export MYSQL_USER=<your-mysql-username>
$ export MYSQL_PASSWORD=<your-mysql-password>
$ export MYSQL_ROOT_PASSWORD=<your-mysql-rootpassword>
$ docker run --name tododb \
  -p 3306:3306 \
  -e MYSQL_USER=$MYSQL_USER \
  -e MYSQL_PASSWORD=$MYSQL_PASSWORD \
  -e MYSQL_DATABASE=tododb \
  -e MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD \
  -d mysql:latest

Start a MySQL client to change the password algorithm to mysql_native_password.
- Assuming the database server is running, start a database CLI client.
```
$ docker exec \
   -it tododb mysql \
   -uroot \
   -p
```
- When prompted for the password, enter the password for the root user.
- After being connected, run the ALTER command at the MySQL prompt.
```
$ ALTER USER '<your-mysql-username>'@'%' identified with mysql_native_password by '<your-mysql-password>';
```

Create a WebLogic Server domain

If you do not have WebLogic Server 12.2.1.4.0 installed, install it now.
- Choose the GENERIC installer from WebLogic Server Downloads and follow the documented installation instructions.
- Be aware of these domain limitations:
  - There are two supported domain types, single server and single cluster.
  - Domains must use:
    - The default value AdminServer for AdminServerName.
    - WebLogic Server listen port for the Administration Server: 7001.
    - WebLogic Server listen port for the Managed Server: 8001.
    - Note that these are all standard WebLogic Server default values.
- Save the installer after you have finished; you will need it to build the Docker image.
- To make copying commands easier, define an environment variable for ORACLE_HOME that points to the directory where you installed WebLogic Server 12.2.1.4.0. For example:
```
$ export ORACLE_HOME=$HOME/Oracle/Middleware/Oracle_Home
```
Use the Oracle WebLogic Server Configuration Wizard to create a domain called tododomain.

NOTE: This example assumes that the on-premises WebLogic Server domain is on Linux.
- Launch $ORACLE_HOME/oracle_common/common/bin/config.sh.
- Select Create a new domain.
- Specify a Domain Location of <oracle home>/user_projects/domains/tododomain and click Next.
- Select the Basic WebLogic Server Domain [wlserver] template and click Next.
- Enter the password for the administrative user and click Next.
- Accept the defaults for Domain Mode and JDK, and click Next.
- Select Administration Server and click Next.
- Ensure that the server name is AdminServer and click Next.
- Click Create.
- After it has completed, click Next, then Finish.

To start the newly created domain, run the domain’s start script.

 $ $ORACLE_HOME/user_projects/domains/tododomain/bin/startWebLogic.sh

Access the Console of the newly started domain with your browser, for example, http://localhost:7001/console, and log in using the administrator credentials you specified.

Add a data source configuration to access the database

Using the WebLogic Server Administration Console, log in and add a data source configuration to access the MySQL database. During the data source configuration, you can accept the default values for most fields, but the following fields are required to match the application and database settings you used when you created the MySQL database.

In the left pane in the Console, expand Services and select Data Sources.
On the Summary of JDBC Data Sources page, click New and select Generic Data Source.
On the JDBC Data Sources page, enter or select the following information:
- Name: tododb
- JNDI Name: jdbc/ToDoDB
- Database Type: MySQL
Click Next and then click Next two more times.
On the Create a New JDBC Data Source page, enter the following information:
- Database Name: tododb
- Host name: localhost
- Database Port: 3306
- Database User Name: <your-mysql-username>
- Password: <your-mysql-password>
- Confirm Password: <your-mysql-password>
Click Next.
Select Test Configuration, and make sure you see “Connection Test Succeeded” in the Messages field of the Console.
Click Next.
On the Select Targets page, select AdminServer.
Click Finish to complete the configuration.

Build and deploy the application

Using Maven, build this project to produce todo.war.

NOTE: You should clone this repo outside of $ORACLE_HOME or copy the WAR file to another location, as WDT may ignore it during the model creation phase.
```
 $ git clone https://github.com/verrazzano/examples.git
 $ cd examples/todo-list/
 $ mvn clean package
```
Using the WebLogic Server Administration Console, deploy the ToDo List application.
- In the left pane in the Console, select Deployments and click Install.
- Use the navigation links or provide the file path to todo.war, typically <repo>/todo-list/target. For example, if you cloned the examples repository in your $HOME directory, the location should be $HOME/examples/examples/todo-list/target/todo.war.
- Click Next twice, then Finish.
NOTE: The remaining steps assume that the application context is todo.

Initialize the database

After the application is deployed and running in WebLogic Server, access the http://localhost:7001/todo/rest/items/init REST service to create the database table used by the application. In addition to creating the application table, the init service also will load four sample items into the table.

If you get an error here, go back to the Select Targets page in the WebLogic Server Administration Console and make sure that you selected AdminServer as the data source target.

Access the application

Access the application at http://localhost:7001/todo/index.html.

Add a few entries or delete some.
After verifying the application and database, you may shut down the local WebLogic Server domain.

Lift and Shift steps

The following steps will move the sample domain to Kubernetes with Verrazzano.

Create a WDT Model

If you have not already done so, download v1.9.15 or later of WebLogic Deploy Tooling (WDT) from GitHub.
Unzip the installer weblogic-deploy.zip file so that you can access bin/discoverDomain.sh.
To make copying commands easier, define an environment variable for WDT_HOME that points to the directory where you installed WebLogic Deploy Tooling.
```
 $ export WDT_HOME=/install/directory
```

For example, to get the latest version:

$ curl -OL https://github.com/oracle/weblogic-deploy-tooling/releases/latest/download/weblogic-deploy.zip
$ unzip  weblogic-deploy.zip
$ cd weblogic-deploy
$ export WDT_HOME=$(pwd)

To create a reusable model of the application and domain, use WDT to create a metadata model of the domain.

First, create an output directory to hold the generated scripts and models.

Then, run WDT discoverDomain.

$ mkdir v8o
$ $WDT_HOME/bin/discoverDomain.sh \
  -oracle_home $ORACLE_HOME \
  -domain_home /path/to/domain/dir \
  -model_file ./v8o/wdt-model.yaml \
  -archive_file ./v8o/wdt-archive.zip \
  -target vz \
  -output_dir v8o

You will find the following files in ./v8o:

create_k8s_secrets.sh - A helper script with kubectl commands to apply the Kubernetes secrets needed for this domain
vz-application.yaml - Verrazzano application configuration and component file
vz_variable.properties - A set of properties extracted from the WDT domain model
wdt-archive.zip - The WDT archive file containing the ToDo List application WAR file
wdt-model.yaml - The WDT model of the WebLogic Server domain

If you chose to skip the Access the application step and did not verify that the ToDo List application was deployed, then you should verify that you see the todo.war file inside the wdt-archive.zip file. If you do not see the WAR file, there was something wrong in your deployment of the application on WebLogic Server that will require additional troubleshooting in your domain.

Create a Docker image

At this point, the Verrazzano model is just a template for the real model. The WebLogic Image Tool will fill in the placeholders for you, or you can edit the model manually to set the image name and domain home directory.

If you have not already done so, download WebLogic Image Tool (WIT) from GitHub.
Unzip the installer imagetool.zip file so that you can access bin/imagetool.sh.
To make copying commands easier, define an environment variable for WIT_HOME that points to the directory where you installed WebLogic Image Tool.
```
 $ export WIT_HOME=/install/directory
```

For example, to get the latest WIT tool:

$ curl -OL https://github.com/oracle/weblogic-image-tool/releases/latest/download/imagetool.zip
$ unzip imagetool.zip
$ cd imagetool
$ export WIT_HOME=$(pwd)

You will need a Docker image to run your WebLogic Server domain in Kubernetes. To use WIT to create the Docker image, run imagetool create. Although WIT will download patches and PSUs for you, it does not yet download installers. Until then, you must download the WebLogic Server and Java Development Kit installer manually and provide their location to the imagetool cache addInstaller command.

# The directory created previously to hold the generated scripts and models.
$ cd v8o

$ $WIT_HOME/bin/imagetool.sh cache addInstaller \
  --path /path/to/installer/jdk-8u231-linux-x64.tar.gz \
  --type jdk \
  --version 8u231

# The installer file name may be slightly different depending on
# which version of the 12.2.1.4.0 installer that you downloaded, slim or generic.
$ $WIT_HOME/bin/imagetool.sh cache addInstaller \
  --path /path/to/installer/fmw_12.2.1.4.0_wls_Disk1_1of1.zip \
  --type wls \
  --version 12.2.1.4.0

$ $WIT_HOME/bin/imagetool.sh cache addInstaller \
  --path /path/to/installer/weblogic-deploy.zip \
  --type wdt \
  --version latest

# Paths for the files in this command assume that you are running it from the
# v8o directory created during the `discoverDomain` step.
$ $WIT_HOME/bin/imagetool.sh create \
  --tag your/repo/todo:1 \
  --version 12.2.1.4.0 \
  --jdkVersion 8u231 \
  --wdtModel ./wdt-model.yaml \
  --wdtArchive ./wdt-archive.zip \
  --wdtVariables ./vz_variable.properties \
  --resourceTemplates=./vz-application.yaml \
  --wdtModelOnly

The imagetool create command will have created a local Docker image and updated the Verrazzano model with the domain home and image name. Check your Docker images for the tag that you used in the create command using docker images from the Docker CLI.

If everything worked correctly, it is time to push that image to the container registry that Verrazzano will use to access the image from Kubernetes. You can use the Oracle Cloud Infrastructure Registry (OCIR) as your repository for this example, but most Docker compliant registries should work.

The variables in the vz-application.yaml resource template should be resolved with information from the image tool build.
Verify this by looking in the v8o/vz-application.yaml file to make sure that the image: {{{imageName}}} value has been set with the given --tag value.

Push the image to your repo.

NOTE: The image name must be the same as what is in the vz-application.yaml file under spec > workload > spec > image for the tododomain-domain component.

$ docker push your/repo/todo:1

Deploy to Verrazzano

After the application image has been created, there are several steps required to deploy the application into a Verrazzano environment.

These include:

Creating and labeling the tododomain namespace.
Creating the necessary secrets required by the ToDo List application.
Creating the Verrazzano components such as Service, Deployment, and ConfigMap required by the MySQL instance in the tododomain namespace.
Updating the vz-application.yaml file to enable the Verrazzano MySQL components in the ToDo List ApplicationConfiguration to deploy as Kubernetes objects.
Updating the vz-application.yaml file to use the Verrazzano MySQL deployment and (optionally) expose the WebLogic Server Administration Console.
Applying the vz-application.yaml file.

The following steps assume that you have a Kubernetes cluster and that Verrazzano is already installed in that cluster.

Label the namespace

Create the tododomain namespace, and add labels to allow the WebLogic Server Kubernetes Operator to manage it and enabled for Istio.

$ kubectl create namespace tododomain
$ kubectl label namespace tododomain verrazzano-managed=true istio-injection=enabled

Create the required secrets

If you haven’t already done so, edit and run the create_k8s_secrets.sh script to generate the Kubernetes secrets. WDT does not discover passwords from your existing domain. Before running the create secrets script, you will need to edit create_k8s_secrets.sh to set the passwords for the WebLogic Server domain and the data source. In this domain, there are a few passwords that you need to enter:

Administrator credentials
ToDo database credentials

For example:

# Update <admin-user> and <admin-password> for weblogic-credentials
$ create_paired_k8s_secret weblogic-credentials <your-WLS-username> <your-WLS-password>

# Update <user> and <password> for tododomain-jdbc-tododb
$ create_paired_k8s_secret jdbc-tododb <your-mysql-username> <your-mysql-password>

Then run the script:

$ sh ./create_k8s_secrets.sh

Verrazzano will need a credential to pull the image that you just created, so you need to create one more secret. The name for this credential can be changed in the vz-application.yaml file to anything you like, but it defaults to tododomain-registry-credentials.

Assuming that you leave the name tododomain-registry-credentials, you will need to run a kubectl create secret command similar to the following:

$ kubectl create secret docker-registry tododomain-registry-credentials \
  --docker-server=phx.ocir.io \
  --docker-email=your.name@example.com \
  --docker-username=tenancy/username \
  --docker-password='passwordForUsername' \
  --namespace=tododomain

Update the application configuration

Update the generated vz-application.yaml file for the todo application to:

Update the tododomain-configmap component to use the in-cluster MySQL service URL jdbc:mysql://mysql.tododomain.svc.cluster.local:3306/tododb to access the database.

        wdt_jdbc.yaml: |
          resources:
            JDBCSystemResource:
              'todo-ds':
                JdbcResource:
                  JDBCDriverParams:
                    # This is the URL of the database used by the WebLogic Server application
                    URL: "jdbc:mysql://mysql.tododomain.svc.cluster.local:3306/tododb"

Update the tododomain-appconf ApplicationConfiguration to enable Verrazzano MySQL components to be deployed as Kubernetes objects.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: tododomain-appconf
  namespace: tododomain
  annotations:
    version: v1.0.0
    description: "tododomain application configuration"
spec:
  components:
    - componentName: tododomain-domain
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
              scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            spec:
              rules:
                - paths:
                    # application todo
                    - path: "/todo"
                      pathType: Prefix
    - componentName: tododomain-configmap
    - componentName: todo-mysql-service
    - componentName: todo-mysql-deployment
    - componentName: todo-mysql-configmap

The file vz-application-modified.yaml is an example of a modified vz-application.yaml file. A diff of these two sample files is shown:

$ diff vz-application.yaml vz-application-modified.yaml
30a31,33
>     - componentName: todo-mysql-service
>     - componentName: todo-mysql-deployment
>     - componentName: todo-mysql-configmap
102c105
<                   URL: "jdbc:mysql://localhost:3306/tododb"
---
>                   URL: "jdbc:mysql://mysql.tododomain.svc.cluster.local:3306/tododb"

Create Verrazzano components for MySQL

As noted previously, moving a production environment to Verrazzano would require migrating the data as well. While data migration is beyond the scope of this guide, we will still need to include a MySQL instance to be deployed with the application in the Verrazzano environment.

To do so, first, we need to create the Verrazzano components for MySQL by applying the mysql-oam.yaml file in the tododomain namespace. The components will be deployed as Kubernetes objects when the ToDo List application is deployed by applying the vz-application.yaml file in the next step.

Download the mysql-oam.yaml file.
Then, apply the YAML file:

$ kubectl apply -f mysql-oam.yaml

# Expected response
component.core.oam.dev/todo-mysql-service created
component.core.oam.dev/todo-mysql-deployment created
component.core.oam.dev/todo-mysql-configmap created

$ kubectl get components -ntododomain

# Expected response
todo-mysql-configmap    ConfigMap       26s
todo-mysql-deployment   Deployment      26s
todo-mysql-service      Service         26s

Deploy the ToDo List application and MySQL instance.

Finally, run kubectl apply to apply the Verrazzano components and Verrazzano application configuration files to start your domain.

$ kubectl apply -f vz-application.yaml

This will:

Create the application Component resources for the ToDo List application.
Deploys the Verrazzano component resources as Kubernetes objects and creates the MySQL instance.
Create the application configuration resources that create the instance of the ToDo List application in the Verrazzano cluster.

Wait for the ToDo List example application to be ready.

$ kubectl wait pod \
    --for=condition=Ready tododomain-adminserver \
    -n tododomain

# Expected response
pod/tododomain-adminserver condition met

Verify that the pods are in the Running state:

$ kubectl get pod -n tododomain

# Sample output
NAME                     READY   STATUS    RESTARTS   AGE
mysql-55bb4c4565-c8zf5   1/1     Running   0          8m
tododomain-adminserver   4/4     Running   0          5m

Access the application from your browser

Get the generated host name for the application.

$ kubectl get gateways.networking.istio.io tododomain-tododomain-appconf-gw \
    -n tododomain \
    -o jsonpath={.spec.servers[0].hosts[0]}

# Sample output
tododomain-appconf.tododomain.11.22.33.44.nip.io

Initialize the database by accessing the init URL.

https://tododomain-appconf.tododomain.11.22.33.44.nip.io/todo/rest/items/init

Access the application.

https://tododomain-appconf.tododomain.11.22.33.44.nip.io/todo

Access the WebLogic Server Administration Console

Set up port forwarding.
```
$ kubectl port-forward pods/tododomain-adminserver 7001:7001 -n tododomain
```
NOTE: If you are using the Oracle Cloud Infrastructure Cloud Shell to run kubectl, in order to access the WebLogic Server Administration Console using port forwarding, you will need to run kubectl on another machine.
Access the WebLogic Server Administration Console from your browser.
```
http://localhost:7001/console
```

NOTE

It is recommended that the WebLogic Server Administration Console not be exposed publicly.

Docs: LoggingTrait Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The LoggingTrait custom resource contains the configuration for an additional logging sidecar with a custom image and Fluentd configuration file. Here is a sample ApplicationConfiguration that includes a LoggingTrait. To deploy an example application with this LoggingTrait, replace the ApplicationConfiguration of the ToDo-List example application with the following sample.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: todo-appconf
  namespace: todo-list
  annotations:
    version: v1.0.0
    description: "ToDo List example application"
spec:
  components:
    - componentName: todo-domain
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: LoggingTrait
            metadata:
              name: logging-trait-example
              namespace: todo-list
            spec:
              loggingImage: fluent/fleuntd-example-image # Replace with custom Fluentd Image
              loggingConfig: |-
                # Replace with Fluentd config file
                <match **>
                @type stdout
                </match>                
    - componentName: todo-jdbc-configmap
    - componentName: todo-mysql-configmap
    - componentName: todo-mysql-service
    - componentName: todo-mysql-deployment

In this sample configuration, the LoggingTrait logging-trait-example is set on the todo-domain application component and defines a logging sidecar with the given Fluentd image and configuration file. This sidecar will be attached to the component’s pod and will gather logs according to the given Fluentd configuration file. In order for the Fluentd DaemonSet to collect the custom logs, the Fluentd configuration file needs to direct the logs to STDOUT, as demonstrated in the previous example.

For example, when the ToDo-List example ApplicationConfiguration is successfully deployed with a LoggingTrait, the tododomain-adminserver pod will have a container named logging-stdout.

$ kubectl get pods tododomain-adminserver -n todo-list -o jsonpath='{.spec.containers[*].name}'
  ... logging-stdout ...

In this example, the logging-stdout container will run the image given in the LoggingTrait and a ConfigMap named logging-stdout-todo-domain-domain will be created with the custom Fluentd configuration file.

LoggingTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	LoggingTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	LoggingTraitSpec	The desired state of a logging trait.	Yes

LoggingTraitSpec

LoggingTraitSpec specifies the desired state of a logging trait.

Field	Type	Description	Required
`loggingConfig`	string	A string representation of the Fluentd configuration.	Yes
`loggingImage`	string	The name of the custom Fluentd image.	Yes

Docs: Metrics Template Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The Metrics Template CRD contains the metrics configuration for default Kubernetes workloads. Here is the default Metrics Template that Verrazzano installs.

apiVersion: app.verrazzano.io/v1alpha1
kind: MetricsTemplate
metadata:
  name: standard-k8s-metrics-template
  namespace: verrazzano-system
spec:
  workloadSelector:
    apiGroups: ["apps", ""]
    apiVersions: ["v1"]
    resources: ["deployment", "statefulset", "replicaset", "pod"]
  prometheusConfig:
    targetConfigMap:
      namespace: verrazzano-system
      name: vmi-system-prometheus-config
    scrapeConfigTemplate: |
      kubernetes_sd_configs:
        - namespaces:
            names:
            - {{`{{.workload.metadata.namespace}}`}}
          role: pod
      relabel_configs:
        - action: replace
          replacement: local
          source_labels: null
          target_label: verrazzano_cluster
        - action: keep
          regex: {{`{{index .workload.metadata.labels "app.verrazzano.io/workload"}}`}};true
          source_labels:
            - __meta_kubernetes_pod_label_app_verrazzano_io_workload
            - __meta_kubernetes_pod_annotation_prometheus_io_scrape
        - action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          source_labels:
            - __address__
            - __meta_kubernetes_pod_annotation_prometheus_io_port
          target_label: __address__
        - action: replace
          regex: (.*)
          source_labels:
            - __meta_kubernetes_pod_annotation_prometheus_io_path
          target_label: __metrics_path__
        - action: replace
          regex: (.*)
          replacement: $1
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod_name
        - action: labeldrop
          regex: (controller_revision_hash)
        - action: replace
          regex: .*/(.*)$
          replacement: $1
          source_labels:
            - name
          target_label: webapp
      {{`{{ if index .namespace.metadata.labels "istio-injection" }}`}}
      {{`{{ if eq (index .namespace.metadata.labels "istio-injection" ) "enabled" }}`}}
      scheme: https
      tls_config:
        ca_file: /etc/istio-certs/root-cert.pem
        cert_file: /etc/istio-certs/cert-chain.pem
        insecure_skip_verify: true
        key_file: /etc/istio-certs/key.pem
      {{`{{ end }}`}}
      {{`{{ end }}`}}

For more information on using the Metrics Template, see Metrics Template.

MetricsTemplate

Field	Type	Description	Required
`apiVersion`	string	`app.verrazzano.io/v1alpha1`	Yes
`kind`	string	MetricsTemplate	Yes
`metadata`	ObjectMeta	Refer to the Kubernetes API documentation for fields of metadata.	No
`spec`	MetricsTemplateSpec	The desired state of a metrics trait.	Yes

MetricsTemplateSpec

Field	Type	Description	Required
`workloadSelector`	WorkloadSelector	Selector for target workloads.	No
`prometheusConfig`	PrometheusConfig	Prometheus configuration details.	No

WorkloadSelector

Field	Type	Description	Required
`namespaceSelector`	LabelSelector	Scopes the template to a namespace.	No
`objectSelector`	LabelSelector	Scopes the template to a specific workload object.	No
`apiGroups`	[]string	Scopes the template to given API Groups.	No
`apiVersions`	[]string	Scopes the template to given API Versions.	No
`resources`	[]string	Scopes the template to given API Resources.	No

PrometheusConfig

Field	Type	Description	Required
`targetConfigMap`	TargetConfigMap	Identity of the ConfigMap to be updated with the scrape configuration specified in `scrapeConfigTemplate`.	Yes
`scrapeConfigTemplate`	string	Scrape configuration template to be added to the Prometheus configuration.	Yes

TargetConfigMap

Field	Type	Description	Required
`namespace`	string	Namespace of the ConfigMap to be updated with the scrape target configuration.	Yes
`name`	string	Name of the ConfigMap to be updated with the scrape target configuration.	Yes

Docs: MetricsTrait Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MetricsTrait custom resource contains the configuration information needed to enable metrics for an application component. Component workloads configured with a MetricsTrait are set up to emit metrics through an endpoint that are scraped by a given Prometheus deployment. Here is a sample ApplicationConfiguration that specifies a MetricsTrait. To deploy an example application that demonstrates a MetricsTrait, see Hello World Helidon.

Note that if an ApplicationConfiguration does not specify a MetricsTrait, then a default MetricsTrait will be generated with values appropriate for the workload type.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
                scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

In the sample configuration, a MetricsTrait is specified for the hello-helidon-component application component.

With the sample application configuration successfully deployed, you can query for metrics from the application component.

$ HOST=$(kubectl get ingress \
     -n verrazzano-system vmi-system-prometheus \
     -o jsonpath={.spec.rules[0].host})
$ echo $HOST

prometheus.vmi.system.default.<ip>.nip.io

$ VZPASS=$(kubectl get secret \
     --namespace verrazzano-system verrazzano \
     -o jsonpath={.data.password} | base64 \
     --decode; echo)
$ curl -sk \
    --user verrazzano:${VZPASS} \
    -X GET https://${HOST}/api/v1/query?query=vendor_requests_count_total

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"vendor_requests_count_total","app":"hello-helidon","app_oam_dev_component":"hello-helidon-component","app_oam_dev_name":"hello-helidon-appconf","app_oam_dev_resourceType":"WORKLOAD","app_oam_dev_revision":"hello-helidon-component-v1","containerizedworkload_oam_crossplane_io":"496df78f-ef8b-4753-97fd-d9218d2f38f1","job":"hello-helidon-appconf_default_helidon-logging_hello-helidon-component","namespace":"helidon-logging","pod_name":"hello-helidon-workload-b7d9d95d8-ht7gb","pod_template_hash":"b7d9d95d8"},"value":[1616535232.487,"4800"]}]}}

MetricsTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	MetricsTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	MetricsTraitSpec	The desired state of a metrics trait.	Yes

MetricsTraitSpec

MetricsTraitSpec specifies the desired state of a metrics trait.

Field	Type	Description	Required
`port`	integer	The HTTP port for the related metrics endpoint. Defaults to `8080`.	No
`ports`	[]PortSpec	The HTTP endpoints for the related metrics.	No
`path`	string	The HTTP path for the related metrics endpoint. Defaults to `/metrics`.	No
`secret`	string	The name of an opaque secret (for example, user name and password) within the workload’s namespace for metrics endpoint access.	No
`scraper`	string	The Prometheus deployment used to scrape the related metrics endpoints. Defaults to `verrazzano-system/vmi-system-prometheus-0`.	No

PortSpec

PortSpec defines an HTTP port and path combination.

Field	Type	Description	Required
`port`	integer	The HTTP port for the related metrics endpoint. Defaults to `8080`.	No
`path`	string	The HTTP path for the related metrics endpoint. Defaults to `/metrics`.	No

Docs: MultiClusterApplicationConfiguration Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterApplicationConfiguration custom resource is an envelope used to distribute core.oam.dev/v1alpha2/ApplicationConfiguration resources in a multicluster environment.

Here is a sample MultiClusterApplicationConfiguration that specifies an ApplicationConfiguration resource to create on the cluster named managed1. To deploy an example application that demonstrates a MultiClusterApplicationConfiguration, see Multicluster ToDo List.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterApplicationConfiguration
metadata:
  name: todo-appconf
  namespace: mc-todo-list
spec:
  template:
    metadata:
      annotations:
        version: v1.0.0
        description: "ToDo List example application"
    spec:
      components:
        - componentName: todo-domain
          traits:
            - trait:
                apiVersion: oam.verrazzano.io/v1alpha1
                kind: MetricsTrait
                spec:
                  scraper: verrazzano-system/vmi-system-prometheus-0
            - trait:
                apiVersion: oam.verrazzano.io/v1alpha1
                kind: IngressTrait
                spec:
                  rules:
                    - paths:
                        - path: "/todo"
                          pathType: Prefix
        - componentName: todo-jdbc-config
        - componentName: mysql-initdb-config
        - componentName: todo-mysql-service
        - componentName: todo-mysql-deployment
  placement:
    clusters:
      - name: managed1
  secrets:
    - tododomain-repo-credentials
    - tododomain-jdbc-tododb
    - tododomain-weblogic-credentials

MultiClusterApplicationConfiguration

A MultiClusterApplicationConfiguration is an envelope to create core.oam.dev/v1alpha2/ApplicationConfiguration resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterApplicationConfiguration	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterApplicationConfigurationSpec	The desired state of a `core.oam.dev/v1alpha2/ApplicationConfiguration` resource.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterApplicationConfigurationSpec

MultiClusterApplicationConfigurationSpec specifies the desired state of a core.oam.dev/v1alpha2/ApplicationConfiguration resource.

Field	Type	Description	Required
`template`	ApplicationConfigurationTemplate	The embedded `core.oam.dev/v1alpha2/ApplicationConfiguration` resource.	Yes
`placement`	Placement	Clusters in which the resource is to be placed.	Yes
`secrets`	string array	List of secrets used by the application. These secrets must be created in the application’s namespace before deploying a MultiClusterApplicationConfiguration resource.	No

ApplicationConfigurationTemplate

ApplicationConfigurationTemplate has the metadata and spec of the core.oam.dev/v1alpha2/ApplicationConfiguration resource.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	ApplicationConfigurationSpec	An instance of the `struct` ApplicationConfigurationSpec defined in core_types.go.	No

Docs: MultiClusterComponent Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterComponent custom resource is an envelope used to distribute core.oam.dev/v1alpha2/Component resources in a multicluster environment.

NOTE

Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterComponent custom resource not be used; instead directly use core.oam.dev/v1alpha2/Component resources in your application. See the example application, Multicluster ToDo List, which directly uses core.oam.dev/v1alpha2/Component resources.

Here is a sample MultiClusterComponent that specifies a OAM Component resource to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterComponent
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  template:
    spec:
      workload:
        apiVersion: oam.verrazzano.io/v1alpha1
        kind: VerrazzanoHelidonWorkload
        metadata:
          name: hello-helidon-workload
          namespace: hello-helidon
          labels:
            app: hello-helidon
        spec:
          deploymentTemplate:
            metadata:
              name: hello-helidon-deployment
            podSpec:
              containers:
                - name: hello-helidon-container
                  image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.12-1-20210409130027-707ecc4"
                  ports:
                    - containerPort: 8080
                      name: http
  placement:
    clusters:
      - name: managed1

MultiClusterComponent

A MultiClusterComponent is an envelope to create core.oam.dev/v1alpha2/Component resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterComponent	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterComponentSpec	The desired state of a `core.oam.dev/v1alpha2/Component` resource.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterComponentSpec

MultiClusterComponentSpec specifies the desired state of a core.oam.dev/v1alpha2/Component resource.

Field	Type	Description	Required
`template`	ComponentTemplate	The embedded `core.oam.dev/v1alpha2/Component` resource.	Yes
`placement`	Placement	Clusters in which the resource is to be placed.	Yes

ComponentTemplate

ComponentTemplate has the metadata and spec of the core.oam.dev/v1alpha2/Component resource.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	ComponentSpec	An instance of the `struct` ComponentSpec defined in core_types.go.	No

Docs: MultiClusterConfigMap Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterConfigMap custom resource is an envelope used to distribute Kubernetes ConfigMap resources in a multicluster environment.

NOTE

Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterConfigMap custom resource not be used; instead directly use core.oam.dev/v1alpha2/Component to define ConfigMap resources in your application. See the example application, Multicluster ToDo List, which uses core.oam.dev/v1alpha2/Component resources to define ConfigMaps.

Here is a sample MultiClusterConfigMap that specifies a Kubernetes ConfigMap to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterConfigMap
metadata:
  name: mymcconfigmap
  namespace: multiclustertest
spec:
  template:
    metadata:
      name: myconfigmap
      namespace: myns
    data:
      simple.key: "simplevalue"
  placement:
    clusters:
      - name: managed1

MultiClusterConfigMap

A MultiClusterConfigMap is an envelope to create Kubernetes ConfigMap resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterConfigMap	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterConfigMapSpec	The desired state of a Kubernetes ConfigMap.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterConfigMapSpec

MultiClusterConfigMapSpec specifies the desired state of a Kubernetes ConfigMap.

Field	Type	Description	Required
`template`	ConfigMapTemplate	The embedded Kubernetes ConfigMap.	Yes
`placement`	Placement	Clusters in which the ConfigMap is to be placed.	Yes

ConfigMapTemplate

ConfigMapTemplate has the metadata and spec of the Kubernetes ConfigMap.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`immutable`	*bool	Corresponds to the `immutable` field of the `struct` ConfigMap defined in types.go.	No
`data`	map[string]string	Corresponds to the `data` field of the `struct` ConfigMap defined in types.go.	No
`binaryData`	map[string][]byte	Corresponds to the `binaryData` field of the `struct` ConfigMap defined in types.go.	No

Docs: MultiClusterResourceStatus Subresource

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterResourceStatus subresource is shared by multicluster custom resources.

MultiClusterResourceStatus

MultiClusterResourceStatus specifies the status portion of a multicluster resource.

Field	Type	Description	Required
`conditions`	Condition array	The current state of a multicluster resource.	No
`state`	string	The state of the multicluster resource. State values are case-sensitive and formatted as follows: `Pending`: deployment to cluster is in progress `Succeeded`: deployment to cluster successfully completed `Failed`: deployment to cluster failed	No
`clusters`	ClusterLevelStatus array	Array of status information for each cluster.	No

Condition

Condition describes current state of a multicluster resource across all clusters.

Field	Type	Description	Required
`type`	string	The condition of the multicluster resource which can be checked with a `kubectl wait` command. Condition values are case-sensitive and formatted as follows: `DeployComplete`: deployment to all clusters completed successfully `DeployFailed`: deployment to all clusters failed	Yes
`status`	ConditionStatus	An instance of the type ConditionStatus that is defined in types.go.	Yes
`lastTransitionTime`	string	The last time the condition transitioned from one status to another.	No
`message`	string	A message with details about the last transition.	No

ClusterLevelStatus

ClusterLevelStatus describes the status of the multicluster resource on an individual cluster.

Field	Type	Description	Required
`name`	string	Name of the cluster.	Yes
`state`	string	The state of the multicluster resource. State values are case-sensitive and formatted as follows: `Pending`: deployment is in progress `Succeeded`: deployment successfully completed `Failed`: deployment failed	No
`message`	string	Message with details about the status in this cluster.	No
`lastUpdateTime`	string	The last time the resource state was updated.	Yes

Docs: MultiClusterSecret Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterSecret custom resource is an envelope used to distribute Kubernetes Secret resources in a multicluster environment.

NOTE

Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterSecret custom resource not be used; instead specify secrets in the MultiClusterApplicationConfiguration resource. See the example application, Multicluster ToDo List where secrets are specified in a MultiClusterApplicationConfiguration resource.

Here is a sample MultiClusterSecret that specifies a Kubernetes secret to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterSecret
metadata:
  name: mymcsecret
  namespace: multiclustertest
spec:
  template:
    data:
      username: dmVycmF6emFubw==
      password: dmVycmF6emFubw==
  spec:
  placement:
    clusters:
      - name: managed1

MultiClusterSecret

A MultiClusterSecret is an envelope to create Kubernetes Secret resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterSecret	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterSecretSpec	The desired state of a Kubernetes Secret.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterSecretSpec

MultiClusterSecretSpec specifies the desired state of a Kubernetes Secret.

Field	Type	Description	Required
`template`	SecretTemplate	The embedded Kubernetes Secret.	Yes
`placement`	Placement	Clusters in which the Secret is to be placed.	Yes

SecretTemplate

SecretTemplate has the metadata and spec of the Kubernetes Secret.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`data`	map[string][]byte	Corresponds to the `data` field of the `struct` Secret defined in types.go.	No
`stringData`	map[string]string	Corresponds to the `stringData` field of the `struct` Secret defined in types.go.	No
`type`	string	Corresponds to the `type` field of the `struct` Secret defined in types.go.	No

Docs: Network Traffic

Mon, 01 Jan 0001 00:00:00 +0000

Network traffic refers to the data flowing across the network. In the context of this document, it is useful to think of network traffic from two perspectives: traffic based on direction and traffic related to component types, system or applications. Traffic direction is either north-south traffic, which enters and leaves the cluster, or east-west traffic, which stays within the cluster.

First is a description of getting traffic into the cluster, then how traffic flows after it is in the cluster.

Ingress

Ingress is an overloaded term, so it needs to be understood in context. Sometimes the term means external access into the cluster, as in “ingress to the cluster.” The term also refers to the Kubernetes Ingress resource. In addition, it might be used to mean network ingress to a container in a Pod. Here, it’s used to refer to both general ingress into the cluster and the Kubernetes Ingress resource.

During installation, Verrazzano creates the necessary network resources to access both system components and applications. The following ingress and load balancers description is in the context of a Verrazzano installation.

LoadBalancer Services

To reach Pods from outside a cluster, an external IP address must be exposed using a LoadBalancer or NodePort service. Verrazzano creates two LoadBalancer services, one for system component traffic and another for application traffic. The specifics of how the service gets traffic into the cluster depends on the underlying Kubernetes platform. With Oracle OKE, creating a LoadBalancer type service will result in an Oracle Cloud Infrastructure load balancer being created and configured to load balance to a set of Pods.

Ingress for system components

To provide ingress to system components, Verrazzano installs a NGINX Ingress Controller, which includes a NGINX load balancer. Verrazzano also creates Kubernetes Ingress resources to configure ingress for each system component that requires ingress. An Ingress resource is used is to specify HTTP/HTTPS routes to Kubernetes services, along with an endpoint hostname and a TLS certificate. An Ingress by itself doesn’t do anything; it is just a resource. An ingress controller is needed to watch Ingress resources and reconcile them, configuring the underlying Kubernetes load balancer to handle the service routing. The NGINX Ingress Controller processes Ingress resources and configures NGINX with the ingress route information, and such.

The NGINX Ingress Controller is a LoadBalancer service, as seen here:

$ kubectl get service -n ingress-nginx

# Sample output
ingress-controller-ingress-nginx-controller           LoadBalancer

Using the OKE example, traffic entering the Oracle Cloud Infrastructure load balancer is routed to the NGINX load balancer, then routed from there to the Pods belonging to the services described in the Ingress.

Ingress for applications

Verrazzano also provides ingress into applications, but uses an Istio ingress gateway, which is an Envoy proxy, instead of NGINX. Istio has a Gateway resource that provides load balancer information, such as hosts, ports, and certificates for traffic coming into the mesh. For more information, see Istio Gateway. Just as an Ingress needs a corresponding Ingress controller, the same is true for the Gateway resource, where there is a corresponding Istio ingress gateway controller. However, unlike the Ingress, the Gateway resource doesn’t have service routing information. That is handled by the Istio VirtualService resource. The combination of Gateway and VirtualService is basically a superset of Ingress, because the combination provides more features than Ingress. In summary, the Istio ingress gateway provides ingress to the cluster using information from both the Gateway and VirtualService resources.

Because Verrazzano doesn’t create any applications during installations, there is no need to create a Gateway and VirtualService at that time. However, during installation, Verrazzano does create the Istio ingress gateway, which is a LoadBalancer service, along with the Istio egress gateway, which is a ClusterIP service.

$ kubectl get service -n istio-system

# Sample output
istio-ingressgateway   LoadBalancer

Again, referring to the OKE use case, this means that there will another Oracle Cloud Infrastructure load balancer created, routing traffic to the Istio ingress gateway Pod, for example, the Envoy proxy.

External DNS

When you install Verrazzano, you can optionally specify an external DNS for your domain. If you do that, Verrazzano will not only create the DNS records, using ExternalDNS, but also it will configure your host name in the Ingress resources. You can then use that host name to access the system components through the NGINX Ingress Controller.

System traffic

System traffic includes all traffic that enters and leaves system Pods.

North-south system traffic

North-south traffic includes all system traffic that enters or leaves a Kubernetes cluster.

Ingress

The following lists the Verrazzano system components which are accessed through the NGINX Ingress Controller from a client external to the cluster:

OpenSearch
Keycloak
OpenSearch Dashboards
Grafana
Prometheus
Rancher
Verrazzano Console
Verrazzano API

Egress

The following table shows Verrazzano system components that initiate requests to a destination outside the cluster.

Component	Destination	Description
cert-manager	Let’s Encrypt	Gets signed certificate.
ExternalDNS	External DNS	Creates and deletes DNS entries in an external DNS.
Fluentd	OpenSearch	Fluentd on the managed cluster calls OpenSearch on the admin cluster.
Prometheus	Prometheus	Prometheus on the admin cluster scrapes metrics from Prometheus on the managed cluster.
Rancher Agent	Rancher	Rancher agent on the managed cluster sends requests to Rancher on the admin cluster.
Verrazzano Authentication Proxy	Keycloak	Calls Keycloak for authentication, which includes redirects.
Verrazzano Platform Operator	Kubernetes API server	Multicluster agent on the managed cluster calls API server on the admin cluster.

East-west system traffic

The following tables show Verrazzano system components that send traffic to a destination inside the cluster, with the following exceptions:

Usage of CoreDNS: It can be assumed that any Pod in the cluster can access CoreDNS for name resolution.
Envoy to Istiod: The Envoy proxies all make requests to the Istio control plane to get dynamic configuration, and such. This includes both the gateways and the mesh sidecar proxies. That traffic is not shown.
Traffic within a component is not shown, for example, traffic between OpenSearch Pods.
Prometheus scraping traffic is shown in the second table.

Component	Destination	Description
cert-manager	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Fluentd	OpenSearch	Fluentd sends data to OpenSearch.
Grafana	Prometheus	UI for Prometheus data.
OpenSearch Dashboards	OpenSearch	UI for OpenSearch.
NGINX Ingress Controller	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Istio	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Rancher	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Authentication Proxy	Keycloak	Calls Keycloak for token authentication.
Verrazzano Authentication Proxy	VMI components	Access UIs for OpenSearch Dashboards, Grafana, and such.
Verrazzano Authentication Proxy	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Application Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Monitoring Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Platform Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Platform Operator	Rancher	Registers the managed cluster with Rancher.

Prometheus scraping traffic

This table shows Prometheus traffic for each system component scrape target.

Target	Description
cadvisor	Kubernetes metrics
OpenSearch	Envoy metrics
Grafana	Envoy metrics
Istiod	Istio control plane metrics
Istiod	Envoy metrics
Istio egress gateway	Envoy metrics
Istio ingress gateway	Envoy metrics
Keycloak	Envoy metrics
OpenSearch Dashboards	Envoy metrics
MySQL	Envoy metrics
NGINX Ingress Controller	Envoy metrics
NGINX Ingress Controller	NGINX metrics
NGINX default back end	Envoy metrics
Node exporter	Node metrics
Prometheus	Envoy metrics
Prometheus	Prometheus metrics
Verrazzano Console	Envoy metrics
Verrazzano API	Envoy metrics
WebLogic operator	Envoy metrics

Webhooks

Several of the system components are controllers, and some of those have webhooks. Webhooks are called by the Kubernetes API server on a component HTTPS port to validate or mutate API payloads before they reach the API server.

The following components use webhooks:

cert-manager
Coherence Operator
Istio
Rancher
Verrazzano Application Operator
Verrazzano Platform Operator

Application traffic

Application traffic includes all traffic to and from Verrazzano applications.

North-south application traffic

After Verrazzano is installed, you can deploy applications into the Istio mesh. When doing so, you will likely need ingress into the application. As previously mentioned, this can be done with Istio using the Gateway and VirtualService resources. Verrazzano will create those resources for you when you use an IngressTrait in your ApplicationConfiguration. The Istio ingress gateway created during installation will be shared by all applications in the mesh, and the Gateway resource is bound to the Istio ingress gateway that was created during installation. This is done by the selector field in the Gateway:

   selector:
     istio: ingressgateway

Verrazzano creates a Gateway/VirtualService pair for each IngressTrait. Following is an example of those two resources created by Verrazzano.

Here is the Gateway; in this case both the host name and certificate were generated by Verrazzano.

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: Gateway
  metadata:
   ...
    name: hello-helidon-hello-helidon-appconf-gw
    namespace: hello-helidon
  ...
  spec:
    selector:
      istio: ingressgateway
    servers:
    - hosts:
      - hello-helidon-appconf.hello-helidon.1.2.3.4.nip.io
      port:
        name: HTTPS
        number: 443
        protocol: HTTPS
      tls:
        credentialName: hello-helidon-hello-helidon-appconf-cert-secret
        mode: SIMPLE

Here is the VirtualService; notice that it refers back to the Gateway and that it contains the service routing information.

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
  ...
    name: hello-helidon-ingress-rule-0-vs
    namespace: hello-helidon
  spec:
    gateways:
    - hello-helidon-hello-helidon-appconf-gw
    hosts:
    - hello-helidon-appconf.hello-helidon.1.2.3.4.nip.io
    HTTP:
    - match:
      - uri:
          prefix: /greet
      route:
      - destination:
          host: hello-helidon
          port:
            number: 8080

East-west application traffic

To manage east-west traffic, each service in the mesh should be routed using a VirtualService and an optional DestinationRule. You can still send east-west traffic without either of these resources, but you won’t get any custom routing or load balancing. Verrazzano doesn’t configure east-west traffic. Consider bobbys-front-end in the Bob’s Books example at bobs-books-comp.yaml. When deploying Bob’s Books, a VirtualService is created for bobbys-front-end, because of the IngressTrait, but there are no VirtualServices for the other services in the application. When bobbys-front-end sends requests to bobbys-helidon-stock-application, this east-west traffic still goes to bobbys-helidon-stock-application through the Envoy sidecar proxies in the source and destination Pods, but there is no VirtualService representing bobbys-helidon-stock-application, where you could specify a canary deployment or custom load balancing. This is something you could configure manually, but it is not configured by Verrazzano.

Proxies

Verrazzano uses network proxies in multiple places. The two proxy products are Envoy and NGINX. The following table shows which proxies are used and in which Pod they run.

Usage	Proxy	Pod	Namespace	Description
System ingress	NGINX	`ingress-controller-ingress-nginx-controller-*`	`ingress-nginx`	Provides external access to Verrazzano system components.
Verrazzano authentication proxy	NGINX	`verrazzano-authproxy-*`	`verrazzano-system`	Verrazzano authentication proxy server for Kubernetes API and Single Sign-On (SSO).
Application ingress	Envoy	`istio-ingressgateway-*`	`istio-system`	Provides external access to Verrazzano applications.
Application egress	Envoy	`istio-egressgateway-*`	`istio-system`	Provides control of application egress traffic.
Istio mesh sidecar	Envoy	`ingress-controller-ingress-nginx-controller-*`	`ingress-nginx`	NGINX Ingress Controller in the Istio mesh.
Istio mesh sidecar	Envoy	`ingress-controller-ingress-nginx-defaultbackend-*`	`ingress-nginx`	NGINX default backend in the Istio mesh.
Istio mesh sidecar	Envoy	`fluentd-*`	`verrazzano-system`	Fluentd in the Istio mesh.
Istio mesh sidecar	Envoy	`keycloak-*`	`keycloak`	Keycloak in the Istio mesh.
Istio mesh sidecar	Envoy	`mysql-*`	`keycloak`	MySQL used by Keycloak in the Istio mesh.
Istio mesh sidecar	Envoy	`verrazzano-api-*`	`verrazzano-system`	Verrazzano API in the Istio mesh.
Istio mesh sidecar	Envoy	`verrazzano-console-*`	`verrazzano-system`	Verrazzano Console in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-master-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-data-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-ingest-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-kibana-*`	`verrazzano-system`	OpenSearch Dashboards in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-prometheus-*`	`verrazzano-system`	Prometheus in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-grafana-*`	`verrazzano-system`	Grafana in the Istio mesh.
Istio mesh sidecar	Envoy	`weblogic-operator-*`	`verrazzano-system`	WebLogic operator in the Istio mesh.

Multicluster

Some Verrazzano components send traffic between Kubernetes clusters. Those components are the Verrazzano agent, Verrazzano authentication proxy, and Prometheus.

Multicluster egress

The following table shows Verrazzano system components that initiate requests between the admin and managed clusters. All of these requests go through the NGINX Ingress Controller on the respective destination cluster.

Source Cluster	Source Component	Destination Cluster	Destination Component	Description
Admin	Prometheus	Managed	Prometheus	Scapes metrics on managed clusters.
Admin	Verrazzano Console	Managed	Verrazzano Authentication Proxy	Admin cluster proxy sends Kubernetes API requests to managed cluster proxy.
Managed	Fluentd	Admin	OpenSearch	Fluentd sends logs to OpenSearch.
Managed	Rancher Agent	Admin	Rancher	Rancher Agent sends requests Rancher.
Managed	Verrazzano Authentication Proxy	Admin	Keycloak	Proxy sends requests to Keycloak.
Managed	Verrazzano Agent	Admin	Kubernetes API server	Agent, in the platform operator, sends requests Kubernetes API server.

Verrazzano agent

In the multicluster topology, the Verrazzano platform operator has an agent thread running on the managed cluster that sends requests to the Kubernetes API server on the admin cluster. The URL for the admin cluster Kubernetes API server is registered on the managed cluster by the user.

Verrazzano authentication proxy

In a multicluster topology, the Verrazzano authentication proxy runs on both the admin and managed clusters.
On the admin cluster, the authentication proxy connects to in-cluster Keycloak, using the Keycloak Service. On the managed cluster, the authentication proxy connects to Keycloak on the admin cluster through the NGINX Ingress Controller running on the admin cluster.

For Single Sign-On (SSO), the authentication proxy also needs to send requests to Keycloak, either in-cluster or through the cluster ingress. When a request comes into the authentication proxy without an authentication header, the proxy sends a request to Keycloak through the NGINX Ingress Controller, so the request exits the cluster. Otherwise, if the authentication proxy is on the admin cluster, then the request is sent directly to Keycloak within the cluster. If the authentication proxy is on the managed cluster, then it must send requests to Keycloak on the admin cluster.

Prometheus

A single Prometheus service in the cluster, scrapes metrics from Pods in system components and applications. It also scrapes Pods in the Istio mesh using HTTPS, and outside the mesh using HTTP. In the multicluster case, the Prometheus on the admin cluster, scrapes metrics from Prometheus on the managed cluster, through the NGINX Ingress Controller on the managed cluster.

Docs: Oracle Cloud Infrastructure Logging Service

Mon, 01 Jan 0001 00:00:00 +0000

The Oracle Cloud Infrastructure Logging service is a highly scalable and fully managed single pane of glass for all the logs in your tenancy. You can configure Verrazzano to send logs to Oracle Cloud Infrastructure Logging instead of OpenSearch. For general information, see Oracle Cloud Infrastructure Logging Overview.

Set up custom logs

Verrazzano can send its logs to Oracle Cloud Infrastructure custom logs. You will need to provide two Oracle Cloud Infrastructure Log identifiers in your Verrazzano installation resource: one for Verrazzano system logs and one for application logs. Follow the steps in Creating Custom Logs to create two custom logs. Do not create an agent configuration when creating a custom log, otherwise the log records will be duplicated.

Configure credentials

The Fluentd plug-in included with Verrazzano will use Oracle Cloud Infrastructure instance principal authentication by default. Optionally, you can configure Verrazzano with a user API signing key. API signing key authentication is required to send logs to Oracle Cloud Infrastructure Logging if the cluster is running outside of Oracle Cloud Infrastructure.

Create a dynamic group that includes the compute instances in your cluster’s node pools and assign the appropriate policy, so that the dynamic group is allowed to send log entries to the custom logs you created earlier. Pay close attention to the required permissions.

If the dynamic group and policy are configured incorrectly, then Fluentd will fail to send logs to Oracle Cloud Infrastructure Logging.

If you do not already have an API signing key, then see Required Keys and OCIDs in the Oracle Cloud Infrastructure documentation. You need to create an Oracle Cloud Infrastructure configuration file with the credential details and then use that configuration file to create a secret.

The following requirements must be met for Fluentd Oracle Cloud Infrastructure Logging to work:

The profile name in the Oracle Cloud Infrastructure configuration file must be DEFAULT.
The key_file path in the Oracle Cloud Infrastructure configuration file must be /root/.oci/key. The actual key file does not need to be in that location, because you will be providing the actual key file location in a secret.
The user associated with the API key must have the appropriate Oracle Cloud Infrastructure Identity and Access Management (IAM) policy in place to allow the Fluentd plug-in to send logs to Oracle Cloud Infrastructure. See Details for Logging in the Oracle Cloud Infrastructure documentation for the IAM policies used by the Oracle Cloud Infrastructure Logging service.

After the Verrazzano platform operator has been installed, create an opaque secret in the verrazzano-install namespace from the Oracle Cloud Infrastructure configuration and private key files. The key for the configuration file must be config and the key for the private key file data must be key.

Here is an example kubectl command that will create the secret.

$ kubectl create secret generic oci-fluentd -n verrazzano-install \
      --from-file=config=/home/myuser/oci_config --from-file=key=/home/myuser/keys/oci_api.pem

The secret should look something like this.

apiVersion: v1
data:
  config: W0RFRkFVTFRdCnVzZXI9b2NpZDEudXN...
  key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS...
kind: Secret
metadata:
  name: oci-fluentd
  namespace: verrazzano-install
type: Opaque

For convenience, there is a helper script available here that you can point at an existing Oracle Cloud Infrastructure configuration file and it will create the secret for you. The script allows you to override the default configuration file location, profile name, and the name of the secret.

Install Verrazzano

Oracle Cloud Infrastructure Logging is enabled in your cluster when installing Verrazzano. The Verrazzano installation custom resource has fields for specifying two custom logs: one for system logs and one for application logs. Here is an example Verrazzano installation YAML file for each type of credential. Note that the API references Kibana, upcoming releases will use OpenSearch Dashboards in the public API.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: vz-oci-logging
spec:
  profile: dev
  components:
    fluentd:
      enabled: true
      oci:
        systemLogId: ocid1.log.oc1.iad.system.example
        defaultAppLogId: ocid1.log.oc1.iad.app.example
    elasticsearch:
      enabled: false
    kibana:
      enabled: false

When using user API credentials, you need to configure the name of the secret in the Verrazzano custom resource, under the Oracle Cloud Infrastructure section of the Fluentd component settings. Your YAML file should look something like this.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: vz-oci-logging
spec:
  profile: dev
  components:
    fluentd:
      enabled: true
      oci:
        systemLogId: ocid1.log.oc1.iad.system.example
        defaultAppLogId: ocid1.log.oc1.iad.app.example
        apiSecret: oci-fluentd
    elasticsearch:
      enabled: false
    kibana:
      enabled: false

The apiSecret value must match the secret you created earlier when configuring the user API credentials.

Override the default log objects

You can override the Oracle Cloud Infrastructure Log object on an individual namespace. To specify a log identifier on a namespace, add an annotation named verrazzano.io/oci-log-id to the namespace. The value of the annotation is the Oracle Cloud Infrastructure Log object identifier.

Here is an example namespace.

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    verrazzano.io/oci-log-id: ocid1.log.oc1.iad.ns.app.example
  creationTimestamp: "2022-01-14T15:09:19Z"
  labels:
    istio-injection: enabled
    verrazzano-managed: "true"
  name: example
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Note that if you add and subsequently remove the annotation, then the logs will revert to the default Oracle Cloud Infrastructure Log object specified in the Verrazzano custom resource.

Search logs

To search Verrazzano logs, you can use the Oracle Cloud Infrastructure Console, Oracle Cloud Infrastructure CLI, or Oracle Cloud Infrastructure SDK.

For example, using the Oracle Cloud Infrastructure CLI to search the system logs for records emitted by the verrazzano-application-operator container.

$ oci logging-search search-logs --search-query=\
     "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \
     where \"data\".\"kubernetes.container_name\" = 'verrazzano-application-operator' | sort by datetime desc" \
     --time-start 2021-12-07 --time-end 2021-12-17

Search for all application log records in the springboot namespace.

$ oci logging-search search-logs --search-query=\
     "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \
     where \"data\".\"kubernetes.namespace_name\" = 'springboot' | sort by datetime desc" \
     --time-start 2021-12-07 --time-end 2021-12-17

For more information on searching logs, see the Logging Query Language Specification.

Troubleshooting

If you are not able to view Verrazzano logs in Oracle Cloud Infrastructure Logging, then check the Fluentd container logs in the cluster to see if there are errors.

$ kubectl logs -n verrazzano-system -l app=fluentd --tail=-1

If you see not authorized error messages, then there is likely a problem with the Oracle Cloud Infrastructure Dynamic Group or IAM policy that is preventing the Fluentd plug-in from communicating with the Oracle Cloud Infrastructure API.

To ensure the appropriate permissions are in place, review the Oracle Cloud Infrastructure Logging required permissions documentation.

Docs: Placement Subresource

Mon, 01 Jan 0001 00:00:00 +0000

The Placement subresource is shared by multicluster custom resources.

Placement

Placement contains the name of each cluster where this resource will be located.

Field	Type	Description	Required
`clusters`	Cluster array	An array of cluster locations.	Yes

Cluster

Cluster contains the name of a single cluster.

Field	Type	Description	Required
`cluster`	string	The name of a cluster.	Yes

Docs: Verrazzano Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano custom resource contains the configuration information for an installation. Here is a sample Verrazzano custom resource file that uses Oracle Cloud Infrastructure DNS. See other examples here.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  environmentName: env
  profile: prod
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: emailAddress@example.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: dnsZoneCompartmentOcid
        dnsZoneOCID: dnsZoneOcid
        dnsZoneName: my.dns.zone.name
    ingress:
      type: LoadBalancer

VerrazzanoSpec

Field	Type	Description	Required
`environmentName`	string	Name of the installation. This name is part of the endpoint access URLs that are generated. The default value is `default`.	No
`profile`	string	The installation profile to select. Valid values are `prod` (production), `dev` (development), and `managed-cluster`. The default is `prod`.	No
`version`	string	The version to install. Valid versions can be found here. Defaults to the current version supported by the Verrazzano platform operator.	No
`components`	Components	The Verrazzano components.	No
`defaultVolumeSource`	VolumeSource	Defines the type of volume to be used for persistence for all components unless overridden, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of an existing `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.	No
`volumeClaimSpecTemplates`	VolumeClaimSpecTemplate	Defines a named set of PVC configurations that can be referenced from components to configure persistent volumes.	No

VolumeClaimSpecTemplate

Field	Type	Description	Required
`metadata`	ObjectMeta	Metadata about the PersistentVolumeClaimSpec template.	No
`spec`	PersistentVolumeClaimSpec	A `PersistentVolumeClaimSpec` template that can be referenced by a Component to override its default storage settings for a profile. At present, only a subset of the `resources.requests` object are honored depending on the component.	No

Components

Field	Type	Description	Required
`authProxy`	AuthProxyComponent	The AuthProxy component configuration.	No
`certManager`	CertManagerComponent	The cert-manager component configuration.	No
`dns`	DNSComponent	The DNS component configuration.	No
`ingress`	IngressComponent	The ingress component configuration.	No
`istio`	IstioComponent	The Istio component configuration.	No
`fluentd`	FluentdComponent	The Fluentd component configuration.	No
`jaegerOperator`	JaegerOperatorComponent	The Jaeger Operator component configuration.	No
`keycloak`	KeycloakComponent	The Keycloak component configuration.	No
`elasticsearch`	OpenSearchComponent	The OpenSearch component configuration.	No
`prometheus`	PrometheusComponent	The Prometheus component configuration.	No
`kibana`	OpenSearchDashboardsComponent	The OpenSearch Dashboards component configuration.	No
`grafana`	GrafanaComponent	The Grafana component configuration.	No
`kiali`	KialiComponent	The Kiali component configuration.	No
`prometheusOperator`	PrometheusOperatorComponent	The Prometheus Operator component configuration.	No
`prometheusAdapter`	PrometheusAdapterComponent	The Prometheus Adapter component configuration.	No
`kubeStateMetrics`	KubeStateMetricsComponent	The kube-state-metrics component configuration.	No

AuthProxy Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then AuthProxy will be installed.	No
`kubernetes`	AuthProxyKubernetes	The Kubernetes resources than can be configured for AuthProxy.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

AuthProxy Kubernetes Configuration

Field	Type	Description	Required
`replicas`	uint32	The number of pods to replicate.	No
`affinity`	Affinity	A Kubernetes affinity definition.	No

CertManager Component

Field	Type	Description	Required
`certificate`	Certificate	The certificate configuration.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Certificate

Field	Type	Description	Required
`acme`	Acme	The ACME configuration. Either `acme` or `ca` must be specified.	No
`ca`	CertificateAuthority	The certificate authority configuration. Either `acme` or `ca` must be specified.	No

Acme

Field	Type	Description	Required
`provider`	string	Name of the Acme provider.	Yes
`emailAddress`	string	Email address of the user.	Yes

CertificateAuthority

Field	Type	Description	Required
`secretName`	string	The secret name.	Yes
`clusterResourceNamespace`	string	The secrete namespace.	Yes

DNS Component

Field	Type	Description	Required
`wildcard`	DNS-Wilcard	Wildcard DNS configuration. This is the default with a domain of `nip.io`.	No
`oci`	DNS-OCI	Oracle Cloud Infrastructure DNS configuration.	No
`external`	DNS-External	External DNS configuration.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

DNS Wildcard

Field	Type	Description	Required
`domain`	string	The type of wildcard DNS domain. For example, `nip.io`, `sslip.io`, and such.	Yes

DNS Oracle Cloud Infrastructure

Field	Type	Description	Required
`ociConfigSecret`	string	Name of the Oracle Cloud Infrastructure configuration secret. Generate a secret based on the Oracle Cloud Infrastructure configuration profile you want to use. You can specify a profile other than DEFAULT and specify the secret name. See instructions by running `./install/create_oci_config_secret.sh`.	Yes
`dnsZoneCompartmentOCID`	string	The Oracle Cloud Infrastructure DNS compartment OCID.	Yes
`dnsZoneOCID`	string	The Oracle Cloud Infrastructure DNS zone OCID.	Yes
`dnsZoneName`	string	Name of Oracle Cloud Infrastructure DNS zone.	Yes
`dnsScope`	string	Scope of the Oracle Cloud Infrastructure DNS zone (`PRIVATE`, `GLOBAL`). If not specified, then defaults to `GLOBAL`.	No

DNS External

Field	Type	Description	Required
`suffix`	string	The suffix for DNS names.	Yes

Ingress Component

Field	Type	Description	Required
`type`	string	The ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`. If the ingress type is `NodePort`, a valid and accessible IP address must be specified using the `controller.service.externalIPs` key in NGINXInstallArgs. For sample usage, see External Load Balancers.	No
`nginxInstallArgs`	NGINXInstallArgs list	A list of values to use during NGINX installation.	No
`ports`	PortConfig list	The list port configurations used by the ingress.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

NGINX Install Args

Name	Type	ValueType	Description	Required
`controller.service.externalIPs`	NameValue	string list	The external IP address used by the NGINX Ingress Controller.	No
`controller.service.externalTrafficPolicy`	NameValue	string	Preserves the client source IP address. See Bare-metal considerations.	No
`controller.service.annotations.*`	NameValue	string	Annotations used for NGINX Ingress Controller. For sample usage, see Customize Ingress.	No
`controller.autoscaling.enabled`	NameValue	Boolean	If true, then enable horizonal pod autoscaler. Default `false`.	No
`controller.autoscaling.minReplicas`	NameValue	string	Minimum replicas used for autoscaling. Default `1`.	No

Port Config

Field	Type	Description	Required
`name`	string	The port name.	No
`port`	string	The port value.	Yes
`targetPort`	string	The target port value. The default is same as the port value.	Yes
`protocol`	string	The protocol used by the port. `TCP` is the default.	No
`nodePort`	string	The `nodePort` value.	No

Name Value

Field	Type	Description	Required
`name`	string	The name of a Helm override for a Verrazzano component chart, specified with a `—set` flag on the Helm command line, for example, `helm install --set name=value`. For more information about chart overrides, see Customize Ingress.	Yes
`value`	string	The value of a Helm override for a Verrazzano component chart, specified with a `—set` flag on the Helm command line, for example, `helm install --set name=value`. Either `value` or `valueList` must be specified. For more information about chart overrides, see Customize Ingress.	No
`valueList`	string list	The list of Helm override values for a Verrazzano component, each specified with a `—set` flag on the Helm command line, for example, `helm install --set name[0]=<first element of valueList> —set name[1]=<second element of valueList>`. Either `value` or `valueList` must be specified. For more information about chart overrides, see Customize Ingress.	No
`setString`	Boolean	Specifies if the argument requires the Helm `--set-string` command-line flag to override a chart value, for example, `helm install --set-string name=value`.	No

Istio Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Istio will be installed.	No
`istioIngress`	IstioIngress	The Istio ingress gateway configuration.	No
`istioEgress`	IstioEgress	The Istio egress gateway configuration.	No
`istioInstallArgs`	IstioInstallArgs list	A list of values to use during Istio installation. Each argument is specified as either a `name/value` or `name/valueList` pair.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for default IstioOperator. Lower Overrides have precedence over the ones above them. You can find all possible values here. Passing through an invalid IstioOperator resource will result in an error.	No

Istio Ingress Configuration

Field	Type	Description	Required
`type`	string	The Istio ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`. If the Istio ingress type is `NodePort`, a valid and accessible IP address must be specified using the `gateways.istio-ingressgateway.externalIPs` key in IstioInstallArgs. For sample usage, see External Load Balancers.	No
`ports`	PortConfig list	The list port configurations used by the Istio ingress.	No
`kubernetes`	IstioKubernetes	The Kubernetes resources than can be configured for an Istio ingress gateway.	No

Istio Egress Configuration

Field	Type	Description	Required
`kubernetes`	IstioKubernetes	The Kubernetes resources than can be configured for an Istio egress gateway.	No

Istio Kubernetes Configuration

Field	Type	Description	Required
`replicas`	uint32	The number of pods to replicate.	No
`affinity`	Affinity	A Kubernetes affinity definition.	No

Istio Install Args

Name	Type	ValueType	Description	Required
`gateways.istio-ingressgateway.externalIPs`	NameValue	string list	The external IP address used by the Istio ingress gateway.	No
`gateways.istio-ingressgateway.serviceAnnotations.*`	NameValue	string	Annotations used for the Istio ingress gateway. For sample usage, see Customize Ingress.	No
`meshConfig.enableTracing`	NameValue	string	If `"true"`, Istio will export tracing when Jaeger is installed. Defaults to `"false"`.	No
`meshConfig.defaultConfig.tracing.sampling`	NameValue	string	Sampling rate for Istio tracing. Defaults to `"1"`, meaning a 1% sampling rate.	No

Fluentd Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Fluentd will be installed.	No
`extraVolumeMounts`	ExtraVolumeMount list	A list of host path volume mounts in addition to `/var/log` into the Fluentd DaemonSet. The Fluentd component collects log files in the `/var/log/containers` directory of Kubernetes worker nodes. The `/var/log/containers` directory may contain symbolic links to files located outside the `/var/log` directory. If the host path directory containing the log files is located outside of `/var/log`, the Fluentd DaemonSet must have the volume mount of that directory to collect the logs.	No
`elasticsearchURL`	string	The target OpenSearch URLs. Specify this option in this format. The default `http://vmi-system-es-ingest-oidc:8775` is the VMI OpenSearch URL.	No
`elasticsearchSecret`	string	The secret containing the credentials for connecting to OpenSearch. This secret needs to be created in the `verrazzano-install` namespace prior to creating the Verrazzano custom resource. Specify the OpenSearch login credentials in the `username` and `password` fields in this secret. Specify the CA for verifying the OpenSearch certificate in the `ca-bundle` field, if applicable. The default `verrazzano` is the secret for connecting to the VMI OpenSearch.	No
`oci`	OCILoggingConfiguration	The Oracle Cloud Infrastructure Logging configuration.	No

Jaeger Operator Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Jaeger Operator will be installed.	No

Extra Volume Mount

Field	Type	Description	Required
`source`	string	The source host path.	Yes
`destination`	string	The destination path on the Fluentd Container, defaults to the `source` host path.	No
`readOnly`	Boolean	Specifies if the volume mount is read-only, defaults to `true`.	No

Oracle Cloud Infrastructure Logging Configuration

Field	Type	Description	Required
`systemLogId`	string	The OCID of the Oracle Cloud Infrastructure Log that will collect system logs.	Yes
`defaultAppLogId`	string	The OCID of the Oracle Cloud Infrastructure Log that will collect application logs.	Yes
`apiSecret`	string	The name of the secret containing the Oracle Cloud Infrastructure API configuration and private key.	No

Keycloak Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Keycloak will be installed.	No
`mysql`	MySQLComponent	Contains the MySQL component configuration needed for Keycloak.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

MySQL Component

Field	Type	Description	Required
`volumeSource`	VolumeSource	Defines the type of volume to be used for persistence for Keycloak/MySQL, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of a `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

OpenSearch Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then OpenSearch will be installed.	No
`installArgs`	OpenSearchInstallArgs list	A list of values to use during OpenSearch installation. Each argument is specified as either a `name/value` or `name/valueList` pair. For sample usage, see Customize OpenSearch.	No
`policies`	Policy list	A list of Index State Management policies to enable on OpenSearch.	No
`nodes`	Node list	A list of OpenSearch node groups.	No

OpenSearch Node Groups

Field	Type	Description	Required
`name`	string	Name of the node group.	Yes
`replicas`	integer	Node group replica count.	No
`roles`	list	Role(s) that nodes in the group will assume. May be `master`, `data`, and/or `ingest`.	Yes
`storage`	Storage	Storage settings for the node group.	No
`resources`	Resources	Kubernetes container resources for nodes in the node group.	No

OpenSearch Node Group Storage

Field	Type	Description	Required
`size`	string	Node group storage size expressed as a Quantity.	Yes

OpenSearch Index Management Policies

Field	Type	Description	Required
`policyName`	string	Name of the Index State Management policy.	Yes
`indexPattern`	string	An Index Pattern is an index name or pattern like `my-index-*`. If an index matches the pattern, the associated policy will attach to the index.	Yes
`minIndexAge`	Time	Amount of time until a managed index is deleted. Default is seven days (`7d`).	No
`rollover`	Rollover	Index rollover settings.	No

OpenSearch Install Args

To configure OpenSearch, instead of using install args, Oracle recommends that you use OpenSearch Node Groups.

Name	Type	ValueType	Description	Required
`nodes.master.replicas`	NameValue	string	The number of master node replicas.	No
`nodes.master.requests.memory`	NameValue	string	The master node memory request amount expressed as a Quantity.	No
`nodes.master.requests.storage`	NameValue	string	The master storage request amount expressed as a Quantity.	No
`nodes.ingest.replicas`	NameValue	string	The number of ingest node replicas.	No
`nodes.ingest.requests.memory`	NameValue	string	The ingest node memory request amount expressed as a Quantity.	No
`nodes.data.replicas`	NameValue	string	The number of data node replicas.	No
`nodes.data.requests.memory`	NameValue	string	The data node memory request amount expressed as a Quantity.	No
`nodes.data.requests.storage`	NameValue	string	The data storage request amount expressed as a Quantity.	No

OpenSearch Index Management Rollover

Field	Type	Description	Required
`minIndexAge`	Time	Amount of time until a managed index is rolled over. Default is 1 day (`1d`).	No
`minSize`	Bytes	The size at which a managed index is rolled over.	No
`minDocCount`	uint32	Amount of documents in a managed index that triggers a rollover.	No

OpenSearch Dashboards Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then OpenSearch Dashboards will be installed.	No

Prometheus Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Prometheus will be installed.	No

Grafana Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Grafana will be installed.	No

Kiali Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Kiali will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Prometheus Operator Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then the Prometheus Operator will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Prometheus Adapter Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then the Prometheus Adapter will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Kube State Metrics Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then kube-state-metrics will be installed.	No
`monitorChanges`	Boolean	If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.	No
`overrides`	Overrides list	List of Overrides for the default `values.yaml` file for the component Helm chart. Lower Overrides have precedence over the ones above them. You can find all possible values here and invalid values will be ignored.	No

Overrides

Field	Type	Description	Required
`configMapRef`	ConfigMapKeySelector	Selector for ConfigMap containing override data.	No
`secretRef`	SecretKeySelector	Selector for Secret containing override data.	No
`values`	JSON	Configure overrides using inline YAML.	No

Docs: Verrazzano Workload Custom Resource Definitions

Mon, 01 Jan 0001 00:00:00 +0000

VerrazzanoCoherenceWorkload

The VerrazzanoCoherenceWorkload custom resource contains the configuration information for a Coherence workload within Verrazzano. Here is a sample component that specifies a VerrazzanoCoherenceWorkload. To deploy an example application that demonstrates this workload type, see Sock Shop.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: carts
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: carts-coh
        spec:
          cluster: SockShop
          role: Carts
          replicas: 1
          image: ghcr.io/helidon-sockshop/carts-coherence:2.2.0
          imagePullPolicy: Always
          application:
            type: helidon
          jvm:
            args:
              - "-Dcoherence.k8s.operator.health.wait.dcs=false"
              - "-Dcoherence.metrics.legacy.names=false"
            memory:
              heapSize: 2g
          coherence:
            logLevel: 9
          ports:
            - name: http
              port: 7001
              service:
                name: carts
                port: 80
              serviceMonitor:
                enabled: true
            - name: metrics
              port: 7001
              serviceMonitor:
                enabled: true

VerrazzanoCoherenceWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoCoherenceWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoCoherenceWorkloadSpec	The desired state of a Verrazzano Coherence workload.	Yes

VerrazzanoCoherenceWorkloadSpec

VerrazzanoCoherenceWorkloadSpec specifies the desired state of a Verrazzano Coherence workload.

Field	Type	Description	Required
`template`	RawExtension	The metadata and spec for the underlying Coherence resource.	Yes

VerrazzanoHelidonWorkload

The VerrazzanoHelidonWorkload custom resource contains the configuration information for a Helidon workload within Verrazzano. Here is a sample component that specifies a VerrazzanoHelidonWorkload. To deploy an example application that demonstrates this workload type, see Hello World Helidon.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.10-3-20201016220428-56fb4d4"
              ports:
                - containerPort: 8080
                  name: http

VerrazzanoHelidonWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoHelidonWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoHelidonWorkloadSpec	The desired state of a Verrazzano Helidon workload.	Yes

VerrazzanoHelidonWorkloadSpec

VerrazzanoHelidonWorkloadSpec specifies the desired state of a Verrazzano Helidon workload.

Field	Type	Description	Required
`deploymentTemplate`	DeploymentTemplate	The embedded deployment.	Yes

DeploymentTemplate

DeploymentTemplate specifies the metadata and pod spec of the underlying deployment.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`strategy`	DeploymentStrategy	The replacement strategy of the underlying deployment.	No
`podSpec`	PodSpec	The pod spec of the underlying deployment.	Yes

VerrazzanoWebLogicWorkload

The VerrazzanoWebLogicWorkload custom resource contains the configuration information for a WebLogic Domain workload within Verrazzano. Here is a sample component that specifies a VerrazzanoWebLogicWorkload. To deploy an example application that demonstrates this workload type, see the ToDo List Lift-and-Shift application.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: todo-domain
  namespace: todo-list
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoWebLogicWorkload
    spec:
      template:
        metadata:
          name: todo-domain
          namespace: todo-list
        spec:
          domainUID: tododomain
          domainHome: /u01/domains/tododomain
          image: container-registry.oracle.com/verrazzano/example-todo:0.8.0
          imagePullSecrets:
            - name: tododomain-repo-credentials
          domainHomeSourceType: "FromModel"
          includeServerOutInPodLog: true
          replicas: 1
          webLogicCredentialsSecret:
            name: tododomain-weblogic-credentials
          configuration:
            introspectorJobActiveDeadlineSeconds: 900
            model:
              configMap: tododomain-jdbc-config
              domainType: WLS
              modelHome: /u01/wdt/models
              runtimeEncryptionSecret: tododomain-runtime-encrypt-secret
            secrets:
              - tododomain-jdbc-tododb
          serverPod:
            env:
              - name: JAVA_OPTIONS
                value: "-Dweblogic.StdoutDebugEnabled=false"
              - name: USER_MEM_ARGS
                value: "-Djava.security.egd=file:/dev/./urandom -Xms64m -Xmx256m "
              - name: WL_HOME
                value: /u01/oracle/wlserver
              - name: MW_HOME
                value: /u01/oracle

VerrazzanoWebLogicWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoWebLogicWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoWebLogicWorkloadSpec	The desired state of a Verrazzano WebLogic workload.	Yes

VerrazzanoWebLogicWorkloadSpec

VerrazzanoWebLogicWorkloadSpec specifies the desired state of a Verrazzano WebLogic workload.

Field	Type	Description	Required
`template`	RawExtension	The metadata and spec for the underlying WebLogic Domain resource.	Yes

Docs: VerrazzanoManagedCluster Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The VerrazzanoManagedCluster custom resource is used to register a managed cluster with an admin cluster. Here is a sample VerrazzanoManagedCluster that registers the cluster named managed1. To deploy an example application that demonstrates a VerrazzanoManagedCluster, see Multicluster Hello World Helidon.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Managed Cluster 1"
  caSecret: ca-secret-managed1

VerrazzanoManagedCluster

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoManagedCluster	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	VerrazzanoManagedClusterSpec	The managed cluster specification.	Yes
`status`	VerrazzanoManagedClusterStatus	The runtime status this resource.	No

VerrazzanoManagedClusterSpec

VerrazzanoManagedClusterSpec specifies a managed cluster to associate with an admin cluster.

Field	Type	Description	Required
`description`	string	The description of the managed cluster.	No
`caSecret`	string	The name of a Secret that contains the CA certificate of the managed cluster. This is used to configure the admin cluster to scrape metrics from the Prometheus endpoint on the managed cluster. See the pre-registration instructions for how to create this Secret.	Yes
`serviceAccount`	string	The name of the ServiceAccount that was generated for the managed cluster. This field is managed by a Verrazzano Kubernetes operator.	No
`managedClusterManifestSecret`	string	The name of the Secret containing generated YAML manifest file to be applied by the user to the managed cluster. This field is managed by a Verrazzano Kubernetes operator.	No

VerrazzanoManagedClusterStatus

Field	Type	Description	Required
`conditions`	Condition array	The current state of this resource.	No
`lastAgentConnectTime`	string	The last time the agent from this managed cluster connected to the admin cluster.	No
`apiUrl`	string	The Verrazzano API server URL for the managed cluster.	No

Condition

Condition describes current state of this resource.

Field	Type	Description	Required
`type`	string	The condition of the multicluster resource which can be checked with a `kubectl wait` command. Condition values are case-sensitive and formatted as follows: `Ready`: the VerrazzanoManagedCluster is ready to be used and all resources needed have been generated.	Yes
`status`	ConditionStatus	An instance of the type ConditionStatus that is defined in types.go.	Yes
`lastTransitionTime`	string	The last time the condition transitioned from one status to another.	No
`message`	string	A message with details about the last transition.	No

Docs: VerrazzanoProject Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The VerrazzanoProject custom resource is used to create the application namespaces and their associated security settings on one or more clusters. The namespaces are always created on the admin cluster. Here is a sample VerrazzanoProject that specifies a namespace to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoProject
metadata:
  name: hello-helidon
  namespace: verrazzano-mc
spec:
  template:
    namespaces:
      - metadata:
          name: hello-helidon
  placement:
    clusters:
      - name: managed1

VerrazzanoProject

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoProject	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	VerrazzanoProjectSpec	The project specification.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

VerrazzanoProjectSpec

VerrazzanoProjectSpec specifies the namespaces to create and on which clusters to create them.

Field	Type	Description	Required
`template`	ProjectTemplate	The project template.	Yes
`placement`	Placement	Clusters on which the namespaces are to be created.	Yes

ProjectTemplate

ProjectTemplate contains the list of namespaces to create and the optional security configuration for each namespace.

Field	Type	Description	Required
`namespaces`	NamespaceTemplate array	The list of application namespaces to create for this project.	Yes
`security`	SecuritySpec	The project security configuration.	No
`networkPolicies`	NetworkPolicyTemplate array	The network policies applied to namespaces in the project.	No

NamespaceTemplate

NamespaceTemplate contains the metadata and specification of a Kubernetes namespace.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	NamespaceSpec	An instance of the `struct` NamespaceSpec defined in types.go.	No

SecuritySpec

SecuritySpec defines the security configuration for a project.

Field	Type	Description	Required
`projectAdminSubjects`	Subject	The subject to bind to the `verrazzano-project-admin` role. Encoded as an instance of the `struct` Subject defined in types.go.	No
`projectMonitorSubjects`	Subject	The subject to bind to the `verrazzano-project-monitoring` role. Encoded as an instance of the `struct` Subject defined in types.go.	No

NetworkPolicyTemplate

NetworkPolicyTemplate contains the metadata and specification of the underlying NetworkPolicy.

NOTE

To add an application NetworkPolicy, see NetworkPolicies for applications.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	NetworkPolicySpec	An instance of the `struct` NetworkPolicySpec defined in types.go.	No

Docs: Hello World Helidon

Mon, 01 Jan 0001 00:00:00 +0000

The Hello World Helidon example is a Helidon-based service that returns a “Hello World” response when invoked. The example application is specified using Open Application Model (OAM) component and application configuration YAML files, and then deployed by applying those files.

The example application has two endpoints, which differ in configuration source:

/greet- uses a microprofile properties file. Deploy this application by using the instructions here.
/config- uses a Kubernetes ConfigMap. Deploy this application by using the instructions here.

For more information and the code of this application, see the Verrazzano examples.

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano metrics stack automates metrics aggregation and consists of Prometheus and Grafana components. Metrics sources expose system and application metrics. The Prometheus components retrieve and store the metrics and Grafana provides dashboards to visualize them.

Metrics sources

The following sections describe metrics sources that Verrazzano provides for OAM and standard Kubernetes applications.

OAM

Metrics sources produce metrics and expose them to the Kubernetes Prometheus system using annotations in the pods. The metrics annotations may differ slightly depending on the resource type. The following is an example of the WebLogic Prometheus-related configuration specified in the todo-list application pod:

$ kubectl describe pod tododomain-adminserver -n todo-list

Annotations:  prometheus.io/path: /wls-exporter/metrics
              prometheus.io/port: 7001
              prometheus.io/scrape: true

For other resource types, such as Coherence or Helidon, the annotations would look similar to this:

Annotations:  verrazzano.io/metricsEnabled: true
              verrazzano.io/metricsPath: /metrics
              verrazzano.io/metricsPort: 8080

To look directly at the metrics that are being made available by the metric source, map the port and then access the path.

For example, for the previous metric source:

Map the port being used to expose the metrics.

$ kubectl port-forward tododomain-adminserver 7001:7001 -n todo-list

Get the user name and password used to access the metrics source from the corresponding secret.

$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.username} | base64 \
    --decode; echo
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

Access the metrics at the exported path, using the user name and password retrieved in the previous step.
```
$ curl -u USERNAME:PASSWORD localhost:7001/wls-exporter/metrics
```

Standard Kubernetes workloads

Verrazzano enables metric sources for Kubernetes workloads deployed without OAM components. Verrazzano supports the following workload types: Deployments, ReplicaSets, StatefulSets, and Pods. To enable metrics for Kubernetes workloads, you must label the workload namespace with verrazzano-managed=true.

Metrics Template

A Metrics Template is a custom resource created by Verrazzano to manage metrics configurations for standard Kubernetes workloads. Metrics templates can be placed in the workload namespace or the verrazzano-system namespace. By default, Verrazzano installs a metrics template named standard-k8s-metrics-template in the verrazzano-system namespace. This metrics template handles all the aforementioned workload types. If the default metrics template does not meet your requirements, then you can create your own metrics templates to extend and alter its functionality.

As outlined in the API, the metrics template contains a workloadSelector field that specifies the resources for which the template applies. If you want to forgo the workload selection and manually specify a template, you can add the annotation app.verrazzano.io/metrics=<template-name> to the namespace of the workload or to the workload itself. Additionally, you can opt out of metrics for your namespace or workload by setting the annotation app.verrazzano.io/metrics=none.

The template matching precedence is as follows:

A workload is annotated.

a. A template matching the annotation value is found in the workload namespace.

b. A template matching the annotation value is found in the verrazzano-system namespace.

c. No template is found, an error is recorded, and metrics are not processed for this workload.
A workload namespace is annotated.

a. A template matching the annotation value is found in the workload namespace.

b. A template matching the annotation value is found in the verrazzano-system namespace.

c. No template is found, an error is recorded, and metrics are not processed for this namespace.
No annotation is present.

a. A template in the workload namespace matches the workload through the workloadSelector field.

b. A template in the verrazzano-system namespace matches the workload through the workloadSelector field.

c. No templates match the workload and metrics are not processed for this workload.

If a workload with no annotations matches multiple templates in a namespace, there is no guaranteed precedence in template matching. If this is the case, it is more reliable to specify the template you require by using an annotation.

To verify that the metrics template process was successful, follow these steps:

Access the Prometheus console.
From the console, use the navigation bar to access Status/Targets.
On this page, you will see a target name with this formatting: <workload-namespace>_<workload-name>_<workload-type>.
Copy this job name for use in future queries.
Verify that the State of this target is UP.
Next, use the navigation bar to access the Graph.
Here, use the job name you copied to construct this expression: {job="<job_name>"}
Use the graph to run this expression and verify that you see application metrics appear.

Prometheus overrides

The standard-k8s-metrics-template metrics template installed by Verrazzano uses the following pod annotations to populate the Prometheus configuration. If not specified, Verrazzano will use these default values:

Annotations:  prometheus.io/path: /metrics
              prometheus.io/port: 8080
              prometheus.io/scrape: true

To alter these values, annotate the workload pod with the corresponding annotations. For example, if you want to change the metrics path, then add the following to a Deployment definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-helidon-deployment
  namespace: hello-helidon
  annotations:
    app.verrazzano.io/metrics: standard-k8s-metrics-template
spec:
  template:
    metadata:
      # add path annotation to the pod template
      annotations:
        prometheus.io/path: "/custom/metrics/path"

Prometheus configuration

If you want to create your own metrics template, you will need to construct a Prometheus scrape config. The scrape config uses Go Templates to generate configuration values based on Kubernetes resources. You can reference values in the workload and namespace definitions for use in the scrape config. For example, the default scrape config references the workload namespace field through this reference: .workload.metadata.namespace. Do not include the job_name field in your scrape config as it will be generated by Verrazzano. For guidance on how to construct a Prometheus scrape config, reference the scrapeConfigTemplate section in the Metrics Template example.

Metrics server

Single pod per cluster.
Named vmi-system-prometheus-* in verrazzano-system namespace.
Discovers exposed metrics source endpoints.
Scrapes metrics from metrics sources.
Responsible for exposing all metrics.

Grafana

Grafana provides visualization for your Prometheus metric data.

Single pod per cluster.
Named vmi-system-grafana-* in verrazzano-system namespace.
Provides dashboards for metrics visualization.

To access Grafana:

Get the host name from the Grafana ingress.

$ kubectl get ingress vmi-system-grafana -n verrazzano-system

# Sample output
NAME                 CLASS    HOSTS                                              ADDRESS          PORTS     AGE
vmi-system-grafana   <none>   grafana.vmi.system.default.123.456.789.10.nip.io   123.456.789.10   80, 443   26h

Get the password for the user verrazzano.

$ kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

Access Grafana in a browser using the previous host name.
Log in using the verrazzano user and the previous password.

From here, you can select an existing dashboard or create a new dashboard. To select an existing dashboard, use the drop-down list in the top left corner. The initial value of this list is Home.

To view host level metrics, select Host Metrics. This will provide system metrics for all of the nodes in your cluster.

To view the application metrics for the todo-list example application, select WebLogic Server Dashboard because the todo-list application is a WebLogic application.

Docs: Network Security

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano manages and secures network traffic between Verrazzano system components and deployed applications. Verrazzano does not manage or secure traffic for the Kubernetes cluster itself, or for non-Verrazzano services or applications running in the cluster. Traffic is secured at two levels in the network stack:

ISO Layer 3/4: Using NetworkPolicies to control IP access to Pods.
ISO Layer 6: Using TLS and mutual TLS authentication (mTLS) to provide authentication, confidentiality, and integrity for connections within the cluster, and for external connections.

NetworkPolicies

By default, all Pods in a Kubernetes cluster have network access to all other Pods in the cluster. Kubernetes has a NetworkPolicy resource that provides network level 3 and 4 security for Pods, restricting both ingress and egress IP traffic for a set of Pods in a namespace. Verrazzano configures all system components with NetworkPolicies to control ingress. Egress is not restricted.

NOTE: A NetworkPolicy resource needs a NetworkPolicy controller to implement the policy, otherwise the policy has no effect. You must install a Kubernetes Container Network Interface (CNI) plug-in that provides a NetworkPolicy controller, such as Calico, before installing Verrazzano, or else the policies are ignored.

NetworkPolicies for system components

Verrazzano installs a set of NetworkPolicies for system components to control ingress into the Pods. A policy is scoped to a namespace and uses selectors to specify the Pods that the policy applies to, along with the ingress and egress rules. For example, the following policy applies to the Verrazzano API Pod in the verrazzano-system namespace. This policy allows network traffic from NGINX Ingress Controller on port 8775, and from Prometheus on port 15090. No other Pods can reach those ports or any other ports of the Verrazzano API Pod. Notice that namespace selectors need to be used; the NetworkPolicy resource does not support specifying the namespace name.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
...
spec:
  PodSelector:
    matchLabels:
      app: verrazzano-api
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: ingress-nginx
      PodSelector:
        matchLabels:
          app.kubernetes.io/instance: ingress-controller
    ports:
    - port: 8775
      protocol: TCP
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      PodSelector:
        matchLabels:
          app: system-prometheus
    ports:
    - port: 15090
      protocol: TCP

The following table shows all of the ingresses that allow network traffic into system components. The ports shown are Pod ports, which is what NetworkPolicies require.

Component	Pod Port	From	Description
Verrazzano Application Operator	9443	Kubernetes API Server	Webhook entrypoint
Verrazzano Platform Operator	9443	Kubernetes API Server	Webhook entrypoint
Verrazzano Console	8000	NGINX Ingress	Access from external client
Verrazzano Console	15090	Prometheus	Prometheus scraping
Verrazzano Authentication Proxy	8775	NGINX Ingress	Access from external client
Verrazzano Authentication Proxy	15090	Prometheus	Prometheus scraping
cert-manager	9402	Prometheus	Prometheus scraping
Coherence Operator	9443	Prometheus	Webhook entrypoint
OpenSearch	8775	NGINX Ingress	Access from external client
OpenSearch	8775	Fluentd	Access from Fluentd
OpenSearch	9200	OpenSearch Dashboards, Internal	OpenSearch data port
OpenSearch	9300	Internal	OpenSearch cluster port
OpenSearch	15090	Prometheus	Envoy metrics scraping
Istio control plane	15012	Envoy	Envoy access to `istiod`
Istio control plane	15014	Prometheus	Prometheus scraping.
Istio control plane	15017	Kubernetes API Server	Webhook entrypoint
Istio ingress gateway	8443	External	Application ingress
Istio ingress gateway	15090	Prometheus	Prometheus scraping
Istio egress gateway	8443	Mesh services	Application egress
Istio egress gateway	15090	Prometheus	Prometheus scraping
Keycloak	8080	NGINX Ingress	Access from external client
Keycloak	15090	Prometheus	Prometheus scraping
MySql	15090	Prometheus	Prometheus scraping
MySql	3306	Keycloak	Keycloak datastore
Node exporter	9100	Prometheus	Prometheus scraping
Rancher	80	NGINX Ingress	Access from external client
Rancher	9443	Kubernetes API Server	Webhook entrypoint
Prometheus	8775	NGINX Ingress	Access from external client
Prometheus	9090	Grafana	Acccess for Grafana UI

NetworkPolicies for applications

By default, applications do not have NetworkPolicies that restrict ingress into the application or egress from it. You can configure them for the application namespaces using the NetworkPolicy section of a Verrazzano project.

NOTE

Verrazzano requires specific ingress to and egress from application pods. If you add a NetworkPolicy for your application namespace or pods, you must add an additional policy to ensure that Verrazzano still has the required access it needs. The ingress policy is only needed if you restrict ingress. Likewise, the egress policy is only needed if you restrict egress. The following are the ingress and egress NetworkPolicies:

ingress NetworkPolicies

  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-ingressgateway
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: system-prometheus
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: weblogic-operator

egress NetworkPolicies

  egress:
  - ports:
    - port: 15012
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-egressgateway
  - ports:
    - port: 53
      protocol: TCP
    - port: 53
      protocol: UDP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: kube-system
  - ports:
    - port: 8000
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator

NetworkPolicies for Envoy sidecar proxies

As mentioned, Envoy sidecar proxies run in both system component pods and application pods. Each proxy sends requests to the Istio control plane pod, istiod, for a variety of reasons. During installation, Verrazzano creates a NetworkPolicy named istiod-access in the istio-system namespace to give ingress to system component and application sidecar proxies.

Mutual TLS authentication (mTLS)

Istio can be enabled to use mTLS between services in the mesh, and also between the Istio gateways and Envoy sidecar proxies. There are various options to customize mTLS usage, for example it can be disabled on a per-port level. The Istio control plane, Istiod, is a CA and provides key and certificate rotation for the Envoy proxies, both gateways and sidecars.

Verrazzano configures Istio to have strict mTLS for the mesh. All components and applications put into the mesh will use mTLS, with the exception of Coherence clusters, which are not in the mesh. Also, all traffic between the Istio ingress gateway and mesh sidecars use mTLS, and the same is true between the proxy sidecars and the egress gateway.

Verrazzano sets up mTLS during installation with the PeerAuthentication resource as follows:

apiVersion: v1
items:
- apiVersion: security.istio.io/v1beta1
  kind: PeerAuthentication
  ...
  spec:
    mtls:
      mode: STRICT

TLS

TLS is used by external clients to access the cluster, both through the NGINX Ingress Controller and the Istio ingress gateway. The certificate used by these TLS connections vary; see Verrazzano security for details. All TLS connections are terminated at the ingress proxy. Traffic between the two proxies and the internal cluster Pods always uses mTLS, because those Pods are all in the Istio mesh.

Istio mesh

Istio provides extensive security protection for both authentication and authorization, as described in Istio Security. Access control and mTLS are two security features that Verrazzano configures. These security features are available in the context of a service mesh.

A service mesh is an infrastructure layer that provides certain capabilities like security, observability, load balancing, and such, for services. Istio defines a service mesh here. In the context of Istio on Kubernetes, a service in the mesh is a Kubernetes Service. Consider the Bob’s Books example application, which has several OAM Components defined. At runtime, there is a Kubernetes Service for each component, and each Service is in the mesh, with one or more Pods associated with the service. All services in the mesh have an Envoy proxy in front of their Pods, intercepting network traffic to and from the Pod. In Kubernetes, that proxy happens to be a sidecar running in each Pod.

There are various ways to put a service in the mesh. Verrazzano uses the namespace label, istio-injection: enabled, to designate that all Pods in a given namespace are in the mesh. When a Pod is created in that namespace, the Istio control plane mutating webhook, changes the Pod spec to add the Envoy proxy sidecar container, causing the Pod to be in the mesh.

Disabling sidecar injection

In certain cases, Verrazzano needs to disable sidecar injection for specific Pods in a namespace. This is done in two ways: first, during installation, Verrazzano modifies the istio-sidecar-injector ConfigMap using a Helm override file for the Istio chart. This excludes several components from the mesh, such as the Verrazzano application operator. Second, certain Pods, such as Coherence Pods, are labeled at runtime with sidecar.istio.io/inject="false" to exclude them from the mesh.

Components in the mesh

The following Verrazzano components are in the mesh and use mTLS for all service to service communication.

OpenSearch
Fluentd
Grafana
Kiali
OpenSearch Dashboards
Keycloak
MySQL
NGINX Ingress Controller
Prometheus
Verrazzano Authentication Proxy
Verrazzano Console
WebLogic Kubernetes Operator

Some of these components, have mesh-related details that are worth noting, as described in the following sections.

NGINX

The NGINX Ingress Controller listens for HTTPS traffic, and provides ingress into the cluster. NGINX is configured to do TLS termination of client connections. All traffic from NGINX to the mesh services use mTLS, which means that traffic is fully encrypted from the client to the target back-end services.

Keycloak and MySQL

Keycloak and MySQL are also in the mesh and use mTLS for network traffic. Because all of the components that use Keycloak are in the mesh, there is end to end mTLS security for all identity management handled by Keycloak. The following components access Keycloak:

Verrazzano Authentication Proxy
Verrazzano Console
OpenSearch
Prometheus
Grafana
Kiali
OpenSearch Dashboards

Prometheus

Although Prometheus is in the mesh, it is configured to use the Envoy sidecar and mTLS only when communicating with Keycloak. All the traffic related to scraping metrics, bypasses the sidecar proxy, doesn’t use the service IP address, but rather connects to the scrape target using the Pod IP address. If the scrape target is in the mesh, then HTTPS is used; otherwise, HTTP is used. For Verrazzano multicluster, Prometheus also connects from the admin cluster to the Prometheus server in the managed cluster by using the managed cluster NGINX Ingress, using HTTPS. Prometheus in the managed cluster and never establishes connections to targets outside the cluster.

Because Prometheus is in the mesh, additional configuration is done to allow the Envoy sidecar to be bypassed when scraping Pods. This is done with the Prometheus Pod annotation traffic.sidecar.istio.io/includeOutboundIPRanges: <keycloak-service-ip>. This causes traffic bound for Keycloak to go through the Envoy sidecar, and all other traffic to bypass the sidecar.

WebLogic Kubernetes Operator

Applications in the mesh

Before you create a Verrazzano application, you should decide if it should be in the mesh. You control sidecar injection, for example, mesh inclusion, by labeling the application namespace with istio-injection=enabled or istio-injection=disabled. By default, applications will not be put in the mesh if that label is missing. If your application uses a Verrazzano project, then Verrazzano will label the namespaces in the project to enable injection. If the application is in the mesh, then mTLS will be used. You can change the PeerAuthentication mTLS mode as desired if you don’t want strict mTLS. Also, if you need to add mTLS port exceptions, you can do this with DestinationRules or by creating another PeerAuthentication resource in the application namespace. Consult the Istio documentation for more information.

WebLogic

When the WebLogic operator creates a domain, it needs to communicate with the Pods in the domain. Verrazzano puts the WebLogic operator in the mesh so that it can communicate with the domain Pods using mTLS. Because of that, the WebLogic domain must be created in the mesh. Also, because mTLS is used, do not configure WebLogic to use TLS. If you want to use a custom certificate for your application, you can specify that in the ApplicationConfiguration, but that TLS connection will be terminated at the Istio ingress gateway, which you configure using a Verrazzano IngressTrait.

Coherence

Coherence clusters are represented by the Coherence resource, and are not in the mesh. When Verrazzano creates a Coherence cluster in a namespace that is annotated to do sidecar injection, it disables injection of the Coherence resource using the sidecar.istio.io/inject="false" label shown previously. Furthermore, Verrazzano will create a DestinationRule in the application namespace to disable mTLS for the Coherence extend port 9000. This allows a service in the mesh to call the Coherence extend proxy. For an example, see Bobs Books.

Here is an example of a DestinationRule created for the Bob’s Books application which includes a Coherence cluster.

API Version:  networking.istio.io/v1beta1
Kind:         DestinationRule
...
Spec:
  Host:  *.bobs-books.svc.cluster.local
  Traffic Policy:
    Port Level Settings:
      Port:
        Number:  9000
      Tls:
    Tls:
      Mode:  ISTIO_MUTUAL

Istio access control

Istio lets you control access to your workload in the mesh, using the AuthorizationPolicy resource. This lets you control which services or Pods can access your workloads. Some of these options require mTLS; for more information, see Authorization Policy.

Verrazzano always creates AuthorizationPolicies for applications, but never for system components. During application deployment, Verrazzano creates the policy in the application namespace and configures it to allow access from the following:

Other Pods in the application
Istio ingress gateway
Prometheus scraper

This prevents other Pods in the cluster from gaining network access to the application Pods.
Istio uses a service identity to determine the identity of the request’s origin; for Kubernetes this identity is a service account. Verrazzano creates a per-application AuthorizationPolicy as follows:

AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
...
spec:
  rules:
    - from:
    - source:
  principals:
    - cluster.local/ns/sales/sa/greeter
    - cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
    - cluster.local/ns/verrazzano-system/sa/verrazzano-monitoring-operator

WebLogic domain access

For WebLogic applications, the WebLogic operator must have access to the domain Pods for two reasons. First, it must access the domain servers to get health status; second it must inject configuration into the Monitoring Exporter sidecar running in the domain server Pods. When a WebLogic domain is created, Verrazzano adds an additional source, cluster.local/ns/verrazzano-system/sa/weblogic-operator-sa to the principals section to permit that access.

Docs: Load Balancers on OCI

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano sets up the following load balancers on Kubernetes at installation:

Load balancer for NGINX ingress
Load balancer for Istio ingress

Verrazzano allows customizing the load balancers allocated by Oracle Cloud Infrastructure (OCI) using annotations defined by the OCI Cloud Controller Manager (OCI-CCM). For a detailed description of different load balancer customization annotations, see the documentation here.

This document describes how to use these annotations to customize the following settings for Verrazzano load balancers:

Load balancer shape
Private IP address and subnet placement

Customize the load balancer shape

At installation, Verrazzano lets you customize the shape and size of the load balancers created. Oracle Cloud Infrastructure offers a flexible load balancer which uses Dynamic Shape:

10 Mbps
100 Mbps
400 Mbps
8,000 Mbps

For more details on service limits and shape, see here.

For example, you can set up an NGINX load balancer with 10Mbps as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
        value: "10Mbps"

For example, you can set up an Istio load balancer with 10Mbps as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      istioInstallArgs:
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
        value: "10Mbps"

Use private IP addresses with a load balancer

At installation, Verrazzano lets you customize the IP address and subnet of the load balancers created. This is achieved using OCI-CCM annotations on the NGINX and Istio load balancer services, as documented here.

The following example configures the NGINX load balancer service to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for Istio:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"    
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures the Istio ingress gateway service to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for NGINX:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer      
    istio:
      istioInstallArgs:
        - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
          value: "true"
        - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
          value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures both NGINX and Istio to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"
    istio:
      istioInstallArgs:
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

Docs: Coherence Workload

Mon, 01 Jan 0001 00:00:00 +0000

A Verrazzano application can contain any number of Coherence component workloads, where each workload is a standalone Coherence cluster, independent from other Coherence clusters in the application.

Verrazzano uses the standard Coherence operator to provision and manage clusters, as documented at Coherence Operator. The Coherence operator uses a CRD, coherence.oracle.com (Coherence resource), to represent a Coherence cluster. When a Verrazzano application with Coherence is provisioned, Verrazzano configures the default logging and metrics for the Coherence cluster. Logs are sent to OpenSearch and metrics to Prometheus.
You can view this telemetry data using the OpenSearch Dashboards and Grafana consoles.

OAM Component

The custom resource YAML file for the Coherence cluster is specified as a VerrazzanoCoherenceWorkload custom resource. In the following example, everything under the spec: section is standard Coherence resource YAML that you would typically use to provision a Coherence cluster. Including this Component reference in your ApplicationConfiguration will result in a new Coherence cluster being provisioned. You can have multiple clusters in the same application with no conflict.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: orders
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: orders-coh
        spec:
          cluster: SockShop
          ...

Life cycle

With Verrazzano, you manage the life cycle of applications using Component and ApplicationConfiguration resources. Typically, you would modify the Coherence cluster resource to make changes or to do lifecycle operations, like scale in and scale out. However, in the Verrazzano environment, the cluster resource is owned by the Verrazzano application operator and will be reconciled to match the Component workload resource. Therefore, you need to manage the cluster configuration by modifying the resource, either by kubectl edit or applying a new YAML file. Verrazzano will notice that the Component resource changed and will update the Coherence resource as needed.

Provisioning

When you apply the Component YAML file shown previously, Kubernetes will create a component.oam.verrazzano.io resource, but the Coherence cluster will not be created until you create the ApplicationConfiguration resource, which references the Coherence component. When the application is created, Verrazzano creates a Coherence custom resource for each cluster, which is subsequently processed by the Coherence operator, resulting in a new cluster. After a cluster is created, the Coherence operator will monitor the Coherence resource to reconcile the state of the cluster. You can add a new Coherence workload to a running application, or remove an existing workload, by modifying the ApplicationConfiguration resource, and adding or removing the Coherence component.

Scaling

Scaling a Coherence cluster is done by modifying the replicas field in the Component resource. Verrazzano will modify the Coherence resource replicas field and the cluster will be scaled accordingly. The following example configuration shows the replicas field that specifies the number of pods in the cluster.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: orders
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: orders-coh
        spec:
          cluster: SockShop
          replicas: 3
          ...

NOTE: A Coherence cluster provisioned with Verrazzano does not support autoscaling with a Horizontal Pod Autoscaler.

Termination

You can terminate the Coherence cluster by removing the Component from the ApplicationConfiguration or by deleting the ApplicationConfiguration resource entirely.

NOTE

Do not delete the Coherence component if the application is still using it.

Logging

When a Coherence cluster is provisioned, Verrazzano configures it to send logs to OpenSearch. This is done by injecting Fluentd sidecar configuration into the Coherence resource. The Coherence operator will create the pod with the Fluentd sidecar. This sidecar periodically copies the Coherence logs from /logs to stdout, enabling the Fluentd DaemonSet in the verrazzano-system namespace to send the logs to OpenSearch. Note that the Fluend sidecar running in the Coherence pod never communicates with OpenSearch or any other network endpoint.

The logs are placed in a per-namespace OpenSearch data stream named verrazzano-application-<namespace>, for example: verrazzano-application-sockshop. All logs from Coherence pods in the same namespace will go into the same data stream, even for different applications. This is standard behavior and there is no way to disable or change it.

Each log record has some Coherence and application fields, along with the log message itself. For example:

 kubernetes.labels.coherenceCluster        SockShop
 kubernetes.labels.app_oam_dev/name        sockshop-appconf
 kubernetes.labels.app_oam_dev/component   orders
 ...

Metrics

Verrazzano uses Prometheus to scrape metrics from Coherence cluster pods. Like logging, metrics scraping is also enabled during provisioning, however, the Coherence resource YAML file must have proper metrics configuration. For details, see Coherence Metrics. In summary, there are two ways to configure the Coherence metrics endpoint. Coherence has a default metrics endpoint that you can enable. If your application serves metrics from its own endpoint, such as a Helidon application, then do not use the native Coherence metrics endpoint. To see the difference, examine the socks-shop and bobs-books examples.

Bobs Books

The bobs-books example uses the default Coherence metrics endpoint, so the configuration must enable this feature, shown in the following metrics section of the roberts-coherence component in the YAML file, bobs-books-comp.yaml.

          coherence:
            metrics:
              enabled: true

Sock Shop

The sock-shop example, which is a Helidon application with embedded Coherence, explicitly specifies the metrics port 7001 and doesn’t enable Coherence metrics. Coherence metrics still will be scraped, but not at the default endpoint.

          ports:
            ...
            - name: metrics
              port: 7001
              serviceMonitor:
                enabled: true

Because sock-shop components are not using the default Coherence metrics port, you must add a MetricsTrait section to the ApplicationConfiguration for each component, specifying the metrics port as follows:

        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            metadata:
              name: carts-metrics
            spec:
              port: 7001

Prometheus configuration

Prometheus is configured to scrape targets using the ConfigMaps in the verrazzano-system namespace. During application deployments, Verrazzano updates the vmi-system-prometheus-config ConfigMap and adds targets for the application pods. Verrazzano also annotates those pods to match the expected annotations in the ConfigMap. When the application is deleted, Verrazzano removes the targets from the ConfigMap. You do not need to manually modify the ConfigMap or annotate the application pods.

Here is an example of thesock-shop Prometheus ConfigMap section for catalog. Notice that pods in the sock-shop namespace with labels app_oam_dev_name and app_oam_dev_component are targeted. Prometheus will find those pods and then look at the pod annotations, verrazzano_io/metricsEnabled, verrazzano_io/metricsPath, and verrazzano_io/metricsPort for scrape configuration.

- job_name: sockshop-appconf_default_sockshop_catalog
  ...
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - sockshop
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_verrazzano_io_metricsEnabled,
      __meta_kubernetes_pod_label_app_oam_dev_name, __meta_kubernetes_pod_label_app_oam_dev_component]
  ...  
  - source_labels: [__meta_kubernetes_pod_annotation_verrazzano_io_metricsPath]
  ...
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_verrazzano_io_metricsPort]

Here is the corresponding catalog pod labels and annotations.

kind: Pod
metadata:
  labels:
    ...
    app.oam.dev/component: catalog
    app.oam.dev/name: sockshop-appconf
  annotations:
    ...
    verrazzano.io/metricsEnabled: "true"
    verrazzano.io/metricsPath: /metrics
    verrazzano.io/metricsPort: "7001"

Istio integration

Verrazzano ensures that Coherence clusters are not included in an Istio mesh, even if the namespace has the istio-injection: enabled label. This is done by adding the sidecar.istio.io/inject: "false" annotation to the Coherence resource, resulting in Coherence pods being created with that label. However, other application components in the mesh using mutual TLS authentication (mTLS) may need to communicate with Coherence. To handle this case, Verrazzano automatically creates an Istio DestinationRule to disable TLS for the Coherence port. This policy disables mTLS for port 9000, which happens to be used as a Coherence extend port for Bob’s Books.

  trafficPolicy:
    portLevelSettings:
    - port:
        number: 9000
      tls: {}
   ...

Currently, port 9000 is the only port where TLS is disabled, so you need to use this as the Coherence extend port if other components in the mesh access Coherence over the extend protocol.

Summary

Verrazzano makes it easy to deploy and observe Coherence clusters in your application, providing seamless integration with other components in your application running in an Istio mesh.

Docs: Helidon Workload

Mon, 01 Jan 0001 00:00:00 +0000

Helidon is a collection of Java libraries for writing microservices. Helidon provides an open source, lightweight, fast, reactive, cloud native framework for developing Java microservices. It is available as two frameworks:

Helidon SE is a compact toolkit that embraces the latest Java SE features: reactive streams, asynchronous and functional programming, and fluent-style APIs.
Helidon MP implements and supports Eclipse MicroProfile, a baseline platform definition that leverages Java EE and Jakarta EE technologies for microservices and delivers application portability across multiple runtimes.

Helidon is designed and built with container-first philosophy.

Small footprint, low memory usage and faster startup times.
All 3rd party dependencies are stored separately to enable Docker layering.
Provides readiness, liveness and customizable health information for container schedulers like Kubernetes.

Containerized Helidon applications are generally deployed as Deployment in Kubernetes.

Verrazzano integration

Verrazzano supports application definition using Open Application Model (OAM). Verrrazzano applications are composed of components and application configurations.

Helidon applications are first class citizens in Verrazzano with specialized Helidon workload support, for example, VerrazzanoHelidonWorkload. VerrazzanoHelidonWorkload is supported as part of verrazzano-application-operator in the Verrazzano installation and no additional operator setup or installation is required. VerrazzanoHelidonWorkload also supports all the traits and scopes defined by Verrazzano along with core ones defined by the OAM specification.

VerrazzanoHelidonWorkload is modeled after ContainerizedWorkload, for example, it is used for long-running workloads in containers. However, VerrazzanoHelidonWorkload closely resembles and directly refers to Kubernetes Deployment schema. This enables an easy lift and shift of existing containerized Helidon applications.

The complete VerrazzanoHelidonWorkload API definition and description is available at VerrazzanoHelidonWorkload.

Verrazzano Helidon application development

With Verrazzano, you manage the life cycle of applications using Component and ApplicationConfiguration resources. A Verrazzano application can contain any number of VerrazzanoHelidonWorkload components, where each workload is a standalone containerized Helidon application, independent of any other in the application.

In the following example, everything under the spec: section is the custom resource YAML file for the containerized Helidon application, as defined by VerrazzanoHelidonWorkload custom resource. Including this Component reference in your ApplicationConfiguration will result in a new containerized Helidon application being provisioned.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              ...
              ...

The Application Development Guide provides end-to-end instructions for developing and deploying the Verrazzano Helidon application.

For more Verrazzano Helidon application examples, see Examples.

Provisioning

When you apply the previous Component YAML file, Kubernetes will create a component.oam.verrazzano.io resource, but the containerized Helidon application will not be created until you create the ApplicationConfiguration resource, which references the VerrazzanoHelidonWorkload component. When the application is created, Verrazzano creates a Deployment and Service resource for each containerized Helidon application.

Typically, you would modify the Deployment and Service resource to make changes or to do lifecycle operations, like scale in and scale out. However, in the Verrazzano environment, the containerized Helidon application resource is owned by the verrazzano-application-operator and will be reconciled to match the component workload resource. Therefore, you need to manage the application configuration by modifying the VerrazzanoHelidonWorkload or ApplicationConfiguration resource, either by kubectl edit or applying new YAML file. Verrazzano will notice that the Component resource change and will update the Deployment and Service resource as needed.

You can add a new VerrazzanoHelidonWorkload to a running application, or remove an existing workload, by modifying the ApplicationConfiguration resource and adding or removing the VerrazzanoHelidonWorkload component.

Scaling

The recommended way to scale containerized Helidon application replicas is to specify ManualScalerTrait with VerrazzanoHelidonWorkload in ApplicationConfiguration. The following example configuration shows the replicaCount field that specifies the number of replicas for the application.

...
    spec:
      components:
      - componentName: hello-helidon-component
        traits:
        - trait:                      
            apiVersion: core.oam.dev/v1alpha2
            kind: ManualScalerTrait
            spec:
              replicaCount: 2
...

Verrazzano will modify the Deployment resource replicas field and the containerized Helidon application replicas will be scaled accordingly.

NOTE

Make sure the replicas defined on the VerrazzanoHelidonWorkload component and that replicaCount defined on ManualScalerTrait for that component matches, or else the DeploymentController in Kubernetes and OAM runtime in verrazzano-application-operator will compete to create a different number of Pods for same containerized Helidon application. To avoid confusion, we recommend that you specify replicaCount defined on ManualScalerTrait and leave replicas undefined on VerrazzanoHelidonWorkload (as it is optional).

Logging

When a containerized Helidon application is provisioned on Verrazzano, Verrazzano will configure the default logging and send logs to OpenSearch. Logs can be viewed using the OpenSearch Dashboards console.

The logs are placed in a per-namespace OpenSearch data stream named verrazzano-application-<namespace>, for example: verrazzano-application-hello-helidon. All logs from containerized Helidon application pods in the same namespace will go into the same data stream, even for different applications. This is standard behavior and there is no way to disable or change it.

Metrics

Verrazzano uses Prometheus to scrape metrics from containerized Helidon application pods. Like logging, metrics scraping is also enabled during provisioning. Metrics can be viewed using the Grafana console.

Verrazzano lets you to customize configuration information needed to enable metrics using MetricsTrait for an application component.

Ingress

Verrazzano lets you to configure traffic routing to a containerized Helidon application, using IngressTrait for an application component.

Troubleshooting

Whenever you have a problem with your Verrazzano Helidon application, there are some basic techniques you can use to troubleshoot. Troubleshooting shows you some simple things to try when troubleshooting, as well as how to solve common problems you may encounter.

Docs: Jaeger Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Jaeger is a distributed tracing system used for monitoring and troubleshooting microservices. For more information on Jaeger, see the Jaeger website.

Install Jaeger Operator

To install the Jaeger Operator, enable the jaegerOperator component in your Verrazzano resource. Here is an example YAML file that enables the Jaeger Operator.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true

Install Jaeger using the Jaeger Operator

Jaeger is installed using the Jaeger Custom Resource Definition. The following example shows you how to install Jaeger inside the Istio mesh using the Verrazzano system OpenSearch cluster as a tracing backend.

Before creating the Jaeger instance, create a secret containing the OpenSearch user name and password. Jaeger will use these credentials to connect to OpenSearch:

$ kubectl create secret generic jaeger-secret \
  --from-literal=ES_PASSWORD=<OPENSEARCH PASSWORD> \
  --from-literal=ES_USERNAME=<OPENSEARCH USERNAME> \
  -n verrazzano-system

Use the following YAML to create the Jaeger resource:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: verrazzano-prod
  namespace: verrazzano-system
spec:
  annotations:
    sidecar.istio.io/inject: "true"
  strategy: production
  storage:
    # Jaeger Elasticsearch storage is compatible with Verrazzano OpenSearch.
    type: elasticsearch
    esIndexCleaner:
      enabled: false
      numberOfDays: 7
      schedule: "* * * * *"
    options:
      es:
        # Enter your OpenSearch cluster endpoint here.
        server-urls: https://elasticsearch.vmi.system.default.172.18.0.151.nip.io
        index-prefix: jaeger
        tls:
          ca: /verrazzano/certificates/ca.crt
    secretName: jaeger-secret
  volumeMounts:
    - name: certificates
      mountPath: /verrazzano/certificates/
      readOnly: true
  volumes:
    - name: certificates
      secret:
        # Jaeger should use the client TLS secret for OpenSearch. This is the default secret name for Verrazzano OpenSearch.
        secretName: system-tls-es-ingest

The Jaeger Operator will create services for query and collection. After applying the example resource, you should see similar output by listing Jaeger resources:

$ kubectl get services,deployments -l app.kubernetes.io/instance=verrazzano-prod -n verrazzano-system

NAME                                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                  AGE
service/verrazzano-prod-collector            ClusterIP   10.96.76.108   <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   52m
service/verrazzano-prod-collector-headless   ClusterIP   None           <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   52m
service/verrazzano-prod-query                ClusterIP   10.96.205.8    <none>        16686/TCP,16685/TCP                      52m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/verrazzano-prod-collector   1/1     1            1           52m
deployment.apps/verrazzano-prod-query       1/1     1            1           52m

Configure an application to export traces to Jaeger

The Jaeger agent sidecar is injected to application pods by the "sidecar.jaegertracing.io/inject": "true" annotation. You may apply this annotation to namespaces or pod controllers such as Deployments. The subsequent snippet shows how to annotate an OAM component for Jaeger agent injection.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-component
spec:
  workload:
    apiVersion: core.oam.dev/v1alpha2
    kind: ContainerizedWorkload
    metadata:
      name: example-workload
      annotations:
        # The component's Deployment will carry the Jaeger annotation.
        "sidecar.jaegertracing.io/inject": "true"

View traces on the Jaeger UI

You can view the UI by port forwarding the Jaeger query service or by configuring an ingress controller for HTTPS access. Explore the Jaeger configuration in more detail using the Jaeger Custom Resource Documentation.

Configure the Istio mesh to use Jaeger tracing

You can view Istio mesh traffic by enabling Istio’s distributed tracing integration. Traces from the Istio mesh provide observability on application traffic that passes through Istio’s ingress and egress gateways.

Istio tracing is disabled by default. To turn on traces, customize your Istio component like the following example:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      istioInstallArgs:
        - name: "meshConfig.enableTracing"
          value: "true"

After enabling tracing, Istio will automatically configure itself with the Jaeger endpoint in your cluster, and any new Istio-injected pods will begin exporting traces to Jaeger. Existing pods require a restart to pull the new Istio configuration and start sending traces.

Istio’s default sampling rate is 1%, meaning 1 in 100 requests will be traced in Jaeger. If you want a different sampling rate, configure your desired rate using the meshConfig.defaultConfig.tracing.sampling Istio installation argument:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      istioInstallArgs:
        - name: "meshConfig.enableTracing"
          value: "true"
        # 25% of Istio traces will be sampled.  
        - name: "meshConfig.defaultConfig.tracing.sampling"
          value: "25.0"

Docs: Sock Shop

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Install Verrazzano by following the installation instructions.

NOTE: The Sock Shop example application deployment files are contained in the Verrazzano project located at <VERRAZZANO_HOME>/examples/sockshop, where <VERRAZZANO_HOME> is the root of the Verrazzano project.

Deploy the application

This example application provides various implementations of the Sock Shop Microservices Demo Application. It uses OAM resources to define the application deployment:

Coherence and Helidon in the helidon subdirectory.
Coherence and Micronaut in the micronaut subdirectory.
Coherence and Spring in the spring subdirectory.

NOTE

To run this application in the default namespace:

$ kubectl label namespace default verrazzano-managed=true

If you chose the default namespace, you can skip Step 1 and ignore the -n option in the rest of the commands.

Create a namespace for the Sock Shop application and add a label identifying the namespace as managed by Verrazzano.
```
$ kubectl create namespace sockshop
$ kubectl label namespace sockshop verrazzano-managed=true
```

To deploy the application, apply the Sock Shop OAM resources. Choose to deploy either the helidon, micronaut, or spring variant.

$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/helidon/sock-shop-comp.yaml -n sockshop
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/helidon/sock-shop-app.yaml -n sockshop

$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/micronaut/sock-shop-comp.yaml -n sockshop
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/micronaut/sock-shop-app.yaml -n sockshop

$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/spring/sock-shop-comp.yaml -n sockshop
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/spring/sock-shop-app.yaml -n sockshop

Wait for the Sock Shop application to be ready.

$ kubectl wait \
   --for=condition=Ready pods \
   --all -n sockshop \
   --timeout=300s

Explore the application

The Sock Shop microservices application implements REST API endpoints including:

/catalogue - Returns the Sock Shop catalog. This endpoint accepts the GET HTTP request method.
/register - POST { "username":"xxx", "password":"***", "email":"foo@example.com", "firstName":"foo", "lastName":"bar" } to create a user. This endpoint accepts the POST HTTP request method.

NOTE: The following instructions assume that you are using a Kubernetes environment, such as OKE. Other environments or deployments may require alternative mechanisms for retrieving addresses, ports, and such.

Follow these steps to test the endpoints.

Get the generated host name for the application.

$ HOST=$(kubectl get gateways.networking.istio.io \
     -n sockshop \
     -o jsonpath={.items[0].spec.servers[0].hosts[0]})
$ echo $HOST

# Sample output
sockshop-appconf.sockshop.11.22.33.44.nip.io

Get the EXTERNAL_IP address of the istio-ingressgateway service.

$ ADDRESS=$(kubectl get service \
     -n istio-system istio-ingressgateway \
     -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $ADDRESS

# Sample output
11.22.33.44

Access the Sock Shop application.

Using the command line

# Get catalogue
$ curl -sk \
   -X GET \
   https://${HOST}/catalogue \
   --resolve ${HOST}:443:${ADDRESS}

# Sample output
[{"count":115,"description":"For all those leg lovers out there....", ...}]

# Add a new user (replace values of username and password)
$ curl -i \
   --header "Content-Type: application/json" \
   --request POST \
   --data '{"username":"foo","password":"****","email":"foo@example.com","firstName":"foo","lastName":"foo"}' \
   -k https://${HOST}/register \
   --resolve ${HOST}:443:${ADDRESS}

# Add an item to the user's cart
$ curl -i \
   --header "Content-Type: application/json" \
   --request POST \
   --data '{"itemId": "a0a4f044-b040-410d-8ead-4de0446aec7e","unitPrice": "7.99"}' \
   -k https://${HOST}/carts/{username}/items \
   --resolve ${HOST}:443:${ADDRESS}

# Get cart items
$ curl -i \
   -k https://${HOST}/carts/{username}/items \
   --resolve ${HOST}:443:${ADDRESS}

# Sample output
[{"itemId":"a0a4f044-b040-410d-8ead-4de0446aec7e","quantity":1,"unitPrice":7.99}]

If you are using nip.io, then you do not need to include --resolve.

Local testing with a browser

Temporarily, modify the /etc/hosts file (on Mac or Linux) or c:\Windows\System32\Drivers\etc\hosts file (on Windows 10), to add an entry mapping the host name to the ingress gateway’s EXTERNAL-IP address. For example:
```
11.22.33.44 sockshop.example.com
```
Then, you can access the application in a browser at https://sockshop.example.com/catalogue.

If you are using nip.io, then you can access the application in a browser using the HOST variable (for example, https://${HOST}/catalogue). If you are going through a proxy, you may need to add *.nip.io to the NO_PROXY list.
Using your own DNS name
- Point your own DNS name to the ingress gateway’s EXTERNAL-IP address.
- In this case, you would need to edit the sock-shop-app.yaml file to use the appropriate value under the hosts section (such as yourhost.your.domain), before deploying the Sock Shop application.
- Then, you can use a browser to access the application at https://<yourhost.your.domain>/catalogue.

A variety of endpoints associated with the deployed application, are available to further explore the logs, metrics, and such. You can access them according to the directions here.

Verify the deployed application

Verify that the application configuration, component, workload, and ingress trait all exist.

$ kubectl get ApplicationConfiguration -n sockshop
$ kubectl get Component -n sockshop
$ kubectl get VerrazzanoCoherenceWorkload -n sockshop
$ kubectl get IngressTrait -n sockshop

Verify that the Sock Shop service pods are successfully created and transition to the READY state. Note that this may take a few minutes and that you may see some of the services terminate and restart.

 $ kubectl get pods -n sockshop

 # Sample output
 NAME             READY   STATUS        RESTARTS   AGE
 carts-coh-0      1/1     Running       0          41s
 catalog-coh-0    1/1     Running       0          40s
 orders-coh-0     1/1     Running       0          39s
 payment-coh-0    1/1     Running       0          37s
 shipping-coh-0   1/1     Running       0          36s
 users-coh-0      1/1     Running       0          35s

Undeploy the application

To undeploy the application, delete the Sock Shop OAM resources. Choose to undeploy either the helidon, micronaut, or spring variant.

$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/helidon/sock-shop-comp.yaml -n sockshop
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/helidon/sock-shop-app.yaml -n sockshop

$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/micronaut/sock-shop-comp.yaml -n sockshop
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/micronaut/sock-shop-app.yaml -n sockshop

$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/spring/sock-shop-comp.yaml -n sockshop
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.3.8/examples/sock-shop/spring/sock-shop-app.yaml -n sockshop

Delete the namespace sockshop after the application pods are terminated.
```
$ kubectl delete namespace sockshop
```

Docs: AuthProxy

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano AuthProxy component enables authentication and authorization for Keycloak users accessing Verrazzano resources. You can customize the AuthProxy component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the AuthProxy component.

Path to Field Description

spec.components.authProxy.kubernetes.replicas The number of pods to replicate. The default is 2 for the prod profile and 1 for all other profiles.

spec.components.authProxy.kubernetes.affinity

The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes.

spec:
  components:
    authProxy:
      kubernetes:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    authproxy:
      kubernetes:
        replicas: 3
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - verrazzano-authproxy
                topologyKey: kubernetes.io/hostname

Docs: Image Pull Back Off

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods that had issues due to failures to pull an image or images.

The analysis was not able to identify a specific root cause, however, it might have supplied data that is related to the failures.

Steps

Review the analysis data. At a minimum, it will indicate which pods are being impacted and might give other clues on the root cause.
If the service is experiencing an outage, then consult the specific service status page. For common service status pages, see Related information.

Docs: Image Pull Not Found

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images where the root cause was that the image was not found.

Steps

Review the analysis data; it enumerates the pods and related messages regarding which images had this issue.
Confirm that the image name, digest, and tag are correctly specified.

Kubernetes Troubleshooting

Docs: Image Pull Rate Limit

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images.

The root cause was rate limit exceeded errors while pulling images.

Steps

Review the analysis data; it enumerates the pods and related messages regarding which images had this issue.
The detailed messages might provide specific instructions for the registry that is involved. For example, it might provide a link to instructions on how to increase the limit.

Increase Rate Limits

Docs: Image Pull Service Issue

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images where the root cause was that the service was not available.

The service might be unreachable or might be incorrect.

Steps

Review the analysis data; it enumerates the pods and related messages about which images had this issue.
Confirm that the registry for the image is correct.
The messages might identify a connectivity issue.
If the service is experiencing an outage, then consult the specific service status page. For common service status pages, see Related information.

Docs: Ingress Controller Load Balancer Service Limit Reached

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that the load balancer service limit has been reached.

Steps

Review the messages from the supporting details for the exact limits, and delete unused load balancers.
If available, use a different load balancer shape. See Customizing Ingress.
Refer to the Oracle Cloud Infrastructure documentation on Service Limits.

Docs: Ingress Controller No Load Balancer IP

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that the load balancer is either missing or unable to set the ingress IP address on the NGINX Ingress service.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Ingress Controller Oracle Cloud Infrastructure IP Limit Exceeded

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that an Oracle Cloud Infrastructure IP non-ephemeral address limit has been reached.

Steps

Review the messages from the supporting details for the exact limit.
Refer to the Oracle Cloud Infrastructure documentation related to managing IP Addresses.

Public IP Addresses

Docs: Install Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation has failed, however, it did not isolate the exact reason for the failure.

Steps

Review the analysis data, which can help identify the issue.

Docs: Install Ingress Controller Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation has failed related to the NGINX Ingress Controller, however, it was unable to isolate the specific root cause.

Steps

Review the analysis data, which might help identify the issue.

Docs: Install Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Before you begin, read this document, Verrazzano in a multicluster environment.

Overview

To set up a multicluster Verrazzano environment, you will need two or more Kubernetes clusters. One of these clusters will the admin cluster; the others will be managed clusters.

The instructions assume an admin cluster and a single managed cluster. For each additional managed cluster, simply repeat the managed cluster instructions.

Install Verrazzano

Install Verrazzano on each Kubernetes cluster.

On one cluster, install Verrazzano using the dev or prod profile; this will be the admin cluster.
On the other cluster, install Verrazzano using the managed-cluster profile; this will be a managed cluster. The managed-cluster profile contains only the components that are required for a managed cluster.

Create the environment variables, KUBECONFIG_ADMIN, KUBECONTEXT_ADMIN, KUBECONFIG_MANAGED1, and KUBECONTEXT_MANAGED1, and point them to the kubeconfig files and contexts for the admin and managed cluster, respectively. You will use these environment variables in subsequent steps when registering the managed cluster. The following shows an example of how to set these environment variables.

$ export KUBECONFIG_ADMIN=/path/to/your/adminclusterkubeconfig
$ export KUBECONFIG_MANAGED1=/path/to/your/managedclusterkubeconfig

# lists the contexts in each kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN config get-contexts -o=name
my-admin-cluster-context
some-other-cluster-context

$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 config get-contexts -o=name
my-managed-cluster-context
some-other-cluster2-context

# Choose the right context name for your admin and managed clusters from the output shown and set the KUBECONTEXT
# environment variables
$ export KUBECONTEXT_ADMIN=<admin-cluster-context-name>
$ export KUBECONTEXT_MANAGED1=<managed-cluster-context-name>

For detailed instructions on how to install and customize Verrazzano on a Kubernetes cluster using a specific profile, see the Installation Guide and Installation Profiles.

Register the managed cluster with the admin cluster

The following sections show you how to register the managed cluster with the admin cluster. As indicated, some of these steps are performed on the admin cluster and some on the managed cluster. The commands provided use the environment variables set previously to connect to the appropriate cluster.

Preregistration setup

Before registering the managed cluster, first you’ll need to set up the following items:

A Secret containing the managed cluster’s CA certificate. Note that the cacrt field in this secret can be empty only if the managed cluster uses a well-known CA. This CA certificate is used by the admin cluster to scrape metrics from the managed cluster, for both applications and Verrazzano components.
A ConfigMap containing the externally reachable address of the admin cluster. This will be provided to the managed cluster during registration so that it can connect to the admin cluster.

Follow these preregistration setup steps:

If needed for the admin cluster, obtain the managed cluster’s CA certificate. The admin cluster scrapes metrics from the managed cluster’s Prometheus endpoint. If the managed cluster Verrazzano installation uses self-signed certificates or LetsEncrypt staging certificates, then the admin cluster will need the managed cluster’s CA certificate to make an https connection.
- Depending on whether the Verrazzano installation on the managed cluster uses self-signed certificates, LetsEncrypt staging certificates, or certificates signed by a well-known certificate authority, choose the appropriate instructions.
- If you are unsure what type of certificates are used, use the following instructions.
  - To check if the verrazzano resource is configured to use LetsEncrypt staging certificates:
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     describe verrazzano
```
    If the output contains the following information, then LetsEncrypt staging certificates are being used.
```
Cert Manager:
  Certificate:
    Acme:
      Environment:    staging
      Provider:       letsEncrypt
```
  - To check the ca.crt field of the verrazzano-tls secret in the verrazzano-system namespace on the managed cluster:
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     -n verrazzano-system get secret verrazzano-tls -o jsonpath='{.data.ca\.crt}'
```
    If this value is empty, then your managed cluster is using certificates signed by a well-known certificate authority. Otherwise, your managed cluster is using self-signed certificates.
  Well-knownCA Self-Signed LetsEncryptStaging
  In this case, no additional configuration is necessary.
  If the managed cluster certificates are self-signed, create a file called managed1.yaml containing the CA certificate of the managed cluster as the value of the cacrt field. In the following commands, the managed cluster’s CA certificate is saved in an environment variable called MGD_CA_CERT. Then use the --dry-run option of the kubectl command to generate the managed1.yaml file.
```
# On the managed cluster
$ export MGD_CA_CERT=$(kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     get secret verrazzano-tls \
     -n verrazzano-system \
     -o jsonpath="{.data.ca\.crt}" | base64 --decode)
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
  create secret generic "ca-secret-managed1" \
  -n verrazzano-mc \
  --from-literal=cacrt="$MGD_CA_CERT" \
  --dry-run=client \
  -o yaml > managed1.yaml
```
  Create a Secret on the admin cluster that contains the CA certificate for the managed cluster. This secret will be used for scraping metrics from the managed cluster. The managed1.yaml file that was created in the previous step provides input to this step.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
     apply -f managed1.yaml

# Once the command succeeds, you may delete the managed1.yaml file
$ rm managed1.yaml
```
  If the managed cluster certificates are LetsEncrypt staging, then create a file called managed1.yaml containing the CA certificate of the managed cluster as the value of the cacrt field. In the following commands, the managed cluster’s CA certificate is saved in an environment variable called MGD_CA_CERT. Then use the --dry-run option of the kubectl command to generate the managed1.yaml file.
```
# On the managed cluster
$ export MGD_CA_CERT=$(kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     get secret tls-ca-additional \
     -n cattle-system \
     -o jsonpath="{.data.ca-additional\.pem}" | base64 --decode)
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
  create secret generic "ca-secret-managed1" \
  -n verrazzano-mc \
  --from-literal=cacrt="$MGD_CA_CERT" \
  --dry-run=client \
  -o yaml > managed1.yaml
```
  Create a Secret on the admin cluster that contains the CA certificate for the managed cluster. This secret will be used for scraping metrics from the managed cluster. The managed1.yaml file that was created in the previous step provides input to this step.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
     apply -f managed1.yaml

# After the command succeeds, you may delete the managed1.yaml file
$ rm managed1.yaml
```
Use the following instructions to obtain the Kubernetes API server address for the admin cluster. This address must be accessible from the managed cluster.

MostKubernetesClusters KindClusters
For most types of Kubernetes clusters, except for Kind clusters, you can find the externally accessible API server address of the admin cluster from its kubeconfig file.
```
# View the information for the admin cluster in your kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN config view --minify

# Sample output
apiVersion: v1
kind: Config
clusters:
- cluster:
  certificate-authority-data: DATA+OMITTED
  server: https://11.22.33.44:6443
  name: my-admin-cluster
contexts:
....
....
```
In the output of this command, you can find the URL of the admin cluster API server from the server entry. Set the value of the ADMIN_K8S_SERVER_ADDRESS variable to this URL.
```
export ADMIN_K8S_SERVER_ADDRESS=<the server address from the config output>
```
Kind clusters run within a Docker container. If your admin and managed clusters are Kind clusters, the API server address of the admin cluster in its kubeconfig file is usually a local address on the host machine, which will not be accessible from the managed cluster. Use the kind command to obtain the “internal” kubeconfig of the admin cluster, which will contain a server address accessible from other Kind clusters on the same machine, and therefore in the same Docker network.
```
$ kind get kubeconfig --internal --name <your-admin-cluster-name> | grep server
```
In the output of this command, you can find the URL of the admin cluster API server from the server entry. Set the value of the ADMIN_K8S_SERVER_ADDRESS variable to this URL.
```
export ADMIN_K8S_SERVER_ADDRESS=<the server address from the config output>
```

On the admin cluster, create a ConfigMap that contains the externally accessible admin cluster Kubernetes server address found in the previous step.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: v1
kind: ConfigMap
metadata:
  name: verrazzano-admin-cluster
  namespace: verrazzano-mc
data:
  server: "${ADMIN_K8S_SERVER_ADDRESS}"
EOF

Registration steps

Perform the first three registration steps on the admin cluster, and the last step, on the managed cluster. The cluster against which to run the command is indicated in each code block.

On the admin cluster

To begin the registration process for a managed cluster named managed1, apply the VerrazzanoManagedCluster object on the admin cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Test VerrazzanoManagedCluster object"
  caSecret: ca-secret-managed1
EOF

Wait for the VerrazzanoManagedCluster resource to reach the Ready status. At that point, it will have generated a YAML file that must be applied on the managed cluster to complete the registration process.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    wait --for=condition=Ready \
    vmc managed1 -n verrazzano-mc
```

Export the YAML file created to register the managed cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register.yaml

On the managed cluster

Apply the registration file exported in the previous step, on the managed cluster.

# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register.yaml

# Once the command succeeds, you may delete the register.yaml file
$ rm register.yaml

After this step, the managed cluster will begin connecting to the admin cluster periodically. When the managed cluster connects to the admin cluster, it will update the Status field of the VerrazzanoManagedCluster resource for this managed cluster, with the following information:

The timestamp of the most recent connection made from the managed cluster, in the lastAgentConnectTime status field.
The host address of the Prometheus instance running on the managed cluster, in the prometheusHost status field. This is then used by the admin cluster to scrape metrics from the managed cluster.
The API address of the managed cluster, in the apiUrl status field. This is used by the admin cluster’s authentication proxy to route incoming requests for managed cluster information, to the managed cluster’s authentication proxy.

Verify that managed cluster registration completed

You can perform all the verification steps on the admin cluster.

Verify that the managed cluster can connect to the admin cluster. View the status of the VerrazzanoManagedCluster resource on the admin cluster, and check whether the lastAgentConnectTime, prometheusUrl, and apiUrl fields are populated. This may take up to two minutes after completing the registration steps.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get vmc managed1 -n verrazzano-mc -o yaml

# Sample output showing the status field
spec:
  ....
  ....
status:
  apiUrl: https://verrazzano.default.172.18.0.211.nip.io
  conditions:
  - lastTransitionTime: "2021-07-07T15:49:43Z"
    message: Ready
    status: "True"
    type: Ready
  lastAgentConnectTime: "2021-07-16T14:47:25Z"
  prometheusHost: prometheus.vmi.system.default.172.18.0.211.nip.io

Verify that the managed cluster is successfully registered with Rancher. When you perform the registration steps, Verrazzano also registers the managed cluster with Rancher. View the Rancher UI on the admin cluster. If the registration with Rancher was successful, then your cluster will be listed in Rancher’s list of clusters, and will be in Active state. You can find the Rancher UI URL for your cluster by following the instructions for Accessing Verrazzano.

Verify that managed cluster metrics are being collected

Verify that the admin cluster is collecting metrics from the managed cluster. The Prometheus output will include records that contain the name of the Verrazzano cluster (labeled as verrazzano_cluster).

You can find the Prometheus UI URL for your cluster by following the instructions for Accessing Verrazzano. Run a query for a metric (for example, node_disk_io_time_seconds_total).

Sample output of a Prometheus query

An alternative approach to using the Prometheus UI is to query metrics from the command line. Here is an example of how to obtain Prometheus metrics from the command line. Search the output of the query for responses that have the verrazzano_cluster field set to the name of the managed cluster.

# On the admin cluster
$ prometheusUrl=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.prometheusUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} "${prometheusUrl}/api/v1/query?query=node_disk_io_time_seconds_total"

Verify that managed cluster logs are being collected

Verify that the admin cluster is collecting logs from the managed cluster. The output will include records which have the name of the managed cluster in the cluster_name field.

You can find the OpenSearch Dashboards UI URL for your cluster by following the instructions for Accessing Verrazzano. Searching the verrazzano-system data stream for log records with the cluster_name set to the managed cluster name yields logs for the managed cluster.

Sample output of a OpenSearch Dashboards screen

An alternative approach to using the OpenSearch Dashboards UI is to query OpenSearch from the command line. Here is an example of how to obtain log records from the command line. Search the output of the query for responses that have the cluster_name field set to the name of the managed cluster.

# On the admin cluster
$ OS_URL=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.elasticUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} -X POST -H 'kbn-xsrf: true' "${OS_URL}/verrazzano-system/_search?size=25"

Run applications in multicluster Verrazzano

The Verrazzano multicluster setup is now complete and you can deploy applications by following the Multicluster Hello World Helidon example application.

Use the admin cluster UI

The admin cluster serves as a central point from which to register and deploy applications to managed clusters.

In the Verrazzano UI on the admin cluster, you can view the following:

The managed clusters registered with this admin cluster.
VerrazzanoProjects located on this admin cluster, or any of its registered managed clusters, or both.
Applications located on this admin cluster, or any of its registered managed clusters, or both.

Docs: Insufficient Memory

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were nodes reporting insufficient memory.

Steps

Review the analysis data to identify the specific nodes involved.
Review the nodes to determine why they do not have sufficient memory.

a. Are the nodes sized correctly for the workload?
1. For the minimum resources required for installing Verrazzano, see the Installation Guide.
2. Refer to documentation for other applications that you are deploying for resource guidelines and take those into account.
b. Is something unexpected running on the nodes or consuming more memory than expected?

Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: Application Deployment Guide

Overview

What you need

Application development

Application deployment

Verrazzano components

Verrazzano application configurations

Deploy the application

Verify the deployment

Explore the application

Access the application’s logs

Access the application’s metrics

Suppress Kiali console warnings

Remove the application

Docs: Application Deployment

Review oam-kubernetes-runtime operator status

Review verrazzano-application-operator operator status

Review oam-kubernetes-runtime operator logs

Review verrazzano-application-operator logs

Review generated workload resources

Review generated Trait resources

Check for RBAC privilege issues

Check resource owners

Check related resources

Docs: DNS

How Verrazzano constructs a DNS domain

Prerequisites

Create an Oracle Cloud Infrastructure API secret in the target cluster

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Installation

Docs: Install Guide

Prerequisites

Prepare for the install

Install the Verrazzano platform operator

Perform the install

NOTE

Install Verrazzano

Verify the install

(Optional) Run the example applications

To get the consoles URLs and credentials, see Access Verrazzano.

Docs: Multicluster Verrazzano

Verify managed cluster registration and connectivity

Verify VerrazzanoProject placement

Check the multicluster resource status

Re-register the managed cluster

Docs: Prerequisites

Supported hardware

Supported software versions

Kubernetes

WebLogic Server

Coherence

Helidon

Installed components

Docs: Verrazzano Analysis Tools

Tools Setup

Linux Instructions

Download the tooling

Verify the downloaded files

Unpack the verrazzano-analysis binary

Mac Instructions

Download the tooling

Verify the downloaded files

Unpack the verrazzano-analysis binary

Use the k8s-dump-cluster.sh tool

Use the verrazzano-analysis tool

Usage information

Docs: Verrazzano and the Open Application Model

How does OAM work?

Docs: Verrazzano in a Multicluster Environment

Docs: Verrazzano Projects

Docs: Certificates

NOTE

Use the Verrazzano self-signed CA

Use a custom CA

Use LetsEncrypt certificates

NOTE

NOTE

LetsEncrypt staging versus production

Docs: IngressTrait Custom Resource Definition

Review `oam-kubernetes-runtime` operator status

Review `verrazzano-application-operator` operator status

Review `oam-kubernetes-runtime` operator logs

Review `verrazzano-application-operator` logs

Unpack the `verrazzano-analysis` binary

Unpack the `verrazzano-analysis` binary

Use the `k8s-dump-cluster.sh` tool

Use the `verrazzano-analysis` tool

Create a database using MySQL called `tododb`