Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: Application Deployment Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Developing and deploying an application to Verrazzano consists of:

Packaging the application as a Docker image.
Publishing the application’s Docker image to a container registry.
Applying the application’s Verrazzano components to the cluster.
Applying the application’s Verrazzano applications to the cluster.

This guide does not provide the full details for the first two steps. An existing example application Docker image has been packaged and published for use.

Verrazzano supports application definition using Open Application Model (OAM). Verrrazzano applications are composed of components and application configurations. This document demonstrates creating OAM resources that define an application as well as the steps required to deploy those resources.

What you need

About 10 minutes.
Access to an existing Kubernetes cluster with Verrazzano installed.
Access to the application’s image in GitHub Container Registry.

Confirm access using this command to pull the example’s Docker image:
```
$ docker pull ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.12-1-20210218160249-d8db8f3
```

Application development

This guide uses an example application which was written with Java and Helidon. For the implementation details, see the Helidon MP tutorial. See the application source code in the Verrazzano examples repository.

The example application is a JAX-RS service and implements the following REST endpoints:

/greet - Returns a default greeting message that is stored in memory. This endpoint accepts the GET HTTP request method.
/greet/{name} - Returns a greeting message including the name provided in the path parameter. This endpoint accepts the GET HTTP request method.
/greet/greeting - Changes the greeting message to be used in future calls to the other endpoints. This endpoint accepts the PUT HTTP request method and a JSON payload.

The following code shows a portion of the application’s implementation. The Verrazzano examples repository contains the complete implementation. An important detail here is that the application contains a single resource exposed on path /greet.

package io.helidon.examples.quickstart.mp;
...
@Path("/greet")
@RequestScoped
public class GreetResource {

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getDefaultMessage() {
        ...
    }

    @Path("/{name}")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getMessage(@PathParam("name") String name) {
        ...
    }

    @Path("/greeting")
    @PUT
    @Consumes(MediaType.APPLICATION_JSON)
    ...
    public Response updateGreeting(JsonObject jsonObject) {
        ...
    }

}

A Dockerfile is used to package the completed application JAR file into a Docker image. The following code shows a portion of the Dockerfile. The Verrazzano examples repository contains the complete Dockerfile. Note that the Docker container exposes a single port 8080.

FROM ghcr.io/oracle/oraclelinux:7-slim
...
CMD java -cp /app/helidon-quickstart-mp.jar:/app/* io.helidon.examples.quickstart.mp.Main
EXPOSE 8080

Application deployment

When you deploy applications with Verrazzano, the platform sets up connections, network policies, and ingresses in the service mesh, and wires up a monitoring stack to capture the metrics, logs, and traces. Verrazzano employs OAM Components to define the functional units of a system that are then assembled and configured by defining associated application configurations.

Verrazzano components

A Verrazzano OAM Component is a Kubernetes Custom Resource describing an application’s general composition and environment requirements. The following code shows the component for the example application used in this guide. This resource describes a component which is implemented by a single Docker image containing a Helidon application exposing a single endpoint.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.10-3-20201016220428-56fb4d4"
              ports:
                - containerPort: 8080
                  name: http

A brief description of each field of the component:

apiVersion - Version of the component custom resource definition
kind - Standard name of the component custom resource definition
metadata.name - The name used to create the component’s custom resource
metadata.namespace - The namespace used to create this component’s custom resource
spec.workload.kind - VerrazzanoHelidonWorkload defines a stateless workload of Kubernetes
spec.workload.spec.deploymentTemplate.podSpec.metadata.name - The name used to create the stateless workload of Kubernetes
spec.workload.spec.deploymentTemplate.podSpec.containers - The implementation containers
spec.workload.spec.deploymentTemplate.podSpec.containers.ports - Ports exposed by the container

Verrazzano application configurations

A Verrazzano application configuration is a Kubernetes Custom Resource which provides environment specific customizations. The following code shows the application configuration for the example used in this guide. This resource specifies the deployment of the application to the hello-helidon namespace. Additional runtime features are specified using traits, or runtime overlays that augment the workload. For example, the ingress trait specifies the ingress host and path, while the metrics trait provides the Prometheus scraper used to obtain the application related metrics.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
                scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

A brief description of each field in the application configuration:

apiVersion - Version of the ApplicationConfiguration custom resource definition
kind - Standard name of the application configuration custom resource definition
metadata.name - The name used to create this application configuration resource
metadata.namespace - The namespace used for this application configuration custom resource
spec.components - Reference to the application’s components leveraged to specify runtime configuration
spec.components[].traits - The traits specified for the application’s components

To explore traits, we can examine the fields of an ingress trait:

apiVersion - Version of the OAM trait custom resource definition
kind - IngressTrait is the name of the OAM application ingress trait custom resource definition
spec.rules.paths - The context paths for accessing the application

Deploy the application

The following steps are required to deploy the example application. Steps similar to the apply steps would be used to deploy any application to Verrazzano.

Create a namespace for the example application and add labels identifying the namespace as managed by Verrazzano and enabled for Istio.
```
$ kubectl create namespace hello-helidon
$ kubectl label namespace hello-helidon verrazzano-managed=true istio-injection=enabled
```
Apply the application’s component.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/hello-helidon/hello-helidon-comp.yaml
```
This step causes the validation and creation of the Component resource. No other resources or objects are created as a result. Application configurations applied in the future may reference this Component resource.
Apply the application configuration.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/hello-helidon/hello-helidon-app.yaml
```
This step causes the validation and creation of the application configuration resource. This operation triggers the activation of a number of Verrazzano operators. These operators create other Kubernetes objects (for example, Deployments, ReplicaSets, Pods, Services, Ingresses) that collectively provide and support the application.
Configure the application’s DNS resolution.

After deploying the application, configure DNS to resolve the application’s ingress DNS name to the application’s load balancer IP address. The generated host name is obtained by querying Kubernetes for the gateway:
```
$ kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-appconf-gw \
    -n hello-helidon \
    -o jsonpath='{.spec.servers[0].hosts[0]}'
```
The load balancer IP is obtained by querying Kubernetes for the Istio ingress gateway status:
```
$ kubectl get service \
    -n istio-system istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```
DNS configuration steps are outside the scope of this guide. For DNS infrastructure that can be configured and used, see the Oracle Cloud Infrastructure DNS documentation. In some small non-production scenarios, DNS configuration using /etc/hosts or an equivalent may be sufficient.

Verify the deployment

Applying the application configuration initiates the creation of several Kubernetes objects. Actual creation and initialization of these objects occurs asynchronously. The following steps provide commands for determining when these objects are ready for use.

Note: Many other Kubernetes objects unrelated to the example application may also exist. Those have been omitted from the lists.

Verify the Helidon application pod is running.

$ kubectl get pods -n hello-helidon -l app=hello-helidon

# Sample output
NAME                                        READY   STATUS    RESTARTS   AGE
hello-helidon-deployment-8664954995-wcb9d   2/2     Running   0          5m5s

Verify that the Verrazzano application operator pod is running.
```
$ kubectl get pod -n verrazzano-system -l app=verrazzano-application-operator

# Sample output
NAME                                               READY   STATUS    RESTARTS   AGE
verrazzano-application-operator-79849b89ff-lr9w6   1/1     Running   0          13m
```
The namespace verrazzano-system is used by Verrazzano for non-application objects managed by Verrazzano. A single verrazzano-application-operator manages the life cycle of all OAM based applications within the cluster.

Verify the Verrazzano monitoring infrastructure is running.

$ kubectl get pods -n verrazzano-system | grep '^NAME\|vmi-system'

# Sample output
NAME                                               READY   STATUS    RESTARTS   AGE
vmi-system-es-master-0                             2/2     Running   0          47m
vmi-system-grafana-799d79648d-wsdp4                2/2     Running   0          47m
vmi-system-kiali-574c6dd94d-f49jv                  2/2     Running   0          51m
vmi-system-kibana-77f8d998f4-zzvqr                 2/2     Running   0          47m
vmi-system-prometheus-0-7f89d54fbf-brg6x           3/3     Running   0          45m

These pods in the verrazzano-system namespace constitute a monitoring stack created by Verrazzano for the deployed applications.

The monitoring infrastructure comprises several components:

vmi-system-es - Elasticsearch for log collection
vmi-system-grafana - Grafana for metric visualization
vms-system-kiali - Kiali for management console of istio service mesh
vmi-system-kibana - Kibana for log visualization
vmi-system-prometheus - Prometheus for metric collection

Diagnose failures.

View the event logs of any pod not entering the Running state within a reasonable length of time, such as five minutes.
```
$ kubectl describe pod -n hello-helidon -l app=hello-helidon
```
Use the specific namespace and name for the pod being investigated.

Explore the application

Follow these steps to explore the application’s functionality. If DNS was not configured, then use the alternative commands.

Save the host name and IP address of the load balancer exposing the application’s REST service endpoints for later.

$ HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-appconf-gw \
      -n hello-helidon \
      -o jsonpath='{.spec.servers[0].hosts[0]}')
$ ADDRESS=$(kubectl get service \
      -n istio-system istio-ingressgateway \
      -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

NOTE:

The value of ADDRESS is used only if DNS has not been configured.
The following alternative commands may not work in conjunction with firewalls that validate HTTP Host headers.

Get the default message.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet"

# Expected response
{"message":"Hello World!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet" \
    --resolve ${HOST}:443:${ADDRESS}

Get a message for Robert.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert"

# Expected response
{"message":"Hello Robert!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET
    "https://${HOST}/greet/Robert" \
    --resolve ${HOST}:443:${ADDRESS}

Update the default greeting.

$ curl -sk \
    -X PUT \
    "https://${HOST}/greet/greeting" \
    -H 'Content-Type: application/json' \
    -d '{"greeting" : "Greetings"}'

If DNS has not been configured, then use this command.

$ curl -sk \
    -X PUT \
    "https://${HOST}/greet/greeting" \
    -H 'Content-Type: application/json' \
    -d '{"greeting" : "Greetings"}' \
    --resolve ${HOST}:443:${ADDRESS}

Get the new message for Robert.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert"

# Expected response
{"message":"Greetings Robert!"}

If DNS has not been configured, then use this command.

$ curl -sk \
    -X GET \
    "https://${HOST}/greet/Robert" \
    --resolve ${HOST}:443:${ADDRESS}

Access the application’s logs

Deployed applications have log collection enabled. These logs are collected using Elasticsearch and can be accessed using Kibana. Elasticsearch and Kibana are examples of infrastructure Verrazzano creates in support of an application as a result of applying an application configuration. For more information on creating an index pattern and visualizing the log data collected in Elasticsearch, see Kibana.

Determine the URL to access Kibana:

$ KIBANA_HOST=$(kubectl get ingress \
     -n verrazzano-system vmi-system-kibana \
     -o jsonpath='{.spec.rules[0].host}')
$ KIBANA_URL="https://${KIBANA_HOST}"
$ echo "${KIBANA_URL}"
$ open "${KIBANA_URL}"

The user name to access Kibana defaults to verrazzano during the Verrazzano installation.

Determine the password to access Kibana:

$ echo $(kubectl get secret \
      -n verrazzano-system verrazzano \
      -o jsonpath={.data.password} | base64 \
      --decode)

Access the application’s metrics

Deployed applications have metric collection enabled. Grafana can be used to access these metrics collected by Prometheus. Prometheus and Grafana are additional components Verrazzano creates as a result of applying an application configuration. For more information on visualizing Prometheus metrics data, see Grafana.

Determine the URL to access Grafana:

$ GRAFANA_HOST=$(kubectl get ingress \
      -n verrazzano-system vmi-system-grafana \
      -o jsonpath='{.spec.rules[0].host}')
$ GRAFANA_URL="https://${GRAFANA_HOST}"
$ echo "${GRAFANA_URL}"
$ open "${GRAFANA_URL}"

The user name to access Grafana is set to the default value verrazzano during the Verrazzano installation.

Determine the password to access Grafana:

$ echo $(kubectl get secret \
      -n verrazzano-system verrazzano \
      -o jsonpath={.data.password} | base64 \
      --decode)

Alternatively, metrics can be accessed directly using Prometheus. Determine the URL for this access:

$ PROMETHEUS_HOST=$(kubectl get ingress \
      -n verrazzano-system vmi-system-prometheus \
      -o jsonpath='{.spec.rules[0].host}')
$ PROMETHEUS_URL="https://${PROMETHEUS_HOST}"
$ echo "${PROMETHEUS_URL}"
$ open "${PROMETHEUS_URL}"

The user name and password for both Prometheus and Grafana are the same.

Remove the application

Run the following commands to delete the application configuration, and optionally the component and namespace.

Delete the application configuration.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/hello-helidon/hello-helidon-app.yaml
```
The deletion of the application configuration will result in the destruction of all application-specific Kubernetes objects.
(Optional) Delete the application’s component.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/hello-helidon/hello-helidon-comp.yaml
```
Note: This step is not required if other application configurations for this component will be applied in the future.

(Optional) Delete the namespace.

$ kubectl delete namespace hello-helidon

Docs: Application Deployment

Mon, 01 Jan 0001 00:00:00 +0000

During application deployment, the oam-kubernetes-runtime and verrazzano-application-operator cooperate through the generation and update of Kubernetes resources. The oam-kubernetes-runtime processes the ApplicationConfiguration and Component resources provided by the user and generates workload and Trait resources. The verrazzano-application-operator processes Verrazzano specific workload and Trait resources. These are then used to generate additional child and related resources.

Troubleshooting application deployments should follow three general steps:

Review the status of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the logs of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the resources generated by the oam-kubernetes-runtime and the verrazzano-application-operator.

Review `oam-kubernetes-runtime` operator status

For application deployment to succeed, the oam-kubernetes-runtime pod must have a status of Running.

Use the following command to get the pod status.

$ kubectl get pods \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

If the pod status is not Running, then see the following instructions for reviewing the oam-kubernetes-runtime pod logs.

Review `verrazzano-application-operator` operator status

For application deployment to succeed, the verrazzano-application-operator pod must have a status of Running.

Use the following command to get the pod status.

$ kubectl get pods \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If the pod status is not Running, then see the following instructions for reviewing the verrazzano-application-operator logs.

Review `oam-kubernetes-runtime` operator logs

Review the oam-kubernetes-runtime pod logs for any indication that pod startup or the generation of workloads or traits has failed.

Use the following command to get the logs.

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

Review `verrazzano-application-operator` logs

Review the verrazzano-application-operator logs for any indication that pod startup or resource generation has failed.

Use the following command to get the logs.

$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

Review generated workload resources

The processing of a Component reference within an ApplicationConfiguration results in the generation of workloads. For example, a referenced Component might result in the generation of a VerrazzanoHelidonWorkload workload resource. In turn, the VerrazzanoHelidonWorkload workload resource will be processed and result in the generation of related Deployment and Service resources.

If the expected workload resource, for example VerrazzanoHelidonWorkload, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Deployment or Service, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to a VerrazzanoHelidonWorkload deployment.

$ kubectl get -n hello-helidon verrazzanohelidonworkload hello-helidon-workload
$ kubectl get -n hello-helidon deployment hello-helidon-deployment
$ kubectl get -n hello-helidon service hello-helidon-deployment

Review generated Trait resources

The processing of traits embedded with an ApplicationConfiguration results in the generation of Trait resources. For example, an IngressTrait embedded within an ApplicationConfiguration will result in the generation of an IngressTrait resource. In turn, the IngressTrait resource will be processed and result in the generation of related Certificate, Gateway, and VirtualService resources.

If the expected Trait resource, for example IngressTrait, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Certificate, Gateway, and VirtualService, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to an IngressTrait.

$ kubectl get -n hello-helidon ingresstrait hello-helidon-ingress
$ kubectl get -n istio-system Certificate hello-helidon-hello-helidon-appconf-cert
$ kubectl get -n hello-helidon gateway hello-helidon-hello-helidon-appconf-gw
$ kubectl get -n hello-helidon virtualservice hello-helidon-ingress-rule-0-vs

Check for RBAC privilege issues

The use of generic Kubernetes resources as workloads and traits can result in deployment failures if privileges are insufficient. In this case, the oam-kubernetes-runtime logs will contain errors containing the term forbidden.

The following command shows how to query for this type of failure message.

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime | grep forbidden

Check resource owners

Kubernetes maintains the child to parent relationship within metadata fields.

The following example returns the parent of the IngressTrait, named hello-helidon-ingress, in the hello-helidon namespace.

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .metadata.ownerReferences[*]}{.name}{"\n"}{end}'

The results of this command can help identify the lineage of a given resource.

Some resources also record the related resources affected during their processing. For example, when processed, an IngressTrait will create related Gateway, VirtualService, and Certificate resources.

The following command is an example of how to obtain the related resources of an IngressTraits.

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .status.resources[*]}{.kind}: {.name}{"\n"}{end}'

The results of this command can help identify which other resources, the given resource affected.

Docs: Customize DNS

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports three DNS choices for Verrazzano services and applications:

Free wildcard DNS services (nip.io and sslip.io)
Oracle Cloud Infrastructure DNS managed by Verrazzano
Custom (user-managed) DNS

How Verrazzano constructs a DNS domain

Regardless of which DNS management you use, the value in the spec.environmentName field in your installation will be prepended to the configured domain in the spec.components.dns section of the custom resource, to form the full DNS domain name used to access Verrazzano endpoints.

For example, if spec.environmentName is set to sales and the domain is configured in spec.components.dns as us.example.com, Verrazzano will create sales.us.example.com as the DNS domain for the installation.

Verrazzano can be configured to use either the nip.io or sslip.io free wildcard DNS services. When queried with a hostname with an embedded IP address, wildcard DNS services return that IP address.

For example, using the nip.io service, the following DNS names all map to the IP address 10.0.0.1:

10.0.0.1.nip.io
app.10.0.0.1.nip.io
customer1.app.10.0.0.1.nip.io

To configure Verrazzano to use one of these services, set the spec.wildcard.domain field in the Verrazzano custom resource to either nip.io or sslip.io; the default is nip.io.

For example, the following configuration uses sslip.io, instead of nip.io, for wildcard DNS with a dev installation profile:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    dns:
      wildcard:
        domain: sslip.io

Verrazzano can directly manage records in Oracle OCI DNS when configured to use the spec.components.dns.oci field. This is achieved through the External DNS Service, which is a component that is conditionally installed when OCI DNS is configured for DNS management in Verrazzano.

Prerequisites

The following prerequisites must be met before using OCI DNS with Verrazzano:

You must have control of a DNS domain.
You must have an OCI DNS Service Zone that is configured to manage records for that domain. Verrazzano also supports the use of both GLOBAL and PRIVATE OCI DNS zones.

A DNS Service Zone is a distinct portion of a domain namespace. You must ensure that the zone is appropriately associated with a parent domain. For example, an appropriate zone name for parent domain example.com is us.example.com.

To create an OCI DNS zone using the OCI CLI:
```
$ oci dns zone create \
    -c <compartment ocid> \
    --name <zone-name-prefix>.example.com \
    --zone-type PRIMARY
```
To create an OCI DNS zone using the OCI Console, see Managing DNS Service Zones.

You must have a valid OCI API signing key that can be used to communicate with OCI DNS in your tenancy.

For example, you can create an API signing key using the OCI CLI:

  $ oci setup keys --key-name myapikey
  Enter a passphrase for your private key (empty for no passphrase):
  Public key written to: /Users/jdoe/.oci/myapikey_public.pem
  Private key written to: /Users/jdoe/.oci/myapikey.pem
  Public key fingerprint: 39:08:44:69:9f:f5:73:86:7a:46:d8:ad:34:4f:95:29


      If you haven't already uploaded your API signing public key through the
      console, follow the instructions on the page linked below in the section
      'How to upload the public key':

          https://docs.cloud.oracle.com/Content/API/Concepts/apisigningkey.htm#How2

After the key pair has been created, you must upload the public key to your account in your OCI tenancy. For details, see the OCI documentation, Required Keys and OCIDs.

Create an OCI API secret in the target cluster

To communicate with OCI DNS to manage DNS records, Verrazzano needs to be made aware of the necessary API credentials.
A generic Kubernetes secret must be created in the cluster’s verrazzano-install namespace with the required credentials. That secret must then be referenced by the custom resource that is used to install Verrazzano.

After you have an OCI API key ready for use, create a YAML file, oci.yaml, with the API credentials in the form:

auth:
  region: <oci-region>
  tenancy: <oci-tenancy-ocid>
  user: <oci-user-ocid>
  key: |
    <oci-api-private-key-file-contents>
  fingerprint: <oci-api-private-key-fingerprint>

This information typically can be found in your OCI CLI config file or in the OCI Console. The <oci-api-private-key-file-contents> contents are the PEM-encoded contents of the key_file value within the OCI CLI configuration profile.

For example, your oci.yaml file will look similar to the following:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1.....
  user: ocid1.user.oc1.....
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:d3:4c:gh:fd:9e:27:g8:b9:0d:9f:00:22:33:c3:gg

Verrazzano also supports the use of instance principals to communicate with OCI in order to create or update OCI DNS records. Instance principal requires some prerequisites that can be found here.

When using instance principals, your oci.yaml file will look as follows:

auth:
  authtype: instance_principal

Then, you can create a generic Kubernetes secret in the cluster’s verrazzano-install namespace using kubectl.

$ kubectl create secret generic -n verrazzano-install <secret-name> --from-file=<path-to-oci-yaml-file>

For example, to create a secret named oci from a file oci.yaml, do the following:

$ kubectl create secret generic -n verrazzano-install oci --from-file=oci.yaml

This secret will later be referenced from the Verrazzano custom resource used during installation.

Use a Verrazzano helper script to create an OCI secret

Verrazzano also provides a helper script to create the necessary Kubernetes secret based on your OCI CLI config file, assuming that you have the OCI CLI installed and a valid OCI CLI profile with the required API key information. The script create_oci_config_secret.sh reads your OCI CLI configuration file to create the secret.

First, download the create_oci_config_secret.sh script:

$ curl \
    -o ./create_oci_config_secret.sh \
    https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/platform-operator/scripts/install/create_oci_config_secret.sh

Next, set your KUBECONFIG environment variable to point to your cluster and run create_oci_config_secret.sh -h to display the script options:

$ chmod +x create_oci_config_secret.sh
$ export KUBECONFIG=<kubeconfig-file>
$ ./create_oci_config_secret.sh  -h
usage: ./create_oci_config_secret.sh [-o oci_config_file] [-s config_file_section]
  -o oci_config_file         The full path to the OCI configuration file (default ~/.oci/config)
  -s config_file_section     The properties section within the OCI configuration file.  Default is DEFAULT
  -k secret_name             The secret name containing the OCI configuration.  Default is oci
  -c context_name            The kubectl context to use
  -a auth_type               The auth_type to be used to access OCI. Valid values are user_principal/instance_principal. Default is user_principal.
  -h                         Help

For example, to have the script create the YAML file using your [DEFAULT] OCI CLI profile and then create a Kubernetes secret named oci, you can run the script with no arguments, as follows:

$ ./create_oci_config_secret.sh
secret/oci created

The following example creates a secret myoci using an OCI CLI profile named [dev]:

$ ./create_oci_config_secret.sh -s dev -k myoci
secret/myoci created

When using instance principals all other parameters will be ignored automatically. The following example creates a secret myoci using OCI instance principal:

$ ./create_oci_config_secret.sh -a instance_principal
secret/myoci created

Installation

After the OCI API secret is created, create a Verrazzano custom resource for the installation that is configured to use OCI DNS, and reference the secret you created.

As a starting point, download the sample Verrazzano custom resource install-oci.yaml file for OCI DNS:

$ curl \
    -o ./install-oci.yaml \
    https://raw.githubusercontent.com/verrazzano/verrazzano/release-1.1/platform-operator/config/samples/install-oci.yaml

Edit the install-oci.yaml file to provide values for the following configuration settings in the custom resource spec:

spec.environmentName
spec.components.dns.oci.ociConfigSecret
spec.components.dns.oci.dnsZoneCompartmentOCID
spec.components.dns.oci.dnsZoneOCID
spec.components.dns.oci.dnsZoneName
spec.components.dns.oci.dnsScope

The field spec.components.dns.oci.ociConfigSecret should reference the secret created earlier. For details on the OCI DNS configuration settings, see spec.components.dns.oci.

For example, a custom resource for a prod installation profile using OCI DNS might look as follows, yielding a domain of myenv.example.com (OCI identifiers redacted):

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com

If using a private DNS zone, then the same prod installation profile using OCI DNS will look as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: my-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com
        dnsScope: PRIVATE

After the custom resource is ready, apply it using kubectl apply -f <path-to-custom-resource-file>.

You can specify your own externally managed, custom DNS domain. In this scenario, you manage your own DNS domain and all DNS records in that domain.

An externally managed DNS domain is specified in the spec.components.dns.external.suffix field of the Verrazzano custom resource.

When using an externally managed DNS domain, you are responsible for:

Configuring A records for Verrazzano ingress points (load balancers)
Configuring CNAME records for hostnames in the domain that point to the A records, as needed

The Verrazzano installer searches the DNS zone you provide for two specific A records.
These are used to configure the cluster and should refer to external addresses of the load balancers provisioned by the user.

The A records need to be created manually.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example, if spec.environmentName is set to myenv, and spec.components.dns.external.suffix is set to example.com, the A records would need to be set up as follows:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

This example assumes that load balancers exist for ingress-mgmt on 198.51.100.10 and for ingress-verrazzano on 203.0.113.10.

For a more complete example, see the documentation for setting up Verrazzano on the OLCNE Platform.

Docs: Elasticsearch Scaling and Resizing

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to recover an Elasticsearch cluster’s health after it becomes unhealthy due to unassigned shards or disk pressure.

It also describes how to scale up the cluster’s data nodes and increase the size of the volumes. Because the volume size change in the Verrazzano operator also affects the master nodes volume size, you must take additional steps to address the volume resizing of a StatefulSet.

First:

# Edit the Verrazzano operator
$ kubectl -n verrazzano-system edit deploy verrazzano-operator

Then, change the following portion by increasing the number of ES_DATA_NODE_REPLICAS to 3, and the ES_DATA_STORAGE to 200Gi:

- name: ES_DATA_NODE_REPLICAS
  value: "3"

- name: ES_DATA_STORAGE
  value: "200"

Scaling Elasticsearch data nodes

Follow this procedure to scale the Elasticsearch data nodes.

Wait for the new data node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When you have a green state, replace the original data node -0 pods:

$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=0
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=0
$ kubectl -n verrazzano-system delete pod/vmi-system-es-data-0-xxxxxxxxx-xxxx pvc/vmi-system-es-data-0
$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=1
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=1

Wait for the new data node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When you have a green state, replace the original data node -1 pods:

$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=0
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=0
$ kubectl -n verrazzano-system delete pod/vmi-system-es-data-1-xxxxxxxxx-xxxx pvc/vmi-system-es-data-1
$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=1
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=1

Wait for the new data node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

You should now have three data nodes that are healthy and at 200GB volumes.

Address the master nodes’ StatefulSet

Now to address the master nodes. Because you cannot directly change the size of the volume associated with a volume template in a StatefulSet, you must follow this procedure:

First:

$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=0
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=0
$ kubectl -n verrazzano-system get sts vmi-system-es-master -o yaml > vmi-system-es-master.yaml

Edit the file created in the previous command, vmi-system-es-master.yaml

a. Remove the lines starting with:
```
creationTimestamp:
generation:
resourceVersion:
selfLink:
uid:
status:
```
b. Remove every line below status:

c. Edit the section to increase the storage to the same value that you used for the Verrazzano operator:
```
storage: 200Gi
```
d. Save that file.
The following command will delete the StatefulSet, but allow the associated pods to continue to run.
```
$ kubectl -n verrazzano-system delete sts vmi-system-es-master --cascade=orphan
```
Then run this command to recreate the StatefulSet with the new volume size defined:
```
$ kubectl -n verrazzano-system apply -f vmi-system-es-master.yaml
```
The next steps are to delete the existing master node pods, one at a time, allowing the cluster to become healthy before moving on to the next node:
```
$ kubectl -n verrazzano-system delete pod/vmi-system-es-master-0 pvc/elasticsearch-master-vmi-system-es-master-0
```

Wait for the new master node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When the cluster is healthy, continue to the next master node:

$ kubectl -n verrazzano-system delete pod/vmi-system-es-master-1 pvc/elasticsearch-master-vmi-system-es-master-1

Wait for the new master node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When the cluster is healthy, continue to the next master node:

$ kubectl -n verrazzano-system delete pod/vmi-system-es-master-2 pvc/elasticsearch-master-vmi-system-es-master-2

Wait for the new master node pod to become ready and then check the health of the cluster:

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/health

$ kubectl -n verrazzano-system exec -it vmi-system-es-master-0 -- curl http://127.0.0.1:9200/_cat/indices

When the cluster is healthy rescale the operators:

$ kubectl -n verrazzano-system scale deploy verrazzano-operator --replicas=1
$ kubectl -n verrazzano-system scale deploy verrazzano-monitoring-operator --replicas=1

Update the Verrazzano Custom Resource

Now you will edit the Verrazzano CR, so that when you upgrade, the above changes will not be overwritten. Follow this procedure:

Get the namespace and name of the Verrazzano CR:
```
$ kubectl get vz -A
```

Edit the Verrazzano CR:

$ kubectl -n <namespace> edit vz <name>

a. Alter to include the following:

spec:
  components:
    elasticsearch:
      installArgs:
      - name: nodes.data.replicas
        value: "3"
      - name: nodes.data.requests.storage
        value: 200Gi

b. Save the changes.

This completes the process.

Docs: Install Guide

Mon, 01 Jan 0001 00:00:00 +0000

The following instructions show you how to install Verrazzano in a single Kubernetes cluster.

Prerequisites

Find the Verrazzano prerequisite requirements here.
Review the list of the software versions supported and installed by Verrazzano.

Prepare for the install

Before installing Verrazzano, see instructions on preparing Kubernetes platforms.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For instructions on how to install a CNI plug-in, see the documentation for your Kubernetes cluster.

Install the Verrazzano platform operator

Verrazzano provides a platform operator to manage the life cycle of Verrazzano installations. Using the Verrazzano custom resource, you can install, uninstall, and upgrade Verrazzano installations.

To install the Verrazzano platform operator:

Deploy the Verrazzano platform operator.

$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/operator.yaml

Wait for the deployment to complete.

$ kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator

# Expected response
deployment "verrazzano-platform-operator" successfully rolled out

Confirm that the operator pod is correctly defined and running.

$ kubectl -n verrazzano-install get pods

# Sample output
NAME                                            READY   STATUS    RESTARTS   AGE
verrazzano-platform-operator-59d5c585fd-lwhsx   1/1     Running   0          114s

Perform the install

Verrazzano supports the following installation profiles: development (dev), production (prod), and managed cluster (managed-cluster). For more information on profiles, see Installation Profiles.

This page shows how to create a basic Verrazzano installation using:

The development (dev) installation profile
Wildcard-DNS, where DNS is provided by nip.io (the default)

NOTE

Because the dev profile installs self-signed certificates, when installing Verrazzano on macOS, you might see: Your connection is not private. For a workaround, see this FAQ.

For a complete description of Verrazzano configuration options, see the Verrazzano Custom Resource Definition.

To use other DNS options, see the Customzing DNS page for more details.

Install Verrazzano

To create a Verrazzano installation as described in the previous section, run the following commands:

$ kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: ${VZ_PROFILE:-dev}
EOF
$ kubectl wait \
    --timeout=20m \
    --for=condition=InstallComplete verrazzano/example-verrazzano

To use a different profile with the above example, set the VZ_PROFILE environment variable to the name of the profile you want to install.

If an error occurs, check the log output of the installation:

$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '"operation":"install"'

For more help troubleshooting the installation, see Analysis Advice.

After the installation is complete, you can use the console URLs. For more information on how to access the Verrazzano consoles, see Access Verrazzano.

Verify the install

Verrazzano installs multiple objects in multiple namespaces. In the verrazzano-system namespaces, all the pods in the Running state, does not guarantee, but likely indicates that Verrazzano is up and running.

$ kubectl get pods -n verrazzano-system

# Sample output
coherence-operator-dcfb446df-24djp                 1/1     Running   1          49m
fluentd-h65xf                                      2/2     Running   1          45m
oam-kubernetes-runtime-6645df49cd-6q96c            1/1     Running   0          49m
verrazzano-application-operator-85ffd7f77b-rhwk7   1/1     Running   0          48m
verrazzano-authproxy-58db5b9484-nhnql              2/2     Running   0          45m
verrazzano-console-5dbdc579bd-hm4rh                2/2     Running   0          45m
verrazzano-monitoring-operator-599654889d-lbb4z    1/1     Running   0          45m
verrazzano-operator-7b6fd64dd5-8j9h8               1/1     Running   0          45m
vmi-system-es-master-0                             2/2     Running   0          45m
vmi-system-grafana-5558d65b46-pxg78                2/2     Running   0          45m
vmi-system-kiali-5949966fb8-465s8                  2/2     Running   0          48m
vmi-system-kibana-86b894d8f6-q4vb5                 2/2     Running   0          45m
vmi-system-prometheus-0-859fcd87dc-m5ws9           3/3     Running   0          44m
weblogic-operator-646756c75c-hgz6j                 2/2     Running   0          49m

(Optional) Run the example applications

Example applications are located here.

To get the consoles URLs and credentials, see Access Verrazzano.

Docs: Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

This document describes some common problems you might encounter when using multicluster Verrazzano, and how to troubleshoot them.

If you created multicluster resources in the admin cluster, and specified a placement value in a managed cluster, then those resources will get created in that managed cluster. If they do not get created in the managed cluster, then use the following steps to troubleshoot:

Verify that the managed cluster is registered correctly and can connect to the admin cluster.
Verify that the VerrazzanoProject for the resource’s namespace, also has a placement in that managed cluster.
Check the multicluster resource’s status field on the admin cluster to know what the status of that resource is on each managed cluster to which it is targeted.

Verify managed cluster registration and connectivity

You can verify that a managed cluster was successfully registered with an admin cluster by viewing the corresponding VerrazzanoManagedCluster (VMC) resource on the admin cluster. For example, to verify that a managed cluster named managed1 was successfully registered:

# on the admin cluster
$ kubectl get verrazzanomanagedcluster managed1 \
    -n verrazzano-mc \
    -o yaml

Partial sample output from the previous command:

  status:
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:27Z"
      message: Ready
      status: "True"
      type: Ready
    lastAgentConnectTime: "2021-06-22T21:06:04Z"
    ... other fields ...

Check the lastAgentConnectTime in the status of the VMC resource. This is the last time at which the managed cluster connected to the admin cluster. If this value is not present, then the managed cluster named managed1 never successfully connected to the admin cluster. This could be due to several reasons:

The managed cluster registration process step of applying the registration YAML on the managed cluster, was not completed. For the complete setup instructions, see here.
The managed cluster does not have network connectivity to the admin cluster. The managed cluster will attempt to connect to the admin cluster at regular intervals, and any errors will be reported in the verrazzano-application-operator pod’s log on the managed cluster. View the logs using the following command.

# on the managed cluster
$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If these logs reveal that there is a connectivity issue, check the admin cluster Kubernetes server address that you provided during registration and ensure that it is correct, and that it is reachable from the managed cluster. If it is incorrect, then you will need to repeat the managed cluster registration process described in the setup instructions here.

Verify VerrazzanoProject placement

For Verrazzano to create an application namespace in a managed cluster, that namespace must be part of a VerrazzanoProject that:

Includes that namespace.
Has a placement value that includes that managed cluster.

View the details of the project that corresponds to your application’s namespace. In the example command that follows, the project name is assumed to be myproject. All projects are expected to be created in the verrazzano-mc namespace.

# on the admin cluster
$ kubectl get verrazzanoproject myproject \
    -n verrazzano-mc \
    -o yaml

The following partial sample output is for a project that will result in the namespace mynamespace being created on the managed cluster managed1.

spec:
  placement:
    clusters:
    - name: managed1
  template:
    namespaces:
    - metadata:
        name: mynamespace
....other fields....

Check the multicluster resource status

On the admin cluster, each multicluster resource’s status field is updated with the status of the underlying resource on each managed cluster in which it is placed.

The following example command shows how to view the status of a MultiClusterApplicationConfiguration named myapp, in the namespace mynamespace, that has a placement value that includes the managed cluster managed1

$ kubectl get multiclusterapplicationconfiguration myapp \
    -n mynamespace \
    -o yaml

The status of the underlying resource in each cluster specified in the placement is shown in the following partial sample output:

  status:
    clusters:
    - lastUpdateTime: "2021-06-22T21:05:04Z"
      message: OAM Application Configuration created
      name: managed1
      state: Succeeded
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:58Z"
      message: OAM Application Configuration created
      status: "True"
      type: DeployComplete
    state: Succeeded

The status message contains additional information on the operation’s success or failure.

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires the following:

A Kubernetes cluster and a compatible kubectl.
At least 2 CPUs, 100GB disk storage, and 16GB RAM available on the Kubernetes worker nodes. This is sufficient to install the development profile of Verrazzano. Depending on the resource requirements of the applications you deploy, this may or may not be sufficient for deploying your applications.

Supported Hardware

Verrazzano requires x86-64; other architectures are not supported.

Supported Software Versions

Verrazzano supports the following software versions.

Kubernetes

You can install Verrazzano on the following Kubernetes versions.

Verrazzano	Kubernetes Versions
1.0	1.18, 1.19, 1.20
1.1	1.19, 1.20, 1.21

For more information, see Kubernetes Release Documentation. For platform specific details, see Verrazzano platform setup.

WebLogic Server

The supported versions of WebLogic Server are dependent on the WebLogic Kubernetes Operator version. See the WebLogic Server versions supported here.

Coherence

The supported versions of Coherence are dependent on the Coherence Operator version. See the Coherence versions supported here.

Helidon

Verrazzano supports all versions of Helidon. For more information, see Helidon and Helidon Commercial Offerings.

Installed Components

Verrazzano installs a curated set of open source components. The following table lists each open source component with its version and a brief description.

Component	Version	Description
cert-manager	1.2.0	Automates the management and issuance of TLS certificates.
Coherence Operator	3.2.3	Assists with deploying and managing Coherence clusters.
Elasticsearch	7.10.2	Provides a distributed, multitenant-capable full-text search engine.
ExternalDNS	0.7.1	Synchronizes exposed Kubernetes Services and ingresses with DNS providers.
Fluentd	1.12.3	Collects logs and sends them to Elasticsearch.
Grafana	7.2.1-2	Tool to help you study, analyze, and monitor metrics.
Istio	1.10.4	Service mesh that layers transparently onto existing distributed applications.
Keycloak	15.0.2	Provides single sign-on with Identity and Access Management.
Kiali	1.34.1	Management console for the Istio service mesh.
Kibana	7.10.2	Provides search and data visualization capabilities for data indexed in Elasticsearch.
MySQL	8.0.26	Open source relational database management system used by Keycloak.
NGINX Ingress Controller	0.46.0	Traffic management solution for cloud‑native applications in Kubernetes.
Node Exporter	1.0.0	Prometheus exporter for hardware and OS metrics.
OAM Kubernetes Runtime	0.3.0	Plug-in for implementing Open Application Model (OAM) control plane with Kubernetes.
Prometheus	2.21.0-1	Provides event monitoring and alerting.
Rancher	2.5.9	Manages multiple Kubernetes clusters.
WebLogic Kubernetes Operator	3.3.7	Assists with deploying and managing WebLogic domains.

Docs: Verrazzano Analysis Tools

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides tooling which assists in troubleshooting issues in your environment:

k8s-dump-cluster.sh
verrazzano-analysis

Tools Setup

These tools are available for Linux and Mac: https://github.com/verrazzano/verrazzano/releases/.

Linux Instructions

Use these instructions to obtain the analysis tools on Linux machines.

Download the tooling:

 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/k8s-dump-cluster.sh
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/k8s-dump-cluster.sh.sha256
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/verrazzano-analysis-linux-amd64.tar.gz
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/verrazzano-analysis-linux-amd64.tar.gz.sha256

Verify the downloaded files:

 $ sha256sum -c k8s-dump-cluster.sh.sha256
 $ sha256sum -c verrazzano-analysis-linux-amd64.tar.gz.sha256

Unpack the `verrazzano-analysis` binary:

 $ tar xvf verrazzano-analysis-linux-amd64.tar.gz

Mac Instructions

Use these instructions to obtain the analysis tools on Mac machines.

Download the tooling:

 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/k8s-dump-cluster.sh
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/k8s-dump-cluster.sh.sha256
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/verrazzano-analysis-darwin-amd64.tar.gz
 $ wget https://github.com/verrazzano/verrazzano/releases/download/v1.1.2/verrazzano-analysis-darwin-amd64.tar.gz.sha256

Verify the downloaded files:

 $ shasum -a 256 -c k8s-dump-cluster.sh.sha256
 $ shasum -a 256 -c verrazzano-analysis-darwin-amd64.tar.gz.sha256

Unpack the `verrazzano-analysis` binary:

 $ tar xvf verrazzano-analysis-darwin-amd64.tar.gz

Use the `k8s-dump-cluster.sh` tool

The k8s-dump-cluster.sh tool is a shell script which runs various kubectl and helm commands against a cluster.

Note that the data captured by this script might include sensitive information. This data is under your control; you can choose whether to share it.

The directory structure created by the k8s-dump-cluster.sh tool, for a specific cluster dump, appears as follows:

$ CAPTURE_DIR
  cluster-dump
    directory per namespace (a directory at this level is assumed to represent a namespace)
      acme-orders.json
      application-configurations.json
      certificate-requests.json
      cluster-role-bindings.json
      cluster-roles.json
      cluster-roles.json
      coherence.json
      components.json
      {CONFIGNAME}.configmap (a file at this level for each configmap in the namespace)
      daemonsets.json
      deployments.json
      events.json
      gateways.json
      ingress-traits.json
      jobs.json
      multicluster-application-configurations.json
      multicluster-components.json
      multicluster-config-maps.json
      multicluster-logging-scopes.json
      multicluster-secrets.json
      namespace.json
      persistent-volume-claims.json
      persistent-volumes.json
      pods.json
      replicasets.json
      replication-controllers.json
      role-bindings.json
      services.json
      verrazzano-managed-clusters.json
      verrazzano-projects.json
      verrazzano_resources.json
      virtualservices.json
      weblogic-domains.json
      directory per pod (a directory at this level is assumed to represent a specific pod)
        logs.txt (includes logs for all containers and initContainers)
    api-resources.out
    application-configurations.json
    cluster-issuers.txt
    coherence.json
    configmap_list.out
    crd.json
    es_indexes.out
    gateways.json
    helm-ls.json
    helm-version.out
    images-on-nodes.csv
    ingress.json
    ingress-traits.json
    kubectl-version.json
    namespace_list.out
    network-policies.json
    network-policies.txt
    nodes.json
    pv.json
    verrazzano_resources.out
    virtualservices.json

The script shows the kubectl and helm commands which are run. The basic structure, shown previously, is formed by running the command, $ kubectl cluster-info dump --all-namespaces, with additional data captured into that directory structure.

To perform a dump of a cluster into a directory named my-cluster-dump:

$ sh k8s-dump-cluster.sh -d my-cluster-dump

Use the `verrazzano-analysis` tool

The verrazzano-analysis tool analyzes data from a cluster dump captured using k8s-dump-cluster.sh, reports the issues found, and prescribes related actions to take. These tools are continually evolving with regard to what may be captured, the knowledge base of issues and actions, and the types of analysis that can be performed.

Users, developers, and Continuous Integration (CI) can use this tooling to quickly identify the root cause of encountered problems, determine mitigation actions, and provide a sharable report with other users or tooling.

The data that the analysis examines follows the structure created by the corresponding capture tooling. For example, k8s-dump-cluster.sh dumps a cluster into a specific structure, which might contain data that you do not want to share. The tooling analyzes the data and provides you with a report, which identifies issues and provides you with actions to take.

The verrazzano-analysis tool will find and analyze all cluster dump directories found under a specified root directory. This lets you create a directory to hold the cluster dumps of related clusters into sub-directories which the tool can analyze.

For example:

my-cluster-dumps
    CAPTURE_DIR-1
        cluster-dump
            ...
    CAPTURE_DIR-2
        cluster-dump
            ...

The tool analyzes each cluster dump directory found; you need to provide only the single root directory.

To perform an analysis of the clusters:

$ verrazzano-analysis my-cluster-dumps

Usage information

Usage: verrazzano-analysis [options] captured-data-directory

Parameter	Definition	Default
`-actions`	Include actions in the report.	`true`
`-help`	Display usage help.
`-info`	Include informational messages.	`true`
`-minConfidence`	Minimum confidence threshold to report for issues, 0-10.	`0`
`-minImpact`	Minimum impact threshold to report for issues, 0-10.	`0`
`-reportFile`	Name of report output file.	Output to stdout.
`-support`	Include support data in the report.	`true`
`-version`	Display tool version.

Docs: Verrazzano and the Open Application Model

Mon, 01 Jan 0001 00:00:00 +0000

Open Application Model (OAM) is a runtime-agnostic specification for defining cloud native applications; it allows developers to focus on the application instead of the complexities of a particular runtime infrastructure. OAM provides the specification for several file formats and rules for a runtime to interpret. Verrazzano uses OAM to enable the definition of a composite application abstraction and makes OAM constructs available within a VerrazzanoApplication YAML file. Verrazzano provides the flexibility to combine what you want into a multicloud enablement. It uses the VerrazzanoApplication as a means to encapsulate a set of components, scopes, and traits, and deploy them on a selected cluster.

OAM’s workload concept makes it easy to use many different workload types. Verrazzano includes specific workload types with special handling to deploy and manage those types, such as WebLogic, Coherence, and Helidon. OAM’s flexibility lets you create a grouping that is managed as a unit, although each component can be scaled or updated independently.

How does OAM work?

OAM has five core concepts:

Workloads - Declarations of the kinds of resources supported by the platform and the OpenAPI schema for that resource. Most Kubernetes CRDs can be exposed as workloads. Standard Kubernetes resource types can also be used (for example, Deployment, Service, Pod, ConfigMap).
Components - Wrap a workload resource’s specification data within OAM specific metadata.
Application Configurations - Describe a collection of components that comprise an application. This is also where customization (such as, environmental) of each component is done. Customization is achieved using scopes and traits.
Scopes - Apply customization to several components.
Traits - Apply customization to a single component.

Docs: Verrazzano in a Multicluster Environment

Mon, 01 Jan 0001 00:00:00 +0000

Review the following key concepts to understand multicluster Verrazzano:

Admin cluster - A Kubernetes cluster that serves as the central management point for deploying and monitoring applications in managed clusters.
Managed clusters - A Kubernetes cluster that has the following characteristics:
- It is registered with an admin cluster with a unique name.
- Verrazzano multicluster applications may be deployed to the managed cluster from the admin cluster.
- Logs and metrics for Verrazzano system components and Verrazzano multicluster applications deployed on the managed cluster are viewable from the admin cluster.
Verrazzano multicluster resources - Custom Kubernetes resources defined by Verrazzano.
- Each multicluster resource serves as a wrapper for an underlying resource type.
- A multicluster resource allows the placement of the underlying resource to be specified as a list of names of the clusters in which the resource must be placed.

For more details, see here.

Docs: Verrazzano Projects

Mon, 01 Jan 0001 00:00:00 +0000

A project provides a way to group application namespaces that are owned or administered by the same user or group of users. When creating a project, you can specify the subjects: users, groups and/or service accounts, that are to be granted access to the namespaces governed by the project. Two types of subjects may be specified:

Project admins, who have both read and write access to the project’s namespaces.
Project monitors, who have read-only access to the project’s namespaces.

For more information, see Projects.

Docs: Customize Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano issues certificates to secure access from external clients to secure system endpoints.
A certificate from a certificate authority (CA) must be configured to issue the endpoint certificates in one of the following ways:

Let Verrazzano generate a self-signed CA (the default).
Configure a CA that you provide.
Configure LetsEncrypt as the certificate issuer (requires OCI DNS).

In all cases, Verrazzano uses CertManager to manage the creation of certificates.

NOTE

Self-signed certificate authorities generate certificates that are NOT signed by a trusted authority; typically, they are not used in production environments.

Use the Verrazzano self-signed CA

By default, Verrazzano creates its own self-signed CA. No configuration is required.

Use a custom CA

If you want to provide your own CA, you must:

(Optional) Create your own signing key pair and CA certificate.

For example, you can use the openssl CLI to create a key pair for the nip.io domain:
```
# Generate a CA private key
$ openssl genrsa -out tls.key 2048

# Create a self signed certificate, valid for 10yrs with the 'signing' option set
$ openssl req -x509 -new -nodes -key tls.key -subj "/CN=*.nip.io" -days 3650 -reqexts v3_req -extensions v3_ca -out tls.crt
```
The output of these commands will be two files, tls.key and tls.crt, the key and certificate for your signing key pair. These files must be named in that manner for the next step.

If you already have generated your own key pair, you must name the private key and certificate, tls.key and tls.crt, respectively. If your issuer represents an intermediate, ensure that tls.crt contains the issuer’s full chain in the correct order.

You can find more details on providing your own CA, in the CertManager CA documentation.

Save your signing key pair as a Kubernetes secret.

$ kubectl create ns mynamespace
$ kubectl create secret tls myca --namespace=mynamespace --cert=tls.crt --key=tls.key

Specify the secret name and namespace location in the Verrazzano custom resource.

The custom CA secret must be provided to CertManager using the following fields in spec.components.certManager.certificate.ca in the Verrazzano custom resource:
- spec.components.certManager.certificate.ca.secretName
- spec.components.certManager.certificate.ca.clusterResourceNamespace

For example, if you created a CA secret named myca in the namespace mynamespace, you would configure it as shown:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: custom-ca-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        ca:
          secretName: myca
          clusterResourceNamespace: mynamespace

Use LetsEncrypt certificates

You can configure Verrazzano to use certificates generated by LetsEncrypt. LetsEncrypt implements the ACME protocol, which provides a standard protocol for the automated issuance of certificates signed by a trusted authority. This is managed through the spec.components.certManager.certificate.acme field in the Verrazzano custom resource.

NOTE

Using LetsEncrypt for certificates also requires using OCI DNS for DNS management. For details, see the Customize DNS page.

To configure CertManager to use LetsEncrypt as the certificates provider, you must configure a CertManager ACME provider with the following values in the Verrazzano custom resource:

Set the spec.components.certManager.certificate.acme.provider field to letsEncrypt.
Set the spec.components.certManager.certificate.acme.emailAddress field to a valid email address for the letsEncrypt account.
(Optional) Set the spec.components.certManager.certificate.acme.environment field to either staging or production (the default).

The following example configures Verrazzano to use the LetsEncrypt production environment by default, with OCI DNS for DNS record management:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

The following example configures Verrazzano to use the LetsEncrypt staging environment with OCI DNS:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
          environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

NOTE

Certificates issued by the LetsEncrypt staging environment are signed by untrusted authorities, similar to self-signed certificates. They are typically not used in production environments.

LetsEncrypt staging versus production

LetsEncrypt provides rate-limits on generated certificates to ensure fair usage across all clients. The production environment limits can be exceeded more frequently in environments where Verrazzano may be being installed or reinstalled frequently (like a test environment). This can result in failed installations due to rate limit exceptions on certificate generation.

In such environments, it is better to use the LetsEncrypt staging environment, which has much higher limits than the production environment. For test environments, the self-signed CA also may be more appropriate to completely avoid LetsEncrypt rate limits.

Docs: IngressTrait Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The IngressTrait custom resource contains the configuration of host and path rules for traffic routing to an application. Here is a sample ApplicationConfiguration that specifies an IngressTrait. To deploy an example application that demonstrates this IngressTrait, see Hello World Helidon.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
                scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

In the sample configuration, the IngressTrait hello-helidon-ingress is set on the hello-helidon-component application component and defines an ingress rule that configures a path and path type. This exposes a route for external access to the application. Note that because no hosts list is given for the IngressRule, a DNS host name is automatically generated.

For example, with the sample application configuration successfully deployed, the application will be accessible with the path specified in the IngressTrait and the generated host name.

$ HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-appconf-gw -n hello-helidon -o jsonpath={.spec.servers[0].hosts[0]})
$ echo $HOST
hello-helidon-appconf.hello-helidon.11.22.33.44.nip.io

$ curl -sk -X GET https://${HOST}/greet

Alternatively, specific host names can be given in an IngressRule. Doing this implies that a secret and certificate have been created for the specific hosts and the secret name has been specified in the associated IngressSecurity secretName field.

IngressTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	IngressTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	IngressTraitSpec	The desired state of an ingress trait.	Yes

IngressTraitSpec

IngressTraitSpec specifies the desired state of an ingress trait.

Field	Type	Description	Required
`rules`	IngressRule array	A list of ingress rules to for an ingress trait.	Yes
`tls`	IngressSecurity	The security parameters for an ingress trait. This is required only if specific hosts are given in an IngressRule.	No

IngressRule

IngressRule specifies a rule for an ingress trait.

Field	Type	Description	Required
`hosts`	string array	One or more hosts exposed by the ingress trait. Wildcard hosts or hosts that are empty are filtered out. If there are no valid hosts provided, then a DNS host name is automatically generated and used.	No
`paths`	IngressPath array	The paths to be exposed for an ingress trait.	Yes
`destination`	IngressDestination	The destination host and port for the ingress paths.	No

IngressPath

IngressPath specifies a specific path to be exposed for an ingress trait.

Field	Type	Description	Required
`path`	string	If no path is provided, it defaults to `/`.	No
`pathType`	string	Path type values are case-sensitive and formatted as follows: `exact`: exact string match `prefix`: prefix-based match `regex`: regex-based match If the provided ingress path doesn’t contain a `pathType`, it defaults to `prefix` if the path is `/` and `exact` otherwise.	No

IngressDestination

IngressDestination specifies a specific destination host and port for the ingress paths.

Field	Type	Description	Required
`host`	string	Destination host.	No
`port`	uint32	Destination port.	No

NOTE

If there are multiple ports defined for a service, then the destination port must be specified OR the service port name must have the prefix “http”.

IngressSecurity

IngressSecurity specifies the secret containing the certificate securing the transport for an ingress trait.

Field	Type	Description	Required
`secretName`	string	The name of a secret containing the certificate securing the transport. The specification of a secret here implies that a certificate was created for specific hosts, as specified in an IngressRule.	Yes

Docs: Installation Profiles

Mon, 01 Jan 0001 00:00:00 +0000

This document describes built-in configuration profiles that you can use to simplify a Verrazzano installation. An installation profile is a well-known configuration of Verrazzano settings that can be referenced by name, which then can be customized as needed.

The following table describes the Verrazzano installation profiles.

Profile	Description	Characteristics
`prod`	Full install, production configuration.	Default profile: - Full installation. - Persistent storage. - Production Elasticsearch cluster topology.
`dev`	Development or evaluation configuration.	Lightweight installation: - For evaluation purposes. - No persistence. - Single-node Elasticsearch cluster topology.
`managed-cluster`	A specialized installation for managed clusters in a multicluster topology.	Minimal installation for a managed cluster: - Cluster must be registered with an admin cluster to use multicluster features.

Use an installation profile

To use a profile to install Verrazzano, set the profile name in the profile field of your Verrazzano custom resource.

For example, to use the dev profile:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev

To use a different profile, simply replace dev with prod or managed-cluster.

Customize an installation profile

You can override the profile settings for any component regardless of the profile. The following example uses a customized dev profile to configure a small 8Gi persistent volume for the MySQL instance used by Keycloak to provide more stability for the Keycloak service:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: custom-dev-example
spec:
  profile: dev
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql
  volumeClaimSpecTemplates:
  - metadata:
      name: mysql      
    spec:
      resources:
        requests:
          storage: 8Gi

For details on how to customize Verrazzano components, see Customize an Installation.

Profile configurations

The following table lists the Verrazzano components that are installed with each profile. Note that you can customize any Verrazzano installation, regardless of the profile.

Component	dev	prod	managed-cluster
Istio	✔️	✔️	✔️
NGINX	✔️	✔️	✔️
Cert-Manager	✔️	✔️	✔️
External-DNS	️	️
Prometheus	✔️	✔️	✔️
Elasticsearch	✔️	✔️
Console	✔️	✔️
Kibana	✔️	✔️
Grafana	✔️	✔️
Rancher	✔️	✔️
Keycloak	✔️	✔️

Prometheus and Grafana configurations

The following table describes the Prometheus and Grafana configurations in each profile.

Profile	Prometheus	Grafana
`prod`	1 replica (128M memory, 50Gi storage)	1 replica (48M memory, 50Gi storage)
`dev`	1 replica (128M memory, ephemeral storage)	1 replica (48M memory, ephemeral storage)
`managed-cluster`	1 replica (128M memory, 50Gi storage)	Not installed

Kibana and Elasticsearch configurations

The following table describes the Kibana and Elasticsearch cluster topology in each profile.

Profile	Elasticsearch	Kibana
`prod`	3 master replicas (1.4Gi memory, 50Gi storage each) 1 ingest replica (2.5Gi memory, no storage) 2 data replicas (4.8Gi memory, 50Gi storage each)	1 replica (192M memory, ephemeral storage)
`dev`	1 master/data/ingest replica (1Gi memory, ephemeral storage)	1 replica (192M memory, ephemeral storage)
`managed-cluster`	Not installed	Not installed

NOTE

Elasticsearch containers are configured to use 75% of the configured request memory for the Java min/max heap settings.

Profile-independent defaults

The following table shows the settings for components that are profile-independent (consistent across all profiles unless overridden).

Component	Default
DNS	Wildcard DNS provider nip.io.
Certificates	Uses the cert-manager self-signed ClusterIssuer for certificates.
Ingress-type	Defaults to `LoadBalancer` service type for the ingress.

For details on how to customize Verrazzano components, see Customize an Installation.

Docs: Kubernetes RBAC

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano uses Kubernetes Role-Based Access Control (RBAC) to protect Verrazzano resources.

Verrazzano includes a set of roles that can be granted to users, enabling access to Verrazzano resources managed by Kubernetes. In addition, Verrazzano creates a number of roles that grant permissions needed by various Verrazzano system components (operators and third-party components).

Verrazzano creates default role bindings during installation and for projects, at project creation or update.

NOTE

Kubernetes RBAC must be enabled in every cluster to which Verrazzano is deployed or access control will not work. RBAC is enabled by default in most Kubernetes environments.

Verrazzano user roles

The following table lists the defined Verrazzano user roles. Each is a ClusterRole intended to be granted directly to users or groups. (In some scenarios, it may be appropriate to grant a user role to a service account.)

Verrazzano Role	Binding Scope	Description
verrazzano-admin	Cluster	Manage Verrazzano system components, clusters, and projects. Install/update Verrazzano.
verrazzano-monitor	Cluster	View/monitor Verrazzano system components, clusters, and projects.
verrazzano-project-admin	Namespace	Deploy/manage applications.
verrazzano-project-monitor	Namespace	View/monitor applications.

Kubernetes user roles

Verrazzano roles do not include permissions for Kubernetes itself. Instead, it relies on the default user roles provided by Kubernetes. This allows Verrazzano to easily grant the Kubernetes access appropriate to a Verrazzano role, without having to maintain a long list of fine-grained Kubernetes permissions in the Verrazzano roles.

The following table shows the default Kubernetes roles that are granted by default for each Verrazzano role.

Verrazzano Role	Kubernetes Role	Binding Scope
verrazzano-admin	admin	Cluster
verrazzano-monitor	view	Cluster
verrazzano-project-admin	admin	Namespace
verrazzano-project-monitor	view	Namespace

Default role bindings

Verrazzano creates role bindings for the system and for projects, binding Verrazzano ClusterRoles to one or more Kubernetes Subjects. By default, each role is bound to a Keycloak group, so all Keycloak users who are members of that group will be granted the role.

Also, Verrazzano creates role bindings for the corresponding Kubernetes user roles. The Kubernetes role appropriate for a given Verrazzano role is bound to the same set of Subjects as the corresponding Verrazzano role.

The default bindings can be overridden by specifying one or more Kubernetes Subjects to which the role should be bound. Any valid Subject can be specified (user, group, or service account), but two caveats should be kept in mind:

It’s generally better to grant a role to a group, rather than a specific user, so that roles can be granted (or withdrawn) by editing a user’s group memberships, rather than deleting a role binding and creating a new one.
If you do want to grant a role directly to a specific user, the user must be specified using its unique ID, not its user name. This is because the authentication proxy impersonates the sub (subject) field from the user’s token, which contains the ID. Keycloak user IDs are guaranteed to be unique, unlike user names.

Default system role bindings

Verrazzano creates role bindings for system users during installation. The default role bindings are listed below.

Role	Default Binding Subject
verrazzano-admin	group: verrazzano-admins
verrazzano-monitor	group: verrazzano-monitors

Default project role bindings

Verrazzano creates role bindings for project users at project creation or update. The default role bindings are listed below.

Role	Default Binding Subject
verrazzano-project-admin	group: verrazzano-project-<proj_name>-admins
verrazzano-project-monitor	group: verrazzano-project-<proj_name>-monitors

NOTE

The role bindings for project roles are created automatically, but the project-specific groups that they refer to are not automatically created. You must create those groups using the Keycloak console or API, or specify different binding subjects for the project.

Override default role bindings

You can override the default role bindings that are created for system and project roles.

Override system role bindings

To override the set of subjects that are bound to Verrazzano (and Kubernetes) roles during installation, add the Subjects to the Verrazzano CR you use to install Verrazzano, as shown in the following example:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  ...
  security:
    adminSubjects:
    - name: admin-group
      kind: Group
    monitorSubjects:
    - name: view-group
      kind: Group
  ...

You can specify multiple subjects for both admin and monitor roles. You can also specify a subject or subjects for one role, but not the other. If no subjects are specified for a role, then the default binding subjects will be used.

Override project role bindings

To override the set of subjects that are bound to Verrazzano (and Kubernetes) roles for a project, add the Subjects to the VerrazzanoProject CR for the project, as shown in the example below.

Note that the generated role bindings will be updated if you update the VerrazzanoProject CR and change the subjects specified for either role.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoProject
metadata:
  name: my-project
spec:
  ...
  security:
    projectAdminSubjects:
    - name: my-project-admin-group
      kind: Group
    projectMonitorSubjects:
    - name: my-project-view-group
      kind: Group
  ...

As with the system role bindings, you can specify multiple subjects for both project-admin and project-monitor roles. You can also specify a subject or subjects for one role, but not the other. If no subjects are specified for a role, then the default binding subjects will be used.

Docs: Lift-and-Shift Guide

Mon, 01 Jan 0001 00:00:00 +0000

This guide describes how to move (“Lift-and-Shift”) an on-premises WebLogic Server domain to a cloud environment running Kubernetes using Verrazzano.

Overview

The Initial steps create a very simple on-premises domain that you will move to Kubernetes. The sample domain is the starting point for the lift and shift process; it contains one application (ToDo List) and one data source. First, you’ll configure the database and the WebLogic Server domain. Then, in Lift and Shift, you will move the domain to Kubernetes with Verrazzano. This guide does not include the setup of the networking that would be needed to access an on-premises database, nor does it document how to migrate a database to the cloud.

What you need

The Git command-line tool and access to GitHub
MySQL Database 8.x - a database server
WebLogic Server 12.2.1.4.0 - an application server; Note that all WebLogic Server installers are supported except the Quick Installer.
Maven - to build the application
WebLogic Deploy Tooling (WDT) - v1.9.15 or later, to convert the WebLogic Server domain to and from metadata
WebLogic Image Tool (WIT) - v1.9.13 or later, to build the Docker image

Initial steps

In the initial steps, you create a sample domain that represents your on-premises WebLogic Server domain.

Create a database using MySQL called `tododb`

Download the MySQL image from Docker Hub.
```
$ docker pull mysql:latest
```

Start the container database (and optionally mount a volume for data).

$ export MYSQL_USER=<your-mysql-username>
$ export MYSQL_PASSWORD=<your-mysql-password>
$ export MYSQL_ROOT_PASSWORD=<your-mysql-rootpassword>
$ docker run --name tododb \
  -p 3306:3306 \
  -e MYSQL_USER=$MYSQL_USER \
  -e MYSQL_PASSWORD=$MYSQL_PASSWORD \
  -e MYSQL_DATABASE=tododb \
  -e MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD \
  -d mysql:latest

Start a MySQL client to change the password algorithm to mysql_native_password.
- Assuming the database server is running, start a database CLI client.
```
$ docker exec \
   -it tododb mysql \
   -uroot \
   -p
```
- When prompted for the password, enter the password for the root user.
- After being connected, run the ALTER command at the MySQL prompt.
```
$ ALTER USER '<your-mysql-username>'@'%' identified with mysql_native_password by '<your-mysql-password>';
```

Create a WebLogic Server domain

If you do not have WebLogic Server 12.2.1.4.0 installed, install it now.
- Choose the GENERIC installer from WebLogic Server Downloads and follow the documented installation instructions.
- Be aware of these domain limitations:
  - There are two supported domain types, single server and single cluster.
  - Domains must use:
    - The default value AdminServer for AdminServerName.
    - WebLogic Server listen port for the Administration Server: 7001.
    - WebLogic Server listen port for the Managed Server: 8001.
    - Note that these are all standard WebLogic Server default values.
- Save the installer after you have finished; you will need it to build the Docker image.
- To make copying commands easier, define an environment variable for ORACLE_HOME that points to the directory where you installed WebLogic Server 12.2.1.4.0. For example:
```
$ export ORACLE_HOME=$HOME/Oracle/Middleware/Oracle_Home
```
Use the Oracle WebLogic Server Configuration Wizard to create a domain called tododomain.

NOTE: This example assumes that the on premises WebLogic Server domain is on Linux.
- Launch $ORACLE_HOME/oracle_common/common/bin/config.sh.
- Select Create a new domain.
- Specify a Domain Location of <oracle home>/user_projects/domains/tododomain and click Next.
- Select the Basic WebLogic Server Domain [wlserver] template and click Next.
- Enter the password for the administrative user and click Next.
- Accept the defaults for Domain Mode and JDK, and click Next.
- Select Administration Server and click Next.
- Ensure that the server name is AdminServer and click Next.
- Click Create.
- After it has completed, click Next, then Finish.

To start the newly created domain, run the domain’s start.

 $ $ORACLE_HOME/user_projects/domains/tododomain/bin/startWebLogic.sh

Access the Console of the newly started domain with your browser, for example, http://localhost:7001/console, and log in using the administrator credentials you specified.

Add a data source configuration to access the database

Using the WebLogic Server Administration Console, log in and add a data source configuration to access the MySQL database. During the data source configuration, you can accept the default values for most fields, but the following fields are required to match the application and database settings you used when you created the MySQL database.

In the left pane in the Console, expand Services and select Data Sources.
On the Summary of JDBC Data Sources page, click New and select Generic Data Source.
On the JDBC Data Sources page, enter or select the following information:
- Name: tododb
- JNDI Name: jdbc/ToDoDB
- Database Type: MySQL
Click Next and then click Next two more times.
On the Create a New JDBC Data Source page, enter the following information:
- Database Name: tododb
- Host name: localhost
- Database Port: 3306
- Database User Name: <your-mysql-username>
- Password: <your-mysql-password>
- Confirm Password: <your-mysql-password>
Click Next.
Select Test Configuration, and make sure you see “Connection Test Succeeded” in the Messages field of the Console.
Click Next.
On the Select Targets page, select AdminServer.
Click Finish to complete the configuration.

Build and deploy the application

Using Maven, build this project to produce todo.war.

NOTE: You should clone this repo outside of $ORACLE_HOME or copy the WAR file to another location, as WDT may ignore it during the model creation phase.
```
 $ git clone https://github.com/verrazzano/examples.git
 $ cd examples/todo-list/
 $ mvn clean package
```
Using the WebLogic Server Administration Console, deploy the ToDo List application.
- In the left pane in the Console, select Deployments and click Install.
- Use the navigation links or provide the file path to todo.war typically <repo>/todo-list/target. For example, if you cloned the examples repository in your $HOME directory, the location should be $HOME/examples/examples/todo-list/target/todo.war.
- Click Next twice, then Finish.
NOTE: The remaining steps assume that the application context is todo.

Initialize the database

After the application is deployed and running in WebLogic Server, access the http://localhost:7001/todo/rest/items/init REST service to create the database table used by the application. In addition to creating the application table, the init service also will load four sample items into the table.

If you get an error here, go back to the Select Targets page in the WebLogic Server Administration Console and make sure that you selected AdminServer as the data source target.

Access the application

Access the application at http://localhost:7001/todo/index.html.

Add a few entries or delete some.
After verifying the application and database, you may shut down the local WebLogic Server domain.

Lift and Shift steps

The following steps will move the sample domain to Kubernetes with Verrazzano.

Create a WDT Model

If you have not already done so, download v1.9.15 or later of WebLogic Deploy Tooling (WDT) from GitHub.
Unzip the installer weblogic-deploy.zip file so that you can access bin/discoverDomain.sh.
To make copying commands easier, define an environment variable for WDT_HOME that points to the directory where you installed WebLogic Deploy Tooling.
```
 $ export WDT_HOME=/install/directory
```

For example, to get the latest version:

$ curl -OL https://github.com/oracle/weblogic-deploy-tooling/releases/latest/download/weblogic-deploy.zip
$ unzip  weblogic-deploy.zip
$ cd weblogic-deploy
$ export WDT_HOME=$(pwd)

To create a reusable model of the application and domain, use WDT to create a metadata model of the domain.

First, create an output directory to hold the generated scripts and models.

Then, run WDT discoverDomain.

$ mkdir v8o
$ $WDT_HOME/bin/discoverDomain.sh \
  -oracle_home $ORACLE_HOME \
  -domain_home /path/to/domain/dir \
  -model_file ./v8o/wdt-model.yaml \
  -archive_file ./v8o/wdt-archive.zip \
  -target vz \
  -output_dir v8o

You will find the following files in ./v8o:

create_k8s_secrets.sh - A helper script with kubectl commands to apply the Kubernetes secrets needed for this domain
vz-application.yaml - Verrazzano application configuration and component file
vz_variable.properties - A set of properties extracted from the WDT domain model
wdt-archive.zip - The WDT archive file containing the ToDo List application WAR file
wdt-model.yaml - The WDT model of the WebLogic Server domain

If you chose to skip the Access the application step and did not verify that the ToDo List application was deployed, then you should verify that you see the todo.war file inside the wdt-archive.zip file. If you do not see the WAR file, there was something wrong in your deployment of the application on WebLogic Server that will require additional troubleshooting in your domain.

Create a Docker image

At this point, the Verrazzano model is just a template for the real model. The WebLogic Image Tool will fill in the placeholders for you, or you can edit the model manually to set the image name and domain home directory.

If you have not already done so, download WebLogic Image Tool (WIT) from GitHub.
Unzip the installer imagetool.zip file so that you can access bin/imagetool.sh.
To make copying commands easier, define an environment variable for WIT_HOME that points to the directory where you installed WebLogic Image Tool.
```
 $ export WIT_HOME=/install/directory
```

For example, to get the latest WIT tool:

$ curl -OL https://github.com/oracle/weblogic-image-tool/releases/latest/download/imagetool.zip
$ unzip imagetool.zip
$ cd imagetool
$ export WIT_HOME=$(pwd)

You will need a Docker image to run your WebLogic Server domain in Kubernetes. To use WIT to create the Docker image, run imagetool create. Although WIT will download patches and PSUs for you, it does not yet download installers. Until then, you must download the WebLogic Server and Java Development Kit installer manually and provide their location to the imagetool cache addInstaller command.

# The directory created previously to hold the generated scripts and models.
$ cd v8o

$ $WIT_HOME/bin/imagetool.sh cache addInstaller \
  --path /path/to/installer/jdk-8u231-linux-x64.tar.gz \
  --type jdk \
  --version 8u231

# The installer file name may be slightly different depending on
# which version of the 12.2.1.4.0 installer that you downloaded, slim or generic.
$ $WIT_HOME/bin/imagetool.sh cache addInstaller \
  --path /path/to/installer/fmw_12.2.1.4.0_wls_Disk1_1of1.zip \
  --type wls \
  --version 12.2.1.4.0

$ $WIT_HOME/bin/imagetool.sh cache addInstaller \
  --path /path/to/installer/weblogic-deploy.zip \
  --type wdt \
  --version latest

# Paths for the files in this command assume that you are running it from the
# v8o directory created during the `discoverDomain` step.
$ $WIT_HOME/bin/imagetool.sh create \
  --tag your/repo/todo:1 \
  --version 12.2.1.4.0 \
  --jdkVersion 8u231 \
  --wdtModel ./wdt-model.yaml \
  --wdtArchive ./wdt-archive.zip \
  --wdtVariables ./vz_variable.properties \
  --resourceTemplates=./vz-application.yaml \
  --wdtModelOnly

The imagetool create command will have created a local Docker image and updated the Verrazzano model with the domain home and image name. Check your Docker images for the tag that you used in the create command using docker images from the Docker CLI.

If everything worked correctly, it is time to push that image to the container registry that Verrazzano will use to access the image from Kubernetes. You can use the Oracle Cloud Infrastructure Registry (OCIR) as your repository for this example, but most Docker compliant registries should work.

The variables in the vz-application.yaml resource template should be resolved with information from the image tool build.
Verify this by looking in the v8o/vz-application.yaml file to make sure that the image: {{{imageName}}} value has been set with the given --tag value.

Push the image to your repo.

NOTE: The image name must be the same as what is in the vz-application.yaml file under spec > workload > spec > image for the tododomain-domain component.

$ docker push your/repo/todo:1

Deploy to Verrazzano

After the application image has been created, there are several steps required to deploy the application into a Verrazzano environment.

These include:

Creating and labeling the tododomain namespace.
Creating the necessary secrets required by the ToDo List application.
Creating the Verrazzano components such as Service, Deployment, and ConfigMap required by the MySQL instance in the tododomain namespace.
Updating the vz-application.yaml file to enable the Verrazzano MySQL components in the ToDo List ApplicationConfiguration to deploy as Kubernetes objects.
Updating the vz-application.yaml file to use the Verrazzano MySQL deployment and (optionally) expose the WebLogic Server Administration Console.
Applying the vz-application.yaml file.

The following steps assume that you have a Kubernetes cluster and that Verrazzano is already installed in that cluster.

Label the namespace

Create the tododomain namespace, and add labels to allow the WebLogic Server Kubernetes Operator to manage it and enabled for Istio.

$ kubectl create namespace tododomain
$ kubectl label namespace tododomain verrazzano-managed=true istio-injection=enabled

Create the required secrets

If you haven’t already done so, edit and run the create_k8s_secrets.sh script to generate the Kubernetes secrets. WDT does not discover passwords from your existing domain. Before running the create secrets script, you will need to edit create_k8s_secrets.sh to set the passwords for the WebLogic Server domain and the data source. In this domain, there are a few passwords that you need to enter:

Administrator credentials
ToDo database credentials

For example:

# Update <admin-user> and <admin-password> for weblogic-credentials
$ create_paired_k8s_secret weblogic-credentials <your-WLS-username> <your-WLS-password>

# Update <user> and <password> for tododomain-jdbc-tododb
$ create_paired_k8s_secret jdbc-tododb <your-mysql-username> <your-mysql-password>

Then run the script:

$ sh ./create_k8s_secrets.sh

Verrazzano will need a credential to pull the image that you just created, so you need to create one more secret. The name for this credential can be changed in the vz-application.yaml file to anything you like, but it defaults to tododomain-registry-credentials.

Assuming that you leave the name tododomain-registry-credentials, you will need to run a kubectl create secret command similar to the following:

$ kubectl create secret docker-registry tododomain-registry-credentials \
  --docker-server=phx.ocir.io \
  --docker-email=your.name@example.com \
  --docker-username=tenancy/username \
  --docker-password='passwordForUsername' \
  --namespace=tododomain

Update the application configuration

Update the generated vz-application.yaml file for the todo application to:

Update the tododomain-configmap component to use the in-cluster MySQL service URL jdbc:mysql://mysql.tododomain.svc.cluster.local:3306/tododb to access the database.

        wdt_jdbc.yaml: |
          resources:
            JDBCSystemResource:
              'todo-ds':
                JdbcResource:
                  JDBCDriverParams:
                    # This is the URL of the database used by the WebLogic Server application
                    URL: "jdbc:mysql://mysql.tododomain.svc.cluster.local:3306/tododb"

Update the tododomain-appconf ApplicationConfiguration to enable Verrazzano MySQL components to be deployed as Kubernetes objects.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: tododomain-appconf
  namespace: tododomain
  annotations:
    version: v1.0.0
    description: "tododomain application configuration"
spec:
  components:
    - componentName: tododomain-domain
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
              scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            spec:
              rules:
                - paths:
                    # application todo
                    - path: "/todo"
                      pathType: Prefix
    - componentName: tododomain-configmap
    - componentName: todo-mysql-service
    - componentName: todo-mysql-deployment
    - componentName: todo-mysql-configmap

The file vz-application-modified.yaml is an example of a modified vz-application.yaml file. A diff of these two sample files is shown:

$ diff vz-application.yaml vz-application-modified.yaml
30a31,33
>     - componentName: todo-mysql-service
>     - componentName: todo-mysql-deployment
>     - componentName: todo-mysql-configmap
102c105
<                   URL: "jdbc:mysql://localhost:3306/tododb"
---
>                   URL: "jdbc:mysql://mysql.tododomain.svc.cluster.local:3306/tododb"

Create Verrazzano components for MySQL

As noted previously, moving a production environment to Verrazzano would require migrating the data as well. While data migration is beyond the scope of this guide, we will still need to include a MySQL instance to be deployed with the application in the Verrazzano environment.

To do so, first, we need to create the Verrazzano components for MySQL by applying the mysql-oam.yaml file in the tododomain namespace. The components will be deployed as Kubernetes objects when the ToDo List application is deployed by applying the vz-application.yaml file in the next step.

Download the mysql-oam.yaml file.
Then, apply the YAML file:

$ kubectl apply -f mysql-oam.yaml

# Expected response
component.core.oam.dev/todo-mysql-service created
component.core.oam.dev/todo-mysql-deployment created
component.core.oam.dev/todo-mysql-configmap created

$ kubectl get components -ntododomain

# Expected response
todo-mysql-configmap    ConfigMap       26s
todo-mysql-deployment   Deployment      26s
todo-mysql-service      Service         26s

Deploy the ToDo List application and MySQL instance.

Finally, run kubectl apply to apply the Verrazzano components and Verrazzano application configuration files to start your domain.

$ kubectl apply -f vz-application.yaml

This will:

Create the application Component resources for the ToDo List application.
Deploys the Verrazzano component resources as Kubernetes objects and creates the MySQL instance.
Create the application configuration resources that create the instance of the ToDo List application in the Verrazzano cluster.

Wait for the ToDo List example application to be ready.

$ kubectl wait pod \
    --for=condition=Ready tododomain-adminserver \
    -n tododomain

# Expected response
pod/tododomain-adminserver condition met

Verify that the pods are in the Running state:

$ kubectl get pod -n tododomain

# Sample output
NAME                     READY   STATUS    RESTARTS   AGE
mysql-55bb4c4565-c8zf5   1/1     Running   0          8m
tododomain-adminserver   4/4     Running   0          5m

Access the application from your browser

Get the generated host name for the application.

$ kubectl get gateways.networking.istio.io tododomain-tododomain-appconf-gw \
    -n tododomain \
    -o jsonpath={.spec.servers[0].hosts[0]}

# Sample output
tododomain-appconf.tododomain.11.22.33.44.nip.io

Initialize the database by accessing the init URL.

https://tododomain-appconf.tododomain.11.22.33.44.nip.io/todo/rest/items/init

Access the application.

https://tododomain-appconf.tododomain.11.22.33.44.nip.io/todo

Access the WebLogic Server Administration Console

Set up port forwarding.
```
$ kubectl port-forward pods/tododomain-adminserver 7001:7001 -n tododomain
```
NOTE: If you are using the OCI Cloud Shell to run kubectl, in order to access the WebLogic Server Administration Console using port forwarding, you will need to run kubectl on another machine.
Access the WebLogic Server Administration Console from your browser.
```
http://localhost:7001/console
```

NOTE

It is recommended that the WebLogic Server Administration Console not be exposed publicly.

Docs: LoggingTrait Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The LoggingTrait custom resource contains the configuration for an additional logging sidecar with a custom image and Fluentd configuration file. Here is a sample ApplicationConfiguration that includes a LoggingTrait. To deploy an example application with this LoggingTrait, replace the ApplicationConfiguration of the ToDo-List example application with the following sample.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: todo-appconf
  namespace: todo-list
  annotations:
    version: v1.0.0
    description: "ToDo List example application"
spec:
  components:
    - componentName: todo-domain
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: LoggingTrait
            metadata:
              name: logging-trait-example
              namespace: todo-list
            spec:
              loggingImage: fluent/fleuntd-example-image # Replace with custom Fluentd Image
              loggingConfig: |-
                # Replace with Fluentd config file
                <match **>
                @type stdout
                </match>                
    - componentName: todo-jdbc-configmap
    - componentName: todo-mysql-configmap
    - componentName: todo-mysql-service
    - componentName: todo-mysql-deployment

In this sample configuration, the LoggingTrait logging-trait-example is set on the todo-domain application component and defines a logging sidecar with the given Fluentd image and configuration file. This sidecar will be attached to the component’s pod and will gather logs according to the given Fluentd configuration file. In order for the Fluentd DaemonSet to collect the custom logs, the Fluentd configuration file needs to direct the logs to STDOUT, as demonstrated in the previous example.

For example, when the ToDo-List example ApplicationConfiguration is successfully deployed with a LoggingTrait, the tododomain-adminserver pod will have a container named logging-stdout.

$ kubectl get pods tododomain-adminserver -n todo-list -o jsonpath='{.spec.containers[*].name}'
  ... logging-stdout ...

In this example, the logging-stdout container will run the image given in the LoggingTrait and a ConfigMap named logging-stdout-todo-domain-domain will be created with the custom Fluentd configuration file.

LoggingTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	LoggingTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	LoggingTraitSpec	The desired state of a logging trait.	Yes

LoggingTraitSpec

LoggingTraitSpec specifies the desired state of a logging trait.

Field	Type	Description	Required
`loggingConfig`	string	A string representation of the Fluentd configuration.	Yes
`loggingImage`	string	The name of the custom Fluentd image.	Yes

Docs: MetricsTrait Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MetricsTrait custom resource contains the configuration information needed to enable metrics for an application component. Component workloads configured with a MetricsTrait are setup to emit metrics through an endpoint that are scraped by a given Prometheus deployment. Here is a sample ApplicationConfiguration that specifies a MetricsTrait. To deploy an example application that demonstrates a MetricsTrait, see Hello World Helidon.

Note that if an ApplicationConfiguration does not specify a MetricsTrait, then a default MetricsTrait will be generated with values appropriate for the workload type.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
                scraper: verrazzano-system/vmi-system-prometheus-0
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

In the sample configuration, a MetricsTrait is specified for the hello-helidon-component application component.

With the sample application configuration successfully deployed, you can query for metrics from the application component.

$ HOST=$(kubectl get ingress \
     -n verrazzano-system vmi-system-prometheus \
     -o jsonpath={.spec.rules[0].host})
$ echo $HOST

prometheus.vmi.system.default.<ip>.nip.io

$ VZPASS=$(kubectl get secret \
     --namespace verrazzano-system verrazzano \
     -o jsonpath={.data.password} | base64 \
     --decode; echo)
$ curl -sk \
    --user verrazzano:${VZPASS} \
    -X GET https://${HOST}/api/v1/query?query=vendor_requests_count_total

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"vendor_requests_count_total","app":"hello-helidon","app_oam_dev_component":"hello-helidon-component","app_oam_dev_name":"hello-helidon-appconf","app_oam_dev_resourceType":"WORKLOAD","app_oam_dev_revision":"hello-helidon-component-v1","containerizedworkload_oam_crossplane_io":"496df78f-ef8b-4753-97fd-d9218d2f38f1","job":"hello-helidon-appconf_default_helidon-logging_hello-helidon-component","namespace":"helidon-logging","pod_name":"hello-helidon-workload-b7d9d95d8-ht7gb","pod_template_hash":"b7d9d95d8"},"value":[1616535232.487,"4800"]}]}}

MetricsTrait

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	MetricsTrait	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	MetricsTraitSpec	The desired state of a metrics trait.	Yes

MetricsTraitSpec

MetricsTraitSpec specifies the desired state of a metrics trait.

Field	Type	Description	Required
`port`	integer	The HTTP port for the related metrics endpoint. Defaults to 8080.	No
`path`	string	The HTTP path for the related metrics endpoint. Defaults to `/metrics`.	No
`secret`	string	The name of an opaque secret (for example, user name and password) within the workload’s namespace for metrics endpoint access.	No
`scraper`	string	The Prometheus deployment used to scrape the related metrics endpoints. Defaults to `verrazzano-system/vmi-system-prometheus-0`.	No

Docs: MultiClusterApplicationConfiguration Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterApplicationConfiguration custom resource is an envelope used to distribute core.oam.dev/v1alpha2/ApplicationConfiguration resources in a multicluster environment.

Here is a sample MultiClusterApplicationConfiguration that specifies an ApplicationConfiguration resource to create on the cluster named managed1. To deploy an example application that demonstrates a MultiClusterApplicationConfiguration, see Multicluster ToDo List.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterApplicationConfiguration
metadata:
  name: todo-appconf
  namespace: mc-todo-list
spec:
  template:
    metadata:
      annotations:
        version: v1.0.0
        description: "ToDo List example application"
    spec:
      components:
        - componentName: todo-domain
          traits:
            - trait:
                apiVersion: oam.verrazzano.io/v1alpha1
                kind: MetricsTrait
                spec:
                  scraper: verrazzano-system/vmi-system-prometheus-0
            - trait:
                apiVersion: oam.verrazzano.io/v1alpha1
                kind: IngressTrait
                spec:
                  rules:
                    - paths:
                        - path: "/todo"
                          pathType: Prefix
        - componentName: todo-jdbc-config
        - componentName: mysql-initdb-config
        - componentName: todo-mysql-service
        - componentName: todo-mysql-deployment
  placement:
    clusters:
      - name: managed1
  secrets:
    - tododomain-repo-credentials
    - tododomain-jdbc-tododb
    - tododomain-weblogic-credentials

MultiClusterApplicationConfiguration

A MultiClusterApplicationConfiguration is an envelope to create core.oam.dev/v1alpha2/ApplicationConfiguration resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterApplicationConfiguration	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterApplicationConfigurationSpec	The desired state of a `core.oam.dev/v1alpha2/ApplicationConfiguration` resource.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterApplicationConfigurationSpec

MultiClusterApplicationConfigurationSpec specifies the desired state of a core.oam.dev/v1alpha2/ApplicationConfiguration resource.

Field	Type	Description	Required
`template`	ApplicationConfigurationTemplate	The embedded `core.oam.dev/v1alpha2/ApplicationConfiguration` resource.	Yes
`placement`	Placement	Clusters in which the resource is to be placed.	Yes
`secrets`	string array	List of secrets used by the application. These secrets must be created in the application’s namespace before deploying a MultiClusterApplicationConfiguration resource.	No

ApplicationConfigurationTemplate

ApplicationConfigurationTemplate has the metadata and spec of the core.oam.dev/v1alpha2/ApplicationConfiguration resource.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	ApplicationConfigurationSpec	An instance of the `struct` ApplicationConfigurationSpec defined in core_types.go.	No

Docs: MultiClusterComponent Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterComponent custom resource is an envelope used to distribute core.oam.dev/v1alpha2/Component resources in a multicluster environment.

NOTE: Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterComponent custom resource not be used; instead directly use core.oam.dev/v1alpha2/Component resources in your application. See the example application, Multicluster ToDo List, which directly uses core.oam.dev/v1alpha2/Component resources.

Here is a sample MultiClusterComponent that specifies a OAM Component resource to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterComponent
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  template:
    spec:
      workload:
        apiVersion: oam.verrazzano.io/v1alpha1
        kind: VerrazzanoHelidonWorkload
        metadata:
          name: hello-helidon-workload
          namespace: hello-helidon
          labels:
            app: hello-helidon
        spec:
          deploymentTemplate:
            metadata:
              name: hello-helidon-deployment
            podSpec:
              containers:
                - name: hello-helidon-container
                  image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.12-1-20210409130027-707ecc4"
                  ports:
                    - containerPort: 8080
                      name: http
  placement:
    clusters:
      - name: managed1

MultiClusterComponent

A MultiClusterComponent is an envelope to create core.oam.dev/v1alpha2/Component resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterComponent	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterComponentSpec	The desired state of a `core.oam.dev/v1alpha2/Component` resource.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterComponentSpec

MultiClusterComponentSpec specifies the desired state of a core.oam.dev/v1alpha2/Component resource.

Field	Type	Description	Required
`template`	ComponentTemplate	The embedded `core.oam.dev/v1alpha2/Component` resource.	Yes
`placement`	Placement	Clusters in which the resource is to be placed.	Yes

ComponentTemplate

ComponentTemplate has the metadata and spec of the core.oam.dev/v1alpha2/Component resource.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	ComponentSpec	An instance of the `struct` ComponentSpec defined in core_types.go.	No

Docs: MultiClusterConfigMap Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterConfigMap custom resource is an envelope used to distribute Kubernetes ConfigMap resources in a multicluster environment.

NOTE: Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterConfigMap custom resource not be used; instead directly use core.oam.dev/v1alpha2/Component to define ConfigMap resources in your application. See the example application, Multicluster ToDo List, which uses core.oam.dev/v1alpha2/Component resources to define ConfigMaps.

Here is a sample MultiClusterConfigMap that specifies a Kubernetes ConfigMap to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterConfigMap
metadata:
  name: mymcconfigmap
  namespace: multiclustertest
spec:
  template:
    metadata:
      name: myconfigmap
      namespace: myns
    data:
      simple.key: "simplevalue"
  placement:
    clusters:
      - name: managed1

MultiClusterConfigMap

A MultiClusterConfigMap is an envelope to create Kubernetes ConfigMap resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterConfigMap	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterConfigMapSpec	The desired state of a Kubernetes ConfigMap.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterConfigMapSpec

MultiClusterConfigMapSpec specifies the desired state of a Kubernetes ConfigMap.

Field	Type	Description	Required
`template`	ConfigMapTemplate	The embedded Kubernetes ConfigMap.	Yes
`placement`	Placement	Clusters in which the ConfigMap is to be placed.	Yes

ConfigMapTemplate

ConfigMapTemplate has the metadata and spec of the Kubernetes ConfigMap.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`immutable`	*bool	Corresponds to the `immutable` field of the `struct` ConfigMap defined in types.go.	No
`data`	map[string]string	Corresponds to the `data` field of the `struct` ConfigMap defined in types.go.	No
`binaryData`	map[string][]byte	Corresponds to the `binaryData` field of the `struct` ConfigMap defined in types.go.	No

Docs: MultiClusterResourceStatus Subresource

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterResourceStatus subresource is shared by multicluster custom resources.

MultiClusterResourceStatus

MultiClusterResourceStatus specifies the status portion of a multicluster resource.

Field	Type	Description	Required
`conditions`	Condition array	The current state of a multicluster resource.	No
`state`	string	The state of the multicluster resource. State values are case-sensitive and formatted as follows: `Pending`: deployment to cluster is in progress `Succeeded`: deployment to cluster successfully completed `Failed`: deployment to cluster failed	No
`clusters`	ClusterLevelStatus array	Array of status information for each cluster.	No

Condition

Condition describes current state of a multicluster resource across all clusters.

Field	Type	Description	Required
`type`	string	The condition of the multicluster resource which can be checked with a `kubectl wait` command. Condition values are case-sensitive and formatted as follows: `DeployComplete`: deployment to all clusters completed successfully `DeployFailed`: deployment to all clusters failed	Yes
`status`	ConditionStatus	An instance of the type ConditionStatus that is defined in types.go.	Yes
`lastTransitionTime`	string	The last time the condition transitioned from one status to another.	No
`message`	string	A message with details about the last transition.	No

ClusterLevelStatus

ClusterLevelStatus describes the status of the multicluster resource on an individual cluster.

Field	Type	Description	Required
`name`	string	Name of the cluster.	Yes
`state`	string	The state of the multicluster resource. State values are case-sensitive and formatted as follows: `Pending`: deployment is in progress `Succeeded`: deployment successfully completed `Failed`: deployment failed	No
`message`	string	Message with details about the status in this cluster.	No
`lastUpdateTime`	string	The last time the resource state was updated.	Yes

Docs: MultiClusterSecret Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The MultiClusterSecret custom resource is an envelope used to distribute Kubernetes Secret resources in a multicluster environment.

NOTE: Starting with Verrazzano v1.1.0, it is preferred that the MultiClusterSecret custom resource not be used; instead specify secrets in the MultiClusterApplicationConfiguration resource. See the example application, Multicluster ToDo List where secrets are specified in a MultiClusterApplicationConfiguration resource.

Here is a sample MultiClusterSecret that specifies a Kubernetes secret to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: MultiClusterSecret
metadata:
  name: mymcsecret
  namespace: multiclustertest
spec:
  template:
    data:
      username: dmVycmF6emFubw==
      password: dmVycmF6emFubw==
  spec:
  placement:
    clusters:
      - name: managed1

MultiClusterSecret

A MultiClusterSecret is an envelope to create Kubernetes Secret resources on the clusters specified in the placement section.

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	MultiClusterSecret	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	MultiClusterSecretSpec	The desired state of a Kubernetes Secret.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

MultiClusterSecretSpec

MultiClusterSecretSpec specifies the desired state of a Kubernetes Secret.

Field	Type	Description	Required
`template`	SecretTemplate	The embedded Kubernetes Secret.	Yes
`placement`	Placement	Clusters in which the Secret is to be placed.	Yes

SecretTemplate

SecretTemplate has the metadata and spec of the Kubernetes Secret.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`data`	map[string][]byte	Corresponds to the `data` field of the `struct` Secret defined in types.go.	No
`stringData`	map[string]string	Corresponds to the `stringData` field of the `struct` Secret defined in types.go.	No
`type`	string	Corresponds to the `type` field of the `struct` Secret defined in types.go.	No

Docs: Network Traffic

Mon, 01 Jan 0001 00:00:00 +0000

Network traffic refers to the data flowing across the network. In the context of this document, it is useful to think of network traffic from two perspectives: traffic based on direction and traffic related to component types, system or applications. Traffic direction is either north-south traffic, which enters and leaves the cluster, or east-west traffic, which stays within the cluster.

First is a description of getting traffic into the cluster, then how traffic flows after it is in the cluster.

Ingress

Ingress is an overloaded term, so it needs to be understood in context. Sometimes the term means external access into the cluster, as in “ingress to the cluster.” The term also refers to the Kubernetes Ingress resource. In addition, it might be used to mean network ingress to a container in a Pod. Here, it’s used to refer to both general ingress into the cluster and the Kubernetes Ingress resource.

During installation, Verrazzano creates the necessary network resources to access both system components and applications. The following ingress and load balancers description is in the context of a Verrazzano installation.

LoadBalancer Services

To reach Pods from outside a cluster, an external IP address must be exposed using a LoadBalancer or NodePort service. Verrazzano creates two LoadBalancer services, one for system component traffic and another for application traffic. The specifics of how the service gets traffic into the cluster depends on the underlying Kubernetes platform. With Oracle OKE, creating a LoadBalancer type service will result in an OCI load balancer being created and configured to load balance to a set of Pods.

Ingress for system components

To provide ingress to system components, Verrazzano installs a NGINX Ingress Controller, which includes a NGINX load balancer. Verrazzano also creates Kubernetes Ingress resources to configure ingress for each system component that requires ingress. An Ingress resource is used is to specify HTTP/HTTPS routes to Kubernetes services, along with an endpoint hostname and a TLS certificate. An Ingress by itself doesn’t do anything; it is just a resource. An ingress controller is needed to watch Ingress resources and reconcile them, configuring the underlying Kubernetes load balancer to handle the service routing. The NGINX Ingress Controller processes Ingress resources and configures NGINX with the ingress route information, and such.

The NGINX Ingress Controller is a LoadBalancer service, as seen here:

$ kubectl get service -n ingress-nginx

# Sample output
ingress-controller-ingress-nginx-controller           LoadBalancer

Using the OKE example, traffic entering the OCI load balancer is routed to the NGINX load balancer, then routed from there to the Pods belonging to the services described in the Ingress.

Ingress for applications

Verrazzano also provides ingress into applications, but uses an Istio ingress gateway, which is an Envoy proxy, instead of NGINX. Istio has a Gateway resource that provides load balancer information, such as hosts, ports, and certificates for traffic coming into the mesh. For more information, see Istio Gateway. Just as an Ingress needs a corresponding Ingress controller, the same is true for the Gateway resource, where there is a corresponding Istio ingress gateway controller. However, unlike the Ingress, the Gateway resource doesn’t have service routing information. That is handled by the Istio VirtualService resource. The combination of Gateway and VirtualService is basically a superset of Ingress, because the combination provides more features than Ingress. In summary, the Istio ingress gateway provides ingress to the cluster using information from both the Gateway and VirtualService resources.

Because Verrazzano doesn’t create any applications during installations, there is no need to create a Gateway and VirtualService at that time. However, during installation, Verrazzano does create the Istio ingress gateway, which is a LoadBalancer service, along with the Istio egress gateway, which is a ClusterIP service.

$ kubectl get service -n istio-system

# Sample output
istio-ingressgateway   LoadBalancer

Again, referring to the OKE use case, this means that there will another OCI load balancer created, routing traffic to the Istio ingress gateway Pod, for example, the Envoy proxy.

External DNS

When you install Verrazzano, you can optionally specify an external DNS for your domain. If you do that, Verrazzano will not only create the DNS records, using ExternalDNS, but also it will configure your host name in the Ingress resources. You can then use that host name to access the system components through the NGINX Ingress Controller.

System traffic

System traffic includes all traffic that enters and leaves system Pods.

North-south system traffic

North-south traffic includes all system traffic that enters or leaves a Kubernetes cluster.

Ingress

The following lists the Verrazzano system components which are accessed through the NGINX Ingress Controller from a client external to the cluster:

Elasticsearch
Keycloak
Kibana
Grafana
Prometheus
Rancher
Verrazzano Console
Verrazzano API

Egress

The following table shows Verrazzano system components that initiate requests to a destination outside the cluster.

Component	Destination	Description
cert-manager	Let’s Encrypt	Get signed certificate.
ExternalDNS	External DNS	Create and delete DNS entries in an external DNS.
Fluentd	Elasticsearch	Fluentd on the managed cluster calls Elasticsearch on the admin cluster.
Prometheus	Prometheus	Prometheus on the admin cluster scrapes metrics from Prometheus on the managed cluster.
Rancher Agent	Rancher	Rancher agent on the managed cluster sends requests to Rancher on the admin cluster.
Verrazzano Authentication Proxy	Keycloak	Calls Keycloak for authentication, which includes redirects.
Verrazzano Platform Operator	Kubernetes API server	Multicluster agent on the managed cluster calls API server on the admin cluster.

East-west system traffic

The following tables show Verrazzano system components that send traffic to a destination inside the cluster, with the following exceptions:

Usage of CoreDNS: It can be assumed that any Pod in the cluster can access CoreDNS for name resolution.
Envoy to Istiod: The Envoy proxies all make requests to the Istio control plane to get dynamic configuration, and such. This includes both the gateways and the mesh sidecar proxies. That traffic is not shown.
Traffic within a component is not shown, for example, traffic between Elasticsearch Pods.
Prometheus scraping traffic is shown in the second table.

Component	Destination	Description
cert-manager	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Fluentd	Elasticsearch	Fluentd sends data to Elasticsearch.
Grafana	Prometheus	UI for Prometheus data.
Kibana	Elasticsearch	UI for Elasticsearch.
NGINX Ingress Controller	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Istio	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Rancher	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Verrazzano Authentication Proxy	Keycloak	Calls Keycloak for token authentication.
Verrazzano Authentication Proxy	VMI components	Access UIs for Kibana, Grafana, and such.
Verrazzano Authentication Proxy	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Verrazzano Application Operator	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Verrazzano Monitoring Operator	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Verrazzano Operator	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Verrazzano Platform Operator	Kubernetes API server	Perform CRUD operations on Kubernetes resources.
Verrazzano Platform Operator	Rancher	Register the managed cluster with Rancher.

Prometheus scraping traffic

This table shows Prometheus traffic for each system component scrape target.

Target	Description
cadvisor	Kubernetes metrics.
Elasticsearch	Envoy metrics.
Grafana	Envoy metrics.
Istiod	Istio control plane metrics.
Istiod	Envoy metrics.
Istio egress gateway	Envoy metrics.
Istio ingress gateway	Envoy metrics.
Keycloak	Envoy metrics.
Kibana	Envoy metrics.
MySQL	Envoy metrics.
NGINX Ingress Controller	Envoy metrics.
NGINX Ingress Controller	NGINX metrics.
NGINX default back end	Envoy metrics.
Node exporter	Node metrics.
Prometheus	Envoy metrics.
Prometheus	Prometheus metrics.
Verrazzano Console	Envoy metrics.
Verrazzano API	Envoy metrics.
WebLogic operator	Envoy metrics.

Webhooks

Several of the system components are controllers, and some of those have webhooks. Webhooks are called by the Kubernetes API server on a component HTTPS port to validate or mutate API payloads before they reach the API server.

The following components use webhooks:

cert-manager
Coherence Operator
Istio
Rancher
Verrazzano Application Operator
Verrazzano Platform Operator

Application traffic

Application traffic includes all traffic to and from Verrazzano applications.

North-south application traffic

After Verrazzano is installed, you can deploy applications into the Istio mesh. When doing so, you will likely need ingress into the application. As previously mentioned, this can be done with Istio using the Gateway and VirtualService resources. Verrazzano will create those resources for you when you use an IngressTrait in your ApplicationConfiguration. The Istio ingress gateway created during installation will be shared by all applications in the mesh, and the Gateway resource is bound to the Istio ingress gateway that was created during installation. This is done by the selector field in the Gateway:

   selector:
     istio: ingressgateway

Verrazzano creates a Gateway/VirtualService pair for each IngressTrait. Following is an example of those two resources created by Verrazzano.

Here is the Gateway; in this case both the host name and certificate were generated by Verrazzano.

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: Gateway
  metadata:
   ...
    name: hello-helidon-hello-helidon-appconf-gw
    namespace: hello-helidon
  ...
  spec:
    selector:
      istio: ingressgateway
    servers:
    - hosts:
      - hello-helidon-appconf.hello-helidon.1.2.3.4.nip.io
      port:
        name: HTTPS
        number: 443
        protocol: HTTPS
      tls:
        credentialName: hello-helidon-hello-helidon-appconf-cert-secret
        mode: SIMPLE

Here is the VirtualService; notice that it refers back to the Gateway and that it contains the service routing information.

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
  ...
    name: hello-helidon-ingress-rule-0-vs
    namespace: hello-helidon
  spec:
    gateways:
    - hello-helidon-hello-helidon-appconf-gw
    hosts:
    - hello-helidon-appconf.hello-helidon.1.2.3.4.nip.io
    HTTP:
    - match:
      - uri:
          prefix: /greet
      route:
      - destination:
          host: hello-helidon
          port:
            number: 8080

East-west application traffic

To manage east-west traffic, each service in the mesh should be routed using a VirtualService and an optional DestinationRule. You can still send east-west traffic without either of these resources, but you won’t get any custom routing or load balancing. Verrazzano doesn’t configure east-west traffic. Consider bobbys-front-end in the Bob’s Books example at bobs-books-comp.yaml. When deploying Bob’s Books, a VirtualService is created for bobbys-front-end, because of the IngressTrait, but there are no VirtualServices for the other services in the application. When bobbys-front-end sends requests to bobbys-helidon-stock-application, this east-west traffic still goes to bobbys-helidon-stock-application through the Envoy sidecar proxies in the source and destination Pods, but there is no VirtualService representing bobbys-helidon-stock-application, where you could specify a canary deployment or custom load balancing. This is something you could configure manually, but it is not configured by Verrazzano.

Proxies

Verrazzano uses network proxies in multiple places. The two proxy products are Envoy and NGINX. The following table shows which proxies are used and in which Pod they run.

Usage	Proxy	Pod	Namespace	Description
System ingress	NGINX	`ingress-controller-ingress-nginx-controller-*`	`ingress-nginx`	Provides external access to Verrazzano system components.
Verrazzano authentication proxy	NGINX	`verrazzano-authproxy-*`	`verrazzano-system`	Verrazzano authentication proxy server for Kubernetes API and SSO.
Application ingress	Envoy	`istio-ingressgateway-*`	`istio-system`	Provides external access to Verrazzano applications.
Application egress	Envoy	`istio-egressgateway-*`	`istio-system`	Provides control of application egress traffic.
Istio mesh sidecar	Envoy	`ingress-controller-ingress-nginx-controller-*`	`ingress-nginx`	NGINX Ingress Controller in the Istio mesh.
Istio mesh sidecar	Envoy	`ingress-controller-ingress-nginx-defaultbackend-*`	`ingress-nginx`	NGINX default backend in the Istio mesh.
Istio mesh sidecar	Envoy	`fluentd-*`	`verrazzano-system`	Fluentd in the Istio mesh.
Istio mesh sidecar	Envoy	`keycloak-*`	`keycloak`	Keycloak in the Istio mesh.
Istio mesh sidecar	Envoy	`mysql-*`	`keycloak`	MySQL used by Keycloak in the Istio mesh.
Istio mesh sidecar	Envoy	`verrazzano-api-*`	`verrazzano-system`	Verrazzano API in the Istio mesh.
Istio mesh sidecar	Envoy	`verrazzano-console-*`	`verrazzano-system`	Verrazzano Console in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-master-*`	`verrazzano-system`	Elasticsearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-data-*`	`verrazzano-system`	Elasticsearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-ingest-*`	`verrazzano-system`	Elasticsearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-kibana-*`	`verrazzano-system`	Kibana in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-prometheus-*`	`verrazzano-system`	Prometheus in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-grafana-*`	`verrazzano-system`	Grafana in the Istio mesh.
Istio mesh sidecar	Envoy	`weblogic-operator-*`	`verrazzano-system`	WebLogic operator in the Istio mesh.

Multicluster

Some Verrazzano components send traffic between Kubernetes clusters. Those components are the Verrazzano agent, Verrazzano authentication proxy, and Prometheus.

Multicluster egress

The following table shows Verrazzano system components that initiate requests between the admin and managed clusters. All of these requests go through the NGINX Ingress Controller on the respective destination cluster.

Source Cluster	Source Component	Destination Cluster	Destination Component	Description
Admin	Prometheus	Managed	Prometheus	Scape metrics on managed clusters.
Admin	Verrazzano Console	Managed	Verrazzano Authentication Proxy	Admin cluster proxy sends Kubernetes API requests to managed cluster proxy.
Managed	Fluentd	Admin	Elasticsearch	Fluentd sends logs to Elasticsearch.
Managed	Rancher Agent	Admin	Rancher	Rancher Agent sends requests Rancher.
Managed	Verrazzano Authentication Proxy	Admin	Keycloak	Proxy sends requests to Keycloak.
Managed	Verrazzano Agent	Admin	Kubernetes API server	Agent, in the platform operator, sends requests Kubernetes API server.

Verrazzano agent

In the multicluster topology, the Verrazzano platform operator has an agent thread running on the managed cluster that sends requests to the Kubernetes API server on the admin cluster. The URL for the admin cluster Kubernetes API server is registered on the managed cluster by the user.

Verrazzano authentication proxy

In a multicluster topology, the Verrazzano authentication proxy runs on both the admin and managed clusters.
On the admin cluster, the authentication proxy connects to in-cluster Keycloak, using the Keycloak Service. On the managed cluster, the authentication proxy connects to Keycloak on the admin cluster through the NGINX Ingress Controller running on the admin cluster.

For SSO, the authentication proxy also needs to send requests to Keycloak, either in-cluster or through the cluster ingress. When a request comes into the authentication proxy without an authentication header, the proxy sends a request to Keycloak through the NGINX Ingress Controller, so the request exits the cluster. Otherwise, if the authentication proxy is on the admin cluster, then the request is sent directly to Keycloak within the cluster. If the authentication proxy is on the managed cluster, then it must send requests to Keycloak on the admin cluster.

Prometheus

A single Prometheus service in the cluster, scrapes metrics from Pods in system components and applications. It also scrapes Pods in the Istio mesh using HTTPS, and outside the mesh using HTTP. In the multicluster case, the Prometheus on the admin cluster, scrapes metrics from Prometheus on the managed cluster, through the NGINX Ingress Controller on the managed cluster.

Docs: Placement Subresource

Mon, 01 Jan 0001 00:00:00 +0000

The Placement subresource is shared by multicluster custom resources.

Placement

Placement contains the name of each cluster where this resource will be located.

Field	Type	Description	Required
`clusters`	Cluster array	An array of cluster locations.	Yes

Cluster

Cluster contains the name of a single cluster.

Field	Type	Description	Required
`cluster`	string	The name of a cluster.	Yes

Docs: Verrazzano Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano custom resource contains the configuration information for an installation. Here is a sample Verrazzano custom resource file that uses OCI DNS. See other examples here.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  environmentName: env
  profile: prod
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: emailAddress@example.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: dnsZoneCompartmentOcid
        dnsZoneOCID: dnsZoneOcid
        dnsZoneName: my.dns.zone.name
    ingress:
      type: LoadBalancer

VerrazzanoSpec

Field	Type	Description	Required
`environmentName`	string	Name of the installation. This name is part of the endpoint access URLs that are generated. The default value is `default`.	No
`profile`	string	The installation profile to select. Valid values are `prod` (production) and `dev` (development). The default is `prod`.	No
`version`	string	The version to install. Valid versions can be found here. Defaults to the current version supported by the Verrazzano platform operator.	No
`components`	Components	The Verrazzano components.	No
`defaultVolumeSource`	VolumeSource	Defines the type of volume to be used for persistence for all components unless overridden, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of an existing `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.	No
`volumeClaimSpecTemplates`	VolumeClaimSpecTemplate	Defines a named set of PVC configurations that can be referenced from components to configure persistent volumes.	No

VolumeClaimSpecTemplate

Field	Type	Description	Required
`metadata`	ObjectMeta	Metadata about the PersistentVolumeClaimSpec template.	No
`spec`	PersistentVolumeClaimSpec	A `PersistentVolumeClaimSpec` template that can be referenced by a Component to override its default storage settings for a profile. At present, only a subset of the `resources.requests` object are honored depending on the component.	No

Components

Field	Type	Description	Required
`certManager`	CertManagerComponent	The cert-manager component configuration.	No
`dns`	DNSComponent	The DNS component configuration.	No
`ingress`	IngressComponent	The ingress component configuration.	No
`istio`	IstioComponent	The Istio component configuration.	No
`fluentd`	FluentdComponent	The Fluentd component configuration.	No
`keycloak`	KeycloakComponent	The Keycloak component configuration.	No
`elasticsearch`	ElasticsearchComponent	The Elasticsearch component configuration.	No
`prometheus`	PrometheusComponent	The Prometheus component configuration.	No
`kibana`	KibanaComponent	The Kibana component configuration.	No
`grafana`	GrafanaComponent	The Grafana component configuration.	No
`kiali`	KialiComponent	The Kiali component configuration.	No

CertManager Component

Field	Type	Description	Required
`certificate`	Certificate	The certificate configuration.	No

Certificate

Field	Type	Description	Required
`acme`	Acme	The ACME configuration. Either `acme` or `ca` must be specified.	No
`ca`	CertificateAuthority	The certificate authority configuration. Either `acme` or `ca` must be specified.	No

Acme

Field	Type	Description	Required
`provider`	string	Name of the Acme provider.	Yes
`emailAddress`	string	Email address of the user.	Yes

CertificateAuthority

Field	Type	Description	Required
`secretName`	string	The secret name.	Yes
`clusterResourceNamespace`	string	The secrete namespace.	Yes

DNS Component

Field	Type	Description	Required
`wildcard`	DNS-Wilcard	Wildcard DNS configuration. This is the default with a domain of `nip.io`.	No
`oci`	DNS-OCI	OCI DNS configuration.	No
`external`	DNS-External	External DNS configuration.	No

DNS Wildcard

Field	Type	Description	Required
`domain`	string	The type of wildcard DNS domain. For example, `nip.io`, `sslip.io`, and such.	Yes

DNS OCI

Field	Type	Description	Required
`ociConfigSecret`	string	Name of the OCI configuration secret. Generate a secret based on the OCI configuration profile you want to use. You can specify a profile other than DEFAULT and specify the secret name. See instructions by running `./install/create_oci_config_secret.sh`.	Yes
`dnsZoneCompartmentOCID`	string	The OCI DNS compartment OCID.	Yes
`dnsZoneOCID`	string	The OCI DNS zone OCID.	Yes
`dnsZoneName`	string	Name of OCI DNS zone.	Yes
`dnsScope`	string	Scope of the OCI DNS zone (PRIVATE, GLOBAL). If not specified, then defaults to GLOBAL.	No

DNS External

Field	Type	Description	Required
`suffix`	string	The suffix for DNS names.	Yes

Ingress Component

Field	Type	Description	Required
`type`	string	The ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`.	Yes
`nginxInstallArgs`	NGINXInstallArgs list	A list of values to use during NGINX installation.	No
`ports`	PortConfig list	The list port configurations used by the ingress.	No

NGINX Install Args

Name	Type	ValueType	Description	Required
`controller.service.externalIPs`	NameValue	string list	The external IP address used by the NGINX Ingress Controller.	No
`controller.service.externalTrafficPolicy`	NameValue	string	Preserves the client source IP address. See Bare-metal considerations.	No
`controller.service.annotations.*`	NameValue	string	Annotations used for NGINX Ingress Controller. For sample usage, see Customize Ingress.	No
`controller.autoscaling.enabled`	NameValue	Boolean	If true, then enable horizonal pod autoscaler. Default false.	No
`controller.autoscaling.minReplicas`	NameValue	string	Minimum replicas used for autoscaling. Default 1.	No

Port Config

Field	Type	Description	Required
`name`	string	The port name.	No
`port`	string	The port value.	Yes
`targetPort`	string	The target port value. The default is same as the port value.	Yes
`protocol`	string	The protocol used by the port. TCP is the default.	No
`nodePort`	string	The `nodePort` value.	No

Name Value

Field	Type	Description	Required
`name`	string	The name of a Helm override for a Verrazzano component chart, specified with a `—set` flag on the Helm command line, for example, `helm install --set name=value`. For more information about chart overrides, see Customize Ingress.	Yes
`value`	string	The value of a Helm override for a Verrazzano component chart, specified with a `—set` flag on the Helm command line, for example, `helm install --set name=value`. Either `value` or `valueList` must be specified. For more information about chart overrides, see Customize Ingress.	No
`valueList`	string list	The list of Helm override values for a Verrazzano component, each specified with a `—set` flag on the Helm command line, for example, `helm install --set name[0]=<first element of valueList> —set name[1]=<second element of valueList>`. Either `value` or `valueList` must be specified. For more information about chart overrides, see Customize Ingress.	No
`setString`	Boolean	Specifies if the argument requires the Helm `--set-string` command-line flag to override a chart value, for example, `helm install --set-string name=value`.	No

Istio Component

Field	Type	Description	Required
`istioInstallArgs`	IstioInstallArgs list	A list of values to use during Istio installation. Each argument is specified as either a `name/value` or `name/valueList` pair.	No

Istio Install Args

Name	Type	ValueType	Description	Required
`gateways.istio-ingressgateway.externalIPs`	NameValue	string list	The external IP address used by the Istio Ingress Gateway.	No
`gateways.istio-ingressgateway.serviceAnnotations.*`	NameValue	string	Annotations used for the Istio Ingress Gateway. For sample usage, see Customize Ingress.	No

Fluentd Component

Field	Type	Description	Required
`extraVolumeMounts`	ExtraVolumeMount list	A list of host path volume mounts in addition to `/var/log` into the Fluentd DaemonSet. The Fluentd component collects log files in the `/var/log/containers` directory of Kubernetes worker nodes. The `/var/log/containers` directory may contain symbolic links to files located outside the `/var/log` directory. If the host path directory containing the log files is located outside of `/var/log`, the Fluentd DaemonSet must have the volume mount of that directory to collect the logs.	No
`elasticsearchURL`	string	The target Elasticsearch URLs. Specify this option in this format. The default `http://vmi-system-es-ingest-oidc:8775` is the VMI Elasticsearch URL.	No
`elasticsearchSecret`	string	The secret containing the credentials for connecting to Elasticsearch. This secret needs to be created in the `verrazzano-install` namespace prior to creating the Verrazzano custom resource. Specify the Elasticsearch login credentials in the `username` and `password` fields in this secret. Specify the CA for verifying the Elasticsearch certificate in the `ca-bundle` field, if applicable. The default `verrazzano` is the secret for connecting to the VMI Elasticsearch.	No

Extra Volume Mount

Field	Type	Description	Required
`source`	string	The source host path.	Yes
`destination`	string	The destination path on the Fluentd Container, defaults to the `source` host path.	No
`readOnly`	Boolean	Specifies if the volume mount is read-only, defaults to `true`.	No

Keycloak Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Keycloak will be installed.	No
`mysql`	MySQLComponent	Contains the MySQL component configuration needed for Keycloak.	No

MySQL Component

Field	Type	Description	Required
`volumeSource`	VolumeSource	Defines the type of volume to be used for persistence for Keycloak/MySQL, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of a `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.	No

Elasticsearch Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Elasticsearch will be installed.	No
`installArgs`	ElasticsearchInstallArgs list	A list of values to use during Elasticsearch installation. Each argument is specified as either a `name/value` or `name/valueList` pair. For sample usage, see Customize Elasticsearch.	No

Elasticsearch Install Args

Name	Type	ValueType	Description	Required
`nodes.master.replicas`	NameValue	string	The number of master node replicas.	No
`nodes.master.requests.memory`	NameValue	string	The master node memory request amount expressed as a Quantity.	No
`nodes.ingest.replicas`	NameValue	string	The number of ingest node replicas.	No
`nodes.ingest.requests.memory`	NameValue	string	The ingest node memory request amount expressed as a Quantity.	No
`nodes.data.replicas`	NameValue	string	The number of data node replicas.	No
`nodes.data.requests.memory`	NameValue	string	The data node memory request amount expressed as a Quantity.	No
`nodes.data.requests.storage`	NameValue	string	The data storage request amount expressed as a Quantity.	No

Kibana Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Kibana will be installed.	No

Prometheus Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Prometheus will be installed.	No

Grafana Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Grafana will be installed.	No

Kiali Component

Field	Type	Description	Required
`enabled`	Boolean	If true, then Kiali will be installed.	No

Docs: Verrazzano Workload Custom Resource Definitions

Mon, 01 Jan 0001 00:00:00 +0000

VerrazzanoCoherenceWorkload

The VerrazzanoCoherenceWorkload custom resource contains the configuration information for a Coherence workload within Verrazzano. Here is a sample component that specifies a VerrazzanoCoherenceWorkload. To deploy an example application that demonstrates this workload type, see Sock Shop.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: carts
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: carts-coh
        spec:
          cluster: SockShop
          role: Carts
          replicas: 1
          image: ghcr.io/helidon-sockshop/carts-coherence:2.2.0
          imagePullPolicy: Always
          application:
            type: helidon
          jvm:
            args:
              - "-Dcoherence.k8s.operator.health.wait.dcs=false"
              - "-Dcoherence.metrics.legacy.names=false"
            memory:
              heapSize: 2g
          coherence:
            logLevel: 9
          ports:
            - name: http
              port: 7001
              service:
                name: carts
                port: 80
              serviceMonitor:
                enabled: true
            - name: metrics
              port: 7001
              serviceMonitor:
                enabled: true

VerrazzanoCoherenceWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoCoherenceWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoCoherenceWorkloadSpec	The desired state of a Verrazzano Coherence workload.	Yes

VerrazzanoCoherenceWorkloadSpec

VerrazzanoCoherenceWorkloadSpec specifies the desired state of a Verrazzano Coherence workload.

Field	Type	Description	Required
`template`	RawExtension	The metadata and spec for the underlying Coherence resource.	Yes

VerrazzanoHelidonWorkload

The VerrazzanoHelidonWorkload custom resource contains the configuration information for a Helidon workload within Verrazzano. Here is a sample component that specifies a VerrazzanoHelidonWorkload. To deploy an example application that demonstrates this workload type, see Hello World Helidon.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.10-3-20201016220428-56fb4d4"
              ports:
                - containerPort: 8080
                  name: http

VerrazzanoHelidonWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoHelidonWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoHelidonWorkloadSpec	The desired state of a Verrazzano Helidon workload.	Yes

VerrazzanoHelidonWorkloadSpec

VerrazzanoHelidonWorkloadSpec specifies the desired state of a Verrazzano Helidon workload.

Field	Type	Description	Required
`deploymentTemplate`	DeploymentTemplate	The embedded deployment.	Yes

DeploymentTemplate

DeploymentTemplate specifies the metadata and pod spec of the underlying deployment.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`strategy`	DeploymentStrategy	The replacement strategy of the underlying deployment.	No
`podSpec`	PodSpec	The pod spec of the underlying deployment.	Yes

VerrazzanoWebLogicWorkload

The VerrazzanoWebLogicWorkload custom resource contains the configuration information for a WebLogic Domain workload within Verrazzano. Here is a sample component that specifies a VerrazzanoWebLogicWorkload. To deploy an example application that demonstrates this workload type, see the ToDo List Lift-and-Shift application.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: todo-domain
  namespace: todo-list
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoWebLogicWorkload
    spec:
      template:
        metadata:
          name: todo-domain
          namespace: todo-list
        spec:
          domainUID: tododomain
          domainHome: /u01/domains/tododomain
          image: container-registry.oracle.com/verrazzano/example-todo:0.8.0
          imagePullSecrets:
            - name: tododomain-repo-credentials
          domainHomeSourceType: "FromModel"
          includeServerOutInPodLog: true
          replicas: 1
          webLogicCredentialsSecret:
            name: tododomain-weblogic-credentials
          configuration:
            introspectorJobActiveDeadlineSeconds: 900
            model:
              configMap: tododomain-jdbc-config
              domainType: WLS
              modelHome: /u01/wdt/models
              runtimeEncryptionSecret: tododomain-runtime-encrypt-secret
            secrets:
              - tododomain-jdbc-tododb
          serverPod:
            env:
              - name: JAVA_OPTIONS
                value: "-Dweblogic.StdoutDebugEnabled=false"
              - name: USER_MEM_ARGS
                value: "-Djava.security.egd=file:/dev/./urandom -Xms64m -Xmx256m "
              - name: WL_HOME
                value: /u01/oracle/wlserver
              - name: MW_HOME
                value: /u01/oracle

VerrazzanoWebLogicWorkload

Field	Type	Description	Required
`apiVersion`	string	`oam.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoWebLogicWorkload	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	No
`spec`	VerrazzanoWebLogicWorkloadSpec	The desired state of a Verrazzano WebLogic workload.	Yes

VerrazzanoWebLogicWorkloadSpec

VerrazzanoWebLogicWorkloadSpec specifies the desired state of a Verrazzano WebLogic workload.

Field	Type	Description	Required
`template`	RawExtension	The metadata and spec for the underlying WebLogic Domain resource.	Yes

Docs: VerrazzanoManagedCluster Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The VerrazzanoManagedCluster custom resource is used to register a managed cluster with an admin cluster. Here is a sample VerrazzanoManagedCluster that registers the cluster named managed1. To deploy an example application that demonstrates a VerrazzanoManagedCluster, see Multicluster Hello World Helidon.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Managed Cluster 1"
  caSecret: ca-secret-managed1

VerrazzanoManagedCluster

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoManagedCluster	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	VerrazzanoManagedClusterSpec	The managed cluster specification.	Yes
`status`	VerrazzanoManagedClusterStatus	The runtime status this resource.	No

VerrazzanoManagedClusterSpec

VerrazzanoManagedClusterSpec specifies a managed cluster to associate with an admin cluster.

Field	Type	Description	Required
`description`	string	The description of the managed cluster.	No
`caSecret`	string	The name of a Secret that contains the CA certificate of the managed cluster. This is used to configure the admin cluster to scrape metrics from the Prometheus endpoint on the managed cluster. See the steps 3 and 4 in instructions for how to create this Secret.	Yes
`serviceAccount`	string	The name of the ServiceAccount that was generated for the managed cluster. This field is managed by a Verrazzano Kubernetes operator.	No
`managedClusterManifestSecret`	string	The name of the Secret containing generated YAML manifest file to be applied by the user to the managed cluster. This field is managed by a Verrazzano Kubernetes operator.	No

VerrazzanoManagedClusterStatus

Field	Type	Description	Required
`conditions`	Condition array	The current state of this resource.	No
`lastAgentConnectTime`	string	The last time the agent from this managed cluster connected to the admin cluster.	No
`apiUrl`	string	The Verrazzano API server URL for the managed cluster.	No

Condition

Condition describes current state of this resource.

Field	Type	Description	Required
`type`	string	The condition of the multicluster resource which can be checked with a `kubectl wait` command. Condition values are case-sensitive and formatted as follows: `Ready`: the VerrazzanoManagedCluster is ready to be used and all resources needed have been generated.	Yes
`status`	ConditionStatus	An instance of the type ConditionStatus that is defined in types.go.	Yes
`lastTransitionTime`	string	The last time the condition transitioned from one status to another.	No
`message`	string	A message with details about the last transition.	No

Docs: VerrazzanoProject Custom Resource Definition

Mon, 01 Jan 0001 00:00:00 +0000

The VerrazzanoProject custom resource is used to create the application namespaces and their associated security settings on one or more clusters. The namespaces are always created on the admin cluster. Here is a sample VerrazzanoProject that specifies a namespace to create on the cluster named managed1.

apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoProject
metadata:
  name: hello-helidon
  namespace: verrazzano-mc
spec:
  template:
    namespaces:
      - metadata:
          name: hello-helidon
  placement:
    clusters:
      - name: managed1

VerrazzanoProject

Field	Type	Description	Required
`apiVersion`	string	`clusters.verrazzano.io/v1alpha1`	Yes
`kind`	string	VerrazzanoProject	Yes
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	VerrazzanoProjectSpec	The project specification.	Yes
`status`	MultiClusterResourceStatus	The runtime status of a multicluster resource.	No

VerrazzanoProjectSpec

VerrazzanoProjectSpec specifies the namespaces to create and on which clusters to create them.

Field	Type	Description	Required
`template`	ProjectTemplate	The project template.	Yes
`placement`	Placement	Clusters on which the namespaces are to be created.	Yes

ProjectTemplate

ProjectTemplate contains the list of namespaces to create and the optional security configuration for each namespace.

Field	Type	Description	Required
`namespaces`	NamespaceTemplate array	The list of application namespaces to create for this project.	Yes
`security`	SecuritySpec	The project security configuration.	No
`networkPolicies`	NetworkPolicyTemplate array	The network policies applied to namespaces in the project.	No

NamespaceTemplate

NamespaceTemplate contains the metadata and specification of a Kubernetes namespace.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	NamespaceSpec	An instance of the `struct` NamespaceSpec defined in types.go.	No

SecuritySpec

SecuritySpec defines the security configuration for a project.

Field	Type	Description	Required
`projectAdminSubjects`	Subject	The subject to bind to the `verrazzano-project-admin` role. Encoded as an instance of the `struct` Subject defined in types.go.	No
`projectMonitorSubjects`	Subject	The subject to bind to the `verrazzano-project-monitoring` role. Encoded as an instance of the `struct` Subject defined in types.go.	No

NetworkPolicyTemplate

NetworkPolicyTemplate contains the metadata and specification of the underlying NetworkPolicy.

NOTE

To add application NetworkPolicy, see NetworkPolicies for applications.

Field	Type	Description	Required
`metadata`	ObjectMeta	Refer to Kubernetes API documentation for fields of metadata.	Yes
`spec`	NetworkPolicySpec	An instance of the `struct` NetworkPolicySpec defined in types.go.	No

Docs: Hello World Helidon

Mon, 01 Jan 0001 00:00:00 +0000

The Hello World Helidon example is a Helidon-based service that returns a “Hello World” response when invoked. The example application is specified using Open Application Model (OAM) component and application configuration YAML files, and then deployed by applying those files.

The example application has two endpoints, which differ in configuration source:

/greet- uses a microprofile properties file. Deploy this application by using the instructions here.
/config- uses a Kubernetes ConfigMap. Deploy this application by using the instructions here.

For more information and the code of this application, see the Verrazzano examples.

Docs: Network Security

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano manages and secures network traffic between Verrazzano system components and deployed applications. Verrazzano does not manage or secure traffic for the Kubernetes cluster itself, or for non-Verrazzano services or applications running in the cluster. Traffic is secured at two levels in the network stack:

ISO Layer 3/4: Using NetworkPolicies to control IP access to Pods.
ISO Layer 6: Using TLS and mTLS to provide authentication, confidentiality, and integrity for connections within the cluster, and for external connections.

NetworkPolicies

By default, all Pods in a Kubernetes cluster have network access to all other Pods in the cluster. Kubernetes has a NetworkPolicy resource that provides network level 3 and 4 security for Pods, restricting both ingress and egress IP traffic for a set of Pods in a namespace. Verrazzano configures all system components with NetworkPolicies to control ingress. Egress is not restricted.

NOTE: A NetworkPolicy resource needs a NetworkPolicy controller to implement the policy, otherwise the policy has no effect. You must install a Kubernetes CNI plug-in that provides a NetworkPolicy controller, such as Calico, before installing Verrazzano, or else the policies are ignored.

NetworkPolicies for system components

Verrazzano installs a set of NetworkPolicies for system components to control ingress into the Pods. A policy is scoped to a namespace and uses selectors to specify the Pods that the policy applies to, along with the ingress and egress rules. For example, the following policy applies to the Verrazzano API Pod in the verrazzano-system namespace. This policy allows network traffic from NGINX Ingress Controller on port 8775, and from Prometheus on port 15090. No other Pods can reach those ports or any other ports of the Verrazzano API Pod. Notice that namespace selectors need to be used; the NetworkPolicy resource does not support specifying the namespace name.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
...
spec:
  PodSelector:
    matchLabels:
      app: verrazzano-api
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: ingress-nginx
      PodSelector:
        matchLabels:
          app.kubernetes.io/instance: ingress-controller
    ports:
    - port: 8775
      protocol: TCP
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      PodSelector:
        matchLabels:
          app: system-prometheus
    ports:
    - port: 15090
      protocol: TCP

The following table shows all of the ingresses that allow network traffic into system components. The ports shown are Pod ports, which is what NetworkPolicies require.

Component	Pod Port	From	Description
Verrazzano Application Operator	9443	Kubernetes API Server	Webhook entrypoint.
Verrazzano Platform Operator	9443	Kubernetes API Server	Webhook entrypoint.
Verrazzano Console	8000	NGINX Ingress	Access from external client.
Verrazzano Console	15090	Prometheus	Prometheus scraping.
Verrazzano Authentication Proxy	8775	NGINX Ingress	Access from external client.
Verrazzano Authentication Proxy	15090	Prometheus	Prometheus scraping.
cert-manager	9402	Prometheus	Prometheus scraping.
Coherence Operator	9443	Prometheus	Webhook entrypoint.
Elasticsearch	8775	NGINX Ingress	Access from external client.
Elasticsearch	8775	Fluentd	Access from Fluentd.
Elasticsearch	9200	Kibana, Internal	Elasticsearch data port.
Elasticsearch	9300	Internal	Elasticsearch cluster port.
Elasticsearch	15090	Prometheus	Envoy metrics scraping.
Istio control plane	15012	Envoy	Envoy access to `istiod`.
Istio control plane	15014	Prometheus	Prometheus scraping.
Istio control plane	15017	Kubernetes API Server	Webhook entrypoint.
Istio ingress gateway	8443	External	Application ingress.
Istio ingress gateway	15090	Prometheus	Prometheus scraping.
Istio egress gateway	8443	Mesh services	Application egress.
Istio egress gateway	15090	Prometheus	Prometheus scraping.
Keycloak	8080	NGINX Ingress	Access from external client.
Keycloak	15090	Prometheus	Prometheus scraping.
MySql	15090	Prometheus	Prometheus scraping.
MySql	3306	Keycloak	Keycloak datastore.
Node exporter	9100	Prometheus	Prometheus scraping.
Rancher	80	NGINX Ingress	Access from external client.
Rancher	9443	Kubernetes API Server	Webhook entrypoint.
Prometheus	8775	NGINX Ingress	Access from external client.
Prometheus	9090	Grafana	Acccess for Grafana UI.

NetworkPolicies for applications

By default, applications do not have NetworkPolicies that restrict ingress into the application or egress from it. You can configure them for the application namespaces using the NetworkPolicy section of a Verrazzano project.

NOTE

Verrazzano requires specific ingress to and egress from application pods. If you add a NetworkPolicy for your application namespace or pods, you must add an additional policy to ensure that Verrazzano still has the required access it needs. The ingress policy is only needed if you restrict ingress. Likewise, the egress policy is only needed if you restrict egress. The following are the ingress and egress NetworkPolicies:

ingress NetworkPolicies

  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-ingressgateway
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: system-prometheus
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: weblogic-operator

egress NetworkPolicies

  egress:
  - ports:
    - port: 15012
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-egressgateway
  - ports:
    - port: 53
      protocol: TCP
    - port: 53
      protocol: UDP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: kube-system
  - ports:
    - port: 8000
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator

NetworkPolicies for Envoy sidecar proxies

As mentioned, Envoy sidecar proxies run in both system component pods and application pods. Each proxy sends requests to the Istio control plane pod, istiod, for a variety of reasons. During installation, Verrazzano creates a NetworkPolicy named istiod-access in the istio-system namespace to give ingress to system component sidecar proxies. For applications, Verrazzano creates a per-application NetworkPolicy in the istio-system namespace to allow the same access to istiod. When the application is deleted, Verrazzano will delete the policy.

mTLS

Istio can be enabled to use mTLS between services in the mesh, and also between the Istio gateways and Envoy sidecar proxies. There are various options to customize mTLS usage, for example it can be disabled on a per-port level. The Istio control plane, Istiod, is a CA and provides key and certificate rotation for the Envoy proxies, both gateways and sidecars.

Verrazzano configures Istio to have strict mTLS for the mesh. All components and applications put into the mesh will use mTLS, with the exception of Coherence clusters, which are not in the mesh. Also, all traffic between the Istio ingress gateway and mesh sidecars use mTLS, and the same is true between the proxy sidecars and the egress gateway.

Verrazzano sets up mTLS during installation with the PeerAuthentication resource as follows:

apiVersion: v1
items:
- apiVersion: security.istio.io/v1beta1
  kind: PeerAuthentication
  ...
  spec:
    mtls:
      mode: STRICT

TLS

TLS is used by external clients to access the cluster, both through the NGINX Ingress Controller and the Istio ingress gateway. The certificate used by these TLS connections vary; see Verrazzano security for details. All TLS connections are terminated at the ingress proxy. Traffic between the two proxies and the internal cluster Pods always uses mTLS, because those Pods are all in the Istio mesh.

Istio mesh

Istio provides extensive security protection for both authentication and authorization, as described in Istio Security. Access control and mTLS are two security features that Verrazzano configures. These security features are available in the context of a service mesh.

A service mesh is an infrastructure layer that provides certain capabilities like security, observability, load balancing, and such, for services. Istio defines a service mesh here. In the context of Istio on Kubernetes, a service in the mesh is a Kubernetes Service. Consider the Bob’s Books example application, which has several OAM Components defined. At runtime, there is a Kubernetes Service for each component, and each Service is in the mesh, with one or more Pods associated with the service. All services in the mesh have an Envoy proxy in front of their Pods, intercepting network traffic to and from the Pod. In Kubernetes, that proxy happens to be a sidecar running in each Pod.

There are various ways to put a service in the mesh. Verrazzano uses the namespace label, istio-injection: enabled, to designate that all Pods in a given namespace are in the mesh. When a Pod is created in that namespace, the Istio control plane mutating webhook, changes the Pod spec to add the Envoy proxy sidecar container, causing the Pod to be in the mesh.

Disabling sidecar injection

In certain cases, Verrazzano needs to disable sidecar injection for specific Pods in a namespace. This is done in two ways: first, during installation, Verrazzano modifies the istio-sidecar-injector ConfigMap using a Helm override file for the Istio chart. This excludes several components from the mesh, such as the Verrazzano application operator. Second, certain Pods, such as Coherence Pods, are labeled at runtime with sidecar.istio.io/inject="false" to exclude them from the mesh.

Components in the mesh

The following Verrazzano components are in the mesh and use mTLS for all service to service communication.

Elasticsearch
Fluentd
Grafana
Kibana
Keycloak
MySQL
NGINX Ingress Controller
Prometheus
Verrazzano Authentication Proxy
Verrazzano Console
WebLogic Kubernetes Operator

Some of these components, have mesh-related details that are worth noting, as described in the following sections.

NGINX

The NGINX Ingress Controller listens for HTTPS traffic, and provides ingress into the cluster. NGINX is configured to do TLS termination of client connections. All traffic from NGINX to the mesh services use mTLS, which means that traffic is fully encrypted from the client to the target back-end services.

Keycloak and MySQL

Keycloak and MySQL are also in the mesh and use mTLS for network traffic. Because all of the components that use Keycloak are in the mesh, there is end to end mTLS security for all identity management handled by Keycloak. The following components access Keycloak:

Verrazzano Authentication Proxy
Verrazzano Console
Elasticsearch
Prometheus
Grafana
Kibana

Prometheus

Although Prometheus is in the mesh, it is configured to use the Envoy sidecar and mTLS only when communicating with Keycloak. All the traffic related to scraping metrics, bypasses the sidecar proxy, doesn’t use the service IP address, but rather connects to the scrape target using the Pod IP address. If the scrape target is in the mesh, then HTTPS is used; otherwise, HTTP is used. For Verrazzano multicluster, Prometheus also connects from the admin cluster to the Prometheus server in the managed cluster by using the managed cluster NGINX Ingress, using HTTPS. Prometheus in the managed cluster and never establishes connections to targets outside the cluster.

Because Prometheus is in the mesh, additional configuration is done to allow the Envoy sidecar to be bypassed when scraping Pods. This is done with the Prometheus Pod annotation traffic.sidecar.istio.io/includeOutboundIPRanges: <keycloak-service-ip>. This causes traffic bound for Keycloak to go through the Envoy sidecar, and all other traffic to bypass the sidecar.

WebLogic Kubernetes Operator

Applications in the mesh

Before you create a Verrazzano application, you should decide if it should be in the mesh. You control sidecar injection, for example, mesh inclusion, by labeling the application namespace with istio-injection=enabled or istio-injection=disabled. By default, applications will not be put in the mesh if that label is missing. If your application uses a Verrazzano project, then Verrazzano will label the namespaces in the project to enable injection. If the application is in the mesh, then mTLS will be used. You can change the PeerAuthentication mTLS mode as desired if you don’t want strict mTLS. Also, if you need to add mTLS port exceptions, you can do this with DestinationRules or by creating another PeerAuthentication resource in the application namespace. Consult the Istio documentation for more information.

WebLogic

When the WebLogic operator creates a domain, it needs to communicate with the Pods in the domain. Verrazzano puts the WebLogic operator in the mesh so that it can communicate with the domain Pods using mTLS. Because of that, the WebLogic domain must be created in the mesh. Also, because mTLS is used, do not configure WebLogic to use TLS. If you want to use a custom certificate for your application, you can specify that in the ApplicationConfiguration, but that TLS connection will be terminated at the Istio ingress gateway, which you configure using a Verrazzano IngressTrait.

Coherence

Coherence clusters are represented by the Coherence resource, and are not in the mesh. When Verrazzano creates a Coherence cluster in a namespace that is annotated to do sidecar injection, it disables injection of the Coherence resource using the sidecar.istio.io/inject="false" label shown previously. Furthermore, Verrazzano will create a DestinationRule in the application namespace to disable mTLS for the Coherence extend port 9000. This allows a service in the mesh to call the Coherence extend proxy. For an example, see Bobs Books.

Here is an example of a DestinationRule created for the Bob’s Books application which includes a Coherence cluster.

API Version:  networking.istio.io/v1beta1
Kind:         DestinationRule
...
Spec:
  Host:  *.bobs-books.svc.cluster.local
  Traffic Policy:
    Port Level Settings:
      Port:
        Number:  9000
      Tls:
    Tls:
      Mode:  ISTIO_MUTUAL

Istio access control

Istio lets you control access to your workload in the mesh, using the AuthorizationPolicy resource. This lets you control which services or Pods can access your workloads. Some of these options require mTLS; for more information, see Authorization Policy.

Verrazzano always creates AuthorizationPolicies for applications, but never for system components. During application deployment, Verrazzano creates the policy in the application namespace and configures it to allow access from the following:

Other Pods in the application
Istio ingress gateway
Prometheus scraper

This prevents other Pods in the cluster from gaining network access to the application Pods.
Istio uses a service identity to determine the identity of the request’s origin; for Kubernetes this identity is a service account. Verrazzano creates a per-application AuthorizationPolicy as follows:

AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
...
spec:
  rules:
    - from:
    - source:
  principals:
    - cluster.local/ns/sales/sa/greeter
    - cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
    - cluster.local/ns/verrazzano-system/sa/verrazzano-monitoring-operator

WebLogic domain access

For WebLogic applications, the WebLogic operator must have access to the domain Pods for two reasons. First, it must access the domain servers to get health status; second it must inject configuration into the Monitoring Exporter sidecar running in the domain server Pods. When a WebLogic domain is created, Verrazzano adds an additional source, cluster.local/ns/verrazzano-system/sa/weblogic-operator-sa to the principals section to permit that access.

Docs: Customize Load Balancers on OKE

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano sets up the following load balancers on Kubernetes at installation:

Load balancer for NGINX ingress
Load balancer for Istio ingress

Verrazzano allows customizing the load balancers allocated by Oracle Container Engine (OKE) using annotations defined by OKE. For a detailed description of different load balancer customization annotations, see the OKE documentation here.

This document describes how to use these annotations to customize the following settings for Verrazzano load balancers:

Load balancer shape
Private IP address and subnet placement

Customize the load balancer shape

At installation, Verrazzano lets you customize the shape and size of the load balancers created. OCI offers a flexible load balancer which uses Dynamic Shape:

10 Mbps
100 Mbps
400 Mbps
8,000 Mbps

For more details on service limits and shape, see here.

For example, you can set up an NGINX load balancer with 10Mbps as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
        value: "10Mbps"

For example, you can set up an Istio load balancer with 10Mbps as follows:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      istioInstallArgs:
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-shape"
        value: "10Mbps"

Use private IP addresses with a load balancer

At installation, Verrazzano lets you customize the IP address and subnet of the load balancers created. This is achieved using OKE annotations on the NGINX and Istio load balancer services, as documented here.

The following example configures the NGINX load balancer service to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for Istio:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"    
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures the Istio ingress gateway service to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for NGINX:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer      
    istio:
      istioInstallArgs:
        - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
          value: "true"
        - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
          value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures both NGINX and Istio to have a private load balancer IP address on the private subnet identified by OCID ocid1.subnet.oc1.phx.aaaa..sdjxa:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  environmentName: default
  components:
    ingress:
      type: LoadBalancer
      nginxInstallArgs:
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"
      - name: controller.service.annotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"
    istio:
      istioInstallArgs:
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-internal"
        value: "true"
      - name: gateways.istio-ingressgateway.serviceAnnotations."service\.beta\.kubernetes\.io/oci-load-balancer-subnet1"
        value: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

Docs: Coherence Workload

Mon, 01 Jan 0001 00:00:00 +0000

A Verrazzano application can contain any number of Coherence component workloads, where each workload is a standalone Coherence cluster, independent from other Coherence clusters in the application.

Verrazzano uses the standard Coherence operator to provision and manage clusters, as documented at Coherence Operator. The Coherence operator uses a CRD, coherence.oracle.com (Coherence resource), to represent a Coherence cluster. When a Verrazzano application with Coherence is provisioned, Verrazzano configures the default logging and metrics for the Coherence cluster. Logs are sent to Elasticsearch and metrics to Prometheus.
You can view this telemetry data using the Kibana and Grafana consoles.

OAM Component

The custom resource YAML file for the Coherence cluster is specified as a VerrazzanoCoherenceWorkload custom resource. In the following example, everything under the spec: section is standard Coherence resource YAML that you would typically use to provision a Coherence cluster. Including this Component reference in your ApplicationConfiguration will result in a new Coherence cluster being provisioned. You can have multiple clusters in the same application with no conflict.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: orders
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: orders-coh
        spec:
          cluster: SockShop
          ...

Life cycle

With Verrazzano, you manage the life cycle of applications using Component and ApplicationConfiguration resources. Typically, you would modify the Coherence cluster resource to make changes or to do lifecycle operations, like scale in and scale out. However, in the Verrazzano environment, the cluster resource is owned by the Verrazzano application operator and will be reconciled to match the Component workload resource. Therefore, you need to manage the cluster configuration by modifying the resource, either by kubectl edit or applying a new YAML file. Verrazzano will notice that the Component resource changed and will update the Coherence resource as needed.

Provisioning

When you apply the Component YAML file shown previously, Kubernetes will create a component.oam.verrazzano.io resource, but the Coherence cluster will not be created until you create the ApplicationConfiguration resource, which references the Coherence component. When the application is created, Verrazzano creates a Coherence custom resource for each cluster, which is subsequently processed by the Coherence operator, resulting in a new cluster. After a cluster is created, the Coherence operator will monitor the Coherence resource to reconcile the state of the cluster. You can add a new Coherence workload to a running application, or remove an existing workload, by modifying the ApplicationConfiguration resource, and adding or removing the Coherence component.

Scaling

Scaling a Coherence cluster is done by modifying the replicas field in the Component resource. Verrazzano will modify the Coherence resource replicas field and the cluster will be scaled accordingly. The following example configuration shows the replicas field that specifies the number of pods in the cluster.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: orders
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: orders-coh
        spec:
          cluster: SockShop
          replicas: 3
          ...

NOTE: A Coherence cluster provisioned with Verrazzano does not support autoscaling with a Horizontal Pod Autoscaler.

Termination

You can terminate the Coherence cluster by removing the Component from the ApplicationConfiguration or by deleting the ApplicationConfiguration resource entirely.

NOTE

Do not delete the Coherence component if the application is still using it.

Logging

When a Coherence cluster is provisioned, Verrazzano configures it to send logs to Elasticsearch. This is done by injecting Fluentd sidecar configuration into the Coherence resource. The Coherence operator will create the pod with the Fluentd sidecar. This sidecar periodically copies the Coherence logs from /logs to stdout, enabling the Fluentd DaemonSet in the verrazzano-system namespace to send the logs to Elasticsearch. Note that the Fluend sidecar running in the Coherence pod never communicates with Elasticsearch or any other network endpoint.

The logs are placed in a per-namespace Elasticsearch index named verrazzano-namespace-<namespace>, for example: verrazzano-namespace-sockshop. All logs from Coherence pods in the same namespace will go into the same index, even for different applications. This is standard behavior and there is no way to disable or change it.

Each log record has some Coherence and application fields, along with the log message itself. For example:

 kubernetes.labels.coherenceCluster        SockShop
 kubernetes.labels.app_oam_dev/name        sockshop-appconf
 kubernetes.labels.app_oam_dev/component   orders
 ...

Metrics

Verrazzano uses Prometheus to scrape metrics from Coherence cluster pods. Like logging, metrics scraping is also enabled during provisioning, however, the Coherence resource YAML file must have proper metrics configuration. For details, see Coherence Metrics. In summary, there are two ways to configure the Coherence metrics endpoint. Coherence has a default metrics endpoint that you can enable. If your application serves metrics from its own endpoint, such as a Helidon application, then do not use the native Coherence metrics endpoint. To see the difference, examine the socks-shop and bobs-books examples.

Bobs Books

The bobs-books example uses the default Coherence metrics endpoint, so the configuration must enable this feature, shown in the following metrics section of the roberts-coherence component in the YAML file, bobs-books-comp.yaml.

          coherence:
            metrics:
              enabled: true

Sock Shop

The sock-shop example, which is a Helidon application with embedded Coherence, explicitly specifies the metrics port 7001 and doesn’t enable Coherence metrics. Coherence metrics still will be scraped, but not at the default endpoint.

          ports:
            ...
            - name: metrics
              port: 7001
              serviceMonitor:
                enabled: true

Because sock-shop components are not using the default Coherence metrics port, you must add a MetricsTrait section to the ApplicationConfiguration for each component, specifying the metrics port as follows:

        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            metadata:
              name: carts-metrics
            spec:
              port: 7001

Prometheus configuration

Prometheus is configured to scrape targets using the ConfigMaps in the verrazzano-system namespace. During application deployments, Verrazzano updates the vmi-system-prometheus-config ConfigMap and adds targets for the application pods. Verrazzano also annotates those pods to match the expected annotations in the ConfigMap. When the application is deleted, Verrazzano removes the targets from the ConfigMap. You do not need to manually modify the ConfigMap or annotate the application pods.

Here is an example of thesock-shop Prometheus ConfigMap section for catalog. Notice that pods in the sock-shop namespace with labels app_oam_dev_name and app_oam_dev_component are targeted. Prometheus will find those pods and then look at the pod annotations, verrazzano_io/metricsEnabled, verrazzano_io/metricsPath, and verrazzano_io/metricsPort for scrape configuration.

- job_name: sockshop-appconf_default_sockshop_catalog
  ...
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - sockshop
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_verrazzano_io_metricsEnabled,
      __meta_kubernetes_pod_label_app_oam_dev_name, __meta_kubernetes_pod_label_app_oam_dev_component]
  ...  
  - source_labels: [__meta_kubernetes_pod_annotation_verrazzano_io_metricsPath]
  ...
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_verrazzano_io_metricsPort]

Here is the corresponding catalog pod labels and annotations.

kind: Pod
metadata:
  labels:
    ...
    app.oam.dev/component: catalog
    app.oam.dev/name: sockshop-appconf
  annotations:
    ...
    verrazzano.io/metricsEnabled: "true"
    verrazzano.io/metricsPath: /metrics
    verrazzano.io/metricsPort: "7001"

Istio integration

Verrazzano ensures that Coherence clusters are not included in an Istio mesh, even if the namespace has the istio-injection: enabled label. This is done by adding the sidecar.istio.io/inject: "false" annotation to the Coherence resource, resulting in Coherence pods being created with that label. However, other application components in the mesh using mutual TLS authentication (mTLS) may need to communicate with Coherence. To handle this case, Verrazzano automatically creates an Istio DestinationRule to disable TLS for the Coherence port. This policy disables mTLS for port 9000, which happens to be used as a Coherence extend port for Bob’s Books.

  trafficPolicy:
    portLevelSettings:
    - port:
        number: 9000
      tls: {}
   ...

Currently, port 9000 is the only port where TLS is disabled, so you need to use this as the Coherence extend port if other components in the mesh access Coherence over the extend protocol.

Summary

Verrazzano makes it easy to deploy and observe Coherence clusters in your application, providing seamless integration with other components in your application running in an Istio mesh.

Docs: Helidon Workload

Mon, 01 Jan 0001 00:00:00 +0000

Helidon is a collection of Java libraries for writing microservices. Helidon provides an open source, lightweight, fast, reactive, cloud native framework for developing Java microservices. It is available as two frameworks:

Helidon SE is a compact toolkit that embraces the latest Java SE features: reactive streams, asynchronous and functional programming, and fluent-style APIs.
Helidon MP implements and supports Eclipse MicroProfile, a baseline platform definition that leverages Java EE and Jakarta EE technologies for microservices and delivers application portability across multiple runtimes.

Helidon is designed and built with container-first philosophy.

Small footprint, low memory usage and faster startup times.
All 3rd party dependencies are stored separately to enable Docker layering.
Provides readiness, liveness and customizable health information for container schedulers like Kubernetes.

Containerized Helidon applications are generally deployed as Deployment in Kubernetes.

Verrazzano integration

Verrazzano supports application definition using Open Application Model (OAM). Verrrazzano applications are composed of components and application configurations.

Helidon applications are first class citizen in Verrazzano with specialized Helidon workload support, for example, VerrazzanoHelidonWorkload. VerrazzanoHelidonWorkload is supported as part of verrazzano-application-operator in the Verrazzano installation and no additional operator setup or installation is required. VerrazzanoHelidonWorkload also supports all the traits and scopes defined by Verrazzano along with core ones defined by the OAM specification.

VerrazzanoHelidonWorkload is modeled after ContainerizedWorkload, for example, it is used for long-running workloads in containers. However, VerrazzanoHelidonWorkload closely resembles and directly refers to Kubernetes Deployment schema. This enables an easy lift and shift of existing containerized Helidon applications.

The complete VerrazzanoHelidonWorkload API definition and description is available at VerrazzanoHelidonWorkload.

Verrazzano Helidon application development

With Verrazzano, you manage the life cycle of applications using Component and ApplicationConfiguration resources. A Verrazzano application can contain any number of VerrazzanoHelidonWorkload components, where each workload is a standalone containerized Helidon application, independent of any other in the application.

In the following example, everything under the spec: section is the custom resource YAML file for the containerized Helidon application, as defined by VerrazzanoHelidonWorkload custom resource. Including this Component reference in your ApplicationConfiguration will result in a new containerized Helidon application being provisioned.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              ...
              ...

The Application Development Guide provides end-to-end instructions for developing and deploying the Verrazzano Helidon application.

For more Verrazzano Helidon application examples, see Examples.

Provisioning

When you apply the previous Component YAML file, Kubernetes will create a component.oam.verrazzano.io resource, but the containerized Helidon application will not be created until you create the ApplicationConfiguration resource, which references the VerrazzanoHelidonWorkload component. When the application is created, Verrazzano creates a Deployment and Service resource for each containerized Helidon application.

Typically, you would modify the Deployment and Service resource to make changes or to do lifecycle operations, like scale in and scale out. However, in the Verrazzano environment, the containerized Helidon application resource is owned by the verrazzano-application-operator and will be reconciled to match the component workload resource. Therefore, you need to manage the application configuration by modifying the VerrazzanoHelidonWorkload or ApplicationConfiguration resource, either by kubectl edit or applying new YAML file. Verrazzano will notice that the Component resource change and will update the Deployment and Service resource as needed.

You can add a new VerrazzanoHelidonWorkload to a running application, or remove an existing workload, by modifying the ApplicationConfiguration resource and adding or removing the VerrazzanoHelidonWorkload component.

Scaling

The recommended way to scale containerized Helidon application replicas is to specify ManualScalerTrait with VerrazzanoHelidonWorkload in ApplicationConfiguration. The following example configuration shows the replicaCount field that specifies the number of replicas for the application.

...
    spec:
      components:
      - componentName: hello-helidon-component
        traits:
        - trait:                      
            apiVersion: core.oam.dev/v1alpha2
            kind: ManualScalerTrait
            spec:
              replicaCount: 2
...

Verrazzano will modify the Deployment resource replicas field and the containerized Helidon application replicas will be scaled accordingly.

NOTE

Make sure the replicas defined on the VerrazzanoHelidonWorkload component and that replicaCount defined on ManualScalerTrait for that component matches, or else the DeploymentController in Kubernetes and OAM runtime in verrazzano-application-operator will compete to create a different number of Pods for same containerized Helidon application. To avoid confusion, we recommend that you specify replicaCount defined on ManualScalerTrait and leave replicas undefined on VerrazzanoHelidonWorkload (as it is optional).

Logging

When a containerized Helidon application is provisioned on Verrazzano, Verrazzano will configure the default logging and send logs to Elasticsearch. Logs can be viewed using the Kibana console.

The logs are placed in a per-namespace Elasticsearch index named verrazzano-namespace-<namespace>, for example: verrazzano-namespace-hello-helidon. All logs from containerized Helidon application pods in the same namespace will go into the same index, even for different applications. This is standard behavior and there is no way to disable or change it.

Metrics

Verrazzano uses Prometheus to scrape metrics from containerized Helidon application pods. Like logging, metrics scraping is also enabled during provisioning. Metrics can be viewed using the Grafana console.

Verrazzano lets you to customize configuration information needed to enable metrics using MetricsTrait for an application component.

Ingress

Verrazzano lets you to configure traffic routing to a containerized Helidon application, using IngressTrait for an application component.

Troubleshooting

Whenever you have a problem with your Verrazzano Helidon application, there are some basic techniques you can use to troubleshoot. Troubleshooting shows you some simple things to try when troubleshooting, as well as how to solve common problems you may encounter.

Docs: Sock Shop

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Install Verrazzano by following the installation instructions.

NOTE: The Sock Shop example application deployment files are contained in the Verrazzano project located at <VERRAZZANO_HOME>/examples/sockshop, where <VERRAZZANO_HOME> is the root of the Verrazzano project.

Deploy the Sock Shop application

This example application provides various implementations of the Sock Shop Microservices Demo Application. It uses OAM resources to define the application deployment.

Coherence and Helidon in the helidon subdirectory.
Coherence and Micronaut in the micronaut subdirectory.
Coherence and Spring in the spring subdirectory.

Create a namespace for the Sock Shop application and add a label identifying the namespace as managed by Verrazzano.
```
$ kubectl create namespace sockshop
$ kubectl label namespace sockshop verrazzano-managed=true
```

To deploy the application, apply the Sock Shop OAM resources. Choose to deploy either the helidon, micronaut, or spring variant.

$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/helidon/sock-shop-comp.yaml
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/helidon/sock-shop-app.yaml

$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/micronaut/sock-shop-comp.yaml
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/micronaut/sock-shop-app.yaml

$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/spring/sock-shop-comp.yaml
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/spring/sock-shop-app.yaml

Wait for the Sock Shop application to be ready.

$ kubectl wait \
   --for=condition=Ready pods \
   --all -n sockshop \
   --timeout=300s

Explore the application

The Sock Shop microservices application implements REST API endpoints including:

/catalogue - Returns the Sock Shop catalog. This endpoint accepts the GET HTTP request method.
/register - POST { "username":"xxx", "password":"***", "email":"foo@example.com", "firstName":"foo", "lastName":"bar" } to create a user. This endpoint accepts the POST HTTP request method.

NOTE: The following instructions assume that you are using a Kubernetes environment, such as OKE. Other environments or deployments may require alternative mechanisms for retrieving addresses, ports, and such.

Follow these steps to test the endpoints:

Get the generated host name for the application.

$ HOST=$(kubectl get gateways.networking.istio.io \
     -n sockshop \
     -o jsonpath={.items[0].spec.servers[0].hosts[0]})
$ echo $HOST

# Sample output
sockshop-appconf.sockshop.11.22.33.44.nip.io

Get the EXTERNAL_IP address of the istio-ingressgateway service.

$ ADDRESS=$(kubectl get service \
     -n istio-system istio-ingressgateway \
     -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $ADDRESS

# Sample output
11.22.33.44

Access the Sock Shop application:

Using the command line

# Get catalogue
$ curl -sk \
   -X GET \
   https://${HOST}/catalogue \
   --resolve ${HOST}:443:${ADDRESS}

# Sample output
[{"count":115,"description":"For all those leg lovers out there....", ...}]

# Add a new user (replace values of username and password)
$ curl -i \
   --header "Content-Type: application/json" \
   --request POST \
   --data '{"username":"foo","password":"****","email":"foo@example.com","firstName":"foo","lastName":"foo"}' \
   -k https://${HOST}/register \
   --resolve ${HOST}:443:${ADDRESS}

# Add an item to the user's cart
$ curl -i \
   --header "Content-Type: application/json" \
   --request POST \
   --data '{"itemId": "a0a4f044-b040-410d-8ead-4de0446aec7e","unitPrice": "7.99"}' \
   -k https://${HOST}/carts/{username}/items \
   --resolve ${HOST}:443:${ADDRESS}

# Get cart items
$ curl -i \
   -k https://${HOST}/carts/{username}/items \
   --resolve ${HOST}:443:${ADDRESS}

# Sample output
[{"itemId":"a0a4f044-b040-410d-8ead-4de0446aec7e","quantity":1,"unitPrice":7.99}]

If you are using nip.io, then you do not need to include --resolve.

Local testing with a browser

Temporarily, modify the /etc/hosts file (on Mac or Linux) or c:\Windows\System32\Drivers\etc\hosts file (on Windows 10), to add an entry mapping the host name to the ingress gateway’s EXTERNAL-IP address. For example:
```
11.22.33.44 sockshop.example.com
```
Then, you can access the application in a browser at https://sockshop.example.com/catalogue.

If you are using nip.io, then you can access the application in a browser using the HOST variable (for example, https://${HOST}/catalogue). If you are going through a proxy, you may need to add *.nip.io to the NO_PROXY list.
Using your own DNS name
- Point your own DNS name to the ingress gateway’s EXTERNAL-IP address.
- In this case, you would need to edit the sock-shop-app.yaml file to use the appropriate value under the hosts section (such as yourhost.your.domain), before deploying the Sock Shop application.
- Then, you can use a browser to access the application at https://<yourhost.your.domain>/catalogue.

A variety of endpoints associated with the deployed application, are available to further explore the logs, metrics, and such. You can access them according to the directions here.

Troubleshooting

Verify that the application configuration, component, workload, and ingress trait all exist.

$ kubectl get ApplicationConfiguration -n sockshop
$ kubectl get Component -n sockshop
$ kubectl get VerrazzanoCoherenceWorkload -n sockshop
$ kubectl get IngressTrait -n sockshop

Verify that the Sock Shop service pods are successfully created and transition to the READY state. Note that this may take a few minutes and that you may see some of the services terminate and restart.

 $ kubectl get pods -n sockshop

 # Sample output
 NAME             READY   STATUS        RESTARTS   AGE
 carts-coh-0      1/1     Running       0          41s
 catalog-coh-0    1/1     Running       0          40s
 orders-coh-0     1/1     Running       0          39s
 payment-coh-0    1/1     Running       0          37s
 shipping-coh-0   1/1     Running       0          36s
 users-coh-0      1/1     Running       0          35s

Undeploy the application

To undeploy the application, delete the Sock Shop OAM resources. Choose to undeploy either the helidon, micronaut, or spring variant.

$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/helidon/sock-shop-comp.yaml
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/helidon/sock-shop-app.yaml

$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/micronaut/sock-shop-comp.yaml
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/micronaut/sock-shop-app.yaml

$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/spring/sock-shop-comp.yaml
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.1.2/examples/sock-shop/spring/sock-shop-app.yaml

Delete the namespace sockshop after the application pods are terminated.
```
$ kubectl delete namespace sockshop
```

Docs: Image Pull Back Off

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods that had issues due to failures to pull an image or images.

The analysis was not able to identify a specific root cause, however, it might have supplied data that is related to the failures.

Steps

Review the analysis data. At a minimum, it will indicate which pods are being impacted and might give other clues on the root cause.
If the service is experiencing an outage, then consult the specific service status page. For common service status pages, see Related information.

Docs: Image Pull Not Found

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images where the root cause was that the image was not found.

Steps

Review the analysis data; it enumerates the pods and related messages regarding which images had this issue.
Confirm that the image name, digest, and tag are correctly specified.

Kubernetes Troubleshooting

Docs: Image Pull Rate Limit

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images.

The root cause was rate limit exceeded errors while pulling images.

Steps

Review the analysis data; it enumerates the pods and related messages regarding which images had this issue.
The detailed messages might provide specific instructions for the registry that is involved. For example, it might provide a link to instructions on how to increase the limit.

Increase Rate Limits

Docs: Image Pull Service Issue

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images where the root cause was that the service was not available.

The service might be unreachable or might be incorrect.

Steps

Review the analysis data; it enumerates the pods and related messages about which images had this issue.
Confirm that the registry for the image is correct.
The messages might identify a connectivity issue.
If the service is experiencing an outage, then consult the specific service status page. For common service status pages, see Related information.

Docs: Ingress Controller Load Balancer Service Limit Reached

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that the load balancer service limit has been reached.

Steps

Review the messages from the supporting details for the exact limits, and delete unused load balancers.
If available, use a different load balancer shape. See Customizing Ingress.
Refer to the OCI documentation on Service Limits.

Docs: Ingress Controller No Load Balancer IP

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that the load balancer is either missing or unable to set the ingress IP address on the NGINX Ingress service.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Ingress Controller OCI IP Limit Exceeded

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that an OCI IP non-ephemeral address limit has been reached.

Steps

Review the messages from the supporting details for the exact limit.
Refer to the OCI documentation related to managing IP Addresses.

Public IP Addresses

Docs: Install Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation has failed, however, it did not isolate the exact reason for the failure.

Steps

Review the analysis data, which can help identify the issue.

Docs: Install Ingress Controller Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation has failed related to the NGINX Ingress Controller, however, it was unable to isolate the specific root cause.

Steps

Review the analysis data, which might help identify the issue.

Docs: Install Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Before you begin, read this document, Verrazzano in a multicluster environment.

Overview

To set up a multicluster Verrazzano environment, you will need two or more Kubernetes clusters. One of these clusters will the admin cluster; the others will be managed clusters.

The instructions here assume an admin cluster and a single managed cluster. For each additional managed cluster, simply repeat the managed cluster instructions.

Install Verrazzano

Install Verrazzano on each Kubernetes cluster.

On one cluster, install Verrazzano using the dev or prod profile; this will be the admin cluster.
On the other cluster, install Verrazzano using the managed-cluster profile; this will be a managed cluster. The managed-cluster profile contains only the components that are required for a managed cluster.

Create the environment variables, KUBECONFIG_ADMIN, KUBECONTEXT_ADMIN, KUBECONFIG_MANAGED1, and KUBECONTEXT_MANAGED1, and point them to the kubeconfig files and contexts for the admin and managed cluster, respectively. You will use these environment variables in subsequent steps when registering the managed cluster. The following shows an example of how to set these environment variables.

$ export KUBECONFIG_ADMIN=/path/to/your/adminclusterkubeconfig
$ export KUBECONFIG_MANAGED1=/path/to/your/managedclusterkubeconfig

# lists the contexts in each kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN config get-contexts -o=name
my-admin-cluster-context
some-other-cluster-context

$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 config get-contexts -o=name
my-managed-cluster-context
some-other-cluster2-context

# Choose the right context name for your admin and managed clusters from the output shown and set the KUBECONTEXT
# environment variables
$ export KUBECONTEXT_ADMIN=<admin-cluster-context-name>
$ export KUBECONTEXT_MANAGED1=<managed-cluster-context-name>

For detailed instructions on how to install and customize Verrazzano on a Kubernetes cluster using a specific profile, see the Installation Guide and Installation Profiles.

Register the managed cluster with the admin cluster

The following sections show you how to register the managed cluster with the admin cluster. As indicated, some of these steps are performed on the admin cluster and some on the managed cluster. The commands provided use the environment variables set previously to connect to the appropriate cluster.

Preregistration setup

Before registering the managed cluster, first you’ll need to set up the following items:

A Secret containing the managed cluster’s CA certificate. Note that the cacrt field in this secret can be empty only if the managed cluster uses a well-known CA. This CA certificate is used by the admin cluster to scrape metrics from the managed cluster, for both applications and Verrazzano components.
A ConfigMap containing the externally reachable address of the admin cluster. This will be provided to the managed cluster during registration so that it can connect to the admin cluster.

Follow these preregistration setup steps:

If needed for the admin cluster, obtain the managed cluster’s CA certificate. The admin cluster scrapes metrics from the managed cluster’s Prometheus endpoint. If the managed cluster Verrazzano installation uses self-signed certificates or LetsEncrypt staging certificates, then the admin cluster will need the managed cluster’s CA certificate to make an https connection.
- Depending on whether the Verrazzano installation on the managed cluster uses self-signed certificates, LetsEncrypt staging certificates, or certificates signed by a well-known certificate authority, choose the appropriate instructions.
- If you are unsure what type of certificates are used, use the following instructions.
  - To check if the verrazzano resource is configured to use LetsEncrypt staging certificates:
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     describe verrazzano
```
    If the output contains the following information, then LetsEncrypt staging certificates are being used.
```
Cert Manager:
  Certificate:
    Acme:
      Environment:    staging
      Provider:       letsEncrypt
```
  - To check the ca.crt field of the system-tls secret in the verrazzano-system namespace on the managed cluster:
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
     -n verrazzano-system get secret system-tls -o jsonpath='{.data.ca\.crt}'
```
    If this value is empty, then your managed cluster is using certificates signed by a well-known certificate authority. Otherwise, your managed cluster is using self-signed certificates.
    In this case, no additional configuration is necessary.
    If the managed cluster certificates are self-signed, create a file called managed1.yaml containing the CA certificate of the managed cluster as the value of the cacrt field. In the following commands, the managed cluster’s CA certificate is saved in an environment variable called MGD_CA_CERT. Then use the --dry-run option of the kubectl command to generate the managed1.yaml file.
    
    # On the managed cluster $ MGD_CA_CERT=$(kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \ get secret system-tls \ -n verrazzano-system \ -o jsonpath="{.data.ca\.crt}" | base64 --decode) $ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \ create secret generic "ca-secret-managed1" \ -n verrazzano-mc \ --from-literal=cacrt="$MGD_CA_CERT" \ --dry-run=client \ -o yaml > managed1.yaml
    Create a Secret on the admin cluster that contains the CA certificate for the managed cluster. This secret will be used for scraping metrics from the managed cluster. The managed1.yaml file that was created in the previous step provides input to this step.
    
    # On the admin cluster $ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \ apply -f managed1.yaml # Once the command succeeds, you may delete the managed1.yaml file $ rm managed1.yaml
    If the managed cluster certificates are LetsEncrypt staging, then create a file called managed1.yaml containing the CA certificate of the managed cluster as the value of the cacrt field. In the following commands, the managed cluster’s CA certificate is saved in an environment variable called MGD_CA_CERT. Then use the --dry-run option of the kubectl command to generate the managed1.yaml file.
    
    # On the admin cluster $ MGD_CA_CERT=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \ get secret tls-ca-additional \ -n cattle-system \ -o jsonpath="{.data.ca-additional\.pem}" | base64 --decode) $ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \ create secret generic "ca-secret-managed1" \ -n verrazzano-mc \ --from-literal=cacrt="$MGD_CA_CERT" \ --dry-run=client \ -o yaml > managed1.yaml
    Create a Secret on the admin cluster that contains the CA certificate for the managed cluster. This secret will be used for scraping metrics from the managed cluster. The managed1.yaml file that was created in the previous step provides input to this step.
    
    # On the admin cluster $ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \ apply -f managed1.yaml # After the command succeeds, you may delete the managed1.yaml file $ rm managed1.yaml
Use the following instructions to obtain the Kubernetes API server address for the admin cluster. This address must be accessible from the managed cluster.
For most types of Kubernetes clusters, except for Kind clusters, you can find the externally accessible API server address of the admin cluster from its kubeconfig file.
```
# View the information for the admin cluster in your kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN config view --minify

# Sample output
apiVersion: v1
kind: Config
clusters:
- cluster:
  certificate-authority-data: DATA+OMITTED
  server: https://11.22.33.44:6443
  name: my-admin-cluster
contexts:
....
....
```
In the output of this command, you can find the URL of the admin cluster API server from the server entry. Set the value of the ADMIN_K8S_SERVER_ADDRESS variable to this URL.
```
export ADMIN_K8S_SERVER_ADDRESS=<the server address from the config output>
```
Kind clusters run within a Docker container. If your admin and managed clusters are Kind clusters, the API server address of the admin cluster in its kubeconfig file is usually a local address on the host machine, which will not be accessible from the managed cluster. Use the kind command to obtain the “internal” kubeconfig of the admin cluster, which will contain a server address accessible from other Kind clusters on the same machine, and therefore in the same Docker network.
```
$ kind get kubeconfig --internal --name <your-admin-cluster-name> | grep server
```
In the output of this command, you can find the URL of the admin cluster API server from the server entry. Set the value of the ADMIN_K8S_SERVER_ADDRESS variable to this URL.
```
export ADMIN_K8S_SERVER_ADDRESS=<the server address from the config output>
```

On the admin cluster, create a ConfigMap that contains the externally accessible admin cluster Kubernetes server address found in the previous step.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: v1
kind: ConfigMap
metadata:
  name: verrazzano-admin-cluster
  namespace: verrazzano-mc
data:
  server: "${ADMIN_K8S_SERVER_ADDRESS}"
EOF

Registration steps

Perform the first three registration steps on the admin cluster, and the last step, on the managed cluster. The cluster against which to run the command is indicated in each code block.

On the admin cluster

To begin the registration process for a managed cluster named managed1, apply the VerrazzanoManagedCluster object on the admin cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Test VerrazzanoManagedCluster object"
  caSecret: ca-secret-managed1
EOF

Wait for the VerrazzanoManagedCluster resource to reach the Ready status. At that point, it will have generated a YAML file that must be applied on the managed cluster to complete the registration process.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    wait --for=condition=Ready \
    vmc managed1 -n verrazzano-mc
```

Export the YAML file created to register the managed cluster.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register.yaml

On the managed cluster

Apply the registration file exported in the previous step, on the managed cluster.

# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register.yaml

# Once the command succeeds, you may delete the register.yaml file
$ rm register.yaml

After this step, the managed cluster will begin connecting to the admin cluster periodically. When the managed cluster connects to the admin cluster, it will update the Status field of the VerrazzanoManagedCluster resource for this managed cluster, with the following information:

The timestamp of the most recent connection made from the managed cluster, in the lastAgentConnectTime status field.
The host address of the Prometheus instance running on the managed cluster, in the prometheusHost status field. This is then used by the admin cluster to scrape metrics from the managed cluster.
The API address of the managed cluster, in the apiUrl status field. This is used by the admin cluster’s authentication proxy to route incoming requests for managed cluster information, to the managed cluster’s authentication proxy.

Verify that managed cluster registration completed

You can perform all the verification steps on the admin cluster.

Verify that the managed cluster can connect to the admin cluster. View the status of the VerrazzanoManagedCluster resource on the admin cluster, and check whether the lastAgentConnectTime, prometheusUrl, and apiUrl fields are populated. This may take up to two minutes after completing the registration steps.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get vmc managed1 -n verrazzano-mc -o yaml

# Sample output showing the status field
spec:
  ....
  ....
status:
  apiUrl: https://verrazzano.default.172.18.0.211.nip.io
  conditions:
  - lastTransitionTime: "2021-07-07T15:49:43Z"
    message: Ready
    status: "True"
    type: Ready
  lastAgentConnectTime: "2021-07-16T14:47:25Z"
  prometheusHost: prometheus.vmi.system.default.172.18.0.211.nip.io

Verify that the managed cluster is successfully registered with Rancher. When you perform the registration steps, Verrazzano also registers the managed cluster with Rancher. View the Rancher UI on the admin cluster. If the registration with Rancher was successful, then your cluster will be listed in Rancher’s list of clusters, and will be in Active state. You can find the Rancher UI URL for your cluster by following the instructions for Accessing Verrazzano.

Verify that managed cluster metrics are being collected

Verify that the admin cluster is collecting metrics from the managed cluster. The Prometheus output will include records that contain the name of the Verrazzano cluster (labeled as verrazzano_cluster).

You can find the Prometheus UI URL for your cluster by following the instructions for Accessing Verrazzano. Execute a query for a metric (for example, node_disk_io_time_seconds_total).

Sample output of a Prometheus query

An alternative approach to using the Prometheus UI is to query metrics from the command line. Here is an example of how to obtain Prometheus metrics from the command line. Search the output of the query for responses that have the verrazzano_cluster field set to the name of the managed cluster.

# On the admin cluster
$ prometheusUrl=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.prometheusUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} "${prometheusUrl}/api/v1/query?query=node_disk_io_time_seconds_total"

Verify that managed cluster logs are being collected

Verify that the admin cluster is collecting logs from the managed cluster. The output will include records which have the name of the managed cluster in the cluster_name field.

You can find the Kibana UI URL for your cluster by following the instructions for Accessing Verrazzano. Create an index for verrazzano-namespace-verrazzano-system. Some log records will have the cluster_name field populated with the name of the managed cluster.

Sample output of a Kibana screen

An alternative approach to using the Kibana UI is to query Elasticsearch from the command line. Here is an example of how to obtain log records from the command line. Search the output of the query for responses that have the cluster_name field set to the name of the managed cluster.

# On the admin cluster
$ KIBANA_URL=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.kibanaUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} -X POST -H 'kbn-xsrf: true' "${KIBANA_URL}/elasticsearch/verrazzano-namespace-verrazzano-system/_search?size=25"

Run applications in multicluster Verrazzano

The Verrazzano multicluster setup is now complete and you can deploy applications by following the Multicluster Hello World Helidon example application.

Use the admin cluster UI

The admin cluster serves as a central point from which to register and deploy applications to managed clusters.

In the Verrazzano UI on the admin cluster, you can view the following:

The managed clusters registered with this admin cluster.
VerrazzanoProjects located on this admin cluster, or any of its registered managed clusters, or both.
Applications located on this admin cluster, or any of its registered managed clusters, or both.

Docs: Insufficient Memory

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were nodes reporting insufficient memory.

Steps

Review the analysis data to identify the specific nodes involved.
Review the nodes to determine why they do not have sufficient memory.

a. Are the nodes sized correctly for the workload?
1. For the minimum resources required for installing Verrazzano, see the Installation Guide.
2. Refer to documentation for other applications that you are deploying for resource guidelines and take those into account.
b. Is something unexpected running on the nodes or consuming more memory than expected?

Docs: Keycloak and SSO

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano can be deployed to a number of different hosted and on-premises Kubernetes environments. Particularly in hosted environments, it may not be possible to choose the authentication providers configured for the Kubernetes API server, and Verrazzano may have no ability to view, manage, or authenticate users.

Verrazzano installs Keycloak to provide a common user store across all Kubernetes environments. The Verrazzano admin user can create and manage user accounts in Keycloak, and Verrazzano can authenticate and authorize Keycloak users.

Also, you can configure Keycloak to delegate authentication to an external user store, such as Active Directory or an LDAP server.

Because Keycloak is not configured as an authentication provider for the Kubernetes API, authenticating Keycloak users to Kubernetes requires the use of a proxy that impersonates Keycloak users when making Kubernetes API requests. For more information about the Verrazzano authentication proxy, see Verrazzano Proxies.

Keycloak is also used when authenticating to the Verrazzano Console and the various Verrazzano Monitoring Instance (VMI) logging and metrics consoles. The Verrazzano Console uses the OpenID Connect (OIDC) PKCE flow to authenticate users against Keycloak and obtain ID and access tokens. Authentication for VMI consoles is provided by the Verrazzano authentication proxy, which also uses PKCE to authenticate users, validates the resulting tokens, and authorizes incoming requests. For more information about the Verrazzano authentication proxy, see Verrazzano Proxies.

Docs: OCI Container Engine for Kubernetes (OKE)

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the OCI install

Create the OKE cluster using the OCI Console or by some other means.
For SHAPE, an OKE cluster with 3 nodes of VM.Standard2.4 OCI compute instance shape has proven sufficient to install Verrazzano and deploy the Bob’s Books example application.
Follow the instructions provided by OKE to download the Kubernetes configuration file for your cluster, and set the following ENV variable:

   $ export KUBECONFIG=<path to valid Kubernetes config>

Optional, if your organization requires the use of a private registry to the Docker images installed by Verrazzano, see Use a Private Registry.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For an example on OKE, see Installing Calico and Setting Up Network Policies.

Next steps

To continue, see the Installation Guide.

Docs: Pending Pods

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which were in a pending state without detecting other specific issues related to them.

Steps

Review the analysis data. At a minimum, this should indicate which pods are being impacted and it might give other clues on the root cause.

Kubernetes Troubleshooting

Docs: Problem Pods

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which were not in a running, succeeded, or pending state.

The analysis was not able to determine a specific root cause, however, it might have supplied data that is related to the pods in question. The root cause might be obvious from the supporting data, but the analysis tool isn’t isolating the specific scenario yet.

Steps

Review the analysis data. At a minimum, it should indicate which pods are being impacted and it might give other clues on the root cause.

Kubernetes Troubleshooting

Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: Application Deployment Guide

Overview

What you need

Application development

Application deployment

Verrazzano components

Verrazzano application configurations

Deploy the application

Verify the deployment

Explore the application

Access the application’s logs

Access the application’s metrics

Remove the application

Docs: Application Deployment

Review oam-kubernetes-runtime operator status

Review verrazzano-application-operator operator status

Review oam-kubernetes-runtime operator logs

Review verrazzano-application-operator logs

Review generated workload resources

Review generated Trait resources

Check for RBAC privilege issues

Check resource owners

Check related resources

Docs: Customize DNS

How Verrazzano constructs a DNS domain

Prerequisites

Create an OCI API secret in the target cluster

Use a Verrazzano helper script to create an OCI secret

Installation

Docs: Elasticsearch Scaling and Resizing

Scaling Elasticsearch data nodes

Address the master nodes’ StatefulSet

Update the Verrazzano Custom Resource

Docs: Install Guide

Prerequisites

Prepare for the install

Install the Verrazzano platform operator

Perform the install

NOTE

Install Verrazzano

Verify the install

(Optional) Run the example applications

To get the consoles URLs and credentials, see Access Verrazzano.

Docs: Multicluster Verrazzano

Verify managed cluster registration and connectivity

Verify VerrazzanoProject placement

Check the multicluster resource status

Docs: Prerequisites

Supported Hardware

Supported Software Versions

Kubernetes

WebLogic Server

Coherence

Helidon

Installed Components

Docs: Verrazzano Analysis Tools

Tools Setup

Linux Instructions

Download the tooling:

Verify the downloaded files:

Unpack the verrazzano-analysis binary:

Mac Instructions

Download the tooling:

Verify the downloaded files:

Unpack the verrazzano-analysis binary:

Use the k8s-dump-cluster.sh tool

Use the verrazzano-analysis tool

Usage information

Docs: Verrazzano and the Open Application Model

How does OAM work?

Docs: Verrazzano in a Multicluster Environment

Docs: Verrazzano Projects

Docs: Customize Certificates

NOTE

Use the Verrazzano self-signed CA

Use a custom CA

Use LetsEncrypt certificates

NOTE

NOTE

Review `oam-kubernetes-runtime` operator status

Review `verrazzano-application-operator` operator status

Review `oam-kubernetes-runtime` operator logs

Review `verrazzano-application-operator` logs

Unpack the `verrazzano-analysis` binary:

Unpack the `verrazzano-analysis` binary:

Use the `k8s-dump-cluster.sh` tool

Use the `verrazzano-analysis` tool

Create a database using MySQL called `tododb`