Verrazzano Enterprise Container Platform – Logging & Monitoring

Docs: Logging

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano logging stack consists of Fluentd, OpenSearch, and OpenSearch Dashboards components.

Fluentd: a log aggregator that collects, processes, and formats logs from Kubernetes clusters.
OpenSearch: a scalable search and analytics engine for storing Kubernetes logs.
OpenSearch Dashboards: a visualization layer that provides a user interface to query and visualize collected logs.

As shown in the following diagram, logs written to stdout by a container running on Kubernetes are picked up by the kubelet service running on that node and written to /var/log/containers.

Fluentd sidecar

For components with multiple log streams or that cannot log to stdout, Verrazzano deploys a Fluentd sidecar which parses and translates the log stream. The resulting log is sent to stdout of the sidecar container and then written to /var/log/containers by the kubelet service.

For example, in a WebLogic deployment, AdminServer.log is consumed, translated, and written to stdout by the Fluentd sidecar. You can view these logs using kubectl on the container named fluentd-stdout-sidecar.

$ kubectl logs tododomain-adminserver \
   -n todo-list \
   -c fluentd-stdout-sidecar

The Verrazzano Fluentd Docker image comes with these plug-ins:

The Verrazzano Fluentd Docker image also has two local default plug-ins, kubernetes_parser and kubernetes_multiline_parser. These plug-ins help to parse Kubernetes management log files.

Here are example use cases for these plug-ins:

# ---- fluentd.conf ----
# kubernetes parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>

# kubernetes multi-line parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>
# ----   EOF      ----

For more details, see the Fluentd plugins folder.

Fluentd DaemonSet

Verrazzano deploys a Fluentd DaemonSet which runs one Fluentd replica per node in the verrazzano-system namespace. Each instance pulls logs from the node’s /var/log/containers directory and writes them to the target OpenSearch data stream. Verrazzano system applications receive special handling, and write their logs to the verrazzano-system data stream. Verrazzano application logs are exported to a data stream based on the application’s namespace, following this format: verrazzano-application-<application namespace>.

For example, vmi-system-kibana logs written to /var/log/containers will be pulled by Fluentd and written to OpenSearch. The logs are exported to the verrazzano-system data stream, because vmi-system-kibana is a Verrazzano system application. For a non-system application, if it is in the myapp namespace, then its logs will be exported to the verrazzano-application-myapp data stream.

OpenSearch

Verrazzano creates an OpenSearch cluster as the store and search engine for the logs processed by Fluentd. Records written by Fluentd can be queried using the OpenSearch REST API.

For example, you can use curl to get all of the OpenSearch data streams. First, you must get the password for the verrazzano user and the host for the VMI OpenSearch.

$ PASS=$(kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo)
$ HOST=$(kubectl get ingress \
    -n verrazzano-system vmi-system-es-ingest \
    -o jsonpath={.spec.rules[0].host})

$ curl -ik \
   --user verrazzano:$PASS https://$HOST/_data_stream

To see all of the records for a specific data stream, do the following:

$ DATA_STREAM=verrazzano-application-todo-list

$ curl -ik \
    --user verrazzano:$PASS https://$HOST/$DATA_STREAM/_search?q=message:*

Verrazzano provides support for Installation Profiles. The production profile (prod), which is the default, provides a 3-node OpenSearch and persistent storage for the Verrazzano Monitoring Instance (VMI). The development profile (dev) provides a single node OpenSearch and no persistent storage for the VMI. The managed-cluster profile does not install OpenSearch or OpenSearch Dashboards in the local cluster; all logs are forwarded to the admin cluster’s OpenSearch instance.

If you want the logs sent to an external OpenSearch, instead of the default VMI OpenSearch, specify elasticsearchURL and elasticsearchSecret in the Fluentd Component configuration in your Verrazzano custom resource.

The following is an example of a Verrazzano custom resource to send the logs to the OpenSearch endpoint https://external-es.default.172.18.0.231.nip.io.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: default
spec:
  components:
    fluentd:
      elasticsearchURL: https://external-es.default.172.18.0.231.nip.io
      elasticsearchSecret: external-es-secret

OpenSearch Dashboards

OpenSearch Dashboards is a visualization dashboard for the content indexed on an OpenSearch cluster. Verrazzano creates a OpenSearch Dashboards deployment to provide a user interface for querying and visualizing the log data collected in OpenSearch.

To access the OpenSearch Dashboards console, read Access Verrazzano.

To see the records of an OpenSearch index or data stream through OpenSearch Dashboards, create an index pattern to filter for records under the desired data stream or index.

For example, to see the log records of a WebLogic application deployed to the todo-list namespace, create an index pattern of verrazzano-application-todo-*.

Log rotation

We recommend configuring log rotation for OpenSearch using Index State Management (ISM) or a periodic job to purge or snapshot old records. For information on configuring OpenSearch ISM, see the ISM setup page.

A basic implementation of job-based log rotation (not using ISM) is provided in the following example, implemented using a Kubernetes CronJob. To install the log rotation example on your cluster, save the snippet to a file and make the following modifications:

Substitue the value of OPENSEARCH_HOST with your specific OpenSearch HTTPS endpoint.
By default, the CronJob deletes the last 7 days of data. You may customize this by modifying the query in the ConfigMap.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: log-rotate
  namespace: verrazzano-system
  labels:
    app: log-rotate
spec:
  # Rotate logs every day at midnight
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: log-rotate
          annotations:
            sidecar.istio.io/inject: "false"
        spec:
          containers:
          - name: log-rotate
            args:
            - /bin/sh
            - -c
            - /opt/script/rotate
            env:
            - name: "OPENSEARCH_HOST"
              value: "https://elasticsearch.vmi.system.default.172.18.0.151.nip.io"
            - name: OPENSEARCH_USER
              valueFrom:
                secretKeyRef:
                  key: username
                  name: verrazzano
                  optional: true
            - name: OPENSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: verrazzano
                  optional: true
            image: ghcr.io/oracle/oraclelinux:7-slim
            imagePullPolicy: IfNotPresent
            volumeMounts:
            - mountPath: /opt/script
              name: log-rotate          
          restartPolicy: OnFailure
          volumes:
          - configMap:
              defaultMode: 0777
              name: log-rotate
            name: log-rotate
---

apiVersion: v1
kind: ConfigMap
metadata:
  name: log-rotate
  namespace: verrazzano-system
  labels:
    app: log-rotate
data:
  rotate: |
    #!/bin/bash
    curl -v --silent -k -u "$OPENSEARCH_USER:$OPENSEARCH_PASSWORD" -X POST "$OPENSEARCH_HOST/verrazzano-*/_delete_by_query" -H 'Content-Type: application/json' -d'
    {
      "query": {
        "bool": {
          "filter": [
            {
              "range": {
                "@timestamp": {
                  "lt": "now-7d"
                }
              }
            }
          ]
        }
      }
    }
    '

Docs: Oracle Cloud Infrastructure Logging Service

Mon, 01 Jan 0001 00:00:00 +0000

The Oracle Cloud Infrastructure Logging service is a highly scalable and fully managed single pane of glass for all the logs in your tenancy. You can configure Verrazzano to send logs to Oracle Cloud Infrastructure Logging instead of OpenSearch. For general information, see Oracle Cloud Infrastructure Logging Overview.

Set up custom logs

Verrazzano can send its logs to Oracle Cloud Infrastructure custom logs. You will need to provide two Oracle Cloud Infrastructure Log identifiers in your Verrazzano installation resource: one for Verrazzano system logs and one for application logs. Follow the steps in Creating Custom Logs to create two custom logs. Do not create an agent configuration when creating a custom log, otherwise the log records will be duplicated.

Configure credentials

The Fluentd plug-in included with Verrazzano will use Oracle Cloud Infrastructure instance principal authentication by default. Optionally, you can configure Verrazzano with a user API signing key. API signing key authentication is required to send logs to Oracle Cloud Infrastructure Logging if the cluster is running outside of Oracle Cloud Infrastructure.

Create a dynamic group that includes the compute instances in your cluster’s node pools and assign the appropriate policy, so that the dynamic group is allowed to send log entries to the custom logs you created earlier. Pay close attention to the required permissions.

If the dynamic group and policy are configured incorrectly, then Fluentd will fail to send logs to Oracle Cloud Infrastructure Logging.

If you do not already have an API signing key, then see Required Keys and OCIDs in the Oracle Cloud Infrastructure documentation. You need to create an Oracle Cloud Infrastructure configuration file with the credential details and then use that configuration file to create a secret.

The following requirements must be met for Fluentd Oracle Cloud Infrastructure Logging to work:

The profile name in the Oracle Cloud Infrastructure configuration file must be DEFAULT.
The key_file path in the Oracle Cloud Infrastructure configuration file must be /root/.oci/key. The actual key file does not need to be in that location, because you will be providing the actual key file location in a secret.
The user associated with the API key must have the appropriate Oracle Cloud Infrastructure Identity and Access Management (IAM) policy in place to allow the Fluentd plug-in to send logs to Oracle Cloud Infrastructure. See Details for Logging in the Oracle Cloud Infrastructure documentation for the IAM policies used by the Oracle Cloud Infrastructure Logging service.

After the Verrazzano platform operator has been installed, create an opaque secret in the verrazzano-install namespace from the Oracle Cloud Infrastructure configuration and private key files. The key for the configuration file must be config and the key for the private key file data must be key.

Here is an example kubectl command that will create the secret.

$ kubectl create secret generic oci-fluentd -n verrazzano-install \
      --from-file=config=/home/myuser/oci_config --from-file=key=/home/myuser/keys/oci_api.pem

The secret should look something like this.

apiVersion: v1
data:
  config: W0RFRkFVTFRdCnVzZXI9b2NpZDEudXN...
  key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS...
kind: Secret
metadata:
  name: oci-fluentd
  namespace: verrazzano-install
type: Opaque

For convenience, there is a helper script available here that you can point at an existing Oracle Cloud Infrastructure configuration file and it will create the secret for you. The script allows you to override the default configuration file location, profile name, and the name of the secret.

Install Verrazzano

Oracle Cloud Infrastructure Logging is enabled in your cluster when installing Verrazzano. The Verrazzano installation custom resource has fields for specifying two custom logs: one for system logs and one for application logs. Here is an example Verrazzano installation YAML file for each type of credential. Note that the API references Kibana, upcoming releases will use OpenSearch Dashboards in the public API.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: vz-oci-logging
spec:
  profile: dev
  components:
    fluentd:
      enabled: true
      oci:
        systemLogId: ocid1.log.oc1.iad.system.example
        defaultAppLogId: ocid1.log.oc1.iad.app.example
    elasticsearch:
      enabled: false
    kibana:
      enabled: false

When using user API credentials, you need to configure the name of the secret in the Verrazzano custom resource, under the Oracle Cloud Infrastructure section of the Fluentd component settings. Your YAML file should look something like this.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: vz-oci-logging
spec:
  profile: dev
  components:
    fluentd:
      enabled: true
      oci:
        systemLogId: ocid1.log.oc1.iad.system.example
        defaultAppLogId: ocid1.log.oc1.iad.app.example
        apiSecret: oci-fluentd
    elasticsearch:
      enabled: false
    kibana:
      enabled: false

The apiSecret value must match the secret you created earlier when configuring the user API credentials.

Override the default log objects

You can override the Oracle Cloud Infrastructure Log object on an individual namespace. To specify a log identifier on a namespace, add an annotation named verrazzano.io/oci-log-id to the namespace. The value of the annotation is the Oracle Cloud Infrastructure Log object identifier.

Here is an example namespace.

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    verrazzano.io/oci-log-id: ocid1.log.oc1.iad.ns.app.example
  creationTimestamp: "2022-01-14T15:09:19Z"
  labels:
    istio-injection: enabled
    verrazzano-managed: "true"
  name: example
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Note that if you add and subsequently remove the annotation, then the logs will revert to the default Oracle Cloud Infrastructure Log object specified in the Verrazzano custom resource.

Search logs

To search Verrazzano logs, you can use the Oracle Cloud Infrastructure Console, Oracle Cloud Infrastructure CLI, or Oracle Cloud Infrastructure SDK.

For example, using the Oracle Cloud Infrastructure CLI to search the system logs for records emitted by the verrazzano-application-operator container.

$ oci logging-search search-logs --search-query=\
     "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \
     where \"data\".\"kubernetes.container_name\" = 'verrazzano-application-operator' | sort by datetime desc" \
     --time-start 2021-12-07 --time-end 2021-12-17

Search for all application log records in the springboot namespace.

$ oci logging-search search-logs --search-query=\
     "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \
     where \"data\".\"kubernetes.namespace_name\" = 'springboot' | sort by datetime desc" \
     --time-start 2021-12-07 --time-end 2021-12-17

For more information on searching logs, see the Logging Query Language Specification.

Troubleshooting

If you are not able to view Verrazzano logs in Oracle Cloud Infrastructure Logging, then check the Fluentd container logs in the cluster to see if there are errors.

$ kubectl logs -n verrazzano-system -l app=fluentd --tail=-1

If you see not authorized error messages, then there is likely a problem with the Oracle Cloud Infrastructure Dynamic Group or IAM policy that is preventing the Fluentd plug-in from communicating with the Oracle Cloud Infrastructure API.

To ensure the appropriate permissions are in place, review the Oracle Cloud Infrastructure Logging required permissions documentation.

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano metrics stack automates metrics aggregation and consists of Prometheus and Grafana components. Metrics sources expose system and application metrics. The Prometheus components retrieve and store the metrics and Grafana provides dashboards to visualize them.

Metrics sources

The following sections describe metrics sources that Verrazzano provides for OAM and standard Kubernetes applications.

OAM

Metrics sources produce metrics and expose them to the Kubernetes Prometheus system using annotations in the pods. The metrics annotations may differ slightly depending on the resource type. The following is an example of the WebLogic Prometheus-related configuration specified in the todo-list application pod:

$ kubectl describe pod tododomain-adminserver -n todo-list

Annotations:  prometheus.io/path: /wls-exporter/metrics
              prometheus.io/port: 7001
              prometheus.io/scrape: true

For other resource types, such as Coherence or Helidon, the annotations would look similar to this:

Annotations:  verrazzano.io/metricsEnabled: true
              verrazzano.io/metricsPath: /metrics
              verrazzano.io/metricsPort: 8080

To look directly at the metrics that are being made available by the metric source, map the port and then access the path.

For example, for the previous metric source:

Map the port being used to expose the metrics.

$ kubectl port-forward tododomain-adminserver 7001:7001 -n todo-list

Get the user name and password used to access the metrics source from the corresponding secret.

$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.username} | base64 \
    --decode; echo
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

Access the metrics at the exported path, using the user name and password retrieved in the previous step.
```
$ curl -u USERNAME:PASSWORD localhost:7001/wls-exporter/metrics
```

Standard Kubernetes workloads

Verrazzano enables metric sources for Kubernetes workloads deployed without OAM components. Verrazzano supports the following workload types: Deployments, ReplicaSets, StatefulSets, and Pods. To enable metrics for Kubernetes workloads, you must label the workload namespace with verrazzano-managed=true.

Metrics Template

A Metrics Template is a custom resource created by Verrazzano to manage metrics configurations for standard Kubernetes workloads. Metrics templates can be placed in the workload namespace or the verrazzano-system namespace. By default, Verrazzano installs a metrics template named standard-k8s-metrics-template in the verrazzano-system namespace. This metrics template handles all the aforementioned workload types. If the default metrics template does not meet your requirements, then you can create your own metrics templates to extend and alter its functionality.

As outlined in the API, the metrics template contains a workloadSelector field that specifies the resources for which the template applies. If you want to forgo the workload selection and manually specify a template, you can add the annotation app.verrazzano.io/metrics=<template-name> to the namespace of the workload or to the workload itself. Additionally, you can opt out of metrics for your namespace or workload by setting the annotation app.verrazzano.io/metrics=none.

The template matching precedence is as follows:

A workload is annotated.

a. A template matching the annotation value is found in the workload namespace.

b. A template matching the annotation value is found in the verrazzano-system namespace.

c. No template is found, an error is recorded, and metrics are not processed for this workload.
A workload namespace is annotated.

a. A template matching the annotation value is found in the workload namespace.

b. A template matching the annotation value is found in the verrazzano-system namespace.

c. No template is found, an error is recorded, and metrics are not processed for this namespace.
No annotation is present.

a. A template in the workload namespace matches the workload through the workloadSelector field.

b. A template in the verrazzano-system namespace matches the workload through the workloadSelector field.

c. No templates match the workload and metrics are not processed for this workload.

If a workload with no annotations matches multiple templates in a namespace, there is no guaranteed precedence in template matching. If this is the case, it is more reliable to specify the template you require by using an annotation.

To verify that the metrics template process was successful, follow these steps:

Access the Prometheus console.
From the console, use the navigation bar to access Status/Targets.
On this page, you will see a target name with this formatting: <workload-namespace>_<workload-name>_<workload-type>.
Copy this job name for use in future queries.
Verify that the State of this target is UP.
Next, use the navigation bar to access the Graph.
Here, use the job name you copied to construct this expression: {job="<job_name>"}
Use the graph to run this expression and verify that you see application metrics appear.

Prometheus overrides

The standard-k8s-metrics-template metrics template installed by Verrazzano uses the following pod annotations to populate the Prometheus configuration. If not specified, Verrazzano will use these default values:

Annotations:  prometheus.io/path: /metrics
              prometheus.io/port: 8080
              prometheus.io/scrape: true

To alter these values, annotate the workload pod with the corresponding annotations. For example, if you want to change the metrics path, then add the following to a Deployment definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-helidon-deployment
  namespace: hello-helidon
  annotations:
    app.verrazzano.io/metrics: standard-k8s-metrics-template
spec:
  template:
    metadata:
      # add path annotation to the pod template
      annotations:
        prometheus.io/path: "/custom/metrics/path"

Prometheus configuration

If you want to create your own metrics template, you will need to construct a Prometheus scrape config. The scrape config uses Go Templates to generate configuration values based on Kubernetes resources. You can reference values in the workload and namespace definitions for use in the scrape config. For example, the default scrape config references the workload namespace field through this reference: .workload.metadata.namespace. Do not include the job_name field in your scrape config as it will be generated by Verrazzano. For guidance on how to construct a Prometheus scrape config, reference the scrapeConfigTemplate section in the Metrics Template example.

Metrics server

Single pod per cluster.
Named vmi-system-prometheus-* in verrazzano-system namespace.
Discovers exposed metrics source endpoints.
Scrapes metrics from metrics sources.
Responsible for exposing all metrics.

Grafana

Grafana provides visualization for your Prometheus metric data.

Single pod per cluster.
Named vmi-system-grafana-* in verrazzano-system namespace.
Provides dashboards for metrics visualization.

To access Grafana:

Get the host name from the Grafana ingress.

$ kubectl get ingress vmi-system-grafana -n verrazzano-system

# Sample output
NAME                 CLASS    HOSTS                                              ADDRESS          PORTS     AGE
vmi-system-grafana   <none>   grafana.vmi.system.default.123.456.789.10.nip.io   123.456.789.10   80, 443   26h

Get the password for the user verrazzano.

$ kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

Access Grafana in a browser using the previous host name.
Log in using the verrazzano user and the previous password.

From here, you can select an existing dashboard or create a new dashboard. To select an existing dashboard, use the drop-down list in the top left corner. The initial value of this list is Home.

To view host level metrics, select Host Metrics. This will provide system metrics for all of the nodes in your cluster.

To view the application metrics for the todo-list example application, select WebLogic Server Dashboard because the todo-list application is a WebLogic application.

Docs: Jaeger Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Jaeger is a distributed tracing system used for monitoring and troubleshooting microservices. For more information on Jaeger, see the Jaeger website.

Install Jaeger Operator

To install the Jaeger Operator, enable the jaegerOperator component in your Verrazzano resource. Here is an example YAML file that enables the Jaeger Operator.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true

Install Jaeger using the Jaeger Operator

Jaeger is installed using the Jaeger Custom Resource Definition. The following example shows you how to install Jaeger inside the Istio mesh using the Verrazzano system OpenSearch cluster as a tracing backend.

Before creating the Jaeger instance, create a secret containing the OpenSearch user name and password. Jaeger will use these credentials to connect to OpenSearch:

$ kubectl create secret generic jaeger-secret \
  --from-literal=ES_PASSWORD=<OPENSEARCH PASSWORD> \
  --from-literal=ES_USERNAME=<OPENSEARCH USERNAME> \
  -n verrazzano-system

Use the following YAML to create the Jaeger resource:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: verrazzano-prod
  namespace: verrazzano-system
spec:
  annotations:
    sidecar.istio.io/inject: "true"
  strategy: production
  storage:
    # Jaeger Elasticsearch storage is compatible with Verrazzano OpenSearch.
    type: elasticsearch
    esIndexCleaner:
      enabled: false
      numberOfDays: 7
      schedule: "* * * * *"
    options:
      es:
        # Enter your OpenSearch cluster endpoint here.
        server-urls: https://elasticsearch.vmi.system.default.172.18.0.151.nip.io
        index-prefix: jaeger
        tls:
          ca: /verrazzano/certificates/ca.crt
    secretName: jaeger-secret
  volumeMounts:
    - name: certificates
      mountPath: /verrazzano/certificates/
      readOnly: true
  volumes:
    - name: certificates
      secret:
        # Jaeger should use the client TLS secret for OpenSearch. This is the default secret name for Verrazzano OpenSearch.
        secretName: system-tls-es-ingest

The Jaeger Operator will create services for query and collection. After applying the example resource, you should see similar output by listing Jaeger resources:

$ kubectl get services,deployments -l app.kubernetes.io/instance=verrazzano-prod -n verrazzano-system

NAME                                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                  AGE
service/verrazzano-prod-collector            ClusterIP   10.96.76.108   <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   52m
service/verrazzano-prod-collector-headless   ClusterIP   None           <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   52m
service/verrazzano-prod-query                ClusterIP   10.96.205.8    <none>        16686/TCP,16685/TCP                      52m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/verrazzano-prod-collector   1/1     1            1           52m
deployment.apps/verrazzano-prod-query       1/1     1            1           52m

Configure an application to export traces to Jaeger

The Jaeger agent sidecar is injected to application pods by the "sidecar.jaegertracing.io/inject": "true" annotation. You may apply this annotation to namespaces or pod controllers such as Deployments. The subsequent snippet shows how to annotate an OAM component for Jaeger agent injection.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-component
spec:
  workload:
    apiVersion: core.oam.dev/v1alpha2
    kind: ContainerizedWorkload
    metadata:
      name: example-workload
      annotations:
        # The component's Deployment will carry the Jaeger annotation.
        "sidecar.jaegertracing.io/inject": "true"

View traces on the Jaeger UI

You can view the UI by port forwarding the Jaeger query service or by configuring an ingress controller for HTTPS access. Explore the Jaeger configuration in more detail using the Jaeger Custom Resource Documentation.

Configure the Istio mesh to use Jaeger tracing

You can view Istio mesh traffic by enabling Istio’s distributed tracing integration. Traces from the Istio mesh provide observability on application traffic that passes through Istio’s ingress and egress gateways.

Istio tracing is disabled by default. To turn on traces, customize your Istio component like the following example:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      istioInstallArgs:
        - name: "meshConfig.enableTracing"
          value: "true"

After enabling tracing, Istio will automatically configure itself with the Jaeger endpoint in your cluster, and any new Istio-injected pods will begin exporting traces to Jaeger. Existing pods require a restart to pull the new Istio configuration and start sending traces.

Istio’s default sampling rate is 1%, meaning 1 in 100 requests will be traced in Jaeger. If you want a different sampling rate, configure your desired rate using the meshConfig.defaultConfig.tracing.sampling Istio installation argument:

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      istioInstallArgs:
        - name: "meshConfig.enableTracing"
          value: "true"
        # 25% of Istio traces will be sampled.  
        - name: "meshConfig.defaultConfig.tracing.sampling"
          value: "25.0"