Verrazzano Enterprise Container Platform – Monitoring & Logging

Docs: Logging

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano logging stack consists of Fluentd, Elasticsearch, and Kibana components.

Fluentd: a log aggregator that collects, processes, and formats logs from Kubernetes clusters.
Elasticsearch: a scalable search and analytics engine for storing Kubernetes logs.
Kibana: a visualization layer that provides a user interface to query and visualize collected logs.

As shown in the following diagram, logs written to stdout by a container running on Kubernetes are picked up by the kubelet service running on that node and written to /var/log/containers.

Fluentd sidecar

For components with multiple log streams or that cannot log to stdout, Verrazzano deploys a Fluentd sidecar which parses and translates the log stream. The resulting log is sent to stdout of the sidecar container and then written to /var/log/containers by the kubelet service.

For example, in a WebLogic deployment, AdminServer.log is consumed, translated, and written to stdout by the Fluentd sidecar. You can view these logs using kubectl on the container named fluentd-stdout-sidecar.

$ kubectl logs tododomain-adminserver \
   -n todo-list \
   -c fluentd-stdout-sidecar

The Verrazzano Fluentd Docker image comes with these plug-ins:

The Verrazzano Fluentd Docker image also has two local default plug-ins, kubernetes_parser and kubernetes_multiline_parser. These plug-ins help to parse Kubernetes management log files.

Here are example use cases for these plug-ins:

# ---- fluentd.conf ----
# kubernetes parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>

# kubernetes multi-line parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>
# ----   EOF      ----

For more details, see the Fluentd plugins folder.

Fluentd DaemonSet

Verrazzano deploys a Fluentd DaemonSet which runs one Fluentd replica per node in the verrazzano-system namespace. Each instance pulls logs from the node’s /var/log/containers directory and writes them to the target Elasticsearch index. The index name is based on the namespace associated with the record, using this format: verrazzano-namespace-<record namespace>.

For example, vmi-system-kibana logs written to /var/log/containers will be pulled by Fluentd and written to Elasticsearch. The index used is named verrazzano-namespace-verrazzano-system because the VMI runs in the verrazzano-system namespace.

The same approach is used for both system and application logs.

Elasticsearch

Verrazzano creates an Elasticsearch deployment as the store and search engine for the logs processed by Fluentd. Records written by Fluentd can be queried using the Elasticsearch REST API.

For example, you can use curl to get all of the Elasticsearch indexes. First, you must get the password for the verrazzano user and the host for the VMI Elasticsearch.

$ PASS=$(kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo)
$ HOST=$(kubectl get ingress \
    -n verrazzano-system vmi-system-es-ingest \
    -o jsonpath={.spec.rules[0].host})

$ curl -ik \
   --user verrazzano:$PASS https://$HOST//_cat/indices

To see all of the records for a specific index, do the following:

$ INDEX=verrazzano-namespace-todo-list

$ curl -ik \
    --user verrazzano:$PASS https://$HOST/$INDEX/_doc/_search?q=message:*

Verrazzano provides support for Installation Profiles. The production profile (prod), which is the default, provides a 3-node Elasticsearch and persistent storage for the Verrazzano Monitoring Instance (VMI). The development profile (dev) provides a single node Elasticsearch and no persistent storage for the VMI. The managed-cluster profile does not install Elasticsearch or Kibana in the local cluster; all logs are forwarded to the admin cluster’s Elasticsearch instance.

If you want the logs sent to an external Elasticsearch, instead of the default VMI Elasticsearch, specify elasticsearchURL and elasticsearchSecret in the Fluentd Component configuration in your Verrazzano custom resource.

The following is an example of a Verrazzano custom resource to send the logs to the Elasticsearch endpoint https://external-es.default.172.18.0.231.nip.io.

apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: default
spec:
  components:
    fluentd:
      elasticsearchURL: https://external-es.default.172.18.0.231.nip.io
      elasticsearchSecret: external-es-secret

Kibana

Kibana is a visualization dashboard for the content indexed on an Elasticsearch cluster. Verrazzano creates a Kibana deployment to provide a user interface for querying and visualizing the log data collected in Elasticsearch.

To access the Kibana console, read Access Verrazzano.

To see the records of an Elasticsearch index through Kibana, create an index pattern to filter for records under the desired index.

For example, to see the log records of a WebLogic application deployed to the todo-list namespace, create an index pattern of verrazzano-namespace-todo-list.

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano metrics stack automates metrics aggregation and consists of Prometheus and Grafana components. Metrics sources expose system and application metrics. The Prometheus components retrieve and store the metrics and Grafana provides dashboards to visualize them.

Metrics sources

Metrics sources produce metrics and expose them to the Kubernetes Prometheus system using annotations in the pods. The metrics annotations may differ slightly depending on the resource type. The following is an example of the WebLogic Prometheus-related configuration specified in the todo-list application pod:

$ kubectl describe pod tododomain-adminserver -n todo-list

Annotations:  prometheus.io/path: /wls-exporter/metrics
              prometheus.io/port: 7001
              prometheus.io/scrape: true

For other resource types, such as Coherence or Helidon, the annotations would look similar to this:

Annotations:  verrazzano.io/metricsEnabled: true
              verrazzano.io/metricsPath: /metrics
              verrazzano.io/metricsPort: 8080

To look directly at the metrics that are being made available by the metric source, map the port and then access the path.

For example, for the previous metric source:

Map the port being used to expose the metrics.

$ kubectl port-forward tododomain-adminserver 7001:7001 -n todo-list

Get the user name and password used to access the metrics source from the corresponding secret.

$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.username} | base64 \
    --decode; echo
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

Access the metrics at the exported path, using the user name and password retrieved in the previous step.
```
$ curl -u USERNAME:PASSWORD localhost:7001/wls-exporter/metrics
```

Metrics server

Single pod per cluster.
Named vmi-system-prometheus-* in verrazzano-system namespace.
Discovers exposed metrics source endpoints.
Scrapes metrics from metrics sources.
Responsible for exposing all metrics.

Grafana

Grafana provides visualization for your Prometheus metric data.

Single pod per cluster.
Named vmi-system-grafana-* in verrazzano-system namespace.
Provides dashboards for metrics visualization.

To access Grafana:

Get the hostname from the Grafana ingress.

$ kubectl get ingress vmi-system-grafana -n verrazzano-system

# Sample output
NAME                 CLASS    HOSTS                                              ADDRESS          PORTS     AGE
vmi-system-grafana   <none>   grafana.vmi.system.default.123.456.789.10.nip.io   123.456.789.10   80, 443   26h

Get the password for the user verrazzano.

$ kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

Access Grafana in a browser using the previous hostname.
Log in using the verrazzano user and the previous password.

From here, you can select an existing dashboard or create a new dashboard. To select an existing dashboard, use the drop-down list in the top left corner. The initial value of this list is Home.

To view host level metrics, select Host Metrics. This will provide system metrics for all of the nodes in your cluster.

To view the application metrics for the todo-list example application, select WebLogic Server Dashboard because the todo-list application is a WebLogic application.