Verrazzano Enterprise Container Platform – Platform Setup

Docs: Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE)

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the Oracle Cloud Infrastructure install

Create the OKE cluster using the Oracle Cloud Infrastructure Console or by some other means.
For SHAPE, an OKE cluster with 3 nodes of VM.Standard2.4 Oracle Cloud Infrastructure compute instance shape has proven sufficient to install Verrazzano and deploy the Bob’s Books example application.
Follow the instructions provided by OKE to download the Kubernetes configuration file for your cluster, and set the following ENV variable:

   $ export KUBECONFIG=<path to valid Kubernetes config>

Optional, if your organization requires the use of a private registry to the Docker images installed by Verrazzano, see Use a Private Registry.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For an example on OKE, see Installing Calico and Setting Up Network Policies.

Next steps

To continue, see the Installation Guide.

Docs: Oracle Cloud Native Environment

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the Oracle Cloud Native Environment installation

Oracle Cloud Native Environment can be installed in several different types of environments. These range from physical, on-premises hardware to virtualized cloud infrastructure. The Oracle Cloud Native Environment installation instructions assume that networking and compute resources already exist. The basic infrastructure requirements are a network with a public and private subnet and a set of hosts connected to those networks.

Oracle Cloud Infrastructure example

The following is an example of Oracle Cloud Infrastructure that can be used to evaluate Verrazzano installed on Oracle Cloud Native Environment. If other environments are used, the capacity and configuration should be similar.

You can use the VCN Wizard of the Oracle Cloud Infrastructure Console to automatically create most of the described network infrastructure. Additional security lists/rules, as detailed below, need to be added manually. All CIDR values provided are examples and can be customized as required.

Virtual Cloud Network (for example, CIDR 10.0.0.0/16)

Public Subnet (for example, CIDR 10.0.0.0/24)

Security List / Ingress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type & Code	Description
No	`0.0.0.0/0`	ICMP			3, 4	ICMP errors
No	`10.0.0.0/16`	ICMP			3	ICMP errors
No	`0.0.0.0/0`	TCP	All	22		SSH
No	`0.0.0.0/0`	TCP	All	80		HTTP load balancer
No	`0.0.0.0/0`	TCP	All	443		HTTPS load balancer

Security List / Egress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Description
No	`10.0.1.0/24`	TCP	All	22	SSH
No	`10.0.1.0/24`	TCP	All	30080	HTTP load balancer
No	`10.0.1.0/24`	TCP	All	30443	HTTPS load balancer
No	`10.0.1.0/24`	TCP	All	31380	HTTP load balancer
No	`10.0.1.0/24`	TCP	All	31390	HTTPS load balancer

Private Subnet (for example, CIDR 10.0.1.0/24)

Security List / Ingress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type & Code	Description
No	`0.0.0.0/0`	ICMP			3, 4	ICMP errors
No	`10.0.0.0/16`	ICMP			3	ICMP errors
No	`10.0.0.0/16`	TCP	All	22		SSH
No	`10.0.0.0/24`	TCP	All	30080		HTTP load balancer
No	`10.0.0.0/24`	TCP	All	30443		HTTPS load balancer
No	`10.0.0.0/24`	TCP	All	31380		HTTP load balancer
No	`10.0.0.0/24`	TCP	All	31390		HTTPS load balancer
No	`10.0.1.0/24`	TCP	All	2379-2380		Kubernetes etcd
No	`10.0.1.0/24`	TCP	All	6443		Kubernetes API Server
No	`10.0.1.0/24`	TCP	All	6446		MySQL
No	`10.0.1.0/24`	TCP	All	8090-8091		Oracle Cloud Native Environment Platform Agent
No	`10.0.1.0/24`	UDP	All	8472		Flannel
No	`10.0.1.0/24`	TCP	All	10250-10255		Kubernetes Kublet

Security List / Egress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type and Code	Description
No	`10.0.0.0/0`	TCP				All egress traffic

DHCP Options

DNS Type
Internet and VCN Resolver

Route Tables

Public Subnet Route Table Rules

Destination	Target
`0.0.0.0/0`	Internet Gateway

Private Subnet Route Table Rules

Destination	Target
`0.0.0.0/0`	NAT Gateway
All Oracle Cloud Infrastructure Services	Service Gateway

Internet Gateway

NAT Gateway

Service Gateway

The following compute resources adhere to the guidelines provided in Oracle Cloud Native Environment: Getting Started. The attributes indicated (for example, Subnet, RAM, Shape, and Image) are recommendations that have been tested. Other values can be used if required.

Compute Instances

Role	Subnet	Suggested RAM	Compatible VM Shape	Compatible VM Image
SSH Jump Host	Public	8GB	VM.Standard.E2.1	Oracle Linux 7.8
Oracle Cloud Native Environment Operator Host	Private	16GB	VM.Standard.E2.2	Oracle Linux 7.8
Kubernetes Control Plane Node	Private	32GB	VM.Standard.E2.4	Oracle Linux 7.8
Kubernetes Worker Node 1	Private	32GB	VM.Standard.E2.4	Oracle Linux 7.8
Kubernetes Worker Node 2	Private	32GB	VM.Standard.E2.4	Oracle Linux 7.8
Kubernetes Worker Node 3	Private	32GB	VM.Standard.E2.4	Oracle Linux 7.8

Install Oracle Cloud Native Environment

Deploy Oracle Cloud Native Environment 1.4 with the Kubernetes module, following instructions from Oracle Cloud Native Environment: Getting Started.

Use a single Kubernetes control plane node.
Skip the Kubernetes API load balancer (3.4.3).
Use private CA certificates (3.5.3).
Install a Container Storage Interface Driver, such as OCI-CSI or Gluster.

Notes

The oci-csi module does not elect a default StorageClass or configure policies for the CSIDrivers that it installs. A reasonable choice is the oci-bv StorageClass with its CSIDriver configured with the File group policy.

kubectl patch sc oci-bv -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: blockvolume.csi.oraclecloud.com
spec:
  fsGroupPolicy: File
EOF

Prepare for the Verrazzano installation

A Verrazzano Oracle Cloud Native Environment deployment requires:

Load balancers in front of the worker nodes in the cluster.
DNS records that reference the load balancers.

NOTE: The target ports for the load balancer backends cannot be determined until you install Verrazzano.
You can create the load balancers before you install, but post-installation configuration is required.

Examples for meeting these requirements follow.

Load Balancers

Verrazzano on Oracle Cloud Native Environment uses external load balancer services. These will not automatically be provided by Verrazzano or Kubernetes. Two load balancers must be deployed outside of the subnet used for the Kubernetes cluster. One load balancer is for management traffic and the other for application traffic.

Specific steps will differ for each load balancer provider, but a generic configuration and an Oracle Cloud Infrastructure example follow.

Generic configuration:

Target Host: Host names of Kubernetes worker nodes
Target Ports: See table
External Ports: See table
Distribution: Round-robin
Health Check: TCP

Backend for management load balancer

You must install Verrazzano to get the target ports for each load balancer backend. In the following table, those ports are marked TBD. Run the following command to get the target ports for the NGINX Ingress Controller:

$ kubectl get service ingress-controller-ingress-nginx-controller -n ingress-nginx

In the PORT(S) column you will see the target port associated with port 80 and 443, for example: 80:30080/TCP,443:30443.
Use these target port values for the NGINX Ingress Controller load balancer backend.

Service Name	Type	External Port	Target Port
`ingress-controller-nginx-ingress-controller`	TCP	80	TBD
`ingress-controller-nginx-ingress-controller`	TCP	443	TBD

Backend for application load balancer

Get the target ports for the Istio ingress gateway service using the following command:

$ kubectl get service  istio-ingressgateway  -n  istio-system

Use these port values for the Istio ingress gateway load balancer backend.

Service Name	Type	External Port	Target Port
`istio-ingressgateway`	TCP	80	TBD
`istio-ingressgateway`	TCP	443	TBD

Oracle Cloud Infrastructure example

The following details can be used to create Oracle Cloud Infrastructure load balancers for accessing application and management user interfaces, respectively. These load balancers will route HTTP/HTTPS traffic from the Internet to the private subnet. If load balancers are desired, then they should be created now even though the application and management endpoints will be installed later.

NOTE: In the following list, the using port 0 for the health check indicates that the backend ports should be used.

Application Load Balancer: Public Subnet
- Listeners
  - HTTP Listener: Protocol TCP, Port 80
  - HTTPS Listener: Protocol TCP, Port 443
- Backend Sets
  - HTTP Backend Sets:
    - Health Check: Protocol TCP, Port 0
    - Backends: Kubernetes Worker Nodes, Port TBD, Distribution Policy Weighted Round Robin
  - HTTPS Backend Sets
    - Health Check: Protocol TCP, Port 0
    - Backends: Kubernetes Worker Nodes, Port TBD, Distribution Policy Weighted Round Robin
Management Load Balancer: Public Subnet
- Listeners
  - HTTP Listener: Protocol TCP, Port 80
  - HTTPS Listener: Protocol TCP, Port 443
- Backend Sets
  - HTTP Backend Sets:
    - Health Check: Protocol TCP, Port 0
    - Backends: Kubernetes Worker Nodes, Port TBD, Distribution Policy Weighted Round Robin
  - HTTPS Backend Sets
    - Health Check: Protocol TCP, Port 0
    - Backends: Kubernetes Worker Nodes, Port TBD, Distribution Policy Weighted Round Robin

DNS

When using the Verrazzanospec.components.dns.external DNS type, the installer searches the DNS zone you provide for two specific A records. These are used to configure the cluster and should refer to external addresses of the load balancers in the previous step. The A records will need to be created manually.

NOTE: At this time, the only supported deployment for Oracle Cloud Native Environment is the external DNS type.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

Verrazzano installation will result in a number of management services that need to point to the ingress-mgmt address.

verrazzano.myenv.example.com                    CNAME   ingress-mgmt.myenv.example.com.
keycloak.myenv.example.com                      CNAME   ingress-mgmt.myenv.example.com.
rancher.myenv.example.com                       CNAME   ingress-mgmt.myenv.example.com.

grafana.vmi.system.myenv.example.com            CNAME   ingress-mgmt.myenv.example.com.
prometheus.vmi.system.myenv.example.com         CNAME   ingress-mgmt.myenv.example.com.
kiali.vmi.system.myenv.example.com              CNAME   ingress-mgmt.myenv.example.com.
kibana.vmi.system.myenv.example.com             CNAME   ingress-mgmt.myenv.example.com.
elasticsearch.vmi.system.myenv.example.com      CNAME   ingress-mgmt.myenv.example.com.

For simplicity, an administrator may want to create wildcard DNS records for the management addresses:

*.system.myenv.example.com                      CNAME   ingress-mgmt.myenv.example.com.

*.myenv.example.com                             CNAME   ingress-mgmt.myenv.example.com.

Oracle Cloud Infrastructure example

DNS is configured in Oracle Cloud Infrastructure by creating DNS zones in the Oracle Cloud Infrastructure Console. When creating a DNS zone, use these values:

Method: Manual
Zone Name: <dns-suffix>
Zone Type: Primary

The value for <dns-suffix> excludes the environment (for example, use the example.com portion of myenv.example.com).

DNS A records must be manually added to the zone and published using values described above. DNS CNAME records, in the same way.

During the Verrazzano install, these steps should be performed on the Oracle Cloud Native Environment operator node.

Edit the sample Verrazzano custom resource install-olcne.yaml file and provide these configuration settings for your Oracle Cloud Native Environment:

The value for spec.environmentName is a unique DNS subdomain for the cluster (for example, myenv in myenv.example.com).
The value for spec.components.dns.external.suffix is the remainder of the DNS domain (for example, example.com in myenv.example.com).
Under spec.components.ingress.nginxInstallArgs, the value for controller.service.externalIPs is the IP address of ingress-mgmt.<myenv>.<example.com> configured during DNS set up.
Under spec.components.istio.istioInstallArgs, the value for gateways.istio-ingressgateway.externalIPs is the IP address of ingress-verrazzano.<myenv>.<example.com> configured during DNS set up.

You will install Verrazzano using the external DNS type (the example custom resource for Oracle Cloud Native Environment is already configured to use spec.components.dns.external).

Set the following environment variable:

The value for <path to valid Kubernetes config> is typically ${HOME}/.kube/config.

$ export KUBECONFIG=$VERRAZZANO_KUBECONFIG

Next steps

To continue, see the Installation Guide.

Docs: Kind

Mon, 01 Jan 0001 00:00:00 +0000

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. Follow these instructions to prepare a Kind cluster for running Verrazzano.

NOTE

Kind is not recommended for use on macOS and Windows because the Docker network is not directly exposed to the host.

Prerequisites

Install Docker
Install Kind

Prepare the Kind cluster

To prepare the Kind cluster for use with Verrazzano, you must create the cluster and then install and configure MetalLB in that cluster.

You can create the Kind cluster in two ways: with or without image caching; image caching can speed up your installation time.

Create a Kind cluster

Kind images are prebuilt for each release. To find images suitable for a given release, check the release notes for your Kind version (check with kind version). There you’ll find a complete listing of images created for a Kind release.

The following example references a Kubernetes v1.21.1-based image built for Kind v0.11.1. Replace that image with one suitable for the Kind release you are using.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
EOF

Create a Kind cluster with image caching

While developing or experimenting with Verrazzano, you might destroy and re-create your Kind cluster multiple times. To speed up Verrazzano installation, follow these steps to ensure that the image cache used by containerd inside a Kind cluster, is preserved across clusters. Subsequent installations will be faster because they will not need to pull the images again.

1. Create a named Docker volume that will be used for the image cache, and note its Mountpoint path. In this example, the volume is named containerd.

$ docker volume create containerd

$ docker volume inspect containerd

#Sample output
{
    "CreatedAt": "2021-01-11T16:27:47Z",
    "Driver": "local",
    "Labels": {},
    "Mountpoint": "/var/lib/docker/volumes/containerd/_data",
    "Name": "containerd",
    "Options": {},
    "Scope": "local"
}

2. Specify the Mountpoint path obtained, as the hostPath under extraMounts in your Kind configuration file, with a containerPath of /var/lib/containerd, which is the default containerd image caching location inside the Kind container. An example of the modified Kind configuration is shown in the following create cluster command:

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
    extraMounts:
      - hostPath: /var/lib/docker/volumes/containerd/_data
        containerPath: /var/lib/containerd #This is the location of the image cache inside the Kind container
EOF

Install and configure MetalLB

By default, Kind does not provide an implementation of network load balancers (Services of type LoadBalancer). MetalLB offers a network load balancer implementation.

To install MetalLB:

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
$ kubectl create secret generic \
    -n metallb-system memberlist \
    --from-literal=secretkey="$(openssl rand -base64 128)"
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

For further details, see the MetalLB installation guide.

MetalLB is idle until configured. Configure MetalLB in Layer 2 mode and give it control over a range of IP addresses in the kind Docker network. In versions v0.7.0 and earlier, Kind uses Docker’s default bridge network; in versions v0.8.0 and later, it creates its own bridge network in Kind.

To determine the subnet of the kind Docker network in Kind v0.8.0 and later:

$ docker inspect kind | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.18.0.0/16

To determine the subnet of the kind Docker network in Kind v0.7.0 and earlier:

$ docker inspect bridge | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.17.0.0/16

For use by MetalLB, assign a range of IP addresses at the end of the kind network’s subnet CIDR range.

$ kubectl apply -f - <<-EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: my-ip-space
      protocol: layer2
      addresses:
      - 172.18.0.230-172.18.0.250
EOF

Next steps

To continue, see the Installation Guide.

Docs: Generic Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the generic install

Verrazzano requires that your Kubernetes cluster provides an implementation of network load balancers (Services of type LoadBalancer) for a production environment. If your generic Kubernetes implementation provides this feature, then you can use a default configuration of the Verrazzano custom resource with no customizations and follow the Installation Guide.

NOTE

Remember to not overlap network Classless Inter-Domain Routing (CIDR) blocks when designing and implementing your Kubernetes cluster, proper routing relies on that.

You can install a load balancer, such as MetalLB. This setup requires knowledge of networking both inside and outside your Kubernetes cluster. This would include specifics of your Container Network Interface (CNI) implementation, IP address allocation schemes, and routing that go beyond the scope of this documentation. For a Kind implementation, see Install and configure MetalLB.

It is possible to use a Kubernetes Service of type NodePort to test aspects of Verrazzano. This requires a good working knowledge of networking and has limited use cases.

Customizations

Verrazzano is highly customizable. If your Kubernetes implementation requires a custom configuration, see Customize Installations.

Next steps

To continue, see the Installation Guide.