Verrazzano Enterprise Container Platform – Prepare an Oracle Cloud Native Environment Cluster

Docs: Configure a VCN for OCNE

Mon, 01 Jan 0001 00:00:00 +0000

Before you can create Oracle Cloud Native Environment (OCNE) clusters on Oracle Cloud Infrastructure (OCI), you’ll need to configure a virtual cloud network (VCN) in your OCI compartment. VCNs are software-defined networks that manage access to your cloud resources.

See Networking Overview in the OCI documentation for more information.

You can use the VCN Wizard in the OCI Console to automatically create most of the required network infrastructure. Additional subnets and security rules (described below) must be added manually.

Within your VCN, you’ll need:

Subnets (with security rules)
Gateways
Route tables

NOTE

In addition to the specifications listed, make sure that the VCN is configured to accept the ports and protocols required by Kubernetes. See Ports and Protocols in the Kubernetes documentation for more information.

Subnets

Subnets are subdivisions within a VCN that help to organize configuration settings. All instances within a subnet use the same route table, security lists, and DHCP options. Subnets can be either public or private. For an OCNE cluster, you’ll need both public and private subnets, with four subnets in total.

See Overview of VCNs and Subnets in the OCI documentation for more information.

Each subnet requires its own set of security rules that establish rules for virtual firewalls. These ingress and egress rules specify the types of traffic (protocol and port) that are allowed in and out of the instances.

See Security Rules in the OCI documentation for more information.

NOTE

You can use either Network Security Groups (NSGs) or security lists to add security rules to your VCN. We recommend using NSGs whenever possible. For more information, see Comparison of Security Lists and Network Security Groups in the OCI documentation.

Subnet 1: control plane endpoint

A public subnet for the control plane endpoint that houses an OCI load balancer. The load balancer acts as a reverse proxy for the Kubernetes API server.

In this subnet, create security rules that cover the following traffic:

Egress: control plane traffic
Ingress: external access to the Kubernetes API endpoint
Ingress: ICMP path discovery

Security rules examples

NOTE

These examples are provided for reference only. Customize your security rules as needed for your environment.

Egress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	10.0.0.0/29	6443	TCP	HTTPS traffic to control plane for Kubernetes API server access

Ingress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	0.0.0.0/0	6443	TCP	Public access to endpoint OCI load balancer
CIDR Block	10.0.0.0/16		ICMP Type 3, Code 4	Path MTU discovery

Subnet 2: control plane nodes

A private subnet that houses the control plane nodes that run Kubernetes control plane components, such as the API Server and the control plane pods.

In this subnet, create security rules that cover the following traffic:

Egress: node internet access
Ingress: east-west traffic, originating from within the VCN
Ingress: control plane endpoint to control plane node access on API endpoint
Ingress: worker nodes to control plane node access on API endpoint
Ingress: ETCD client and peer
Ingress: SSH traffic
Ingress: control plane to control plane kubelet communication
Ingress:
Ingress: Calico rules for control plane and worker nodes for
- BGP
- IP-in-IP

Security rules examples

NOTE

These examples are provided for reference only. Customize your security rules as needed for your environment.

Egress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	0.0.0.0/0	All	All	Control plane node access to the internet to pull images

Ingress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	10.0.0.8/29	6443	TCP	Kubernetes API endpoint to Kubernetes control plane communication
CIDR Block	10.0.0.0/29	6443	TCP	Control plane to control plane (API server port) communication
CIDR Block	10.0.64.0/20	6443	TCP	Worker node to Kubernetes control plane (API Server) communication
CIDR Block	10.0.0.0/29	10250	TCP	Control plane to control plane node kubelet communication
CIDR Block	10.0.0.0/29	2379	TCP	etcd client communication
CIDR Block	10.0.0.0/29	2380	TCP	etcd peer communication
CIDR Block	10.0.0.0/29	179	TCP	Calico networking (BGP)
CIDR Block	10.0.64.0/20	179	TCP	Calico networking (BGP)
CIDR Block	10.0.0.0/29		IP-in-IP	Calico networking with IP-in-IP enabled
CIDR Block	10.0.64.0/20		IP-in-IP	Calico networking with IP-in-IP enabled
CIDR Block	10.0.0.0/16		ICMP Type 3, Code 4	Path MTU discovery
CIDR Block	0.0.0.0/0	22	TCP	Inbound SSH traffic to worker nodes
CIDR Block	10.0.0.0/16	All	TCP	East-West communication for Kubernetes API server access / DNS access

Subnet 3: service load balancers

A public subnet that houses the service load balancers.

In this subnet, create security rules that cover the following traffic:

Egress: service load balancer to NodePort on worker nodes
Ingress: ICMP path discovery
Ingress: HTTP and HTTPS traffic

Security rules examples

NOTE

These examples are provided for reference only. Customize your security rules as needed for your environment.

Egress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	10.0.64.0/20	32000-32767	TCP	Access to NodePort services from service load balancers

Ingress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	0.0.0.0/0	80, 443	TCP	Incoming traffic to services
CIDR Block	10.0.0.0/16		ICMP Type 3, Code 4	Path MTU discovery

Subnet 4: worker nodes

A private subnet that houses the worker nodes.

In this subnet, create security rules that cover the following traffic:

Egress: node internet access
Ingress: east-west traffic, originating from within the VCN
Ingress: SSH traffic
Ingress: ICMP path discovery
Ingress: control plane to kubelet on worker nodes
Ingress: worker node to worker node
Ingress: Calico rules for control plane and worker nodes for
- BGP
- IP-in-IP
Ingress: worker nodes to default NodePort ingress

Security rules examples

NOTE

These examples are provided for reference only. Customize your security rules as needed for your environment.

Egress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	0.0.0.0/0	All	All	Worker node access to the internet to pull images

Ingress rules

Destination Type	Destination	Destination Port	Protocol	Description
CIDR Block	10.0.0.32/27	32000-32767	TCP	Incoming traffic from service load balancers (NodePort communication)
CIDR Block	10.0.0.0/29	10250	TCP	Control plane node to worker node (kubelet communication)
CIDR Block	10.0.64.0/20	10250	TCP	Worker node to worker node (kubelet communication)
CIDR Block	10.0.0.0/29	179	TCP	Calico networking (BGP)
CIDR Block	10.0.64.0/20	179	TCP	Calico networking (BGP)
CIDR Block	10.0.0.0/29		IP-in-IP	Calico networking with IP-in-IP enabled
CIDR Block	10.0.64.0/20		IP-in-IP	Calico networking with IP-in-IP enabled
CIDR Block	10.0.0.0/16		ICMP Type 3, Code 4	Path MTU discovery
CIDR Block	0.0.0.0/0	22	22	Inbound SSH traffic to worker nodes
CIDR Block	10.0.0.0/16	All	TCP	East-West communication for Kubernetes API server access / DNS access

Gateways

Gateways control access from your VCN to other networks. You’ll need to configure three different types of gateways:

You may need to perform some additional configuration to expose the VCN’s subnets directly to the internet. See Access to the Internet in the OCI documentation for details.

Route tables

Route tables send traffic out of the VCN (for example, to the internet, to your on-premises network, or to a peered VCN) using rules that are similar to traditional network route rules.

See VCN Route Tables in the OCI documentation for more information.

For OCNE clusters, you’ll need to create two route tables:

A route table for public subnets that will route stateful traffic to and from the internet gateway. Assign this route table to both public subnets.
A route table for private subnets that will route stateful traffic to and from the NAT and service gateways. Assign this route table to both private subnets.

Docs: Cluster API Provider for Oracle Cloud Native Environment

Mon, 01 Jan 0001 00:00:00 +0000

The Cluster API (CAPI) project seeks to develop and standardize Kubernetes-style APIs specific to cluster management. External organizations then can use these standard APIs to develop cluster management solutions built to their preferred requirements.

Learn more about CAPI at Kubernetes Cluster API Documentation.

NOTE

The terminology around clusters differs between CAPI and Verrazzano though the underlying concepts are the same. What CAPI calls Management and Workload clusters are equivalent to Admin and Managed clusters, respectively, in Verrazzano.

CAPI spreads the various cluster management tasks across three types of providers:

Infrastructure providers standardize the host environment by provisioning any infrastructure or computational resources required by the cluster or machine.
Bootstrap providers streamline the node creation process by converting servers into Kubernetes nodes.
Control plane providers work with the Kubernetes API to regulate your clusters, ensuring that they always strive toward a desired state.

The CAPI provider for Oracle Cloud Native Environment (CAPOCNE) includes both a bootstrap and a control plane provider. When you enable Verrazzano with CAPOCNE on an Oracle Cloud Native Environment, you can use it to rapidly design and deploy clusters and then continue managing your clusters throughout their life cycle.

During the setup process, the bootstrap provider converts a cluster into an admin cluster - a Kubernetes cluster that controls any other, subordinate or ‘managed’ clusters. It generates certificates, starts and manages the creation of additional nodes, and handles the addition of control plane and worker nodes to the cluster.

Next, a CAPI infrastructure provider will provision the first instance on the cloud provider and generate a provider ID, a unique identifier that any future nodes and clusters will use to associate with the instance. It will also create a kubeconfig file. The first control plane node is ready after these are created.

After the admin cluster is up and running, you can use CAPOCNE to create additional managed clusters.

CAPOCNE currently works only with the CAPOCI infrastructure provider offered by Oracle Cloud Infrastructure (OCI).

Docs: Configure NFS Storage

Mon, 01 Jan 0001 00:00:00 +0000

Complete the following steps to configure NFS storage in an Oracle Cloud Native Environment.

Create an OLCNE cluster. See OLCNE cluster.

The cluster must have at least 3 worker nodes.
Create an NFS server. For an example that uses an NFS server on Oracle Linux, see Create an NFS server on Oracle Linux.

a. Install the NFS utility package on the server and client instances.
```
$ sudo dnf install -y nfs-utils
```
b. Create a directory for your shared files. Make sure that the server does not have root ownership.

c. Define the shared directory in /etc/exports with the correct permissions. Make sure to disable root squashing.
```
$ <path to directory> <ip-address/subnet-mask>(rw,sync,no_root_squash,no_subtree_check)
```
d. Set the firewall to allow NFS traffic.
```
$ sudo firewall-cmd --permanent --zone=public --add-service=nfs
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --list-all
```
e. Enable and start the NFS service.
```
$ sudo systemctl enable --now nfs-server
```
Deploy an NFS provisioner to your cluster.

a. Install an NFS client provisioner of your choice. For an example, see Kubernetes NFS Subdir External Provisioner.

b. Add the required Helm repo.
```
$ helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
```
c. Install the provisioner. Set your storage class as a default and create a service account.
```
$ helm install nfs-test \
   --set nfs.server=<server ip address> \
   --set nfs.path=<path> \
   --set storageClass.name=<name> \
   --set storageClass.defaultClass=true,rbac.create=true \
   --set storageClass.provisionerName=nfsclientprov/nfs \
   --set serviceAccount.create=true \
   --set serviceAccount.name=nfs-svc-acc-nfs nfs-subdir-external-provisioner/nfs-subdir-external-provisioner
```
d. Only one storage class should be listed as the default. If required, edit the other storage classes and delete the following annotation:
```
$ storageclass.kubernetes.io/is-default-class: "true"
```

Docs: Configure Access to a Private Registry

Mon, 01 Jan 0001 00:00:00 +0000

A private Docker registry is called an insecure registry when it is configured for access using a self-signed certificate or over an unencrypted HTTP connection. For example, for the Oracle Cloud Native Environment platform, insecure registries must be configured in /etc/containers/registries.conf as follows on the worker nodes:

 [registries]
    [registries.insecure]
      registries = ["insecure-registry-1:1001/registry1","insecure-registry-2:1001/registry2"]