CNI Providers

CNI Providers for Kubernetes

Networking is a critical component that ensures smooth communication between containers, services, and applications. To make this process simpler and more flexible, Kubernetes clusters support various Container Network Interface (CNI) providers. These providers enable the configuration of networking plugins that help manage networking resources and communication across containers.

EKS clusters, in particular, support three prominent CNI providers: AWS VPC CNI, Cilium, and Calico. Each of these providers has its own unique set of features, allowing users to select the best networking solution for their specific needs.

💡 Tip: Users can add any CNI when creating an addon; however, Cilium, Calico, and AWS VPC are thoroughly qualified and tested to work seamlessly.

Users can choose and customize their preferred CNI provider's configuration through multiple interfaces, including UI, Terraform, RCTL, and API, ensuring seamless integration with their Kubernetes environments.

AWS VPC

AWS uses Amazon VPC Container Network Interface (CNI) plugin for the pod networking in EKS clusters provisioned by the controller. AWS CNI is the default CNI Provider. AWS VPC ensures that CNI plugin daemons are scheduled on all nodes in a cluster and adds elastic network interface (ENI) resource tagging

Calico

Calico is a popular networking solution used to interconnect virtual machines or Linux containers with the help of virtual routers. Calico provides a Cluster Network Interface (CNI) plugin that can be used for integration with Kubernetes. Users who prefer flexible IP address management capabilities can select the Calico CNI provider. On selecting Calico, all the pods related to Calico gets installed

Cilium

Cilium uses eBPF (Extended Berkeley Packet Filter) technology to provide high-performance networking, enhanced security, and deep network visibility. It is especially beneficial for users looking for advanced load balancing, security policies, and observability in cloud-native environments.

---

CNI Providers in Controller

During EKS Cluster provisioning, user must select a CNI Provider from the CNI Providers drop-down available in Advance Settings. If none of the CNI is selected, AWS is set as the default CNI provider

---

Why Is This Useful?

The introduction of multiple CNI providers allows users to choose a networking solution that best fits their specific requirements, whether it’s for performance, security, or flexibility. For instance, users who prioritize high security and advanced network observability might prefer Cilium, while those requiring simplified IP address management could lean toward Calico.

---

CNI Providers through CLI

Users can use the cluster config yaml file to provision the cluster with Calico CNI. Below is an illustrative example of a cluster config file with the CNI Provider Calico

kind: Cluster
metadata:
  name: demo-ekscluster
  project: project1
spec:
  blueprint: bp-with-co-addon
  blueprintversion: v1
  cloudprovider: demo-provider
  clusterConfig: null
  cniprovider: Calico-v3.19.1
  proxyconfig: {}
  type: eks
---
apiVersion: rafay.io/v1alpha5
kind: ClusterConfig
metadata:
  name: demo-ekscluster
  region: us-west-1
  version: "1.19"
nodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: 2
  iam:
    withAddonPolicies:
      autoScaler: true
      imageBuilder: true
  instanceType: t3.xlarge
  maxSize: 2
  minSize: 2
  name: ng-2220fc4d
  volumeSize: 80
  volumeType: gp3
vpc:
  cidr: 192.168.0.0/16
  clusterEndpoints:
    privateAccess: true
    publicAccess: true
  nat:
    gateway: Single

Use the below command in the terminal to apply the changes

./rctl apply -f <filename>

Important

• All the pods that implement admission webhooks should have hostnetwork to true when using Calico CNI provider

---

CNI Customization

CNI Customization

Previously, users were limited to selecting Container Network Interfaces (CNIs) from a predefined CNI list, which did not allow for any customization of CNI values. With the latest enhancement, users now have the flexibility to customize Calico and Cilium CNI values by leveraging addons and blueprints.

⚠️ Important Note: Required Labels for CNI Add-ons

When creating a new add-on, it is mandatory to include the following labels to ensure proper configuration:

1. key: rafay.type and value: cni
2. key: rafay.cni.name and one of the following values:
   • value: cilium
   • value: calico

Calico CNI Customization

Users can upload the Calico Helm chart and value files of the required version through any interface—UI, RCTL, API, or Terraform. This method provides flexibility to customize the CNI values based on specific environment requirements. It’s important to note that adding labels is mandatory when creating an CNI add-on, as these labels are essential for proper configuration.

Here is an example of creating an add-on with the Calico CNI. Similarly, add-ons can be created for Cilium CNIs

To customize Calico CNI values using Helm charts and value files, follow these steps:

Step 1: Create a Namespace

• Create a Namespace

Step 2: Create a Add-On

• Create a New Add-on with the previously created namespace and click Create

Example Labels for This Case: - key: rafay.type and value: cni - key: rafay.cni.name and value: calico

• Once the labels are added, click New Version
• Provide the version name, and upload the calico helm chart along with the values file

• To customize the values of the Calico CNI, click the edit icon and modify the required values in the editor, as shown below

⚠️ Important Note: Calico CNI Installation Configuration

It is mandatory to update the values file with the following configuration for Calico CNI: installation: kubernetesProvider: EKS cni: type: Calico calicoNetwork: bgp: Disabled

• Click Update if any modifications are made
• Click Save Changes to complete the add-on creation

Here is the newly created add-on with the Calico CNI Helm charts and value files.

Step 3: Create a Blueprint

• Once the add-on is created, create a blueprint
• Provide the required details in the General section and click Configure Add-Ons
• Select the newly created add-on calico and the corresponding version from the drop-down
• Click Save Changes

The blueprint is now created with the calico CNI add-on

Step 4: Create a Cluster

During cluster creation, select the newly created blueprint calico, provide the other configuration details as required, and proceed with cluster provisioning.

Once the cluster is created, run the command kubectl get pods -A to view the Calico CNI running status, as shown in the image below.

⚠️ Key Note

When a CNI-based blueprint is added, the cluster is initially provisioned using the default AWS CNI. Once the cluster provisioning is complete, including the deployment of all resources, the AWS CNI is automatically removed. Only the CNI specified in the blueprint will remain deployed and active within the cluster

---

Cilium CNI Customization

Similar to Calico, users can add a CNI-based blueprint with Cilium CNI, but only on Day 2. To achieve this:

• Create an EKS cluster with any blueprint using the default AWS CNI
• Once the EKS cluster is created, navigate to the Cluster Configuration page and click View More to retrieve the endpoint

• Create a namespace and an add-on using the Cilium Helm chart and value files, along with the required labels

  • key: rafay.type and value: cni
  • key: rafay.cni.name and value: calico

• Update the Cilium value files with the endpoint retrieved from the Cluster Configuration page and update the changes

• Create a new custom blueprint using this add-on and update the cluster blueprint with the newly created blueprint. This will deploy the CNI-based blueprint to the cluster and automatically remove the AWS VPC CNI

---

Day-2 Operations

In addition, users now have the option to modify Calico and Cilium CNI values as part of Day 2 operations. To do this, create a new version of the addon by uploading an updated Helm chart and value files or by editing the existing value file. Below is an example of cilium values file where the ui automount value is modified from true to false. Click Update to apply the changes

Update the blueprint with the new addon version

Now, update the cluster blueprint to deploy the latest or modified Cilium CNI to the cluster. This enhancement provides added flexibility for ongoing network configuration adjustments that were previously unavailable.

This will deploy the updated version of the blueprint to the cluster, and the deployment status is shown below.

---

Migration to Blueprint-Based CNI Configuration

🚀 For older upstream clusters currently using Cilium or Calico without a blueprint-based setup, migrating to a blueprint-based configuration for upgrades or other Day 2 operations requires attention to the following:

Key Steps for Migration

To ensure a seamless migration, follow these highlighted steps:

• Maintain Addon Compatibility: Ensure the addon name remains unchanged to maintain compatibility.
• Add the Required Labels: Include the following labels in the configuration:
• key: rafay.type and value: cni
• key: rafay.cni.name and value: cilium (or) calico
• Set the Namespace: Configure the namespace for the addon as kube-system

---