Overcoming IP Address Limitations in Amazon EKS For Linux: A Guide to Scaling Efforts

Introduction:

One of the most common challenges faced by users of Amazon EKS (Elastic Kubernetes Service) is the limitation on the number of pods that can be deployed within a cluster. This limitation arises primarily due to the constraints related to IP address allocation. In EKS, each pod requires its own IP address for communication, and as clusters grow in size and complexity, managing these addresses becomes increasingly challenging. This issue often leads to scaling bottlenecks, longer launch times, and additional management overhead, hindering the seamless expansion of applications and workloads.

Amazon VPC CNI plugin increases pods per node limits | Containers

In this guide, we will delve into the intricacies of addressing this critical issue by exploring methods to increase the available IP addresses for Amazon EC2 nodes in Amazon EKS. By implementing the strategies outlined in this guide, users can effectively overcome IP address limitations and unlock the full scalability potential of their EKS clusters, enabling smoother deployment and operation of containerized applications.

What we are going to do?

In this segment, we'll utilize a CloudFormation stack to establish a self-managed node group within Amazon EKS, while also maximizing the pod limit for enhanced scalability and resource allocation.

Prerequisites:

An existing Amazon EKS cluster.
Subnets with sufficient contiguous CIDR blocks. For IPv4 clusters, this requires at least a /28 CIDR block, while IPv6 clusters need a /80 CIDR block. This means there must be enough consecutive IP addresses within the subnet to accommodate the needs of the EKS nodes and their associated pods.
AWS Nitro-based instances for assigning IP prefixes.
If your Linux cluster is configured for the IPv4 family, you must have version 1.9.0 or later of the Amazon VPC CNI plugin for Kubernetes add-on installed.
If your Linux cluster is configured for the IPv6 family, you must have version 1.10.1 of the add-on installed. If your plugin version is earlier than the required versions, you must update it.

What is the Prefix Mode?

Prefix mode in Amazon EKS refers to a method of IP address allocation where IP address prefixes are assigned to nodes rather than individual IP addresses. This allows for more efficient utilization of available IP addresses and facilitates scaling of workloads within Kubernetes clusters. By assigning prefixes, clusters can reduce the number of API calls needed to configure network interfaces and IP addresses, leading to faster pod and instance launch times. This mode is particularly beneficial for larger clusters and helps manage IP address allocation more effectively.

Steps to Increase Available IP Addresses:

1. Configure Cluster for IP Prefix Assignment:

Enable prefix delegation for the Amazon VPC CNI DaemonSet:

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Ensure CIDR blocks are contiguous to avoid fragmentation errors.

2. Caclulation Of Number of Pods that can run on each Instance Type:

Amazon EKS offers a script to determine the maximum number of Pods that can run on each instance type based on hardware attributes and configuration options. You can download and run the script to get this information.

Here's a simplified explanation of the steps:

Download the script: Use the curl command to download the script from the provided URL.

 curl -O https://raw.githubusercontent.com/awslabs/amazon-eks-ami/master/files/max-pods-calculator.sh

Make the script executable: Use the chmod command to mark the script as executable on your computer.
```
 chmod +x max-pods-calculator.sh
```
Run the script: Execute the script, specifying the instance type and Amazon VPC CNI add-on version. This will output the maximum number of Pods supported for that instance type.
```
 ./max-pods-calculator.sh --instance-type m5.large --cni-version 1.9.0-eksbuild.1
```
An example output is as follows.
```
 29
```

Optional capabilities:

Use --cni-custom-networking-enabled to assign IP addresses from a different subnet.
Use --cni-prefix-delegation-enabled to assign more IP addresses to each network interface.

Note: The script limits the return value to 110 or 250 based on Kubernetes scalability thresholds and recommended settings, ensuring optimal performance and resource utilization.

3.Configuring Parameters for Amazon EKS Nodes

If you're using Amazon EKS version 1.21 or later and your cluster is configured for IPv6, you can skip this step. Otherwise, you'll need to specify parameters to optimize IP address allocation for your nodes. Read More About this

Parameter Options:

WARM_PREFIX_TARGET: This option sets the target number of IP address prefixes for each node. Use the following command to specify this parameter:
```
 kubectl set env ds aws-node -n kube-system WARM_PREFIX_TARGET=1
```
WARM_IP_TARGET or MINIMUM_IP_TARGET: These options override the value set for WARM_PREFIX_TARGET if specified. They set the target number of individual IP addresses for each node. Use the following commands to set these parameters:
```
 kubectl set env ds aws-node -n kube-system WARM_IP_TARGET=5
```
```
 kubectl set env ds aws-node -n kube-system MINIMUM_IP_TARGET=2
```

Make sure to replace the example values with values greater than zero according to your specific requirements. These parameters help optimize IP address allocation and enhance the efficiency of your Amazon EKS cluster.

4. Launching Self-Managed Linux Nodes

To effectively deploy self-managed node groups with maximum pod limits in Amazon EKS using a CloudFormation stack, follow these detailed steps:

Download CloudFormation Template:

Obtain the latest version of the AWS CloudFormation template by running the following command:
```
 curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2022-12-23/amazon-eks-nodegroup.yaml
```
Wait for Cluster Activation:

Ensure your cluster status is ACTIVE before launching nodes to prevent registration failures.
Access AWS CloudFormation Console:
- Open the AWS CloudFormation console at console.aws.amazon.com/cloudformation.
- Choose "Create stack" and select "With new resources (standard)".
Upload Template:
- Select "Upload a template file" and choose the downloaded amazon-eks-nodegroup.yaml file.
- Proceed to the next step.
Specify Stack Details:

Enter parameters as follows:
- Stack name: Choose a unique name.
- ClusterName: Enter your Amazon EKS cluster name.
- ClusterControlPlaneSecurityGroup: Select the appropriate security group.
- NodeGroupName: Provide a name for your node group.
- NodeAutoScalingGroupMinSize, NodeAutoScalingGroupDesiredCapacity, NodeAutoScalingGroupMaxSize: Define scaling parameters.
- NodeInstanceType: Select an instance type for your nodes.
- NodeImageIdSSMParam: Pre-populated with a recent Amazon EKS optimized AMI parameter.
- NodeVolumeSize, NodeVolumeType, KeyName: Configure node volume and SSH key.
- BootstrapArguments: Specify optional arguments for the node bootstrap script.
  --use-max-pods false --kubelet-extra-args '--max-pods=110'
Configure Stack Options:

Select desired configuration options and proceed.
Create Stack:
- Acknowledge AWS CloudFormation IAM resource creation.
- Choose "Create stack" to initiate stack creation.
Record Outputs:

After stack creation, note the NodeInstanceRole for later configuration.

5. Enabling Nodes to Join Your Cluster

If you launched nodes inside a private VPC without outbound internet access, make sure to enable nodes to join your cluster from within the VPC.

Check aws-auth ConfigMap:

Check for an existing aws-auth ConfigMap:
```
 kubectl describe configmap -n kube-system aws-auth
```
Edit ConfigMap:

If the ConfigMap exists, open it for editing:
```
 kubectl edit -n kube-system configmap/aws-auth
```
Add a new mapRoles entry with the rolearn value set to the NodeInstanceRole recorded earlier.

data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

6. Monitor Node Status:

Watch node status until they reach the Ready state:
```
 kubectl get nodes --watch
```

7. Additional Actions (Optional)

Deploy Sample Application:

Optionally, deploy a sample application to test the cluster and Linux nodes.
Adjust IAM Policies:

If necessary, adjust IAM policies associated with the Amazon EKS node IAM role.
Restrict Pod Access to IMDS:

Consider restricting Pod access to the Instance Metadata Service (IMDS) based on your security requirements.

By following these steps, you can efficiently deploy self-managed node groups with maximum pod limits in Amazon EKS using CloudFormation stacks.

8. View Cluster Nodes:

Confirm nodes are deployed successfully:

kubectl get nodes

Describe a node to view maximum Pods and available IP addresses:

kubectl describe node <node_name> | grep 'pods\|PrivateIPv4Address'

Example Output:

NAME                                             STATUS   ROLES    AGE    VERSION
ip-192-168-22-103.region-code.compute.internal   Ready    <none>   19m    v1.XX.X-eks-6b7464
ip-192-168-97-94.region-code.compute.internal    Ready    <none>   19m    v1.XX.X-eks-6b7464

pods:                                  110
vpc.amazonaws.com/PrivateIPv4Address:  144

The maximum number of partitions that could operate on a t3.medium before these settings was 17, now it is 110.

Conclusion:

Increasing available IP addresses for Amazon EC2 nodes in Amazon EKS is crucial for scaling applications efficiently. By configuring IP prefix assignment and setting appropriate parameters, clusters can handle larger workloads without delays or management overhead. Following these steps ensures smoother operation and scalability in Amazon EKS environments.

This comprehensive guide outlines the necessary steps to expand IP address availability in Amazon EKS clusters, facilitating smoother scaling and enhanced resource utilization. By carefully following these steps, users can optimize their infrastructure for larger workloads and evolving application requirements.

References:

Watch this debate : https://github.com/kubernetes/kubernetes/issues/23349

Documentation: https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html