Written by Arunava Mukherjee, Operations Manager
Kubernetes supports a high-level abstraction called Ingress. Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. The traffic routing is controlled by rules defined on the Ingress resource. An ingress is a core concept of Kubernetes but is always implemented by a third-party proxy. These implementations are known as ingress controllers.
An ingress controller is responsible for reading the Ingress Resource information and processing that data accordingly. Different ingress controllers have extended the specification in different ways to support additional use cases.
Running HA Nginx Ingress on AWS EKS with TLS(AWS ACM)
The most popular ones are the following:
Problem Statement
The first problem was that the default Nginx Ingress Deployment for AWS launches CLB (Classic Loadbalancer) instead of NLB(Network Loadbalancer) when we wanted to use SSL/TLS. We needed NLB (as customer requirements the application should be able to handle millions of requests per second) along with HTTP to HTTPS traffic redirections. This was resolved by changing the service.beta.kubernetes.io/aws-load-balancer-type from “elb” to “nlb”, but we also made several changes to make HTTP to HTTPS redirection was possible.
When we tried to use NLB we got a Second problem. we found that only one of the worker-node among the all was healthy. When we investigated we found that only the Node in which the Nginx Ingress Controller pod was running was healthy.
Here in this article, we have a solution in which we have figured out how we can solve both problems:
Assumptions and Pre-requisites:
- This article assumes you are familiar with AWS EKS and K8S, Create IAM Policy with access to AWS Services (EC2, ELB, IAM, Cognito, WAF, Shield, Certificate Manager, etc – All AWS Services in relation with AWS ELB)
- Associate the k8s service account, AWS IAM Policy by creating an AWS IAM Role.
The Solution
- Download the Ingress controller manifest file: Download Here
- Edit the manifest file
- Annotation section of ingress controller-service.yaml – to support NLB instead of ALB / CLB
- controller-configmap.yaml section – Removed proxy-real-ip-cidr: XXX.XXX.XXX/XX
- controller-deployment.yaml section – Changed deployment kind from Deployment to DaemonSet to run the controller on all the worker nodes
- Annotation section of ingress controller-service.yaml – to support NLB instead of ALB / CLB
A DaemonSet is a controller that ensures that the pod runs on all the nodes of the cluster. If a node is added/removed from a cluster, DaemonSet automatically adds/deletes the pod. Which resolved our problem, as daemonset the number of pod = no of nodes, this bought all our worker nodes healthy back again the added advantages are better HA and better traffic management: the traffic is distributed to all nginx controller pods rather than just one.
Let’s also understand how an ACM can be used to provide TLS at the Ingress level
You might face difficulties on your application performance suffered as encryption/decryption is compute-intensive when you needed to provide extra capacity to your backends to handle TLS termination at the application or instance level. Apart from that, managing certificates on your backends became complex as your applications grew. Applying security patches or upgrading TLS stacks on your backends was hard and time-consuming.
IF you are on the Kubernetes and using Ingress or Service to expose your application, the TLS Termination feature of NLB will address these challenges easily. Request a public ACM certificate for your custom domain and specify the Amazon Resource Name (ARN) of your ACM certificate on your Kubernetes service/Ingress using the “service.beta.kubernetes.io/aws-load-balancer-ssl-cert” annotation from the Kubernetes website. The annotation allows the Kubernetes API server to associate that certificate with the Classic Load Balancer when it’s created.to the Ingress(ALB/NLB) you want to use.
By offloading TLS from the backend servers to a high performant and scalable Network Load Balancer, you can now simplify certificate management, run backend servers optimally, support TLS connections at scale and keep your workloads always secure. Network Load Balancer also offers predefined security policies, which ensures that secured ciphers and protocols are being negotiated between the user and Network Load Balancer.
Running HA Nginx Ingress on AWS EKS with TLS(AWS ACM)
3. Create an Ingress object to route nginx traffic to the respective service.
**Note – Replace host field content with your NLB DNS Name or the Route53 record pointing to this NLB which will be invoked by the end-user client.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: xxxxxx.elb.us-west-2.amazonaws.com
http:
paths:
- path: /api/catalog/v1
backend:
serviceName: catalog-service
servicePort: 8080
- path: /api/product/v1
backend:
serviceName: product -service
servicePort: 8080
Conclusion:
I hope the above information are helpful to move forward with the similar use case you might have.
Hope you found it useful. Keep following our Kubernetes series for more interesting articles. Please free to reach out to us if you need any assistance regarding AWS and Kubernetes consulting. Happy Kubernetizing !!!
