Amazon Elastic Kubernetes Service (EKS) protection with AWS GuardDuty

Containerization has become the go-to solution for deploying and managing applications, and Amazon Elastic Kubernetes Service (EKS) is a popular choice for running containerized workloads on AWS. However, with the benefits of containers come new security challenges. 

Amazon Elastic Kubernetes Service (EKS) protection with AWS GuardDuty

Container security is a critical aspect of deploying and managing containerized applications, ensuring that container workloads remain protected from potential threats, vulnerabilities, and unauthorized access. Here are some basic concepts of container security:

1. Isolation: Containers are designed to be isolated from each other and the host system. Each container runs in its own environment, and there are multiple layers of isolation, such as namespaces and cgroups, that prevent one container from interfering with or accessing the resources of another.
2. Immutable Infrastructure: Containers are typically built from immutable images. This means that once an image is created, it is not modified or patched directly. Instead, any updates or changes are made by creating a new image. This promotes consistency and security because you can ensure that the image contains only known, trusted components.
3. Image Scanning: Before deploying containers, it’s essential to scan container images for known vulnerabilities and malware. Vulnerabilities in software packages can be exploited by attackers, so using tools like Clair, Trivy, or integrated scanning features in container registries helps identify and mitigate potential risks.
4. Access Control: Proper access control is crucial to container security. Use role-based access control (RBAC) and least privilege principles to limit who can interact with containers, both within and outside the cluster. Ensure that only authorized users and processes can access container resources.
5. Container Orchestration Security: If you are using container orchestration platforms like Kubernetes, security considerations extend to the control plane as well. Protect the control plane with strong authentication and authorization mechanisms. Regularly update and patch the orchestrator to address any known vulnerabilities.
6. Network Security: Containers communicate with each other and external services over the network. Implement network policies and security groups to control traffic between containers and between containers and external services. Encourage the use of TLS for secure communication.
7. Secrets Management: Managing sensitive information, such as API keys, passwords, and certificates, is essential. Use dedicated tools like Kubernetes Secrets or Vault to store and manage secrets securely, ensuring they are not exposed in plain text.
8. Runtime Security: Monitor containers at runtime to detect suspicious activities and potential security incidents. Intrusion detection systems and container security solutions can help identify unauthorized access, changes in container behavior, or unusual network traffic.

Container security tools are essential for safeguarding your containerized applications and ensuring the integrity and protection of your container environments. Here’s a list of some popular container security tools and solutions that can help you enhance the security of your containerized workloads:

1. Docker Bench for Security: A script that checks for dozens of common best-practices around deploying Docker containers securely. It can be used for both Docker and containerd runtimes.
2. Clair: An open-source vulnerability scanner for container images. Clair can be integrated with container registries to provide visibility into the security of the images you plan to use.
3. Trivy: An open-source vulnerability scanner for containers that supports various image formats (Docker, OCI, etc.). Trivy is fast and easy to use, making it a popular choice for vulnerability scanning.
4. Aqua Security: A comprehensive container security platform that provides runtime protection, vulnerability scanning, and compliance checking. It’s suitable for securing containers and serverless functions.
5. OpenSCAP: The Security Content Automation Protocol (SCAP) provides a collection of open-source tools for automated vulnerability management, measurement, and policy compliance.
6. Falco: An open-source container security monitoring tool for detecting abnormal behavior and threats at the application layer.
7. Sysdig Falco: Builds on Falco to provide enhanced security and compliance capabilities. It offers real-time threat detection and response for containers and Kubernetes.
8. Amazon GuardDuty: If you’re using Amazon EKS, you can leverage Amazon GuardDuty to enhance threat detection and protect your container workloads in the AWS environment.

These tools offer a range of features, from image scanning and vulnerability detection to runtime protection, threat detection, and compliance checking. The choice of tools may depend on your specific requirements and the container orchestration platform you use (e.g., Docker, Kubernetes). Integrating a combination of these tools into your container security strategy can help you establish a robust defense against security threats.

Amazon GuardDuty is a managed threat detection service offered by Amazon Web Services (AWS). It is designed to help protect your AWS resources by continuously monitoring for malicious or unauthorized activities within your AWS environment. GuardDuty uses machine learning, anomaly detection, and integrated threat intelligence to identify potential security threats, making it a valuable asset for enhancing the security of your AWS workloads, including those hosted on Amazon Elastic Kubernetes Service (Amazon EKS).

Key features and functionalities of Amazon GuardDuty include:

1. Threat Detection: GuardDuty analyzes events and data from various AWS services, such as AWS CloudTrail (for auditing), Amazon VPC Flow Logs (for network traffic), and DNS logs. It uses this data to detect anomalies and potential security threats, such as unauthorized access, unusual API calls, or malicious IP addresses.
2. Machine Learning: The service employs machine learning models to identify patterns of behavior and deviations from the norm. This allows it to detect threats without relying solely on known attack signatures.
3. Integrated Threat Intelligence: GuardDuty leverages threat intelligence from AWS, as well as third-party sources, to identify known malicious IP addresses and domains. This helps in the early detection of common threats.
4. Security Findings: When GuardDuty detects a potential security issue, it generates security findings. These findings provide detailed information about the threat, including affected AWS resources and recommended remediation steps.
5. Automated Remediation: You can integrate GuardDuty findings with other AWS services, such as AWS Lambda, to automate the response to security threats. For example, you can automatically block malicious IP addresses using security groups or WAF rules.
6. Integration: It can be integrated with various AWS services and third-party security information and event management (SIEM) solutions for centralized security management and alerting.

Amazon GuardDuty can play a valuable role in enhancing the security of Amazon Elastic Kubernetes Service (EKS) clusters by providing threat detection and monitoring capabilities. While GuardDuty is not specific to EKS, it can help protect your EKS workloads running within the AWS environment. Here’s how you can use GuardDuty to enhance EKS protection:

1. Enable GuardDuty in Your AWS Account:
   • To get started, you need to enable GuardDuty in your AWS account. This can be done through the AWS Management Console or AWS Command Line Interface (CLI).
2. Integrate with Amazon EKS:
   • GuardDuty can monitor events and data from various AWS services, including AWS CloudTrail, VPC Flow Logs, and DNS logs. Configure your EKS clusters to log relevant activities and resource access by enabling these services. This will allow GuardDuty to analyze the data generated by your EKS clusters.
3. Security Findings and Alerts:
   • GuardDuty will generate security findings based on its analysis of the logged data. These findings will highlight potential security threats and anomalies detected within your EKS clusters, such as unauthorized access attempts or suspicious behavior.
4. Automated Remediation:
   • You can set up automated responses to GuardDuty findings in EKS. For example, if GuardDuty detects a malicious IP address trying to access your EKS cluster, you can use AWS Lambda functions to automatically update security groups or Network ACL rules to block that IP address.
5. Customizing Threat Lists:
   • Customize GuardDuty’s threat lists to include any known malicious IP addresses or domains specific to your EKS workloads. This can improve the service’s ability to detect threats relevant to your containers.
6. Integration with Other AWS Services:
   • Consider integrating GuardDuty findings with other AWS services like AWS CloudWatch, AWS Security Hub, or your SIEM solution for centralized alerting and incident management.

Getting Started Amazon Elastic Kubernetes Service (EKS) protection with AWS GuardDuty

You can enable Guardduty and use 30days free for first time use in AWS account.

Once you enable Guardduty in AWS account EKS protection is automatically enabled.

You can check the status of EKS integration status, usually it takes some for the coverage status to be in healthy state.

You can check in EKS console in the Add-ons tab , there is one add-ons will be reflecting for Amazon GuardDuty EKS Runtime Monitoring

Next , you can check that there are 2 pods running in amazon-guardduty namespace.

Now let’s check how to verify that EKS Cluster protection finding is working by intentionally triggering detectors and checking for findings

Execution:Kubernetes/ExecInKubeSystemPod

We will use sample nginx and ubuntu pod to use some attack method to get the findings in GuardDuty 

A command was executed inside a pod within the kube-system namespace

We will get finding in guard-duty like below

Execution:Runtime/NewBinaryExecuted

• Feature: Runtime Monitoring

A newly created or recently modified binary file in a container has been executed.

We have installed additional binary wget in the container which gets detected in Guardduty.

Finding summary 

This finding informs you that a newly created or a recently modified binary file in a container was executed. It is the best practice to keep containers immutable at runtime, and binary files, scripts, or libraries should not be created or modified during the lifetime of the container. It is highly suspicious that the newly created binaries were executed in the container environment. This behavior is indicative of a malicious actor that has gained access to the workload, and has downloaded and executed malware or other software as part of the potential compromise.

Execution:Runtime/ReverseShell

A process in a container or an Amazon EC2 instance has created a reverse shell.

In this reverse shell example it starts with /bin/echo with redirection to the Google DNS IP 8.8.8.8 on the DNS port, nothing is printed when you run this example and at the end container is deleted after execution.

We can also investigate further with AWS detective, we will cover this in a separate blog for detective.

Finding summary:A reverse shell is a shell session created on a connection that is initiated from the target host to the actor’s host. This is opposite to a normal shell that is initiated from the actor’s host to the target’s host. Threat actors create a reverse shell to execute commands on the target after gaining initial access to the target. This finding identifies a potential attempt to create a reverse shell.

Conclusion:

By integrating Amazon GuardDuty with Amazon EKS clusters and following best practices for threat detection and incident response, we can significantly enhance the security of containerized workloads running on AWS. GuardDuty’s automated threat detection and response capabilities can help to identify and mitigate security risks in real-time, providing a layer of protection for EKS environments. However,Amazon GuardDuty, while a valuable service for enhancing the security of Amazon Elastic Kubernetes Service (EKS) and other AWS resources, does have certain limitations and considerations when used in the context of EKS protection like it has no capability of image scanning and container-specific threat detection as of now. To overcome some of these limitations, you may consider using additional container-specific security tools and practices alongside GuardDuty.

Amazon GuardDuty announces a new capability in GuardDuty EKS Runtime Monitoring that allows you to selectively configure which Amazon Elastic Kubernetes Service (Amazon EKS) clusters are to be monitored for threat detection. Previously, configurability was at the account level only. With this added cluster-level configurability, customers can now selectively monitor EKS clusters for threat detection or continue to use account level configurability to monitor all EKS clusters in a given account and region.