In the first section, we use facts and figures to illustrate the rise of Kubernetes technology and the importance of container technology security. We then present an approach for prioritizing the security hardening of the technology. Specific implementation examples for vulnerability testing in a Kubernetes environment are also provided. Finally, we briefly discuss remediation measures that will be detailed in future articles.
Kubernetes, an open-source orchestration tool, has become pivotal in deploying scalable and efficient containerized applications. This shift towards cloud-native technologies and microservices-based architectures enhances not only flexibility and scalability but also introduces significant security challenges. With over 60% of organizations projected to integrate Kubernetes into their operations by 2022 ( Statista ), understanding its security implications are crucial.
Similarly, our search using Shodan - a search engine for Internet-connected devices - with the search
string product:"Kubernetes" port:10250 identifies more than 240,000 Kubernetes instances with an available kubelet port
on the public internet.
Reports, such as VMware’s “State of Kubernetes 2023” [1] and the RedHat Kubernetes Report [2] , highlight misconfigurations and exposures as primary security concerns, with these issues significantly impacting deployment strategies.
“Maintaining security is a growing pain point” [1]
![Kubernetes Security Concerns [1]](assets/vmw-kube-sec-concerns.png)
![Kubernetes Security Key Findings [2]](assets/rh-kube-sec-key-findings.png)
![Kubernetes Security Top Concerns [2]](assets/rh-kube-sec-top-concerns.png)
As Kubernetes continues to be adopted rapidly, the need for robust security practices becomes a critical requirement for organizations in the cloud-native landscape. This article delves into the security risks associated with Kubernetes and discusses strategic approaches to mitigating these risks through recognized frameworks and tools.
To effectively navigate the complex landscape of Kubernetes security, it’s important to rely on authoritative resources that provide actionable insights and guidance. We’ve compiled a list of these trusted and comprehensive frameworks and tools that we’ve used in our work. This is just a sampling:
Frameworks:
Tools and Exploitaton Frameworks:
Think like an attacker
Adopting a threat-based perspective is crucial for effectively prioritizing security measures within Kubernetes clusters, as outlined in the Threat Modeling Manifesto [3] . This approach requires a deep understanding of potential attack vectors and strategic countermeasure implementations. Below, we explore three relevant resources that provide structured methodologies for getting started securing Kubernetes clusters based on accepted threat models.
The “Threat Matrix for Kubernetes,” developed and updated by Microsoft in 2021 and 2022, uses the MITRE ATT&CK framework , a globally recognized knowledge base of adversary tactics and techniques observed in real-world attacks. This matrix offers a structured approach to enhancing container orchestration security. The latest version is accessible via a web interface at Microsoft’s Kubernetes Threat Matrix .
![Kubernetes Threat Matrix by Microsoft [4]](assets/k8s-matrix.png)
The Cloud Native Computing Foundation’s (CNCF) Financial Services User Group has developed a comprehensive threat model for a generic Kubernetes installation. Published on their GitHub repository [5] , this model employs the STRIDE methodology to analyze each component of Kubernetes architecture, identifying potential security issues at trust boundaries. The overview of high-level architecture and associated trust boundaries illustrated below highlights key potential attack vectors that need protection.
Following Main Attack Vectors for Kubernetes instances were identified by the FSUG group. Detailed Attack Trees 1 developed by the group systematically outline the progression from an initial foothold attempt in the cluster to the ultimate attacker goal.
Well-known top ten list of OWASP does not bypass Kubernetes and aimed at helping security practitioners, system administrators, and software developers prioritize risks around the Kubernetes ecosystem [7] . The 2022 edition of this list is based on data from various organizations and details the most critical security risks to Kubernetes environments.
Top 10 Kubernetes Risks - 2022
Each framework takes a unique approach to identifying risks within Kubernetes clusters. However, collectively, they highlight similar categories of vulnerabilities across environments:
SELinux, AppArmor, or seccomp.Identifying misconfigurations in Kubernetes clusters requires a combination of automated tools and manual analysis to
detect both common and complex security issues. While automated tools can quickly uncover common, known issues, deep
security vulnerabilities often require careful manual inspection using kubectl commands and other Linux utilities.
This section provides a methodology for effectively identifying critical security issues from the three categories mentioned above:
However, it is not a complete guide to uncovering all possible security issues in Kubernetes environments. Our goal with this article is still to provide an introduction to the methodology of identifying security-related misconfiguration issues.
Start with automated tools designed for Kubernetes to quickly identify the most common configuration issues. Tools such
as kube-bench, kubescape, and RBAC Police Scanner scan Kubernetes clusters against known and widely used best practices
such as the CIS Kubernetes Benchmark and the NSA Kubernetes Hardening Guide.
To automate this task, we build a Kubernetes Security Docker image and publish it to the public Docker repository [8] , which supports the most commonly automated analysis tools for Kubernetes security such as:
The configuration and capabilities of this Docker image are documented and can be extended by accessing its description on GitHub [9] .
In addition to automated tools, detailed manual inspection is essential to detect more subtle or complex misconfigurations.
This includes examining user roles and permissions within the cluster, as well as using commands to deeply analyze resource
configurations and states. A key component of this manual inspection is the kubectl command.
Checking the network policies for the security issues involves inspecting the rules defined in each
policy. Use kubectl commands to assess the rules defined in each policy, focusing on:
Check if there are network policies that insufficiently restrict traffic, which could expose the cluster to risks. For example, to find policies that allow all ingress traffic:
kubectl get networkpolicies --all-namespaces -o json | jq -r '.items[] | select(.spec.ingress[]?.ports == null) | "\(.metadata.namespace) \(.metadata.name)"'
Modify the command to assess egress traffic by replacing .spec.ingress[]?.ports with .spec.egress[]?.ports.
You can look for network policies that allow traffic from all pods with a certain label. Here is a command to find network policies that allow ingress traffic from all pods with a certain label:
kubectl get networkpolicies --all-namespaces -o json | jq -r '.items[] | select(.spec.ingress[]?.from[].podSelector.matchLabels != null) | "\(.metadata.namespace) \(.metadata.name)"'
While these commands provide a foundational check, they do not encompass all potential scenarios and should not replace comprehensive security reviews.
RBAC (Role-Based Access Control) is crucial for ensuring secure access control in Kubernetes clusters. This section provides practical commands for effectively managing and auditing RBAC configurations, focusing on critical aspects like cluster administrator roles, role bindings, and permissions for creating essential resources such as pods.
Implementing secure RBAC in Kubernetes can be challenging due to the complexity of permissions. However, Prisma’s whitepaper on “Kubernetes Role-Based Access Control (RBAC)” [10] offers valuable insights into common misconfigurations. The whitepaper categorizes overprivileged Kubernetes permissions by attack class, such as Manipulate AuthN/Z, Acquire Tokens, RCE, Steal Pods, and Meddler-in-the-Middle.
These attack classes are linked to concrete powerful permissions in Kubernetes, which should be carefully reviewed in a security assessment. The figure below illustrates the summary of critical permissions in a Kubernetes cluster:
![Powerful Kubernetes Permissions by Attack Class [10]](assets/prisma-rbac-summary.png)
This section demonstrates how to identify some of the previously highlighted critical permissions within a Kubernetes cluster.
For deeper insights and best practices, please consult the official Kubernetes RBAC and Kubernetes RBAC Good Practices documentation.
Risk: Unauthorized users or service accounts with cluster-admin roles have unrestricted access to all resources within the cluster, which can lead to potential misuse or exploitation.
Purpose: Lists all entities with cluster-admin privileges, helping ensure only authorized users have such high-level access.
kubectl get clusterrolebinding -o json | jq -r '.items[] | select(.roleRef.name=="cluster-admin") | "\(.metadata.name): \(.subjects)"'
RBAC - RoleBindings and ClusterRoleBindings
Risk: Privileged containers can access critical host resources, potentially leading to data breaches or system compromises.
Purpose: This command identifies roles that have permissions to manage role bindings, critical for maintaining proper access control.
kubectl get clusterrole,role --all-namespaces -o json | jq -r '.items[] | select(.rules[] | select(.resources[]? | contains("rolebindings","clusterrolebindings")) and select(.verbs[]? | contains("create"))) | .metadata.name' | while read -r role; do kubectl get rolebinding,clusterrolebinding --all-namespaces -o json | jq -r --arg role "$role" '.items[] | select(.roleRef.name == $role) | "\(.metadata.name)\t\(.metadata.namespace)\t\(.subjects[]? | .kind + "/" + .name)"'; done
Risk: Allowing broad permissions to create pods can expose the cluster to potential security risks, such as the deployment of unauthorized or malicious pods.
Purpose: These commands identify roles and cluster roles with permissions to create pods, which can be crucial for tightening security controls.
kubectl get roles,clusterroles --all-namespaces -o json | jq -r '.items[] | select(.rules[] | select(.resources[]? | contains("pods")) and select(.verbs[]? | contains("create"))) | .metadata.name' | sort | uniq
kubectl get clusterrole,role --all-namespaces -o json | jq -r '.items[] | select(.rules[] | select(.resources[]? | contains("pods")) and select(.verbs[]? | contains("create"))) | .metadata.name' | while read -r role; do kubectl get rolebinding,clusterrolebinding --all-namespaces -o json | jq -r --arg role "$role" '.items[] | select(.roleRef.name == $role) | "\(.metadata.name)\t\(.metadata.namespace)\t\(.subjects[]? | .kind + "/" + .name)"'; done
Risk: The system:unauthenticated group includes all users and services that have not authenticated to the cluster.
Granting any permissions to this group can expose the cluster to unauthorized access.
Purpose: This command retrieves cluster role bindings that grant permissions to the system:unauthenticated group,
helping to ensure such access is intentionally granted and monitored.
kubectl get clusterrolebindings -o json | jq -r '.items[] | select(.subjects != null) | select(.subjects[].name == "system:unauthenticated") | .metadata.name'
Risk: Automatically mounting service account tokens can lead to unintentional exposure of authentication credentials to deployed pods.
Purpose: This command lists all Pods across namespaces, displaying their automountServiceAccountToken setting to
manage and restrict automatic token mounting.
kubectl get pods --all-namespaces -o json | jq -r '.items[] | "\(.metadata.namespace):\(.metadata.name)\t\(.spec.automountServiceAccountToken)"'
This section outlines how to audit and manage the Pod Security Admission (PSA) controls at both the cluster level and within individual namespaces, ensuring Kubernetes clusters maintain stringent security standards.
To review the current PSA and Pod Security Standards (PSS) configurations at both the cluster
level and namespace level, as well as to check any exemptions, the kubectl command can be used effectively.
Confirming whether PSA is enabled is crucial for enforcing pod-level security policies.
Identify the Control-Plane Node: First, list all nodes to find the control-plane node which hosts the API server configurations. Following command provides details such as the node role, helping to pinpoint the control-plane.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane 1337d v1.23.5
Next, access the API server on the controlplane node to verify the enabled admission plugins, ensuring PodSecurity is active.
This command directly queries the API server’s help menu to list all enabled admission plugins, providing real-time configuration data.
kubectl -n kube-system exec kube-apiserver-controlplane -it -- kube-apiserver -h | grep "default enabled ones"
--enable-admission-plugins strings admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
Use the following command to view the PSA configuration for a specific namespace, which retrieves and displays the enforcement level and any exceptions set for each namespace.
kubectl get namespaces -o=jsonpath="{range .items[*]}{'Namespace: '}{.metadata.name}{', Enforce: '}{.metadata.labels['pod-security\.kubernetes\.io/enforce']}{', Audit: '}{.metadata.labels['pod-security\.kubernetes\.io/audit']}{', Warn: '}{.metadata.labels['pod-security\.kubernetes\.io/warn']}{'\n'}{end}" | sed 's/, Enforce: ,/, Enforce: Not set,/g' | sed 's/, Audit: ,/, Audit: Not set,/g' | sed 's/, Warn: $/, Warn: Not set/g'
For a detailed view of the configuration of a specific namespace (replace your-namespace with the namespace you want to
inspect), which displays the labels for the specified namespace, including PSA settings such as enforce, audit, and
warn levels, along with any exceptions.
kubectl get namespace your-namespace -o json | jq '.metadata.labels'
Use the following command to list all namespaces and any PSA exceptions associated with them.
kubectl get namespaces -o=jsonpath="{range .items[*]}{'Namespace: '}{.metadata.name}{', Exemptions: '}{.metadata.labels['pod-security.kubernetes.io/exemptions']}{'\n'}{end}"
In Kubernetes, the securityContext of a pod defines its security parameters, such as privilege levels, access controls,
and resource constraints. Properly configuring the security context is essential for ensuring the security and isolation
of workloads within a Kubernetes cluster, as illustrated below:
apiVersion: v1
kind: Pod
metadata:
name: root-user
spec:
containers:
...
securityContext:
#non-root user:
runAsUser: 5554
#non-privileged
privileged: false
#read-only fs explicitly defined
readOnlyRootFilesystem: true
However, insecure configurations within pod definitions can introduce vulnerabilities that compromise the integrity and confidentiality of the entire cluster. These vulnerabilities often result from misconfigurations or overly permissive settings that grant excessive privileges to containers or allow unauthorized access to critical resources.
Risk: Privileged containers can access critical host resources, potentially leading to data breaches or system compromises.
Purpose: Identifies all containers running in privileged mode to mitigate risks associated with excessive permissions.
kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.containers[].securityContext.privileged==true) | "\(.spec.containers[].name) in \(.metadata.namespace)"'
Risk: Containers with dangerous capabilities like CAP_ALL, SYS_ADMIN, or SYS_PTRACE may bypass kernel-enforced security restrictions.
Purpose: Finds containers with elevated capabilities that pose significant security risks.
kubectl get pods --all-namespaces -o json | jq -r '.items[] | . as $parent | .spec.containers[] | select(.securityContext.capabilities.add[]? | contains("ALL","SYS_ADMIN","SYS_PTRACE")) | "\(.name) in \($parent.metadata.namespace)"'
Risk: Access to the Docker daemon via docker.sock can allow containers to control Docker, leading to unauthorized actions such as starting or stopping other containers.
Purpose: Detects pods with Docker socket bindings to prevent potential abuse or privilege escalation.
kubectl get pods --all-namespaces -o json | jq -r '.items[] | . as $pod | .spec.containers[] | . as $container | .volumeMounts[]? | select(.name | test("docker.sock"; "i")) | "\($pod.metadata.namespace)/\($pod.metadata.name)/\($container.name)/\(.name)"'
Risk: Mounting the host’s /var/log directory inside a container can allow an attacker to tamper with log files, which can be used to obscure their tracks or facilitate further attacks like log injection.
Purpose: This command finds pods that have direct access to the host’s /var/log directory. Such configurations need to be carefully controlled to prevent potential security breaches.
kubectl get pods --all-namespaces -o json | jq -r '.items[] | . as $pod | .spec.containers[] | . as $container | .volumeMounts[]? | select($pod.spec.volumes[]?.hostPath.path == .mountPath) | "\($pod.metadata.namespace)/ -- \($pod.metadata.name)/ -- \($container.name)/\(.mountPath)"' | sort -u | grep "var/log"
This methodology, combining both automated tools and detailed manual inspections, offers a robust approach to identifying
misconfigurations in Kubernetes clusters. In-depth manual analysis, supplemented by direct kubectl interactions, provides
a granular view of the operational and security configurations within a Kubernetes cluster. By combining these manual
techniques with automated tools, security practitioners can comprehensively assess their clusters’ security posture,
uncovering and mitigating potential vulnerabilities that automated scans may overlook.
There are many attack vectors and potential to take over scenarios. However, we can reduce the mitigations to following three main actions to address most of the common attacks against Kubernetes clusters and the discussed threat scenarios.
A detailed explanation and practical examples of each of these mitigations is too complex for this article, but will be part of this series on Kubernetes security. If you want to get started on securing your cluster immediately, we recommend the following resources from the official Kubernetes documentation: [11] [12] .
[1] The State of Kubernetes 2023, VMware, https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/docs/vmware-ebook-state-of-kubernetes.pdf
[2] Kubernetes’ adoption, security, and market trends report 2023, Red Hat, Available at: https://www.redhat.com/rhdc/managed-files/cl-state-kubernetes-security-report-262667-202304-en.pdf
[3] Threat Modeling Manifesto, Threat Modeling Manifesto Working Group, Available at: https://www.threatmodelingmanifesto.org/
[4] Threat Matrix for Kubernetes, Microsoft, Available at: https://www.microsoft.com/en-us/security/blog/2021/03/23/secure-containerized-environments-with-updated-threat-matrix-for-kubernetes/
[5] CNCF k8s Threat Model, CNCF, Available at: https://github.com/cncf/financial-user-group/tree/main/projects/k8s-threat-model
[6] Attack Trees, B. Schneier, Available at: https://www.schneier.com/academic/archives/1999/12/attack_trees.html
[7] Kubernetes Top 10, OWASP, Available at: https://owasp.org/www-project-kubernetes-top-ten/
[8] Docker Image Kubernetes Security, SmartTECS Cyber Security, Available at: https://hub.docker.com/r/marko4smart/kube-security
[9] Kubesec Docker Image, SmartTECS Cyber Security, Available at: https://github.com/security-smarttecs/kube-security
[10] Kubernetes Privilege Escalation: Excessive Permissions in Popular Platforms, Prisma by Palo Alto Networks, Available at https://www.paloaltonetworks.com/apps/pan/public/downloadResource?pagePath=/content/pan/en_US/resources/whitepapers/kubernetes-privilege-escalation-excessive-permissions-in-popular-platforms
[11] Security Checklist, Kubernetes Authors, Available at: https://kubernetes.io/docs/concepts/security/security-checklist/
[12] Securing a Cluster, Kubernetes Authors, Available at: https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/