on October 20, 2025 at 3:30 pm

It’s no secret that Kubernetes can be both powerful and frustrating at times. When I first started dabbling with container orchestration, I made more than my fair share of mistakes enough to compile a whole list of pitfalls. In this post, I want to walk through seven big gotchas I’ve encountered (or seen others run into) and share some tips on how to avoid them. Whether you’re just kicking the tires on Kubernetes or already managing production clusters, I hope these insights help you steer clear of a little extra stress. 1. Skipping resource requests and limits The pitfall: Not specifying CPU and memory requirements in Pod specifications. This typically happens because Kubernetes does not require these fields, and workloads can often start and run without them—making the omission easy to overlook in early configurations or during rapid deployment cycles. Context: In Kubernetes, resource requests and limits are critical for efficient cluster management. Resource requests ensure that the scheduler reserves the appropriate amount of CPU and memory for each pod, guaranteeing that it has the necessary resources to operate. Resource limits cap the amount of CPU and memory a pod can use, preventing any single pod from consuming excessive resources and potentially starving other pods. When resource requests and limits are not set: Resource Starvation: Pods may get insufficient resources, leading to degraded performance or failures. This is because Kubernetes schedules pods based on these requests. Without them, the scheduler might place too many pods on a single node, leading to resource contention and performance bottlenecks. Resource Hoarding: Conversely, without limits, a pod might consume more than its fair share of resources, impacting the performance and stability of other pods on the same node. This can lead to issues such as other pods getting evicted or killed by the Out-Of-Memory (OOM) killer due to lack of available memory. How to avoid it: Start with modest requests (for example 100m CPU, 128Mi memory) and see how your app behaves. Monitor real-world usage and refine your values; the HorizontalPodAutoscaler can help automate scaling based on metrics. Keep an eye on kubectl top pods or your logging/monitoring tool to confirm you’re not over- or under-provisioning. My reality check: Early on, I never thought about memory limits. Things seemed fine on my local cluster. Then, on a larger environment, Pods got OOMKilled left and right. Lesson learned. For detailed instructions on configuring resource requests and limits for your containers, please refer to Assign Memory Resources to Containers and Pods (part of the official Kubernetes documentation). 2. Underestimating liveness and readiness probes The pitfall: Deploying containers without explicitly defining how Kubernetes should check their health or readiness. This tends to happen because Kubernetes will consider a container “running” as long as the process inside hasn’t exited. Without additional signals, Kubernetes assumes the workload is functioning—even if the application inside is unresponsive, initializing, or stuck. Context: Liveness, readiness, and startup probes are mechanisms Kubernetes uses to monitor container health and availability. Liveness probes determine if the application is still alive. If a liveness check fails, the container is restarted. Readiness probes control whether a container is ready to serve traffic. Until the readiness probe passes, the container is removed from Service endpoints. Startup probes help distinguish between long startup times and actual failures. How to avoid it: Add a simple HTTP livenessProbe to check a health endpoint (for example /healthz) so Kubernetes can restart a hung container. Use a readinessProbe to ensure traffic doesn’t reach your app until it’s warmed up. Keep probes simple. Overly complex checks can create false alarms and unnecessary restarts. My reality check: I once forgot a readiness probe for a web service that took a while to load. Users hit it prematurely, got weird timeouts, and I spent hours scratching my head. A 3-line readiness probe would have saved the day. For comprehensive instructions on configuring liveness, readiness, and startup probes for containers, please refer to Configure Liveness, Readiness and Startup Probes in the official Kubernetes documentation. 3. “We’ll just look at container logs” (famous last words) The pitfall: Relying solely on container logs retrieved via kubectl logs. This often happens because the command is quick and convenient, and in many setups, logs appear accessible during development or early troubleshooting. However, kubectl logs only retrieves logs from currently running or recently terminated containers, and those logs are stored on the node’s local disk. As soon as the container is deleted, evicted, or the node is restarted, the log files may be rotated out or permanently lost. How to avoid it: Centralize logs using CNCF tools like Fluentd or Fluent Bit to aggregate output from all Pods. Adopt OpenTelemetry for a unified view of logs, metrics, and (if needed) traces. This lets you spot correlations between infrastructure events and app-level behavior. Pair logs with Prometheus metrics to track cluster-level data alongside application logs. If you need distributed tracing, consider CNCF projects like Jaeger. My reality check: The first time I lost Pod logs to a quick restart, I realized how flimsy “kubectl logs” can be on its own. Since then, I’ve set up a proper pipeline for every cluster to avoid missing vital clues. 4. Treating dev and prod exactly the same The pitfall: Deploying the same Kubernetes manifests with identical settings across development, staging, and production environments. This often occurs when teams aim for consistency and reuse, but overlook that environment-specific factors—such as traffic patterns, resource availability, scaling needs, or access control—can differ significantly. Without customization, configurations optimized for one environment may cause instability, poor performance, or security gaps in another. How to avoid it: Use environment overlays or kustomize to maintain a shared base while customizing resource requests, replicas, or config for each environment. Extract environment-specific configuration into ConfigMaps and / or Secrets. You can use a specialized tool such as Sealed Secrets to manage confidential data. Plan for scale in production. Your dev cluster can probably get away with minimal CPU/memory, but prod might need significantly more. My reality check: One time, I scaled up replicaCount from 2 to 10 in a tiny dev environment just to “test.” I promptly ran out of resources and spent half a day cleaning up the aftermath. Oops. 5. Leaving old stuff floating around The pitfall: Leaving unused or outdated resources—such as Deployments, Services, ConfigMaps, or PersistentVolumeClaims—running in the cluster. This often happens because Kubernetes does not automatically remove resources unless explicitly instructed, and there is no built-in mechanism to track ownership or expiration. Over time, these forgotten objects can accumulate, consuming cluster resources, increasing cloud costs, and creating operational confusion, especially when stale Services or LoadBalancers continue to route traffic. How to avoid it: Label everything with a purpose or owner label. That way, you can easily query resources you no longer need. Regularly audit your cluster: run kubectl get all -n <namespace> to see what’s actually running, and confirm it’s all legit. Adopt Kubernetes’ Garbage Collection: K8s docs show how to remove dependent objects automatically. Leverage policy automation: Tools like Kyverno can automatically delete or block stale resources after a certain period, or enforce lifecycle policies so you don’t have to remember every single cleanup step. My reality check: After a hackathon, I forgot to tear down a “test-svc” pinned to an external load balancer. Three weeks later, I realized I’d been paying for that load balancer the entire time. Facepalm. 6. Diving too deep into networking too soon The pitfall: Introducing advanced networking solutions—such as service meshes, custom CNI plugins, or multi-cluster communication—before fully understanding Kubernetes’ native networking primitives. This commonly occurs when teams implement features like traffic routing, observability, or mTLS using external tools without first mastering how core Kubernetes networking works: including Pod-to-Pod communication, ClusterIP Services, DNS resolution, and basic ingress traffic handling. As a result, network-related issues become harder to troubleshoot, especially when overlays introduce additional abstractions and failure points. How to avoid it: Start small: a Deployment, a Service, and a basic ingress controller such as one based on NGINX (e.g., Ingress-NGINX). Make sure you understand how traffic flows within the cluster, how service discovery works, and how DNS is configured. Only move to a full-blown mesh or advanced CNI features when you actually need them, complex networking adds overhead. My reality check: I tried Istio on a small internal app once, then spent more time debugging Istio itself than the actual app. Eventually, I stepped back, removed Istio, and everything worked fine. 7. Going too light on security and RBAC The pitfall: Deploying workloads with insecure configurations, such as running containers as the root user, using the latest image tag, disabling security contexts, or assigning overly broad RBAC roles like cluster-admin. These practices persist because Kubernetes does not enforce strict security defaults out of the box, and the platform is designed to be flexible rather than opinionated. Without explicit security policies in place, clusters can remain exposed to risks like container escape, unauthorized privilege escalation, or accidental production changes due to unpinned images. How to avoid it: Use RBAC to define roles and permissions within Kubernetes. While RBAC is the default and most widely supported authorization mechanism, Kubernetes also allows the use of alternative authorizers. For more advanced or external policy needs, consider solutions like OPA Gatekeeper (based on Rego), Kyverno, or custom webhooks using policy languages such as CEL or Cedar. Pin images to specific versions (no more :latest!). This helps you know what’s actually deployed. Look into Pod Security Admission (or other solutions like Kyverno) to enforce non-root containers, read-only filesystems, etc. My reality check: I never had a huge security breach, but I’ve heard plenty of cautionary tales. If you don’t tighten things up, it’s only a matter of time before something goes wrong. Final thoughts Kubernetes is amazing, but it’s not psychic, it won’t magically do the right thing if you don’t tell it what you need. By keeping these pitfalls in mind, you’ll avoid a lot of headaches and wasted time. Mistakes happen (trust me, I’ve made my share), but each one is a chance to learn more about how Kubernetes truly works under the hood. If you’re curious to dive deeper, the official docs and the community Slack are excellent next steps. And of course, feel free to share your own horror stories or success tips, because at the end of the day, we’re all in this cloud native adventure together. Happy Shipping!

on October 18, 2025 at 12:00 am

(Note: The Policy Working Group has completed its mission and is no longer active. This article reflects its work, accomplishments, and insights into how a working group operates.) In the complex world of Kubernetes, policies play a crucial role in managing and securing clusters. But have you ever wondered how these policies are developed, implemented, and standardized across the Kubernetes ecosystem? To answer that, let’s take a look back at the work of the Policy Working Group. The Policy Working Group was dedicated to a critical mission: providing an overall architecture that encompasses both current policy-related implementations and future policy proposals in Kubernetes. Their goal was both ambitious and essential: to develop a universal policy architecture that benefits developers and end-users alike. Through collaborative methods, this working group strove to bring clarity and consistency to the often complex world of Kubernetes policies. By focusing on both existing implementations and future proposals, they ensured that the policy landscape in Kubernetes remains coherent and accessible as the technology evolves. This blog post dives deeper into the work of the Policy Working Group, guided by insights from its former co-chairs: Jim Bugwadia Poonam Lamba Andy Suderman Interviewed by Arujjwal Negi. These co-chairs explained what the Policy Working Group was all about. Introduction Hello, thank you for the time! Let’s start with some introductions, could you tell us a bit about yourself, your role, and how you got involved in Kubernetes? Jim Bugwadia: My name is Jim Bugwadia, and I am a co-founder and the CEO at Nirmata which provides solutions that automate security and compliance for cloud-native workloads. At Nirmata, we have been working with Kubernetes since it started in 2014. We initially built a Kubernetes policy engine in our commercial platform and later donated it to CNCF as the Kyverno project. I joined the CNCF Kubernetes Policy Working Group to help build and standardize various aspects of policy management for Kubernetes and later became a co-chair. Andy Suderman: My name is Andy Suderman and I am the CTO of Fairwinds, a managed Kubernetes-as-a-Service provider. I began working with Kubernetes in 2016 building a web conferencing platform. I am an author and/or maintainer of several Kubernetes-related open-source projects such as Goldilocks, Pluto, and Polaris. Polaris is a JSON-schema-based policy engine, which started Fairwinds’ journey into the policy space and my involvement in the Policy Working Group. Poonam Lamba: My name is Poonam Lamba, and I currently work as a Product Manager for Google Kubernetes Engine (GKE) at Google. My journey with Kubernetes began back in 2017 when I was building an SRE platform for a large enterprise, using a private cloud built on Kubernetes. Intrigued by its potential to revolutionize the way we deployed and managed applications at the time, I dove headfirst into learning everything I could about it. Since then, I’ve had the opportunity to build the policy and compliance products for GKE. I lead and contribute to GKE CIS benchmarks. I am involved with the Gatekeeper project as well as I have contributed to Policy-WG for over 2 years and served as a co-chair for the group. Responses to the following questions represent an amalgamation of insights from the former co-chairs. About Working Groups One thing even I am not aware of is the difference between a working group and a SIG. Can you help us understand what a working group is and how it is different from a SIG? Unlike SIGs, working groups are temporary and focused on tackling specific, cross-cutting issues or projects that may involve multiple SIGs. Their lifespan is defined, and they disband once they’ve achieved their objective. Generally, working groups don’t own code or have long-term responsibility for managing a particular area of the Kubernetes project. (To know more about SIGs, visit the list of Special Interest Groups) You mentioned that Working Groups involve multiple SIGS. What SIGS was the Policy WG closely involved with, and how did you coordinate with them? The group collaborated closely with Kubernetes SIG Auth throughout our existence, and more recently, the group also worked with SIG Security since its formation. Our collaboration occurred in a few ways. We provided periodic updates during the SIG meetings to keep them informed of our progress and activities. Additionally, we utilize other community forums to maintain open lines of communication and ensured our work aligned with the broader Kubernetes ecosystem. This collaborative approach helped the group stay coordinated with related efforts across the Kubernetes community. Policy WG Why was the Policy Working Group created? To enable a broad set of use cases, we recognize that Kubernetes is powered by a highly declarative, fine-grained, and extensible configuration management system. We’ve observed that a Kubernetes configuration manifest may have different portions that are important to various stakeholders. For example, some parts may be crucial for developers, while others might be of particular interest to security teams or address operational concerns. Given this complexity, we believe that policies governing the usage of these intricate configurations are essential for success with Kubernetes. Our Policy Working Group was created specifically to research the standardization of policy definitions and related artifacts. We saw a need to bring consistency and clarity to how policies are defined and implemented across the Kubernetes ecosystem, given the diverse requirements and stakeholders involved in Kubernetes deployments. Can you give me an idea of the work you did in the group? We worked on several Kubernetes policy-related projects. Our initiatives included: We worked on a Kubernetes Enhancement Proposal (KEP) for the Kubernetes Policy Reports API. This aims to standardize how policy reports are generated and consumed within the Kubernetes ecosystem. We conducted a CNCF survey to better understand policy usage in the Kubernetes space. This helped gauge the practices and needs across the community at the time. We wrote a paper that will guide users in achieving PCI-DSS compliance for containers. This is intended to help organizations meet important security standards in their Kubernetes environments. We also worked on a paper highlighting how shifting security down can benefit organizations. This focuses on the advantages of implementing security measures earlier in the development and deployment process. Can you tell us what were the main objectives of the Policy Working Group and some of your key accomplishments? The charter of the Policy WG was to help standardize policy management for Kubernetes and educate the community on best practices. To accomplish this we updated the Kubernetes documentation (Policies | Kubernetes), produced several whitepapers (Kubernetes Policy Management, Kubernetes GRC), and created the Policy Reports API (API reference) which standardizes reporting across various tools. Several popular tools such as Falco, Trivy, Kyverno, kube-bench, and others support the Policy Report API. A major milestone for the Policy WG was promoting the Policy Reports API to a SIG-level API or finding it a stable home. Beyond that, as ValidatingAdmissionPolicy and MutatingAdmissionPolicy approached GA in Kubernetes, a key goal of the WG was to guide and educate the community on the tradeoffs and appropriate usage patterns for these built-in API objects and other CNCF policy management solutions like OPA/Gatekeeper and Kyverno. Challenges What were some of the major challenges that the Policy Working Group worked on? During our work in the Policy Working Group, we encountered several challenges: One of the main issues we faced was finding time to consistently contribute. Given that many of us have other professional commitments, it can be difficult to dedicate regular time to the working group’s initiatives. Another challenge we experienced was related to our consensus-driven model. While this approach ensures that all voices are heard, it can sometimes lead to slower decision-making processes. We valued thorough discussion and agreement, but this can occasionally delay progress on our projects. We’ve also encountered occasional differences of opinion among group members. These situations require careful navigation to ensure that we maintain a collaborative and productive environment while addressing diverse viewpoints. Lastly, we’ve noticed that newcomers to the group may find it difficult to contribute effectively without consistent attendance at our meetings. The complex nature of our work often requires ongoing context, which can be challenging for those who aren’t able to participate regularly. Can you tell me more about those challenges? How did you discover each one? What has the impact been? What were some strategies you used to address them? There are no easy answers, but having more contributors and maintainers greatly helps! Overall the CNCF community is great to work with and is very welcoming to beginners. So, if folks out there are hesitating to get involved, I highly encourage them to attend a WG or SIG meeting and just listen in. It often takes a few meetings to fully understand the discussions, so don’t feel discouraged if you don’t grasp everything right away. We made a point to emphasize this and encouraged new members to review documentation as a starting point for getting involved. Additionally, differences of opinion were valued and encouraged within the Policy-WG. We adhered to the CNCF core values and resolve disagreements by maintaining respect for one another. We also strove to timebox our decisions and assign clear responsibilities to keep things moving forward. This is where our discussion about the Policy Working Group ends. The working group, and especially the people who took part in this article, hope this gave you some insights into the group’s aims and workings. You can get more info about Working Groups here.

on October 6, 2025 at 12:00 am

Headlamp is an open‑source, extensible Kubernetes SIG UI project designed to let you explore, manage, and debug cluster resources. Karpenter is a Kubernetes Autoscaling SIG node provisioning project that helps clusters scale quickly and efficiently. It launches new nodes in seconds, selects appropriate instance types for workloads, and manages the full node lifecycle, including scale-down. The new Headlamp Karpenter Plugin adds real-time visibility into Karpenter’s activity directly from the Headlamp UI. It shows how Karpenter resources relate to Kubernetes objects, displays live metrics, and surfaces scaling events as they happen. You can inspect pending pods during provisioning, review scaling decisions, and edit Karpenter-managed resources with built-in validation. The Karpenter plugin was made as part of a LFX mentor project. The Karpenter plugin for Headlamp aims to make it easier for Kubernetes users and operators to understand, debug, and fine-tune autoscaling behavior in their clusters. Now we will give a brief tour of the Headlamp plugin. Map view of Karpenter Resources and how they relate to Kubernetes resources Easily see how Karpenter Resources like NodeClasses, NodePool and NodeClaims connect with core Kubernetes resources like Pods, Nodes etc. Visualization of Karpenter Metrics Get instant insights of Resource Usage v/s Limits, Allowed disruptions, Pending Pods, Provisioning Latency and many more . Scaling decisions Shows which instances are being provisioned for your workloads and understand the reason behind why Karpenter made those choices. Helpful while debugging. Config editor with validation support Make live edits to Karpenter configurations. The editor includes diff previews and resource validation for safer adjustments. Real time view of Karpenter resources View and track Karpenter specific resources in real time such as “NodeClaims” as your cluster scales up and down. Dashboard for Pending Pods View all pending pods with unmet scheduling requirements/Failed Scheduling highlighting why they couldn’t be scheduled. Karpenter Providers This plugin should work with most Karpenter providers, but has only so far been tested on the ones listed in the table. Additionally, each provider gives some extra information, and the ones in the table below are displayed by the plugin. Provider Name Tested Extra provider specific info supported AWS ✅ ✅ Azure ✅ ✅ AlibabaCloud ❌ ❌ Bizfly Cloud ❌ ❌ Cluster API ❌ ❌ GCP ❌ ❌ Proxmox ❌ ❌ Oracle Cloud Infrastructure (OCI) ❌ ❌ Please submit an issue if you test one of the untested providers or if you want support for this provider (PRs also gladly accepted). How to use Please see the plugins/karpenter/README.md for instructions on how to use. Feedback and Questions Please submit an issue if you use Karpenter and have any other ideas or feedback. Or come to the Kubernetes slack headlamp channel for a chat.

on September 25, 2025 at 1:00 pm

We’re excited to announce the alpha support for a changed block tracking mechanism. This enhances the Kubernetes storage ecosystem by providing an efficient way for CSI storage drivers to identify changed blocks in PersistentVolume snapshots. With a driver that can use the feature, you could benefit from faster and more resource-efficient backup operations. If you’re eager to try this feature, you can skip to the Getting Started section. What is changed block tracking? Changed block tracking enables storage systems to identify and track modifications at the block level between snapshots, eliminating the need to scan entire volumes during backup operations. The improvement is a change to the Container Storage Interface (CSI), and also to the storage support in Kubernetes itself. With the alpha feature enabled, your cluster can: Identify allocated blocks within a CSI volume snapshot Determine changed blocks between two snapshots of the same volume Streamline backup operations by focusing only on changed data blocks For Kubernetes users managing large datasets, this API enables significantly more efficient backup processes. Backup applications can now focus only on the blocks that have changed, rather than processing entire volumes. Note:As of now, the Changed Block Tracking API is supported only for block volumes and not for file volumes. CSI drivers that manage file-based storage systems will not be able to implement this capability. Benefits of changed block tracking support in Kubernetes As Kubernetes adoption grows for stateful workloads managing critical data, the need for efficient backup solutions becomes increasingly important. Traditional full backup approaches face challenges with: Long backup windows: Full volume backups can take hours for large datasets, making it difficult to complete within maintenance windows. High resource utilization: Backup operations consume substantial network bandwidth and I/O resources, especially for large data volumes and data-intensive applications. Increased storage costs: Repetitive full backups store redundant data, causing storage requirements to grow linearly even when only a small percentage of data actually changes between backups. The Changed Block Tracking API addresses these challenges by providing native Kubernetes support for incremental backup capabilities through the CSI interface. Key components The implementation consists of three primary components: CSI SnapshotMetadata Service API: An API, offered by gRPC, that provides volume snapshot and changed block data. SnapshotMetadataService API: A Kubernetes CustomResourceDefinition (CRD) that advertises CSI driver metadata service availability and connection details to cluster clients. External Snapshot Metadata Sidecar: An intermediary component that connects CSI drivers to backup applications via a standardized gRPC interface. Implementation requirements Storage provider responsibilities If you’re an author of a storage integration with Kubernetes and want to support the changed block tracking feature, you must implement specific requirements: Implement CSI RPCs: Storage providers need to implement the SnapshotMetadata service as defined in the CSI specifications protobuf. This service requires server-side streaming implementations for the following RPCs: GetMetadataAllocated: For identifying allocated blocks in a snapshot GetMetadataDelta: For determining changed blocks between two snapshots Storage backend capabilities: Ensure the storage backend has the capability to track and report block-level changes. Deploy external components: Integrate with the external-snapshot-metadata sidecar to expose the snapshot metadata service. Register custom resource: Register the SnapshotMetadataService resource using a CustomResourceDefinition and create a SnapshotMetadataService custom resource that advertises the availability of the metadata service and provides connection details. Support error handling: Implement proper error handling for these RPCs according to the CSI specification requirements. Backup solution responsibilities A backup solution looking to leverage this feature must: Set up authentication: The backup application must provide a Kubernetes ServiceAccount token when using the Kubernetes SnapshotMetadataService API. Appropriate access grants, such as RBAC RoleBindings, must be established to authorize the backup application ServiceAccount to obtain such tokens. Implement streaming client-side code: Develop clients that implement the streaming gRPC APIs defined in the schema.proto file. Specifically: Implement streaming client code for GetMetadataAllocated and GetMetadataDelta methods Handle server-side streaming responses efficiently as the metadata comes in chunks Process the SnapshotMetadataResponse message format with proper error handling The external-snapshot-metadata GitHub repository provides a convenient iterator support package to simplify client implementation. Handle large dataset streaming: Design clients to efficiently handle large streams of block metadata that could be returned for volumes with significant changes. Optimize backup processes: Modify backup workflows to use the changed block metadata to identify and only transfer changed blocks to make backups more efficient, reducing both backup duration and resource consumption. Getting started To use changed block tracking in your cluster: Ensure your CSI driver supports volume snapshots and implements the snapshot metadata capabilities with the required external-snapshot-metadata sidecar Make sure the SnapshotMetadataService custom resource is registered using CRD Verify the presence of a SnapshotMetadataService custom resource for your CSI driver Create clients that can access the API using appropriate authentication (via Kubernetes ServiceAccount tokens) The API provides two main functions: GetMetadataAllocated: Lists blocks allocated in a single snapshot GetMetadataDelta: Lists blocks changed between two snapshots What’s next? Depending on feedback and adoption, the Kubernetes developers hope to push the CSI Snapshot Metadata implementation to Beta in the future releases. Where can I learn more? For those interested in trying out this new feature: Official Kubernetes CSI Developer Documentation The enhancement proposal for the snapshot metadata feature. GitHub repository for implementation and release status of external-snapshot-metadata Complete gRPC protocol definitions for snapshot metadata API: schema.proto Example snapshot metadata client implementation: snapshot-metadata-lister End-to-end example with csi-hostpath-driver: example documentation How do I get involved? This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. On behalf of SIG Storage, I would like to offer a huge thank you to the contributors who helped review the design and implementation of the project, including but not limited to the following: Ben Swartzlander (bswartz) Carl Braganza (carlbraganza) Daniil Fedotov (hairyhum) Ivan Sim (ihcsim) Nikhil Ladha (Nikhil-Ladha) Prasad Ghangal (PrasadG193) Praveen M (iPraveenParihar) Rakshith R (Rakshith-R) Xing Yang (xing-yang) Thank also to everyone who has contributed to the project, including others who helped review the KEP and the CSI spec PR For those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the Kubernetes Storage Special Interest Group (SIG). We always welcome new contributors. The SIG also holds regular Data Protection Working Group meetings. New attendees are welcome to join our discussions.

on September 22, 2025 at 6:30 pm

On behalf of the Kubernetes community, I am thrilled to announce that the Pod Level Resources feature has graduated to Beta in the Kubernetes v1.34 release and is enabled by default! This significant milestone introduces a new layer of flexibility for defining and managing resource allocation for your Pods. This flexibility stems from the ability to specify CPU and memory resources for the Pod as a whole. Pod level resources can be combined with the container-level specifications to express the exact resource requirements and limits your application needs. Pod-level specification for resources Until recently, resource specifications that applied to Pods were primarily defined at the individual container level. While effective, this approach sometimes required duplicating or meticulously calculating resource needs across multiple containers within a single Pod. As a beta feature, Kubernetes allows you to specify the CPU, memory and hugepages resources at the Pod-level. This means you can now define resource requests and limits for an entire Pod, enabling easier resource sharing without requiring granular, per-container management of these resources where it’s not needed. Why does Pod-level specification matter? This feature enhances resource management in Kubernetes by offering flexible resource management at both the Pod and container levels. It provides a consolidated approach to resource declaration, reducing the need for meticulous, per-container management, especially for Pods with multiple containers. Pod-level resources enable containers within a pod to share unused resoures amongst themselves, promoting efficient utilization within the pod. For example, it prevents sidecar containers from becoming performance bottlenecks. Previously, a sidecar (e.g., a logging agent or service mesh proxy) hitting its individual CPU limit could be throttled and slow down the entire Pod, even if the main application container had plenty of spare CPU. With pod-level resources, the sidecar and the main container can share Pod’s resource budget, ensuring smooth operation during traffic spikes – either the whole Pod is throttled or all containers work. When both pod-level and container-level resources are specified, pod-level requests and limits take precedence. This gives you – and cluster administrators – a powerful way to enforce overall resource boundaries for your Pods. For scheduling, if a pod-level request is explicitly defined, the scheduler uses that specific value to find a suitable node, insteaf of the aggregated requests of the individual containers. At runtime, the pod-level limit acts as a hard ceiling for the combined resource usage of all containers. Crucially, this pod-level limit is the absolute enforcer; even if the sum of the individual container limits is higher, the total resource consumption can never exceed the pod-level limit. Pod-level resources are prioritized in influencing the Quality of Service (QoS) class of the Pod. For Pods running on Linux nodes, the Out-Of-Memory (OOM) score adjustment calculation considers both pod-level and container-level resources requests. Pod-level resources are designed to be compatible with existing Kubernetes functionalities, ensuring a smooth integration into your workflows. How to specify resources for an entire Pod Using PodLevelResources feature gate requires Kubernetes v1.34 or newer for all cluster components, including the control plane and every node. This feature gate is in beta and enabled by default in v1.34. Example manifest You can specify CPU, memory and hugepages resources directly in the Pod spec manifest at the resources field for the entire Pod. Here’s an example demonstrating a Pod with both CPU and memory requests and limits defined at the Pod level: apiVersion: v1 kind: Pod metadata: name: pod-resources-demo namespace: pod-resources-example spec: # The ‘resources’ field at the Pod specification level defines the overall # resource budget for all containers within this Pod combined. resources: # Pod-level resources # ‘limits’ specifies the maximum amount of resources the Pod is allowed to use. # The sum of the limits of all containers in the Pod cannot exceed these values. limits: cpu: “1” # The entire Pod cannot use more than 1 CPU core. memory: “200Mi” # The entire Pod cannot use more than 200 MiB of memory. # ‘requests’ specifies the minimum amount of resources guaranteed to the Pod. # This value is used by the Kubernetes scheduler to find a node with enough capacity. requests: cpu: “1” # The Pod is guaranteed 1 CPU core when scheduled. memory: “100Mi” # The Pod is guaranteed 100 MiB of memory when scheduled. containers: – name: main-app-container image: nginx … # This container has no resource requests or limits specified. – name: auxiliary-container image: fedora command: [“sleep”, “inf”] … # This container has no resource requests or limits specified. In this example, the pod-resources-demo Pod as a whole requests 1 CPU and 100 MiB of memory, and is limited to 1 CPU and 200 MiB of memory. The containers within will operate under these overall Pod-level constraints, as explained in the next section. Interaction with container-level resource requests or limits When both pod-level and container-level resources are specified, pod-level requests and limits take precedence. This means the node allocates resources based on the pod-level specifications. Consider a Pod with two containers where pod-level CPU and memory requests and limits are defined, and only one container has its own explicit resource definitions: apiVersion: v1 kind: Pod metadata: name: pod-resources-demo namespace: pod-resources-example spec: resources: limits: cpu: “1” memory: “200Mi” requests: cpu: “1” memory: “100Mi” containers: – name: main-app-container image: nginx resources: requests: cpu: “0.5” memory: “50Mi” – name: auxiliary-container image: fedora command: [ “sleep”, “inf”] # This container has no resource requests or limits specified. Pod-Level Limits: The pod-level limits (cpu: “1”, memory: “200Mi”) establish an absolute boundary for the entire Pod. The sum of resources consumed by all its containers is enforced at this ceiling and cannot be surpassed. Resource Sharing and Bursting: Containers can dynamically borrow any unused capacity, allowing them to burst as needed, so long as the Pod’s aggregate usage stays within the overall limit. Pod-Level Requests: The pod-level requests (cpu: “1”, memory: “100Mi”) serve as the foundational resource guarantee for the entire Pod. This value informs the scheduler’s placement decision and represents the minimum resources the Pod can rely on during node-level contention. Container-Level Requests: Container-level requests create a priority system within the Pod’s guaranteed budget. Because main-app-container has an explicit request (cpu: “0.5”, memory: “50Mi”), it is given precedence for its share of resources under resource pressure over the auxiliary-container, which has no such explicit claim. Limitations First of all, in-place resize of pod-level resources is not supported for Kubernetes v1.34 (or earlier). Attempting to modify the pod-level resource limits or requests on a running Pod results in an error: the resize is rejected. The v1.34 implementation of Pod level resources focuses on allowing initial declaration of an overall resource envelope, that applies to the entire Pod. That is distinct from in-place pod resize, which (despite what the name might suggest) allows you to make dynamic adjustments to container resource requests and limits, within a running Pod, and potentially without a container restart. In-place resizing is also not yet a stable feature; it graduated to Beta in the v1.33 release. Only CPU, memory, and hugepages resources can be specified at pod-level. Pod-level resources are not supported for Windows pods. If the Pod specification explicitly targets Windows (e.g., by setting spec.os.name: “windows”), the API server will reject the Pod during the validation step. If the Pod is not explicitly marked for Windows but is scheduled to a Windows node (e.g., via a nodeSelector), the Kubelet on that Windows node will reject the Pod during its admission process. The Topology Manager, Memory Manager and CPU Manager do not align pods and containers based on pod-level resources as these resource managers don’t currently support pod-level resources. Getting started and providing feedback Ready to explore Pod Level Resources feature? You’ll need a Kubernetes cluster running version 1.34 or later. Remember to enable the PodLevelResources feature gate across your control plane and all nodes. As this feature moves through Beta, your feedback is invaluable. Please report any issues or share your experiences via the standard Kubernetes communication channels: Slack: #sig-node Mailing list Open Community Issues/PRs