on September 12, 2025 at 6:30 pm

Historically, configuring the correct cgroup driver has been a pain point for users running new Kubernetes clusters. On Linux systems, there are two different cgroup drivers: cgroupfs and systemd. In the past, both the kubelet and CRI implementation (like CRI-O or containerd) needed to be configured to use the same cgroup driver, or else the kubelet would misbehave without any explicit error message. This was a source of headaches for many cluster admins. Now, we’ve (almost) arrived at the end of that headache. Automated cgroup driver detection In v1.28.0, the SIG Node community introduced the feature gate KubeletCgroupDriverFromCRI, which instructs the kubelet to ask the CRI implementation which cgroup driver to use. You can read more here. After many releases of waiting for each CRI implementation to have major versions released and packaged in major operating systems, this feature has gone GA as of Kubernetes 1.34.0. In addition to setting the feature gate, a cluster admin needs to ensure their CRI implementation is new enough: containerd: Support was added in v2.0.0 CRI-O: Support was added in v1.28.0 Announcement: Kubernetes is deprecating containerd v1.y support While CRI-O releases versions that match Kubernetes versions, and thus CRI-O versions without this behavior are no longer supported, containerd maintains its own release cycle. containerd support for this feature is only in v2.0 and later, but Kubernetes 1.34 still supports containerd 1.7 and other LTS releases of containerd. The Kubernetes SIG Node community has formally agreed upon a final support timeline for containerd v1.y. The last Kubernetes release to offer this support will be the last released version of v1.35, and support will be dropped in v1.36.0. To assist administrators in managing this future transition, a new detection mechanism is available. You are able to monitor the kubelet_cri_losing_support metric to determine if any nodes in your cluster are using a containerd version that will soon be outdated. The presence of this metric with a version label of 1.36.0 will indicate that the node’s containerd runtime is not new enough for the upcoming requirements. Consequently, an administrator will need to upgrade containerd to v2.0 or a later version before, or at the same time as, upgrading the kubelet to v1.36.0.

on September 11, 2025 at 6:30 pm

The functionality for CSI drivers to update information about attachable volume count on the nodes, first introduced as Alpha in Kubernetes v1.33, has graduated to Beta in the Kubernetes v1.34 release! This marks a significant milestone in enhancing the accuracy of stateful pod scheduling by reducing failures due to outdated attachable volume capacity information. Background Traditionally, Kubernetes CSI drivers report a static maximum volume attachment limit when initializing. However, actual attachment capacities can change during a node’s lifecycle for various reasons, such as: Manual or external operations attaching/detaching volumes outside of Kubernetes control. Dynamically attached network interfaces or specialized hardware (GPUs, NICs, etc.) consuming available slots. Multi-driver scenarios, where one CSI driver’s operations affect available capacity reported by another. Static reporting can cause Kubernetes to schedule pods onto nodes that appear to have capacity but don’t, leading to pods stuck in a ContainerCreating state. Dynamically adapting CSI volume limits With this new feature, Kubernetes enables CSI drivers to dynamically adjust and report node attachment capacities at runtime. This ensures that the scheduler, as well as other components relying on this information, have the most accurate, up-to-date view of node capacity. How it works Kubernetes supports two mechanisms for updating the reported node volume limits: Periodic Updates: CSI drivers specify an interval to periodically refresh the node’s allocatable capacity. Reactive Updates: An immediate update triggered when a volume attachment fails due to exhausted resources (ResourceExhausted error). Enabling the feature To use this beta feature, the MutableCSINodeAllocatableCount feature gate must be enabled in these components: kube-apiserver kubelet Example CSI driver configuration Below is an example of configuring a CSI driver to enable periodic updates every 60 seconds: apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: name: example.csi.k8s.io spec: nodeAllocatableUpdatePeriodSeconds: 60 This configuration directs kubelet to periodically call the CSI driver’s NodeGetInfo method every 60 seconds, updating the node’s allocatable volume count. Kubernetes enforces a minimum update interval of 10 seconds to balance accuracy and resource usage. Immediate updates on attachment failures When a volume attachment operation fails due to a ResourceExhausted error (gRPC code 8), Kubernetes immediately updates the allocatable count instead of waiting for the next periodic update. The Kubelet then marks the affected pods as Failed, enabling their controllers to recreate them. This prevents pods from getting permanently stuck in the ContainerCreating state. Getting started To enable this feature in your Kubernetes v1.34 cluster: Enable the feature gate MutableCSINodeAllocatableCount on the kube-apiserver and kubelet components. Update your CSI driver configuration by setting nodeAllocatableUpdatePeriodSeconds. Monitor and observe improvements in scheduling accuracy and pod placement reliability. Next steps This feature is currently in beta and the Kubernetes community welcomes your feedback. Test it, share your experiences, and help guide its evolution to GA stability. Join discussions in the Kubernetes Storage Special Interest Group (SIG-Storage) to shape the future of Kubernetes storage capabilities.

on September 10, 2025 at 6:30 pm

Kubernetes typically uses ConfigMaps and Secrets to set environment variables, which introduces additional API calls and complexity, For example, you need to separately manage the Pods of your workloads and their configurations, while ensuring orderly updates for both the configurations and the workload Pods. Alternatively, you might be using a vendor-supplied container that requires environment variables (such as a license key or a one-time token), but you don’t want to hard-code them or mount volumes just to get the job done. If that’s the situation you are in, you now have a new (alpha) way to achieve that. Provided you have the EnvFiles feature gate enabled across your cluster, you can tell the kubelet to load a container’s environment variables from a volume (the volume must be part of the Pod that the container belongs to). this feature gate allows you to load environment variables directly from a file in an emptyDir volume without actually mounting that file into the container. It’s a simple yet elegant solution to some surprisingly common problems. What’s this all about? At its core, this feature allows you to point your container to a file, one generated by an initContainer, and have Kubernetes parse that file to set your environment variables. The file lives in an emptyDir volume (a temporary storage space that lasts as long as the pod does), Your main container doesn’t need to mount the volume. The kubelet will read the file and inject these variables when the container starts. How It Works Here’s a simple example: apiVersion: v1 kind: Pod spec: initContainers: – name: generate-config image: busybox command: [‘sh’, ‘-c’, ‘echo “CONFIG_VAR=HELLO” > /config/config.env’] volumeMounts: – name: config-volume mountPath: /config containers: – name: app-container image: gcr.io/distroless/static env: – name: CONFIG_VAR valueFrom: fileKeyRef: path: config.env volumeName: config-volume key: CONFIG_VAR volumes: – name: config-volume emptyDir: {} Using this approach is a breeze. You define your environment variables in the pod spec using the fileKeyRef field, which tells Kubernetes where to find the file and which key to pull. The file itself resembles the standard for .env syntax (think KEY=VALUE), and (for this alpha stage at least) you must ensure that it is written into an emptyDir volume. Other volume types aren’t supported for this feature. At least one init container must mount that emptyDir volume (to write the file), but the main container doesn’t need to—it just gets the variables handed to it at startup. A word on security While this feature supports handling sensitive data such as keys or tokens, note that its implementation relies on emptyDir volumes mounted into pod. Operators with node filesystem access could therefore easily retrieve this sensitive data through pod directory paths. If storing sensitive data like keys or tokens using this feature, ensure your cluster security policies effectively protect nodes against unauthorized access to prevent exposure of confidential information. Summary This feature will eliminate a number of complex workarounds used today, simplifying apps authoring, and opening doors for more use cases. Kubernetes stays flexible and open for feedback. Tell us how you use this feature or what is missing.

on September 9, 2025 at 6:30 pm

For years, the Kubernetes community has been on a mission to improve the stability and performance predictability of the API server. A major focus of this effort has been taming list requests, which have historically been a primary source of high memory usage and heavy load on the etcd datastore. With each release, we’ve chipped away at the problem, and today, we’re thrilled to announce the final major piece of this puzzle. The snapshottable API server cache feature has graduated to Beta in Kubernetes v1.34, culminating a multi-release effort to allow virtually all read requests to be served directly from the API server’s cache. Evolving the cache for performance and stability The path to the current state involved several key enhancements over recent releases that paved the way for today’s announcement. Consistent reads from cache (Beta in v1.31) While the API server has long used a cache for performance, a key milestone was guaranteeing consistent reads of the latest data from it. This v1.31 enhancement allowed the watch cache to be used for strongly-consistent read requests for the first time, a huge win as it enabled filtered collections (e.g. “a list of pods bound to this node”) to be safely served from the cache instead of etcd, dramatically reducing its load for common workloads. Taming large responses with streaming (Beta in v1.33) Another key improvement was tackling the problem of memory spikes when transmitting large responses. The streaming encoder, introduced in v1.33, allowed the API server to send list items one by one, rather than buffering the entire multi-gigabyte response in memory. This made the memory cost of sending a response predictable and minimal, regardless of its size. The missing piece Despite these huge improvements, a critical gap remained. Any request for a historical LIST—most commonly used for paginating through large result sets—still had to bypass the cache and query etcd directly. This meant that the cost of retrieving the data was still unpredictable and could put significant memory pressure on the API server. Kubernetes 1.34: snapshots complete the picture The snapshottable API server cache solves this final piece of the puzzle. This feature enhances the watch cache, enabling it to generate efficient, point-in-time snapshots of its state. Here’s how it works: for each update, the cache creates a lightweight snapshot. These snapshots are “lazy copies,” meaning they don’t duplicate objects but simply store pointers, making them incredibly memory-efficient. When a list request for a historical resourceVersion arrives, the API server now finds the corresponding snapshot and serves the response directly from its memory. This closes the final major gap, allowing paginated requests to be served entirely from the cache. A new era of API Server performance 🚀 With this final piece in place, the synergy of these three features ushers in a new era of API server predictability and performance: Get Data from Cache: Consistent reads and snapshottable cache work together to ensure nearly all read requests—whether for the latest data or a historical snapshot—are served from the API server’s memory. Send data via stream: Streaming list responses ensure that sending this data to the client has a minimal and constant memory footprint. The result is a system where the resource cost of read operations is almost fully predictable and much more resiliant to spikes in request load. This means dramatically reduced memory pressure, a lighter load on etcd, and a more stable, scalable, and reliable control plane for all Kubernetes clusters. How to get started With its graduation to Beta, the SnapshottableCache feature gate is enabled by default in Kubernetes v1.34. There are no actions required to start benefiting from these performance and stability improvements. Acknowledgements Special thanks for designing, implementing, and reviewing these critical features go to: Ahmad Zolfaghari (@ah8ad3) Ben Luddy (@benluddy) – Red Hat Chen Chen (@z1cheng) – Microsoft Davanum Srinivas (@dims) – Nvidia David Eads (@deads2k) – Red Hat Han Kang (@logicalhan) – CoreWeave haosdent (@haosdent) – Shopee Joe Betz (@jpbetz) – Google Jordan Liggitt (@liggitt) – Google Łukasz Szaszkiewicz (@p0lyn0mial) – Red Hat Maciej Borsz (@mborsz) – Google Madhav Jivrajani (@MadhavJivrajani) – UIUC Marek Siarkowicz (@serathius) – Google NKeert (@NKeert) Tim Bannister (@lmktfy) Wei Fu (@fuweid) – Microsoft Wojtek Tyczyński (@wojtek-t) – Google …and many others in SIG API Machinery. This milestone is a testament to the community’s dedication to building a more scalable and robust Kubernetes.

on September 8, 2025 at 6:30 pm

The VolumeAttributesClass API, which empowers users to dynamically modify volume attributes, has officially graduated to General Availability (GA) in Kubernetes v1.34. This marks a significant milestone, providing a robust and stable way to tune your persistent storage directly within Kubernetes. What is VolumeAttributesClass? At its core, VolumeAttributesClass is a cluster-scoped resource that defines a set of mutable parameters for a volume. Think of it as a “profile” for your storage, allowing cluster administrators to expose different quality-of-service (QoS) levels or performance tiers. Users can then specify a volumeAttributesClassName in their PersistentVolumeClaim (PVC) to indicate which class of attributes they desire. The magic happens through the Container Storage Interface (CSI): when a PVC referencing a VolumeAttributesClass is updated, the associated CSI driver interacts with the underlying storage system to apply the specified changes to the volume. This means you can now: Dynamically scale performance: Increase IOPS or throughput for a busy database, or reduce it for a less critical application. Optimize costs: Adjust attributes on the fly to match your current needs, avoiding over-provisioning. Simplify operations: Manage volume modifications directly within the Kubernetes API, rather than relying on external tools or manual processes. What is new from Beta to GA There are two major enhancements from beta. Cancel support from infeasible errors To improve resilience and user experience, the GA release introduces explicit cancel support when a requested volume modification becomes infeasible. If the underlying storage system or CSI driver indicates that the requested changes cannot be applied (e.g., due to invalid arguments), users can cancel the operation and revert the volume to its previous stable configuration, preventing the volume from being left in an inconsistent state. Quota support based on scope While VolumeAttributesClass doesn’t add a new quota type, the Kubernetes control plane can be configured to enforce quotas on PersistentVolumeClaims that reference a specific VolumeAttributesClass. This is achieved by using the scopeSelector field in a ResourceQuota to target PVCs that have .spec.volumeAttributesClassName set to a particular VolumeAttributesClass name. Please see more details here. Drivers support VolumeAttributesClass Amazon EBS CSI Driver: The AWS EBS CSI driver has robust support for VolumeAttributesClass and allows you to modify parameters like volume type (e.g., gp2 to gp3, io1 to io2), IOPS, and throughput of EBS volumes dynamically. Google Compute Engine (GCE) Persistent Disk CSI Driver (pd.csi.storage.gke.io): This driver also supports dynamic modification of persistent disk attributes, including IOPS and throughput, via VolumeAttributesClass. Contact For any inquiries or specific questions related to VolumeAttributesClass, please reach out to the SIG Storage community.