Kubernetes – DevOps Global Solutions

Kubernetes Blog The Kubernetes blog is used by the project to communicate new features, community reports, and any news that might be relevant to the Kubernetes community.

Spotlight on SIG Storage
on June 15, 2026 at 12:00 am
In our ongoing SIG Spotlight series, we shine a light on the groups that keep the Kubernetes project moving forward. This time, we catch up with SIG Storage, the group responsible for persistent data, volume management, and the interfaces that connect Kubernetes workloads to the storage systems beneath them. We spoke with Xing Yang, Co-Chair of SIG Storage and Software Engineer at VMware by Broadcom, about the SIG’s history, the features shipping in recent Kubernetes releases, and where storage in Kubernetes is headed as AI workloads become the norm. IntroductionsCould you introduce yourself and share your role(s) within SIG Storage? My name is Xing Yang, a software engineer at VMware by Broadcom. I’m a co-chair in SIG Storage, alongside another co-chair Saad Ali from Google. There are also two Tech Leads in SIG Storage: Michelle Au from Google and Jan Šafránek from Red Hat. What first drew you to storage in Kubernetes, and how did you start contributing? I have always been working in the storage domain, so SIG Storage was a natural place for me to get started when I began to learn Kubernetes. I started attending SIG Storage meetings, trying to figure out what I could do to help. This was before the first Container Storage Interface (CSI) release — lots of things were still evolving. It was a very exciting time. What subprojects or areas do you actively maintain or review today? I’m a maintainer in Kubernetes CSI. There are multiple CSI sidecars — such as csi-provisioner, csi-attacher, csi-resizer, and csi-snapshotter — that we need to release following every Kubernetes release. I’m also a co-chair for a Data Protection Working Group co-sponsored by SIG Storage and SIG Apps. Several features have come out of that WG aimed at filling gaps in data protection support within Kubernetes. One is Volume Group Snapshot, which provides crash-consistent group snapshots for multiple volumes used by an application. Changed Block Tracking (CBT) is another critical feature from the DP WG designed to support efficient backups. About SIG StorageFor folks who are new: what is SIG Storage, in your own words? What problems in Kubernetes are you trying to solve? SIG Storage is a Special Interest Group focused on how to provide storage to containers running in your Kubernetes cluster. We define standard interfaces so that a storage vendor can write a driver and have its underlying storage system consumed by containers in Kubernetes. Why does Kubernetes need a dedicated storage SIG? What makes storage hard in a distributed system? When Kubernetes was first introduced, it was meant for stateless workloads only. Container applications were regarded as ephemeral and therefore did not need to persist data. However, that changed drastically. Stateful workloads started running in Kubernetes, and we needed a dedicated SIG to tackle the associated storage challenges. PersistentVolumeClaims, PersistentVolumes, and StorageClasses were all introduced to provision data volumes for applications running in Kubernetes. How did SIG Storage originally form, and how has its mission changed over time? SIG Storage was formed to address the challenges of handling persistent data within Kubernetes. Initially, PersistentVolumes were implemented as in-tree plugins, and the SIG managed those plugins while developing core storage primitives like PersistentVolumes and PersistentVolumeClaims. Container Storage Interface (CSI) was introduced later and played a crucial role in simplifying storage integration, enabling third-party storage providers to develop and maintain their own out-of-tree plugins without modifying Kubernetes core code. With basic integration addressed by CSI, the SIG’s mission expanded to include advanced storage features that leverage the new interface. The SIG has also expanded its scope to support object storage through the Container Object Storage Interface (COSI). Current work and roadmapWhat are the top features SIG Storage is actively working on right now? The Data Protection WG has been working on a couple of exciting features: VolumeGroupSnapshot is a Kubernetes feature enabling a crash-consistent, point-in-time snapshot of multiple PersistentVolumes simultaneously. This ensures data integrity for applications — like databases — that rely on multiple volumes by capturing all volumes in the group atomically, at the exact same point in time. It just moved to GA in Kubernetes v1.36. CSI Changed Block Tracking (CBT) enables efficient, incremental backups. By allowing storage systems to report only the blocks that have changed since the last snapshot, it significantly reduces the amount of data that needs to be transferred. It just moved to Beta in Kubernetes v1.36. Another feature worth highlighting is Container Object Storage Interface (COSI). COSI provides a standard interface for provisioning and consuming object storage buckets in Kubernetes — standardizing object storage for containerized applications much like CSI did for block and file storage. COSI is now transitioning to v1alpha2, with plans for promotion to Beta in a future release. What recent work from SIG Storage do you consider a “win” for users? The graduation of VolumeAttributesClass to GA in Kubernetes v1.34 is a major win for users managing stateful workloads. Previously, changing volume attributes like IOPS or throughput required out-of-band actions or disruptive operations. Now, users can dynamically tune storage properties such as IOPS or throughput directly through the Kubernetes API — scaling up for peak loads or down to optimize costs — without external processes or downtime. VolumeAttributesClass enables dynamic modification of storage characteristics without recreating the volume. This completes the picture by allowing users to tune both capacity and other storage properties dynamically, just as they can now tune both CPU and memory for compute. Looking ahead one or two releases, what’s on the roadmap that people should watch for? I’d like to draw attention to the Volume Health feature. This feature is designed to offer critical visibility into the operational status and integrity of persistent volumes. By enabling storage drivers and the Kubernetes control plane to report issues, it allows for proactive monitoring and identification of volume-related problems. Currently, volume health information is reported via non-persistent events. We are actively investigating enhancements to this feature with the goal of supporting automated remediation capabilities in the future. Are there areas where you’d really like more discussion or help from the community? We always need help from the community to fix bugs, add tests, and help with reviews. We’d also like to get feedback on the Alpha feature Mutable PV Affinity, which was introduced in Kubernetes v1.35. Use cases include migrating volumes from zonal to regional storage or migrating from one disk type to another. Another topic is volume replication. It was raised at KubeCon Atlanta and has been discussed in the Data Protection WG. Community members interested in this topic are encouraged to join the DP WG meetings. What are the biggest challenges users face today when running stateful workloads on Kubernetes? While Kubernetes has moved stateful workloads — like databases and AI pipelines — into the mainstream, managing “state” in a system designed for ephemerality remains difficult: Data Gravity and Storage Locality: Pods move in seconds, but data has gravity. If a node fails, a pod using local storage is stuck. Operators must decide whether the failure is transient or permanent — a high-stakes call. This is why we are enhancing the Volume Health feature to provide the visibility needed to automate recovery choices. Day 2 Complexity: Setting up a database is easy; maintaining its health over time is the real challenge. Standard Kubernetes objects like StatefulSets offer a baseline, but they lack the operational logic needed for tasks such as schema upgrades, engine patching, or cluster-wide Kubernetes upgrades. Data Mobility: Moving persistent data remains a significant hurdle — whether migrating between storage tiers, shifting workloads across availability zones, or moving to a different cluster. This challenge includes ongoing synchronization and replication for high availability and disaster recovery across a distributed system. Storage and AIHow do you see storage evolving in Kubernetes over the next few years, especially as AI/ML workloads grow? I see several trends shaping storage in Kubernetes as it evolves from a container orchestrator into the “Operating System” for AI: More Intelligent Data Management: We’ll see a shift toward smarter CSI drivers and data management tools offering advanced features like automatic tiering, snapshots, migration, and replication — optimized specifically for high-performance AI/ML workflows and large data platforms. Object Storage as a First-Class Citizen: AI datasets now frequently reach exabyte scale, making object storage the preferred choice for AI workloads. COSI is standardizing bucket management just as CSI did for disks, allowing data scientists to use a BucketClaim to provision S3-compatible storage natively and unifying object, file, and block storage into a single workflow. Performance and Low Latency: For AI/ML, storage needs to keep up with GPU processing speeds. This will accelerate adoption of high-performance parallel file systems and NVMe-over-Fabrics (NVMe-oF) technologies managed natively via Kubernetes. The line between traditional block/file and memory-speed storage will continue to blur. Data-Aware Scheduling: Instead of just considering CPU and RAM, the Kubernetes scheduler will increasingly prioritize placing Pods based on data locality — calculating the cost of moving data versus moving compute to keep massive data platforms performant. SIG Storage continues to tackle some of the hardest problems in Kubernetes: keeping stateful applications running reliably, making storage operations transparent and composable, and now scaling up to meet the demands of AI-era workloads. Whether you’re a user managing databases in production or a developer curious about storage internals, there’s a place for you in SIG Storage. If you’d like to get involved, check out the SIG Storage community page and join the bi-weekly meetings. You can also find the SIG on Slack at #sig-storage. SIG Storage Mailing List SIG Storage on Slack Data Protection WG
From Kubernetes Dashboard to Headlamp: Understanding the Transition
on June 1, 2026 at 6:00 pm
For many people, Kubernetes Dashboard was their first window into Kubernetes. It offered a simple visual way to see what was running in a cluster, inspect resources, and build confidence without relying on the command line. For years, it helped developers, students, and operators make sense of Kubernetes, and it served as an important onramp into the ecosystem. The Kubernetes Dashboard project has now been archived. We deeply respect the work the team did and the role Dashboard played in making Kubernetes more approachable for so many users. Headlamp builds on that foundation and carries it forward. It keeps the clarity of a visual interface while adding capabilities that match how Kubernetes is used today. This includes multi-cluster visibility, application-centric views, extensibility through plugins, and flexible deployment options that work both in-cluster and on the desktop. This guide is meant to help you navigate that transition with confidence. Before diving into the mechanics of migration, we start with familiar ground by looking at how common Kubernetes Dashboard workflows map to Headlamp. We also cover what stays the same and what improves after the switch. The goal is not just to replace a tool, but to honor a user-centered legacy and help you land in a UI that can grow with you as your Kubernetes usage evolves. Mapping Kubernetes Dashboard workloads to HeadlampIf you have used Kubernetes Dashboard before, many workflows in Headlamp will feel familiar. Headlamp does not introduce a new way of thinking. Instead, it builds on workloads users already know and extends them in practical ways. The focus is continuity. What worked before still works, with more room to grow. Viewing workloads and resourcesIn Kubernetes Dashboard, most users started by browsing workloads like pods, deployments, services, and namespaces. Headlamp keeps this same starting point. Workloads are easy to find and inspect, and moving between namespaces and clusters is simpler. Resources are still organized in familiar ways, and navigation feels smoother, especially when you work across multiple environments. Editing and interacting with resourcesLike Kubernetes Dashboard, Headlamp lets you view and edit manifests directly in the UI based on your permissions. You can delete resources, scale workloads, or update configurations from the interface. All actions follow standard Kubernetes RBAC. If you could perform an action in Dashboard, you will find the same capability in Headlamp, with the same respect for access controls. Understanding relationshipsWhere Headlamp begins to expand the experience is in how it presents relationships between resources. In addition to list views, Headlamp offers visual ways to see how workloads, services, and configurations connect. This helps provide context without changing the underlying workloads users already rely on. At a high level, the tasks you performed in Kubernetes Dashboard are still there. Headlamp keeps familiar workflows while making it easier to scale as clusters, teams, and applications grow. Where Headlamp goes beyond Kubernetes DashboardExpanding from single cluster to multi-cluster workflowsKubernetes Dashboard was designed to work with one cluster at a time. That model worked well for simple setups, but it became limiting as teams adopted multiple environments. Headlamp expands this view by letting you work with multiple clusters from a single interface without switching tools or losing context. This makes it easier to manage development, staging, and production environments side by side. For teams running Kubernetes in more than one place, this shift reduces friction. You can stay oriented and move between clusters with confidence. From resource lists to application context with ProjectsProjects give you an application-centered way to view Kubernetes. Instead of jumping between lists, you can group related workloads, services, and supporting resources in one place. This makes applications easier to understand. You can see what belongs together, track changes in context, and troubleshoot without scanning the cluster piece by piece. Projects are built on native Kubernetes concepts. Namespaces, labels, and RBAC continue to work the same way they always have. Headlamp adds a visual layer that brings related resources together. Projects are optional. You can still work at the individual resource level when that fits your task. When you need more context, Projects help you step back and see the bigger picture. Extend the Headlamp UI with pluginsHeadlamp can be extended through plugins that bring common workflows directly into the UI. Instead of switching tools, you work in one place with the same context. For example, the Flux plugin brings GitOps workflows into Headlamp. It allows teams to view application state alongside the Kubernetes resources that Flux manages, making it easier to understand how changes in Git relate to what is running in the cluster. The AI Assistant follows a similar pattern. It adds a conversational layer to the UI that helps users understand what they are seeing, troubleshoot issues, or take action. All of this happens in the same screen where the problem appears. Building your own pluginsPlugins are optional and not limited to community-built extensions. Platform and project teams can also create their own plugins. This allows organizations to add custom integrations that match their specific workflows and internal tooling, while keeping the user experience consistent. Choosing how and where Headlamp runsHeadlamp gives teams flexibility in how they use a Kubernetes UI. You can run it directly in a cluster, use it as a desktop application, or combine both approaches based on your needs. Running Headlamp in-cluster works well for shared environments. It provides a centrally managed UI with controlled access and fits naturally into Kubernetes setups, following the same authentication and RBAC rules as other in-cluster components. The desktop application is often a better fit for local development and onboarding. It also works well when you need to manage multiple clusters from one place. Users can connect using their existing kubeconfig without deploying anything into the cluster. These options are not mutually exclusive. Many teams use the desktop app for day-to-day work, while relying on an in-cluster deployment for shared or production environments. Preparing for the MigrationBefore moving from Kubernetes Dashboard to Headlamp, it can be helpful to pause and take stock of how you use the Dashboard today. A little reflection up front can go a long way toward making the transition feel smooth and familiar. Start by noting which clusters and namespaces you access and how authentication works. Headlamp relies on standard Kubernetes authentication and RBAC. In most cases, existing access models carry over without change. If users already connect using kubeconfig files or service accounts, they will be able to access the same resources in Headlamp. It is also useful to think about the workflows that matter most to your team. Some users rely on Dashboard for quick inspection or troubleshooting, while others use it for lightweight edits or validation. Headlamp supports these same workflows and adds optional capabilities on top. Knowing what you rely on today helps the transition feel predictable and confidence building. If you would like to explore Headlamp or try it out before migrating, you can learn more at headlamp.dev. This blog focused on understanding the transition and what to expect. A step by step migration guide is coming soon and will walk through installation and migration in detail.
Reconciling the Past: Correcting Records for Unfixed Kubernetes CVEs
on May 26, 2026 at 5:30 pm
The Kubernetes project relies on transparency to empower cluster administrators and security researchers. One important way we do that is by publishing CVE records into the Common Vulnerabilities and Exposures database. As part of our ongoing effort to mature the official Kubernetes CVE Feed, we have identified some discrepancies. CVE records for a few older, unfixed issues incorrectly include a fixed version field. The Kubernetes Security Response Committee (SRC) will correct the affected CVE records on June 1, 2026. This may result in vulnerability scanners identifying these vulnerabilities in places where they were previously not detected. To help reduce confusion, this post provides a technical update on three vulnerabilities that were disclosed in previous years but remain unfixed: CVE-2020-8561, CVE-2020-8562, and CVE-2021-25740. Why we are updating these records nowWhile these vulnerabilities have been public for several years, the recent work to generate official Open Source Vulnerabilities (OSV) files revealed that their corresponding CVE records did not accurately reflect their status. Specifically, some records suggested a fixed version existed, when in reality, these issues are architectural design trade-offs that cannot be fully remediated through code without breaking fundamental Kubernetes functionality. Correcting these records is vital for the community for: Automation Fidelity: Modern vulnerability scanners depend on precise version ranges. Inaccurate fixed tags lead to false negatives, giving users a false sense of security. Risk Documentation: By formalizing these as unfixed, we ensure that platform providers and administrators are aware of the persistent need for administrative mitigations. For completeness, we should also mention that CVE-2020-8554 is an unfixed CVE with a correct CVE record stating that it affects all versions. That record will also be updated to use a more-standardized version number format. Technical analysis of unfixed architectural risksThe following vulnerabilities will not be fixed by the Kubernetes project. GitHub issues remain the best reference for the technical mechanics of these flaws. CVE-2020-8561: Webhook redirect in kube-apiserver Severity: Medium (4.1). The Issue: The kube-apiserver follows HTTP redirects when communicating with admission webhooks. An actor capable of configuring an AdmissionWebhookConfiguration can redirect API server requests to internal, private networks. Why it remains unfixed: Restricting this behavior would require breaking the standard HTTP client behavior that many legitimate integrations rely on. Mitigation: Set the API server log level to less than 10 (to prevent logging response bodies) and disable dynamic profiling (–profiling=false) to prevent unauthorized log-level changes. CVE-2020-8562: Proxy bypass via DNS TOCTOU Severity: Low (3.1). The Issue: A Time-of-Check to Time-of-Use (TOCTOU) race condition in the API server proxy allows users to bypass IP restrictions. The system performs a DNS check to validate an IP, but then performs a second resolution for the actual connection, which an attacker can manipulate. Why it remains unfixed: Fixing this requires pinning resolved IPs in a way that breaks complex split-horizon DNS or dynamic IP environments. Mitigation: Use a local DNS caching server like dnsmasq for the API server and configure min-cache-ttl to enforce consistent responses between the check and the connection. CVE-2021-25740: Cross-namespace forwarding via Endpoints Severity: Low (3.1). The Issue: A design flaw in the Endpoints and EndpointSlice API objects allows users to manually specify IP addresses, which can be used to point a LoadBalancer or Ingress toward backends in other namespaces. Why it remains unfixed: This is a fundamental feature of the Endpoints API used by many networking tools and operators. Mitigation: Restrict write access to Endpoints (legacy) and EndpointSlices. Since Kubernetes 1.22, Kubernetes RBAC authorization mode no longer includes those permissions in the default edit and admin ClusterRoles. That removal applies to clusters created using Kubernetes v1.22; for clusters upgraded from older versions, administrators should manually audit and reconcile the system:aggregate-to-edit ClusterRole. Note:On June 1, 2026, these CVE records will be updated to correctly reflect the fact that all versions are affected. You may see them begin to appear in vulnerability scanner results. Required actions for administratorsThe Kubernetes project recommends a secure by configuration approach to manage these persistent risks: Vulnerability Action item Severity score (Rating) Command / configuration CVE-2020-8561 Restrict Log Verbosity 4.1 (Medium) Ensure –v is set to < 10 and –profiling=false. CVE-2020-8562 Enforce DNS Consistency 3.1 (Low) Deploy dnsmasq or a similar caching resolver on control plane nodes. CVE-2021-25740 Hardened RBAC 3.1 (Low) kubectl auth reconcile to remove Endpoints write access from broad roles. The RBAC action for CVE-2021-25740 applies when your cluster uses RBAC authorization mode, which is the default for clusters created with standard Kubernetes tooling. Administrators should independently test and validate these configurations in a non-production environment, assessing the architectural risks against their specific threat model and risk tolerance. Conclusion: maturity through transparencyThe effort to reconcile these records is a sign of a maturing security ecosystem. By moving away from the “patch-only” mindset and accurately documenting architectural debt, the Kubernetes project provides the community with the high-fidelity data needed to secure modern cloud native infrastructure. We would like to thank the security researchers—QiQi Xu, Javier Provecho, and others—who identified these risks, and the SIG Security Tooling contributors who continue to refine our official feeds. Special shoutout to Rory McCune for sharing information around these CVEs through his blog posts. Update 2026/06/01: Today, the Kubernetes SRC has updated the CVE records for CVE-2020-8554, CVE-2020-8561, CVE-2020-8562, and CVE-2021-25740.
Announcing etcd 3.7.0-beta.0
on May 20, 2026 at 12:00 am
SIG-Etcd announces the availability of the first beta release of etcd v3.7.0. This new version of the popular distributed database and key Kubernetes component includes the long-requested RangeStream feature, as well as a refactoring and cleanup of multiple legacy components and interfaces. v3.7 will deliver improved security, better operational reliability, and an improved experience for working with large resultsets. First, however, the project needs users to test the beta. You can find v3.7.0-beta.0 here: Source code Binaries Official container images Please try it out and report issues in the etcd repo. This beta also determines the EOL of version 3.4. RangeStreamIn etcd v3.6 and earlier, it is challenging to work with requests that return large resultsets. The client or requesting application is forced to wait for the full result set, leading to unpredictable latency and memory usage. The RangeStream RPC lets calling applications accept result sets in chunks, reducing latency and making buffering memory usage more predictable. Much of the work on RangeStream was done by a relatively new contributor to etcd, Jeffrey Ying, a software engineer at Google. New contributors can have a substantial impact on etcd development. “I’ve always been fascinated by database internals, and building RangeStream was a great opportunity to solve a bottleneck we were hitting in production with Kubernetes. It was the perfect opportunity to collaborate across projects and improve the ecosystem as a whole. Jumping into etcd as a new contributor had a bit of a learning curve, but the community is incredibly welcoming. The leads were very receptive to my ideas and helped me iterate quickly, while maintaining the project’s high bar for reliability and code quality,” said Jeffrey. Instructions on how to use RangeStream in gRPC calls and in etcdctl can be found in the etcd documentation. Users should try it out for their own applications. Removal of v2storeThe last vestiges of etcd v2store have been removed in v3.7, making this the first release that is 100% on v3store. This includes discovery, bootstrap, v2 requests, and the v2 client. Our team has also removed multiple deprecated experimental flags. All of these changes may create some breakage for users, particularly those who have not already updated to v3.6.11. We are interested in hearing about blockers encountered by users and dependent applications; please report anything you find that can’t be remedied or needs better upgrade documentation. etcd v3.7.0-beta.0 also includes bbolt v1.5.0 and raft v3.7.0. 3.4 EOLAccording to our community support policy, we typically maintain only the latest two minor versions, currently v3.6 and v3.5. Etcd v3.5 will be supported for 1 year after v3.7.0 final release. As mentioned in extended support for v3.4 in the etcd v3.6.0 release announcement, etcd v3.4 has been EOL since May 15, 2026. SIG-etcd may release one more security patch for that version at the end of May, if warranted by patched vulnerabilities. In any case, it will cease being updated after the end of May. Users on v3.4 should be planning to upgrade their clusters. Feedback and Future BetasReach the etcd contributors with your feedback about v3.7.0-beta.0 in any of the following places: Github issues #SIG-etcd slack channel in Kubernetes Slack etcd-dev mailing list SIG-etcd may release additional betas of version v3.7.0 with additional refactoring, particularly of our use of protobuf libraries. Release candidates and the final release will probably happen through June, possibly into early July.
Kubernetes v1.36: New Metric for Route Sync in the Cloud Controller Manager
on May 15, 2026 at 6:35 pm
This article was originally published with the wrong date. It was later republished, dated the 15th of May 2026. Kubernetes v1.36 introduces a new alpha counter metric route_controller_route_sync_total to the Cloud Controller Manager (CCM) route controller implementation at k8s.io/cloud-provider. This metric increments each time routes are synced with the cloud provider. A/B testing watch-based route reconciliationThis metric was added to help operators validate the CloudControllerManagerWatchBasedRoutesReconciliation feature gate introduced in Kubernetes v1.35. That feature gate switches the route controller from a fixed-interval loop to a watch-based approach that only reconciles when nodes actually change. This reduces unnecessary API calls to the infrastructure provider, lowering pressure on rate-limited APIs and allowing operators to make more efficient use of their available quota. To A/B test this, compare route_controller_route_sync_total with the feature gate disabled (default) versus enabled. In clusters where node changes are infrequent, you should see a significant drop in the sync rate with the feature gate turned on. Example: expected behaviorWith the feature gate disabled (the default fixed-interval loop), the counter increments steadily regardless of whether any node changes occurred: # After 10 minutes with no node changes route_controller_route_sync_total 60 # After 20 minutes, still no node changes route_controller_route_sync_total 120 With the feature gate enabled (watch-based reconciliation), the counter only increments when nodes are actually added, removed, or updated: # After 10 minutes with no node changes route_controller_route_sync_total 1 # After 20 minutes, still no node changes — counter unchanged route_controller_route_sync_total 1 # A new node joins the cluster — counter increments route_controller_route_sync_total 2 The difference is especially visible in stable clusters where nodes rarely change. Where can I give feedback?If you have feedback, feel free to reach out through any of the following channels: The #sig-cloud-provider channel on Kubernetes Slack The KEP-5237 issue on GitHub The SIG Cloud Provider community page for other communication channels How can I learn more?For more details, refer to KEP-5237.