on March 30, 2026 at 12:00 am

Kubernetes v1.36 is coming at the end of April 2026. This release will include removals and deprecations, and it is packed with an impressive number of enhancements. Here are some of the features we are most excited about in this cycle! Please note that this information reflects the current state of v1.36 development and may change before release. The Kubernetes API removal and deprecation process The Kubernetes project has a well-documented deprecation policy for features. This policy states that stable APIs may only be deprecated when a newer, stable version of that same API is available and that APIs have a minimum lifetime for each stability level. A deprecated API has been marked for removal in a future Kubernetes release. It will continue to function until removal (at least one year from the deprecation), but usage will result in a warning being displayed. Removed APIs are no longer available in the current version, at which point you must migrate to using the replacement. Generally available (GA) or stable API versions may be marked as deprecated but must not be removed within a major version of Kubernetes. Beta or pre-release API versions must be supported for 3 releases after the deprecation. Alpha or experimental API versions may be removed in any release without prior deprecation notice; this process can become a withdrawal in cases where a different implementation for the same feature is already in place. Whether an API is removed as a result of a feature graduating from beta to stable, or because that API simply did not succeed, all removals comply with this deprecation policy. Whenever an API is removed, migration options are communicated in the deprecation guide. A recent example of this principle in action is the retirement of the ingress-nginx project, announced by SIG-Security on March 24, 2026. As stewardship shifts away from the project, the community has been encouraged to evaluate alternative ingress controllers that align with current security and maintenance best practices. This transition reflects the same lifecycle discipline that underpins Kubernetes itself, ensuring continued evolution without abrupt disruption. Ingress NGINX retirement To prioritize the safety and security of the ecosystem, Kubernetes SIG Network and the Security Response Committee have retired Ingress NGINX on March 24, 2026. Since that date, there have been no further releases, no bugfixes, and no updates to resolve any security vulnerabilities discovered. Existing deployments of Ingress NGINX will continue to function, and installation artifacts like Helm charts and container images will remain available. For full details, see the official retirement announcement. Deprecations and removals for Kubernetes v1.36 Deprecation of .spec.externalIPs in Service The externalIPs field in Service spec is being deprecated, which means you’ll soon lose a quick way to route arbitrary externalIPs to your Services. This field has been a known security headache for years, enabling man-in-the-middle attacks on your cluster traffic, as documented in CVE-2020-8554. From Kubernetes v1.36 and onwards, you will see deprecation warnings when using it, with full removal planned for v1.43. If your Services still lean on externalIPs, consider using LoadBalancer services for cloud-managed ingress, NodePort for simple port exposure, or Gateway API for a more flexible and secure way to handle external traffic. For more details on this enhancement, refer to KEP-5707: Deprecate service.spec.externalIPs Removal of gitRepo volume driver The gitRepo volume type has been deprecated since v1.11. Starting Kubernetes v1.36, the gitRepo volume plugin is permanently disabled and cannot be turned back on. This change protects clusters from a critical security issue where using gitRepo could let an attacker run code as root on the node. Although gitRepo has been deprecated for years and better alternatives have been recommended, it was still technically possible to use it in previous releases. From v1.36 onward, that path is closed for good, so any existing workloads depending on gitRepo will need to migrate to supported approaches such as init containers or external git-sync style tools. For more details on this enhancement, refer to KEP-5040: Remove gitRepo volume driver Featured enhancements of Kubernetes v1.36 The following list of enhancements is likely to be included in the upcoming v1.36 release. This is not a commitment and the release content is subject to change. Faster SELinux labelling for volumes (GA) Kubernetes v1.36 makes the SELinux volume mounting improvement generally available. This change replaced recursive file relabeling with mount -o context=XYZ option, applying the correct SELinux label to the entire volume at mount time. It brings more consistent performance and reduces Pod startup delays on SELinux-enforcing systems. This feature was introduced as beta in v1.28 for ReadWriteOncePod volumes. In v1.32, it gained metrics and an opt-out option (securityContext.seLinuxChangePolicy: Recursive) to help catch conflicts. Now in v1.36, it reaches stable and defaults to all volumes, with Pods or CSIDrivers opting in via spec.SELinuxMount. However, we expect this feature to create the risk of breaking changes in the future Kubernetes releases, due to the potential for mixing of privileged and unprivileged pods. Setting the seLinuxChangePolicy field and SELinux volume labels on Pods, correctly, is the responsibility of the Pod author Developers have that responsibility whether they are writing a Deployment, StatefulSet, DaemonSet or even a custom resource that includes a Pod template. Being careless with these settings can lead to a range of problems when Pods share volumes. For more details on this enhancement, refer to KEP-1710: Speed up recursive SELinux label change External signing of ServiceAccount tokens As a beta feature, Kubernetes already supports external signing of ServiceAccount tokens. This allows clusters to integrate with external key management systems or signing services instead of relying only on internally managed keys. With this enhancement, the kube-apiserver can delegate token signing to external systems such as cloud key management services or hardware security modules. This improves security and simplifies key management services for clusters that rely on centralized signing infrastructure. We expect that this will graduate to stable (GA) in Kubernetes v1.36. For more details on this enhancement, refer to KEP-740: Support external signing of service account tokens DRA Driver support for Device taints and tolerations Kubernetes v1.33 introduced support for taints and tolerations for physical devices managed through Dynamic Resource Allocation (DRA). Normally, any device can be used for scheduling. However, this enhancement allows DRA drivers to mark devices as tainted, which ensures that they will not be used for scheduling purposes. Alternatively, cluster administrators can create a DeviceTaintRule to mark devices that match a certain selection criteria(such as all devices of a certain driver) as tainted. This improves scheduling control and helps ensure that specialized hardware resources are only used by workloads that explicitly request them. In Kubernetes v1.36, this feature graduates to beta with more comprehensive testing complete, making it accessible by default without the need for a feature flag and open to user feedback. To learn about taints and tolerations, see taints and tolerations. For more details on this enhancement, refer to KEP-5055: DRA: device taints and tolerations. DRA support for partitionable devices Kubernetes v1.36 expands Dynamic Resource Allocation (DRA) by introducing support for partitionable devices, allowing a single hardware accelerator to be split into multiple logical units that can be shared across workloads. This is especially useful for high-cost resources like GPUs, where dedicating an entire device to a single workload can lead to underutilization. With this enhancement, platform teams can improve overall cluster efficiency by allocating only the required portion of a device to each workload, rather than reserving it entirely. This makes it easier to run multiple workloads on the same hardware while maintaining isolation and control, helping organizations get more value out of their infrastructure. To learn more about this enhancement, refer to KEP-4815: DRA Partitionable Devices Want to know more? New features and deprecations are also announced in the Kubernetes release notes. We will formally announce what’s new in Kubernetes v1.36 as part of the CHANGELOG for that release. Kubernetes v1.36 release is planned for Wednesday, April 22, 2026. Stay tuned for updates! You can also see the announcements of changes in the release notes for: Kubernetes v1.35 Kubernetes v1.34 Kubernetes v1.33 Kubernetes v1.32 Kubernetes v1.31 Kubernetes v1.30 Get involved The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests. Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly community meeting, and through the channels below. Thank you for your continued feedback and support. Follow us on Bluesky @kubernetes.io for the latest updates Join the community discussion on Discuss Join the community on Slack Post questions (or answer questions) on Server Fault or Stack Overflow Share your Kubernetes story Read more about what’s happening with Kubernetes on the blog Learn more about the Kubernetes Release Team

on March 20, 2026 at 7:00 pm

With the Ingress-NGINX retirement scheduled for March 2026, the Kubernetes networking landscape is at a turning point. For most organizations, the question isn’t whether to migrate to Gateway API, but how to do so safely. Migrating from Ingress to Gateway API is a fundamental shift in API design. Gateway API provides a modular, extensible API with strong support for Kubernetes-native RBAC. Conversely, the Ingress API is simple, and implementations such as Ingress-NGINX extend the API through esoteric annotations, ConfigMaps, and CRDs. Migrating away from Ingress controllers such as Ingress-NGINX presents the daunting task of capturing all the nuances of the Ingress controller, and mapping that behavior to Gateway API. Ingress2Gateway is an assistant that helps teams confidently move from Ingress to Gateway API. It translates Ingress resources/manifests along with implementation-specific annotations to Gateway API while warning you about untranslatable configuration and offering suggestions. Today, SIG Network is proud to announce the 1.0 release of Ingress2Gateway. This milestone represents a stable, tested migration assistant for teams ready to modernize their networking stack. Ingress2Gateway 1.0 Ingress-NGINX annotation support The main improvement for the 1.0 release is more comprehensive Ingress-NGINX support. Before the 1.0 release, Ingress2Gateway only supported three Ingress-NGINX annotations. For the 1.0 release, Ingress2Gateway supports over 30 common annotations (CORS, backend TLS, regex matching, path rewrite, etc.). Comprehensive integration testing Each supported Ingress-NGINX annotation, and representative combinations of common annotations, is backed by controller-level integration tests that verify the behavioral equivalence of the Ingress-NGINX configuration and the generated Gateway API. These tests exercise real controllers in live clusters and compare runtime behavior (routing, redirects, rewrites, etc.), not just YAML structure. The tests: spin up an Ingress-NGINX controller spin up multiple Gateway API controllers apply Ingress resources that have implementation-specific configuration translate Ingress resources to Gateway API with ingress2gateway and apply generated manifests verify that the Gateway API controllers and the Ingress controller exhibit equivalent behavior. A comprehensive test suite not only catches bugs in development, but also ensures the correctness of the translation, especially given surprising edge cases and unexpected defaults, so that you don’t find out about them in production. Notification & error handling Migration is not a “one-click” affair. Surfacing subtleties and untranslatable behavior is as important as translating supported configuration. The 1.0 release cleans up the formatting and content of notifications, so it is clear what is missing and how you can fix it. Using Ingress2Gateway Ingress2Gateway is a migration assistant, not a one-shot replacement. Its goal is to migrate supported Ingress configuration and behavior identify unsupported configuration and suggest alternatives reevaluate and potentially discard undesirable configuration The rest of the section shows you how to safely migrate the following Ingress-NGINX configuration apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/proxy-body-size: “1G” nginx.ingress.kubernetes.io/use-regex: “true” nginx.ingress.kubernetes.io/proxy-send-timeout: “1” nginx.ingress.kubernetes.io/proxy-read-timeout: “1” nginx.ingress.kubernetes.io/enable-cors: “true” nginx.ingress.kubernetes.io/configuration-snippet: | more_set_headers “Request-Id: $req_id”; name: my-ingress namespace: my-ns spec: ingressClassName: nginx rules: – host: my-host.example.com http: paths: – backend: service: name: website-service port: number: 80 path: /users/(\d+) pathType: ImplementationSpecific tls: – hosts: – my-host.example.com secretName: my-secret 1. Install Ingress2Gateway If you have a Go environment set up, you can install Ingress2Gateway with go install github.com/kubernetes-sigs/ingress2gateway@v1.0.0 Otherwise, brew install ingress2gateway You can also download the binary from GitHub or build from source. 2. Run Ingress2Gateway You can pass Ingress2Gateway Ingress manifests, or have the tool read directly from your cluster. # Pass it files ingress2gateway print –input-file my-manifest.yaml,my-other-manifest.yaml –providers=ingress-nginx > gwapi.yaml # Use a namespace in your cluster ingress2gateway print –namespace my-api –providers=ingress-nginx > gwapi.yaml # Or your whole cluster ingress2gateway print –providers=ingress-nginx –all-namespaces > gwapi.yaml Note:You can also pass –emitter <agentgateway|envoy-gateway|kgateway> to output implementation-specific extensions. 3. Review the output This is the most critical step. The commands from the previous section output a Gateway API manifest to gwapi.yaml, and they also emit warnings that explain what did not translate exactly and what to review manually. apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: gateway.networking.k8s.io/generator: ingress2gateway-dev name: nginx namespace: my-ns spec: gatewayClassName: nginx listeners: – hostname: my-host.example.com name: my-host-example-com-http port: 80 protocol: HTTP – hostname: my-host.example.com name: my-host-example-com-https port: 443 protocol: HTTPS tls: certificateRefs: – group: “” kind: Secret name: my-secret — apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: annotations: gateway.networking.k8s.io/generator: ingress2gateway-dev name: my-ingress-my-host-example-com namespace: my-ns spec: hostnames: – my-host.example.com parentRefs: – name: nginx port: 443 rules: – backendRefs: – name: website-service port: 80 filters: – cors: allowCredentials: true allowHeaders: – DNT – Keep-Alive – User-Agent – X-Requested-With – If-Modified-Since – Cache-Control – Content-Type – Range – Authorization allowMethods: – GET – PUT – POST – DELETE – PATCH – OPTIONS allowOrigins: – ‘*’ maxAge: 1728000 type: CORS matches: – path: type: RegularExpression value: (?i)/users/(\d+).* name: rule-0 timeouts: request: 10s — apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: annotations: gateway.networking.k8s.io/generator: ingress2gateway-dev name: my-ingress-my-host-example-com-ssl-redirect namespace: my-ns spec: hostnames: – my-host.example.com parentRefs: – name: nginx port: 80 rules: – filters: – requestRedirect: scheme: https statusCode: 308 type: RequestRedirect Ingress2Gateway successfully translated some annotations into their Gateway API equivalents. For example, the nginx.ingress.kubernetes.io/enable-cors annotation was translated into a CORS filter. But upon closer inspection, the nginx.ingress.kubernetes.io/proxy-{read,send}-timeout and nginx.ingress.kubernetes.io/proxy-body-size annotations do not map perfectly. The logs show the reason for these omissions as well as reasoning behind the translation. ┌─ WARN ──────────────────────────────────────── │ Unsupported annotation nginx.ingress.kubernetes.io/configuration-snippet │ source: INGRESS-NGINX │ object: Ingress: my-ns/my-ingress └─ ┌─ INFO ──────────────────────────────────────── │ Using case-insensitive regex path matches. You may want to change this. │ source: INGRESS-NGINX │ object: HTTPRoute: my-ns/my-ingress-my-host-example-com └─ ┌─ WARN ──────────────────────────────────────── │ ingress-nginx only supports TCP-level timeouts; i2gw has made a best-effort translation to Gateway API timeouts.request. Please verify that this meets your needs. See documentation: https://gateway-api.sigs.k8s.io/guides/http-timeouts/ │ source: INGRESS-NGINX │ object: HTTPRoute: my-ns/my-ingress-my-host-example-com └─ ┌─ WARN ──────────────────────────────────────── │ Failed to apply my-ns.my-ingress.metadata.annotations.”nginx.ingress.kubernetes.io/proxy-body-size” from my-ns/my-ingress: Most Gateway API implementations have reasonable body size and buffering defaults │ source: STANDARD_EMITTER │ object: HTTPRoute: my-ns/my-ingress-my-host-example-com └─ ┌─ WARN ──────────────────────────────────────── │ Gateway API does not support configuring URL normalization (RFC 3986, Section 6). Please check if this matters for your use case and consult implementation-specific details. │ source: STANDARD_EMITTER └─ There is a warning that Ingress2Gateway does not support the nginx.ingress.kubernetes.io/configuration-snippet annotation. You will have to check your Gateway API implementation documentation to see if there is a way to achieve equivalent behavior. The tool also notified us that Ingress-NGINX regex matches are case-insensitive prefix matches, which is why there is a match pattern of (?i)/users/(\d+).*. Most organizations will want to change this behavior to be an exact case-sensitive match by removing the leading (?i) and the trailing .* from the path pattern. Ingress2Gateway made a best-effort translation from the nginx.ingress.kubernetes.io/proxy-{send,read}-timeout annotations to a 10 second request timeout in our HTTP route. If requests for this service should be much shorter, say 3 seconds, you can make the corresponding changes to your Gateway API manifests. Also, nginx.ingress.kubernetes.io/proxy-body-size does not have a Gateway API equivalent, and was thus not translated. However, most Gateway API implementations have reasonable defaults for maximum body size and buffering, so this might not be a problem in practice. Further, some emitters might offer support for this annotation through implementation-specific extensions. For example, adding the –emitter agentgateway, –emitter envoy-gateway, or –emitter kgateway flag to the previous ingress2gateway print command would have resulted in additional implementation-specific configuration in the generated Gateway API manifests that attempted to capture the body size configuration. We also see a warning about URL normalization. Gateway API implementations such as Agentgateway, Envoy Gateway, Kgateway, and Istio have some level of URL normalization, but the behavior varies across implementations and is not configurable through standard Gateway API. You should check and test the URL normalization behavior of your Gateway API implementation to ensure it is compatible with your use case. To match Ingress-NGINX default behavior, Ingress2Gateway also added a listener on port 80 and a HTTP Request redirect filter to redirect HTTP traffic to HTTPS. You may not want to serve HTTP traffic at all and remove the listener on port 80 and the corresponding HTTPRoute. Caution:Always thoroughly review the generated output and logs. After manually applying these changes, the Gateway API manifests might look as follows. — apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: gateway.networking.k8s.io/generator: ingress2gateway-dev name: nginx namespace: my-ns spec: gatewayClassName: nginx listeners: – hostname: my-host.example.com name: my-host-example-com-https port: 443 protocol: HTTPS tls: certificateRefs: – group: “” kind: Secret name: my-secret — apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: annotations: gateway.networking.k8s.io/generator: ingress2gateway-dev name: my-ingress-my-host-example-com namespace: my-ns spec: hostnames: – my-host.example.com parentRefs: – name: nginx port: 443 rules: – backendRefs: – name: website-service port: 80 filters: – cors: allowCredentials: true allowHeaders: – DNT … allowMethods: – GET … allowOrigins: – ‘*’ maxAge: 1728000 type: CORS matches: – path: type: RegularExpression value: /users/(\d+) name: rule-0 timeouts: request: 3s 4. Verify Now that you have Gateway API manifests, you should thoroughly test them in a development cluster. In this case, you should at least double-check that your Gateway API implementation’s maximum body size defaults are appropriate for you and verify that a three-second timeout is enough. After validating behavior in a development cluster, deploy your Gateway API configuration alongside your existing Ingress. We strongly suggest that you then gradually shift traffic using weighted DNS, your cloud load balancer, or traffic-splitting features of your platform. This way, you can quickly recover from any misconfiguration that made it through your tests. Finally, when you have shifted all your traffic to your Gateway API controller, delete your Ingress resources and uninstall your Ingress controller. Conclusion The Ingress2Gateway 1.0 release is just the beginning, and we hope that you use Ingress2Gateway to safely migrate to Gateway API. As we approach the March 2026 Ingress-NGINX retirement, we invite the community to help us increase our configuration coverage, expand testing, and improve UX. Resources about Gateway API The scope of Gateway API can be daunting. Here are some resources to help you work with Gateway API: Listener sets allow application developers to manage gateway listeners. gwctl gives you a comprehensive view of your Gateway resources, such as attachments and linter errors. Gateway API Slack: #sig-network-gateway-api on Kubernetes Slack Ingress2Gateway Slack: #sig-network-ingress2gateway on Kubernetes Slack GitHub: kubernetes-sigs/ingress2gateway

on March 20, 2026 at 6:00 pm

The landscape of artificial intelligence is undergoing a massive architectural shift. In the early days of generative AI, interacting with a model was often treated as a transient, stateless function call: a request that spun up, executed for perhaps 50 milliseconds, and terminated. Today, the world is witnessing AI v2 eating AI v1. The ecosystem is moving from short-lived, isolated tasks to deploying multiple, coordinated AI agents that run constantly. These autonomous agents need to maintain context, use external tools, write and execute code, and communicate with one another over extended periods. As platform engineering teams look for the right infrastructure to host these new AI workloads, one platform stands out as the natural choice: Kubernetes. However, mapping these unique agentic workloads to traditional Kubernetes primitives requires a new abstraction. This is where the new Agent Sandbox project (currently in development under SIG Apps) comes into play. The Kubernetes advantage (and the abstraction gap) Kubernetes is the de facto standard for orchestrating cloud-native applications precisely because it solves the challenges of extensibility, robust networking, and ecosystem maturity. However, as AI evolves from short-lived inference requests to long-running, autonomous agents, we are seeing the emergence of a new operational pattern. AI agents, by contrast, are typically isolated, stateful, singleton workloads. They act as a digital workspace or execution environment for an LLM. An agent needs a persistent identity and a secure scratchpad for writing and executing (often untrusted) code. Crucially, because these long-lived agents are expected to be mostly idle except for brief bursts of activity, they require a lifecycle that supports mechanisms like suspension and rapid resumption. While you could theoretically approximate this by stringing together a StatefulSet of size 1, a headless Service, and a PersistentVolumeClaim for every single agent, managing this at scale becomes an operational nightmare. Because of these unique properties, traditional Kubernetes primitives don’t perfectly align. Introducing Kubernetes Agent Sandbox To bridge this gap, SIG Apps is developing agent-sandbox. The project introduces a declarative, standardized API specifically tailored for singleton, stateful workloads like AI agent runtimes. At its core, the project introduces the Sandbox CRD. It acts as a lightweight, single-container environment built entirely on Kubernetes primitives, offering: Strong isolation for untrusted code: When an AI agent generates and executes code autonomously, security is paramount. The Sandbox custom resource natively supports different runtimes, like gVisor or Kata Containers. This provides the necessary kernel and network isolation required for multi-tenant, untrusted execution. Lifecycle management: Unlike traditional web servers optimized for steady, stateless traffic, an AI agent operates as a stateful workspace that may be idle for hours between tasks. Agent Sandbox supports scaling these idle environments to zero to save resources, while ensuring they can resume exactly where they left off. Stable identity: Coordinated multi-agent systems require stable networking. Every Sandbox is given a stable hostname and network identity, allowing distinct agents to discover and communicate with each other seamlessly. Scaling agents with extensions Because the AI space is moving incredibly quickly, we built an Extensions API layer that enables even faster iteration and development. Starting a new pod adds about a second of overhead. That’s perfectly fine when deploying a new version of a microservice, but when an agent is invoked after being idle, a one-second cold start breaks the continuity of the interaction. It forces the user or the orchestrating service to wait for the environment to provision before the model can even begin to think or act. SandboxWarmPool solves this by maintaining a pool of pre-provisioned Sandbox pods, effectively eliminating cold starts. Users or orchestration services can simply issue a SandboxClaim against a SandboxTemplate, and the controller immediately hands over a pre-warmed, fully isolated environment to the agent. Quick start Ready to try it yourself? You can install the Agent Sandbox core components and extensions directly into your learning or sandbox cluster, using your chosen release. We recommend you use the latest release as the project is moving fast. # Replace “vX.Y.Z” with a specific version tag (e.g., “v0.1.0″) from # https://github.com/kubernetes-sigs/agent-sandbox/releases export VERSION=”vX.Y.Z” # Install the core components: kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/manifest.yaml # Install the extensions components (optional): kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/extensions.yaml # Install the Python SDK (optional): # Create a virtual Python environment python3 -m venv .venv source .venv/bin/activate # Install from PyPI pip install k8s-agent-sandbox Once installed, you can try out the Python SDK for AI agents or deploy one of the ready-to-use examples to see how easy it is to spin up an isolated agent environment. The future of agents is cloud native Whether it’s a 50-millisecond stateless task, or a multi-week, mostly-idle collaborative process, extending Kubernetes with primitives designed specifically for isolated stateful singletons allows us to leverage all the robust benefits of the cloud-native ecosystem. The Agent Sandbox project is open source and community-driven. If you are building AI platforms, developing agentic frameworks, or are interested in Kubernetes extensibility, we invite you to get involved: Check out the project on GitHub: kubernetes-sigs/agent-sandbox Join the discussion in the #sig-apps and #agent-sandbox channels on the Kubernetes Slack.

on March 18, 2026 at 6:00 pm

During production debugging, the fastest route is often broad access such as cluster-admin (a ClusterRole that grants administrator-level access), shared bastions/jump boxes, or long-lived SSH keys. It works in the moment, but it comes with two common problems: auditing becomes difficult, and temporary exceptions have a way of becoming routine. This post offers my recommendations for good practices applicable to existing Kubernetes environments with minimal tooling changes: Least privilege with RBAC Short-lived, identity-bound credentials An SSH-style handshake model for cloud native debugging A good architecture for securing production debugging workflows is to use a just-in-time secure shell gateway (often deployed as an on demand pod in the cluster). It acts as an SSH-style “front door” that makes temporary access actually temporary. You can authenticate with short-lived, identity-bound credentials, establish a session to the gateway, and the gateway uses the Kubernetes API and RBAC to control what they can do, such as pods/log, pods/exec, and pods/portforward. Sessions expire automatically, and both the gateway logs and Kubernetes audit logs capture who accessed what and when without shared bastion accounts or long-lived keys. 1) Using an access broker on top of Kubernetes RBAC RBAC controls who can do what in Kubernetes. Many Kubernetes environments rely primarily on RBAC for authorization, although Kubernetes also supports other authorization modes such as Webhook authorization. You can enforce access directly with Kubernetes RBAC, or put an access broker in front of the cluster that still relies on Kubernetes permissions under the hood. In either model, Kubernetes RBAC remains the source of truth for what the Kubernetes API allows and at what scope. An access broker adds controls that RBAC does not cover well. For example, it can decide whether a request is auto-approved or requires manual approval, whether a user can run a command, and which commands are allowed in a session. It can also manage group membership so that you grant permissions to groups instead of individual users. Kubernetes RBAC can allow actions such as pods/exec, but it cannot restrict which commands run inside an exec session. With that model, Kubernetes RBAC defines the allowed actions for a user or group (for example, an on-call team in a single namespace). I recommend you only define access rules that grant rights to groups or to ServiceAccounts – never to individual users. The broker or identity provider then adds or removes users from that group as needed. The broker can also enforce extra policy on top, like which commands are permitted in an interactive session and which requests can be auto-approved versus require manual approval. That policy can live in a JSON or XML file and be maintained through code review, so updates go through a formal pull request and are reviewed like any other production change. Example: a namespaced on-call debug Role apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: oncall-debug namespace: <namespace> rules: # Discover what’s running – apiGroups: [“”] resources: [“pods”, “events”] verbs: [“get”, “list”, “watch”] # Read logs – apiGroups: [“”] resources: [“pods/log”] verbs: [“get”] # Interactive debugging actions – apiGroups: [“”] resources: [“pods/exec”, “pods/portforward”] verbs: [“create”] # Understand rollout/controller state – apiGroups: [“apps”] resources: [“deployments”, “replicasets”] verbs: [“get”, “list”, “watch”] # Optional: allow kubectl debug ephemeral containers – apiGroups: [“”] resources: [“pods/ephemeralcontainers”] verbs: [“update”] Bind the Role to a group (rather than individual users) so membership can be managed through your identity provider: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: oncall-debug namespace: <namespace> subjects: – kind: Group name: oncall-<team-name> apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: oncall-debug apiGroup: rbac.authorization.k8s.io 2) Short-lived, identity-bound credentials The goal is to use short-lived, identity-bound credentials that clearly tie a session to a real person and expire quickly. These credentials can include the user’s identity and the scope of what they’re allowed to do. They’re typically signed using a private key that stays with the engineer, such as a hardware-backed key (for example, a YubiKey), so they can not be forged without access to that key. You can implement this with Kubernetes-native authentication (for example, client certificates or an OIDC-based flow), or have the access broker from the previous section issue short-lived credentials on the user’s behalf. In many setups, Kubernetes still uses RBAC to enforce permissions based on the authenticated identity and groups/claims. If you use an access broker, it can also encode additional scope constraints in the credential and enforce them during the session, such as which cluster or namespace the session applies to and which actions (or approved commands) are allowed against pods or nodes. In either case, the credentials should be signed by a certificate authority (CA), and that CA should be rotated on a regular schedule (for example, quarterly) to limit long-term risk. Option A: short-lived OIDC tokens A lot of managed Kubernetes clusters already give you short-lived tokens. The main thing is to make sure your kubeconfig refreshes them automatically instead of copying a long-lived token into the file. For example: users: – name: oncall user: exec: apiVersion: client.authentication.k8s.io/v1 command: cred-helper args: [“–cluster=prod”, “–ttl=30m”] Option B: Short-lived client certificates (X.509) If your API server (or your access broker from the previous section) is set up to trust a client CA, you can use short-lived client certificates for debugging access. The idea is: The private key is created and kept under the engineer’s machine (ideally hardware-backed, like a non-exportable key in a YubiKey/PIV token) A short-lived certificate is issued (often via the CertificateSigningRequest API, or your access broker from the previous section, with a TTL). RBAC maps the authenticated identity to a minimal Role This is straightforward to operationalize with the Kubernetes CertificateSigningRequest API. Generate a key and CSR locally: # Generate a private key. # This could instead be generated within a hardware token; # OpenSSL and several similar tools include support for that. openssl genpkey -algorithm Ed25519 -out oncall.key openssl req -new -key oncall.key -out oncall.csr \ -subj “/CN=user/O=oncall-payments” Create a CertificateSigningRequest with a short expiration: apiVersion: certificates.k8s.io/v1 kind: CertificateSigningRequest metadata: name: oncall-<user>-20260218 spec: request: <base64-encoded oncall.csr> signerName: kubernetes.io/kube-apiserver-client expirationSeconds: 1800 # 30 minutes usages: – client auth After the CSR is approved and signed, you extract the issued certificate and use it together with the private key to authenticate, for example via kubectl. 3) Use a just-in-time access gateway to run debugging commands Once you have short-lived credentials, you can use them to open a secure shell session to a just-in-time access gateway, often exposed over SSH and created on demand. If the gateway is exposed over SSH, a common pattern is to issue the engineer a short-lived OpenSSH user certificate for the session. The gateway trusts your SSH user CA, authenticates the engineer at connection time, and then applies the approved session policy before making Kubernetes API calls on the user’s behalf. OpenSSH certificates are separate from Kubernetes X.509 client certificates, so these are usually treated as distinct layers. The resulting session should also be scoped so it cannot be reused outside of what was approved. For example, the gateway or broker can limit it to a specific cluster and namespace, and optionally to a narrower target such as a pod or node. That way, even if someone tries to reuse the access, it will not work outside the intended scope. After the session is established, the gateway executes only the allowed actions and records what happened for auditing. Example: Namespace-scoped role bindings apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: jit-debug namespace: <namespace> annotations: kubernetes.io/description: > Colleagues performing semi-privileged debugging, with access provided just in time and on demand. rules: – apiGroups: [“”] resources: [“pods”, “pods/log”] verbs: [“get”, “list”, “watch”] – apiGroups: [“”] resources: [“pods/exec”] verbs: [“create”] — apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jit-debug namespace: <namespace> subjects: – kind: Group name: jit:oncall:<namespace> # mapped from the short-lived credential (cert/OIDC) apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: jit-debug apiGroup: rbac.authorization.k8s.io These RBAC objects, and the rules they define, allow debugging only within the specified namespace; attempts to access other namespaces are not allowed. Example: Cluster-scoped role binding apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: jit-cluster-read rules: – apiGroups: [“”] resources: [“nodes”, “namespaces”] verbs: [“get”, “list”, “watch”] — apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: jit-cluster-read subjects: – kind: Group name: jit:oncall:cluster apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: jit-cluster-read apiGroup: rbac.authorization.k8s.io These RBAC rules grant cluster-wide read access (for example, to nodes and namespaces) and should be used only for workflows that truly require cluster-scoped resources. Finer-grained restrictions like “only this pod/node” or “only these commands” are typically enforced by the access gateway/broker during the session, but Kubernetes also offers other options, such as ValidatingAdmissionPolicy for restricting writes and webhook authorization for custom authorization across verbs. In environments with stricter access controls, you can add an extra, short-lived session mediation layer to separate session establishment from privileged actions. Both layers are ephemeral, use identity-bound expiring credentials, and produce independent audit trails. The mediation layer handles session setup/forwarding, while the execution layer performs only RBAC-authorized Kubernetes actions. This separation can reduce exposure by narrowing responsibilities, scoping credentials per step, and enforcing end-to-end session expiry. References Authorization Using RBAC Authorization Authenticating Certificates and Certificate Signing Requests Issue a Certificate for a Kubernetes API Client Using a CertificateSigningRequest Role Based Access Control Good Practices Disclaimer: The views expressed in this post are solely those of the author and do not reflect the views of the author’s employer or any other organization.

on March 17, 2026 at 12:00 am

Every container image you pull from registry.k8s.io got there through kpromo, the Kubernetes image promoter. It copies images from staging registries to production, signs them with cosign, replicates signatures across more than 20 regional mirrors, and generates SLSA provenance attestations. If this tool breaks, no Kubernetes release ships. Over the past few weeks, we rewrote its core from scratch, deleted 20% of the codebase, made it dramatically faster, and nobody noticed. That was the whole point. A bit of history The image promoter started in late 2018 as an internal Google project by Linus Arver. The goal was simple: replace the manual, Googler-gated process of copying container images into k8s.gcr.io with a community-owned, GitOps-based workflow. Push to a staging registry, open a PR with a YAML manifest, get it reviewed and merged, and automation handles the rest. KEP-1734 formalized this proposal. In early 2019, the code moved to kubernetes-sigs/k8s-container-image-promoter and grew quickly. Over the next few years, Stephen Augustus consolidated multiple tools (cip, gh2gcs, krel promote-images, promobot-files) into a single CLI called kpromo. The repository was renamed to promo-tools. Adolfo Garcia Veytia (Puerco) added cosign signing and SBOM support. Tyler Ferrara built vulnerability scanning. Carlos Panato kept the project in a healthy and releasable state. 42 contributors made about 3,500 commits across more than 60 releases. It worked. But by 2025 the codebase carried the weight of seven years of incremental additions from multiple SIGs and subprojects. The README said it plainly: you will see duplicated code, multiple techniques for accomplishing the same thing, and several TODOs. The problems we needed to solve Production promotion jobs for Kubernetes core images regularly took over 30 minutes and frequently failed with rate limit errors. The core promotion logic had grown into a monolith that was hard to extend and difficult to test, making new features like provenance or vulnerability scanning painful to add. On the SIG Release roadmap, two work items had been sitting for a while: “Rewrite artifact promoter” and “Make artifact validation more robust”. We had discussed these at SIG Release meetings and KubeCons, and the open research spikes on project board #171 captured eight questions that needed answers before we could move forward. One issue to answer them all In February 2026, we opened issue #1701 (“Rewrite artifact promoter pipeline”) and answered all eight spikes in a single tracking issue. The rewrite was deliberately phased so that each step could be reviewed, merged, and validated independently. Here is what we did: Phase 1: Rate Limiting (#1702). Rewrote rate limiting to properly throttle all registry operations with adaptive backoff. Phase 2: Interfaces (#1704). Put registry and auth operations behind clean interfaces so they can be swapped out and tested independently. Phase 3: Pipeline Engine (#1705). Built a pipeline engine that runs promotion as a sequence of distinct phases instead of one large function. Phase 4: Provenance (#1706). Added SLSA provenance verification for staging images. Phase 5: Scanner and SBOMs (#1709). Added vulnerability scanning and SBOM support. Flipped the default to the new pipeline engine. At this point we cut v4.2.0 and let it soak in production before continuing. Phase 6: Split Signing from Replication (#1713). Separated image signing from signature replication into their own pipeline phases, eliminating the rate limit contention that caused most production failures. Phase 7: Remove Legacy Pipeline (#1712). Deleted the old code path entirely. Phase 8: Remove Legacy Dependencies (#1716). Deleted the audit subsystem, deprecated tools, and e2e test infrastructure. Phase 9: Delete the Monolith (#1718). Removed the old monolithic core and its supporting packages. Thousands of lines deleted across phases 7 through 9. Each phase shipped independently. v4.3.0 followed the next day with the legacy code fully removed. With the new architecture in place, a series of follow-up improvements landed: parallelized registry reads (#1736), retry logic for all network operations (#1742), per-request timeouts to prevent pipeline hangs (#1763), HTTP connection reuse (#1759), local registry integration tests (#1746), the removal of deprecated credential file support (#1758), a rework of attestation handling to use cosign’s OCI APIs and the removal of deprecated SBOM support (#1764), and a dedicated promotion record predicate type registered with the in-toto attestation framework (#1767). These would have been much harder to land without the clean separation the rewrite provided. v4.4.0 shipped all of these improvements and enabled provenance generation and verification by default. The new pipeline The promotion pipeline now has seven clearly separated phases: graph LR Setup –> Plan –> Provenance –> Validate –> Promote –> Sign –> Attest Phase What it does Setup Validate options, prewarm TUF cache. Plan Parse manifests, read registries, compute which images need promotion. Provenance Verify SLSA attestations on staging images. Validate Check cosign signatures, exit here for dry runs. Promote Copy images server-side, preserving digests. Sign Sign promoted images with keyless cosign. Attest Generate promotion provenance attestations using a dedicated in-toto predicate type. Phases run sequentially, so each one gets exclusive access to the full rate limit budget. No more contention. Signature replication to mirror registries is no longer part of this pipeline and runs as a dedicated periodic Prow job instead. Making it fast With the architecture in place, we turned to performance. Parallel registry reads (#1736): The plan phase reads 1,350 registries. We parallelized this and the plan phase dropped from about 20 minutes to about 2 minutes. Two-phase tag listing (#1761): Instead of checking all 46,000 image groups across more than 20 mirrors, we first check only the source repositories. About 57% of images have no signatures at all because they were promoted before signing was enabled. We skip those entirely, cutting API calls roughly in half. Source check before replication (#1727): Before iterating all mirrors for a given image, we check if the signature exists on the primary registry first. In steady state where most signatures are already replicated, this reduced the work from about 17 hours to about 15 minutes. Per-request timeouts (#1763): We observed intermittent hangs where a stalled connection blocked the pipeline for over 9 hours. Every network operation now has its own timeout and transient failures are retried automatically. Connection reuse (#1759): We started reusing HTTP connections and auth state across operations, eliminating redundant token negotiations. This closed a long-standing request from 2023. By the numbers Here is what the rewrite looks like in aggregate. Over 40 PRs merged, 3 releases shipped (v4.2.0, v4.3.0, v4.4.0) Over 10,000 lines added and over 16,000 lines deleted, a net reduction of about 5,000 lines (20% smaller codebase) Performance drastically improved across the board Robustness improved with retry logic, per-request timeouts, and adaptive rate limiting 19 long-standing issues closed The codebase shrank by a fifth while gaining provenance attestations, a pipeline engine, vulnerability scanning integration, parallelized operations, retry logic, integration tests against local registries, and a standalone signature replication mode. No user-facing changes This was a hard requirement. The kpromo cip command accepts the same flags and reads the same YAML manifests. The post-k8sio-image-promo Prow job continued working throughout. The promotion manifests in kubernetes/k8s.io did not change. Nobody had to update their workflows or configuration. We caught two regressions early in production. One (#1731) caused a registry key mismatch that made every image appear as “lost” so that nothing was promoted. Another (#1733) set the default thread count to zero, blocking all goroutines. Both were fixed within hours. The phased release strategy (v4.2.0 with the new engine, v4.3.0 with legacy code removed) gave us a clear rollback path that we fortunately never needed. What comes next Signature replication across all mirror registries remains the most expensive part of the promotion cycle. Issue #1762 proposes eliminating it entirely by having archeio (the registry.k8s.io redirect service) route signature tag requests to a single canonical upstream instead of per-region backends. Another option would be to move signing closer to the registry infrastructure itself. Both approaches need further discussion with the SIG Release and infrastructure teams, but either one would remove thousands of API calls per promotion cycle and simplify the codebase even further. Thank you This project has been a community effort spanning seven years. Thank you to Linus, Stephen, Adolfo, Carlos, Ben, Marko, Lauri, Tyler, Arnaud, and many others who contributed code, reviews, and planning over the years. The SIG Release and Release Engineering communities provided the context, the discussions, and the patience for a rewrite of infrastructure that every Kubernetes release depends on. If you want to get involved, join us in #release-management on the Kubernetes Slack or check out the repository.