Does KubeOn GPU change our clusters?

Only through the GPU optimizer, and only where you install it. It is a separate Helm release with its own published permissions, and it acts only on the node pools and namespaces a policy names, in the mode you choose. The cost agent stays read-only.

Can we start without any automation?

Yes. Every policy starts in Recommend mode, where each action is a proposal your pipeline applies. Move a policy to Approve, then Automatic, when its proposals have earned it. Without the optimizer installed, KubeOn GPU still measures and prices every GPU.

Do we need the NVIDIA GPU Operator?

For sharing, yes: MIG and time-slicing are applied through the GPU Operator. Utilization needs the DCGM exporter scraped by Prometheus. GPU cost and idle GPU time only need the Kubernetes API and your billing or rate card.

Which GPUs are supported?

NVIDIA GPUs exposed through the NVIDIA device plugin, in the cloud or in your own data center. MIG is limited to GPUs that support it, such as A100 and H100; time-slicing works on any NVIDIA GPU.

How does burst respect data residency?

A burst policy lists the locations it may use, such as EU regions and your own data center, and the optimizer never uses a location that is not on the list. Only pods that tolerate the burst taint are scheduled on burst nodes.

KubeOn GPU

Every GPU busy, shared or switched off

KubeOn GPU measures every GPU-hour, then its opt-in optimizer acts on what it finds: MIG and time-slicing for workloads that need part of a card, GPU nodes that scale and pack with demand, burst capacity from other regions, clouds or your data center, and idle GPUs scaled to zero. Each policy runs as a proposal, with an approval, or automatically within limits you set.

Book a demo See the optimizer

GPUs

Data readyUpdated 12 min ago

Last 7 days▾

GPU optimizer

Policies, the mode each one runs in, and what it did last

Policies

5

2 Automatic

Waiting for you

2

Approve mode

Actions, 7 days

18

0 reverted

Policy	Kind	Mode	Last action
ml-inference-autoscalingprod-use1-a / ml-inference	Autoscaling	Automatic	Removed 1 node, 19:15
notebooks-scale-to-zerodev-use2 / ml-notebooks	Scale to zero	Automatic	Scaled to 0, 20:30
dc1-migdc1-prod / gpu	Sharing	Approve	1 repartition waiting
inference-burst-euprod-euw1-a / ml-inference-eu	Burst	Approve	Burst of 4 GPUs waiting
rerank-gpu-typeprod-use1-a / rerank-svc	GPU type	Recommend	Proposal: 24 GB class

The optimizer runs only in clusters where it is installed. The read-only agent is unchanged.

The problem

Allocated is not used

GPUs are the fastest-growing line on many cloud bills, and published research on shared clusters shows how much of that capacity sits idle or barely busy.

Read the research on idle GPUs

14%

of EC2 compute spend went to GPU instances across Datadog's customers, up from 10% a year earlier

Datadog, State of Cloud Costs 2024

~52%

average utilization of the GPUs in use in a large multi-tenant training cluster at Microsoft

Jeon et al., USENIX ATC 2019

0.042

of a GPU: the median task's use in Alibaba's production cluster of over 6,000 GPUs

Weng et al., USENIX NSDI 2022

74% / 18%

of GPUs allocated, against average GPU utilization, on one group of Alibaba machines

Weng et al., USENIX NSDI 2022

The optimizer

Four ways it puts GPUs to work

Step through each one on sample clusters: what the optimizer saw, what it did, and what it logged.

Whole GPUs, barely used

dc1-gpu-07embed-svc6%
dc1-gpu-08classify-svc9%
dc1-gpu-09ocr-svc5%

GPU optimizer

Policy dc1-mig, Approve mode

1Find GPUs under 10%
2Choose the slice that fits
3Wait for your approval
4Repartition in the change window

After

dc1-gpu-07Three services on MIG slicesShared
dc1-gpu-08Back in the poolFree
dc1-gpu-09Back in the poolFree
SavedTwo A100s at the rate card-$1,723/mo

Figures are from a sample company. The repartition waits for a reviewer, then runs in the Saturday change window.

Control

You choose how far each policy goes

Every policy starts as a proposal. Move it to Approve, then Automatic, when its proposals have earned your trust.

Installed only in clusters you choose, as its own Helm release, separate from the read-only agent
Acts only on the node pools and namespaces a policy names; everything else is out of reach
Hard limits per policy: minimum and maximum nodes, monthly cost, GPU-hour price
Disruptive steps such as drains and repartitions run only in the policy's change window
Evicts through the Eviction API, so PodDisruptionBudgets are always respected
No access to Secrets or ConfigMaps outside the GPU Operator config it manages
Pulls policies and approvals from the hub over HTTPS; the hub never connects into your cluster
Every action in the audit log with the policy that allowed it, and a one-click revert where the change can be undone

Mode 1

Recommend · A proposal, nothing else

The optimizer writes the change as a proposal with its evidence and patch. Your pipeline applies it, like other KubeOn proposals. Every policy starts here.

Mode 2

Approve · Approve once, the optimizer applies

Each change waits for a named reviewer. Once approved, the optimizer applies it in the policy's change window and records the outcome.

Mode 3

Automatic · Acts within the limits you set

The optimizer acts on its own, but only on the node pools and namespaces the policy names, within its node, cost and time limits.

KubeOn GPU modules

Five areas, one GPU optimizer

Part of KubeOn

GPU sharing

MIG partitions and time-slicing, applied for workloads that use part of a GPU

GPU autoscaling

GPU nodes added on demand, spot first, packed tight and scaled to zero when idle

GPU burst

GPUs from other regions, clouds or your data center when the home region runs out

GPU cost and utilization

GPU node cost by team, idle GPU time and real utilization

GPU data centers

GPU-hour rates for servers you own, shared and used before you buy more

Where it works

Cloud GPUs and the ones you own

Cloud GPUs are priced from your billing export, owned GPUs from a rate card. The optimizer drives Karpenter or your cloud's node groups and the NVIDIA GPU Operator, and reports every action to KubeOn.

GPU data centers

Amazon EKS
Azure Kubernetes Service
Google Kubernetes Engine
Red Hat OpenShift
Kubernetes on VMware or bare metal
NVIDIA GPU Operator and device plugin
NVIDIA DCGM exporter
Karpenter and managed node groups

Published permissions

What the optimizer may change, in the open

The optimizer is the only KubeOn component with write access, and only in clusters where you install it. Its ClusterRole is published here and on the security page, so your security team can review it first. It never reads Secrets.

Security and trust center

GPU optimizer ClusterRole

apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  name: kubeon-gpu-optimizerrules:  # Read what it optimizes  - apiGroups: [""]    resources: [nodes, pods, namespaces]    verbs: [get, list, watch]  - apiGroups: [apps]    resources: [deployments, statefulsets, replicasets]    verbs: [get, list, watch]  - apiGroups: [policy]    resources: [poddisruptionbudgets]    verbs: [get, list, watch]  # Label, taint and cordon GPU nodes; drain through the Eviction API  - apiGroups: [""]    resources: [nodes]    verbs: [patch]  - apiGroups: [""]    resources: [pods/eviction]    verbs: [create]  # Add and remove GPU capacity through Karpenter  - apiGroups: [karpenter.sh]    resources: [nodepools, nodeclaims]    verbs: [get, list, watch, create, patch, delete]  # Its own policies, synced from the hub, and their status  - apiGroups: [gpu.kubeon.io]    resources: [gpupolicies, gpupolicies/status]    verbs: [get, list, watch, create, update, patch, delete]---# Created only in namespaces named by a ScaleToZero policykind: Rolerules:  - apiGroups: [apps]    resources: [deployments/scale, statefulsets/scale]    verbs: [get, patch]---# In the gpu-operator namespace, for time-slicing onlykind: Rolerules:  - apiGroups: [""]    resources: [configmaps]    resourceNames: [time-slicing-config]    verbs: [get, patch]

KubeOn GPU: your questions answered

See what KubeOn GPU does for your fleet.

Tell us about your GPU fleet and we will plan the rollout with you, starting with what the optimizer would propose.

Book a demo GPU pricing