How the GPU optimizer decides
The measurements, rules and limits behind GPU allocation, sharing, autoscaling, scale to zero, burst and data center GPUs.
On this page
This page explains the rule behind every GPU finding and every optimizer action. To install the optimizer, see install the GPU optimizer; for every policy field, see the GPU policy reference.
What KubeOn reads
| Input | Source | Needed for |
|---|---|---|
| GPU capacity per node | nvidia.com/gpu and MIG resources in node status, from the Kubernetes API | Cost, idle GPU time, every optimizer policy |
| GPU requests per pod | Container limits for nvidia.com/gpu and nvidia.com/mig-* resources | Cost, idle GPU time, packing |
| GPU node cost | Net amortized cost from your billing export, or a GPU-hour rate card | Cost, burst location prices, policy cost limits |
| Utilization and GPU memory | NVIDIA DCGM exporter series in your Prometheus. See connect GPU metrics | Utilization, sharing, scale to zero |
Allocation and utilization
- Each GPU node's billed, net amortized cost is split by the GPUs pods requested, hour by hour. A MIG slice carries the share of the GPU's compute slices it holds: a
2g.10gbslice on an A100 40GB carries two sevenths of that GPU's hourly cost. - GPU capacity no pod requested is idle GPU time, priced at the node's rate and kept in its own column.
- GPU utilization counts the time any kernel ran, so where SM activity is available it is shown alongside as the closer measure of how hard a GPU worked.
Modes
| Mode | What happens |
|---|---|
| Recommend | The optimizer writes the change as a proposal with its evidence and patch. Your pipeline applies it, like other KubeOn proposals. Every policy starts here. |
| Approve | Each change waits for a named reviewer. Once approved, the optimizer applies it in the policy's change window and records the outcome. |
| Automatic | The optimizer acts on its own, but only on the node pools and namespaces the policy names, within its node, cost and time limits. |
Sharing
| Action | Rule |
|---|---|
| MIG partition | Workloads holding a whole MIG-capable GPU whose peak GPU memory plus headroom fits a smaller profile allowed by the policy. Workloads are packed onto as few GPUs as possible. |
| Time-slicing | Workloads in the namespaces the policy lists, holding a whole GPU at low utilization, on any NVIDIA GPU. Replicas per GPU are chosen so the workloads' peak GPU memory fits together, up to maxReplicas. |
| How it is applied | In the policy's window: cordon the node, evict its GPU pods within PodDisruptionBudgets, set nvidia.com/mig.config or the device plugin config, then let pods reschedule onto the new resources. |
Time-slicing has no isolation
Autoscaling
The optimizer adds and removes GPU capacity through Karpenter NodePools or your cloud's managed node groups, and leaves CPU node pools to the autoscaler you already run.
| Action | When | Limit |
|---|---|---|
| Add a GPU node | GPU pods pending that an allowed GPU type can hold | maxNodes and maxMonthlyCost |
| Replace a spot node | Interruption notice (two minutes on AWS, 30 seconds on Azure and Google Cloud) or no spot capacity | Only if on-demand is in the capacity order |
| Pack and remove a node | Pods fit on fewer nodes | The packing window and PodDisruptionBudgets |
| Smaller GPU type | Peak GPU memory plus 30% fits a smaller GPU class | Always a proposal: latency needs a benchmark |
Scale to zero
A workload in a listed namespace whose GPU utilization stays under the idle threshold, 2% by default, for the idle period is scaled to zero replicas; its volumes and configuration stay. Empty GPU nodes are then removed. The workload returns on the policy's wake schedule, or as soon as someone scales it up.
Burst
- Trigger: pods that tolerate
gpu.kubeon.io/burstare pending because no allowed GPU type is available in the home region. - Order: idle owned GPUs first, then the cheapest listed cloud location with capacity, priced with an estimate of data transfer back to the home region.
- Attach: burst nodes are created with the credentials you grant for each location and join the cluster over an encrypted tunnel to the burst gateway. See set up GPU burst.
- Return: when home capacity is back, pods drain home within PodDisruptionBudgets and the burst nodes are removed.
- Residency: a location that is not in
locationsis never used.
Data centers
GPU servers you own are priced with a rate card: a gpuHour rate per node pool or label, or amortized hardware. See on-premises pricing. Owned GPUs can be shared with MIG like cloud GPUs, and idle owned GPUs are the first location a burst policy considers.
Audit and revert
- Every optimizer action is written to the audit log with the policy and mode that allowed it, and the reviewer for Approve mode.
- Node removals, scale-to-zero and MIG repartitions can be reverted from the audit log: the optimizer restores the previous node count, replica count or profile.
- Setting a policy back to Recommend stops it acting at once. Uninstalling the optimizer leaves the last applied state in place; see remove KubeOn.