Skip to content
NewFinOps, Kubernetes and AI, on one platform
Book a demo

GPU autoscaling

GPU nodes that follow demand, in both directions

The GPU optimizer adds GPU nodes when pods wait for them, uses spot GPUs with an on-demand fallback, packs GPU pods so empty nodes can leave, and scales idle GPU workloads to zero. It never goes past a policy's node, cost or GPU-type limits, and every action can be reviewed and reverted.

Book a demoAutoscaling rules

GPU pools sized for the busiest hour, all day

  • Fixed pools:

    GPU node pools are sized once for the peak and left running through nights and weekends.

  • Spread-out pods:

    GPU pods land on whichever node is free, so no node ever empties enough to be removed.

  • Spot feels risky:

    Spot GPUs cost less, but a reclaimed node without a fallback means failed requests.

  • Idle notebooks:

    An interactive session holds a GPU while nobody is using it, often overnight.

Capacity on demand

GPU nodes added when GPU pods wait

When GPU pods are pending, the optimizer adds the cheapest allowed GPU node that fits them, through Karpenter or your cloud's node groups. It checks the policy's maximum nodes and monthly cost first, and stops at the limit instead of passing it.

  • Allowed GPU types and sizes per policy
  • Minimum and maximum nodes, monthly cost cap
  • Pending pods that would breach a limit are reported, not hidden
Autoscaling policy
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata:  name: ml-inference-autoscalingspec:  kind: Autoscaling  mode: Automatic                 # Recommend | Approve | Automatic  nodePools: [ml-inference]  limits:    minNodes: 1    maxNodes: 6    maxMonthlyCost: 40000         # USD at billed rates  capacity:    order: [spot, on-demand]      # spot first, on-demand when spot is gone  packing:    enabled: true    window: "Mon-Fri 19:00-23:00 UTC"    respectPodDisruptionBudgets: true

Spot GPUs

Spot first, on-demand when spot goes away

Policies can prefer spot GPUs and fall back to on-demand when spot capacity is unavailable or reclaimed. On an interruption notice, two minutes on AWS and 30 seconds on Azure and Google Cloud, the optimizer cordons the node, starts a replacement and lets pods drain in time.

  • Capacity order per policy: spot, then on-demand
  • Replacement started on the interruption notice
  • Spot only for workloads that tolerate restarts

Packing and consolidation

Pods packed so empty GPU nodes can leave

GPU pods are placed on the fullest node that fits them, aware of MIG slices and time-sliced GPUs. In the policy's window, when the pods would fit on fewer nodes, the optimizer drains the emptiest node through the Eviction API, within PodDisruptionBudgets, and removes it.

  • GPU-aware placement, MIG and time-slicing included
  • Consolidation only inside the change window
  • Each removal priced at the node's billed rate

Scale to zero

Idle GPU workloads scaled to zero, nodes removed

In the namespaces a policy lists, a GPU workload whose GPU utilization stays under 2% for the idle period is scaled to zero replicas. Its volumes and configuration stay. When its node is empty, the node is removed, and the workload comes back on schedule or when someone scales it up.

  • Only namespaces named in the policy
  • Idle measured from DCGM, not guessed
  • Wake on a schedule or on demand
Scale-to-zero policy
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata:  name: notebooks-scale-to-zerospec:  kind: ScaleToZero  mode: Automatic  namespaces: [ml-notebooks]      # only namespaces you list  idle:    gpuUtilizationBelow: 2%       # DCGM; 2% is the default    for: 30m  wake:    schedule: "Mon-Fri 07:00 UTC"  removeEmptyNodes: true

Where the GPU savings are

52%

average use of the GPUs in a large Microsoft training cluster

Jeon et al., USENIX ATC 2019
18%

average GPU use on Alibaba machines with 74% of GPUs allocated

Weng et al., USENIX NSDI 2022
$8,433/mo

saved by removing one GPU node after the peak at our sample company

Sample company, not a customer result

KubeOn GPU overview

Autoscaling rules

What the optimizer does, and the limit on each action

ActionWhenLimit
Add a GPU nodeGPU pods pending that an allowed GPU type can holdMaximum nodes and monthly cost per policy
Replace a spot nodeInterruption notice, or spot unavailable when scaling upOn-demand only if the policy allows it
Pack and remove a nodePods fit on fewer nodesChange window and PodDisruptionBudgets
Scale a workload to zeroGPU utilization under 2% for the idle periodOnly the namespaces the policy lists
Smaller GPU typePeak GPU memory plus 30% fits a smaller GPU classAlways a proposal: latency needs a benchmark

Commitments still apply: removing a node covered by a Savings Plan or reservation moves the commitment to other usage rather than cutting the bill, so KubeOn prices removals at your net amortized rate.

Autoscaling with limits you set

Scale up

GPU nodes added for pending GPU pods.

Spot with fallback

Spot first, on-demand when spot is gone.

GPU-aware packing

Fewer, fuller GPU nodes, MIG included.

Scale to zero

Idle GPU workloads off, empty nodes removed.

Hard limits

Nodes, monthly cost and GPU types per policy.

Revert

Undo a removal or a scale-to-zero from the log.

GPU autoscaling: your questions answered

KubeOn GPU

Keep exploring

Product

GPU sharing

Finds workloads whose GPU compute and memory fit in part of a card, then partitions supported GPUs with MIG or time-slices development GPUs, drains the node in your change window and reschedules the pods onto their slices.

Product

GPU burst

When a region has no GPUs left, attaches GPU nodes from the regions, clouds and data centers a policy allows, priced with data transfer, and moves the pods back home when capacity returns.

Product

GPU cost and utilization

Splits each GPU node's billed cost by the GPUs pods requested, reports GPUs nobody requested as idle GPU time, and shows how busy the allocated GPUs really are, from NVIDIA DCGM metrics.

Evaluate GPU autoscaling with your team

Bring your platform and FinOps priorities. We will walk through the data, controls and evidence for your evaluation.

Book a demoRead the guide