Skip to content
NewFinOps, Kubernetes and AI, on one platform
Book a demo

GPU cost and utilization

Every GPU-hour, with an owner and a utilization

KubeOn splits each GPU node's cost by the GPUs pods requested, hour by hour, reports the GPUs nobody requested as idle GPU time, and shows how busy the allocated GPUs really were. Cloud GPUs at your billed rate, owned GPUs at your rate card.

Book a demoGPU guide

The most expensive nodes have the least visibility

  • Shared GPU nodes, no owner:

    Several teams' pods run on the same GPU nodes, so the bill shows one large line nobody owns.

  • Allocated is not used:

    A pod holding a GPU looks fully utilized to the scheduler, even when its kernels run a few percent of the time.

  • Idle cards hide in totals:

    GPUs no pod requested disappear into a cluster average, at several dollars an hour each.

  • Owned GPUs look free:

    On-premises GPU servers have no hourly bill, so their idle hours never reach a budget.

GPU cost allocation

GPU node cost split by the GPUs each pod requested

Kubernetes hands out GPUs as whole devices in a container's limits. KubeOn reads each node's GPU capacity and every pod's GPU request from the API each hour, and splits the node's billed, net amortized cost by the GPUs held. Teams see GPU cost next to the rest of their spend, by namespace, label and cost center.

  • Net amortized node cost, commitments and discounts applied
  • Hourly GPU requests per pod, including MIG slices
  • Same owners and reports as CPU and memory cost

Idle GPU time

GPUs no pod requested, priced on their own

GPU capacity that no pod held is idle GPU time, priced at the node's rate and kept in its own column. It is a scheduling problem, with a different fix from low utilization: fewer nodes, a smaller pool, or a schedule. Each GPU node pool shows its allocated and idle GPUs and what the idle ones cost.

  • Idle GPU-hours per node pool, per day
  • Kept out of team totals unless your idle policy spreads it
  • The input to packing and scale-to-zero policies
Idle GPU time, one month
prod-use1-a / ml-inference        g6e.12xlarge x 3, 4 x L40S eachGPU capacity                      12 GPUs x 730 h  = 8,760 GPU-hoursGPU requests (pods)                7 GPUs x 730 h  = 5,110 GPU-hoursidle GPU time                                        3,650 GPU-hoursidle share of the pool's cost     3,650 / 8,760    = 41.7%

Real utilization

How busy the allocated GPUs were, from DCGM

With the NVIDIA DCGM exporter scraped by Prometheus, KubeOn reads GPU utilization and memory per GPU and maps them to pods. GPU utilization counts the time any kernel was running, so it overstates how hard a GPU worked; where profiling metrics are on, SM activity is reported next to it.

  • GPU utilization and memory used, per pod, per hour
  • SM activity where profiling metrics are enabled
  • Without DCGM, cost and idle GPU time still work from requests
Series KubeOn reads
# Prometheus scrape for the NVIDIA DCGM exporter (GPU Operator)# KubeOn reads these series per GPU, per podDCGM_FI_DEV_GPU_UTIL      # % of time a kernel was runningDCGM_FI_DEV_FB_USED       # GPU memory in use, MiBDCGM_FI_DEV_FB_FREE       # GPU memory free, MiBDCGM_FI_PROF_SM_ACTIVE    # SM activity, where profiling metrics are on

Where the GPU savings are

52%

average use of the GPUs in a large Microsoft training cluster

Jeon et al., USENIX ATC 2019
18%

average GPU use on Alibaba machines with 74% of GPUs allocated

Weng et al., USENIX NSDI 2022
$8,433/mo

saved by removing one GPU node after the peak at our sample company

Sample company, not a customer result

KubeOn GPU overview

Three numbers per GPU node

Allocated, idle and used are different problems

MeasureWhat it tells youSourceTypical fix
Allocated GPU-hoursHow many GPUs pods held, and for how longPod GPU requests from the Kubernetes APIOwner and chargeback
Idle GPU-hoursGPU capacity no pod heldNode GPU capacity minus allocated GPUsFewer or smaller GPU nodes, schedules
Utilization of allocated GPUsWhether the GPUs pods hold are doing workDCGM exporter through PrometheusSharing, a smaller GPU type, or a hosted model

GPU cost uses the same allocation method as the rest of KubeOn: billed, net amortized node cost, hourly requests, and idle and shared cost in their own columns.

GPU spend your teams can see and own

GPU allocation

Node cost split by GPU requests, per pod and per hour.

Idle GPU time

Unrequested GPUs priced at the node's rate.

DCGM utilization

GPU utilization, memory and SM activity per pod.

Team GPU cost

By namespace, label and cost center, effective-dated.

Billed rates

Commitments and discounts applied; rate cards on premises.

MIG aware

MIG slices allocated by the share of the GPU they hold.

GPU cost and utilization: your questions answered

KubeOn GPU

Keep exploring

Product

GPU autoscaling

Adds GPU nodes when GPU pods wait, uses spot GPUs with an on-demand fallback, packs pods so empty nodes leave, and scales idle GPU workloads to zero, all within the limits each policy sets.

Product

GPU sharing

Finds workloads whose GPU compute and memory fit in part of a card, then partitions supported GPUs with MIG or time-slices development GPUs, drains the node in your change window and reschedules the pods onto their slices.

Product

GPU burst

When a region has no GPUs left, attaches GPU nodes from the regions, clouds and data centers a policy allows, priced with data transfer, and moves the pods back home when capacity returns.

Evaluate GPU cost and utilization with your team

Bring your platform and FinOps priorities. We will walk through the data, controls and evidence for your evaluation.

Book a demoRead the guide