Skip to content
NewFinOps, Kubernetes and AI, on one platform
Book a demo

GPU sharing

Workloads that use a slice of a GPU, given a slice

KubeOn finds workloads whose GPU compute and memory fit in part of a card. The GPU optimizer then partitions supported GPUs with MIG where isolation matters, or time-slices development GPUs, in your change window, and moves the pods onto their slices. Run each policy as a proposal, with an approval, or automatically.

Book a demoSharing rules

One small service, one whole GPU

  • Whole-device allocation:

    By default the device plugin gives each GPU to one container, however little of it the container uses.

  • Sharing has trade-offs:

    MIG isolates memory and faults but has fixed sizes; time-slicing is flexible but isolates nothing.

  • Not every GPU can partition:

    MIG runs on data center GPUs such as A100 and H100, not on L40S, L4 or A10G.

  • Repartitioning is disruptive:

    Applying a MIG profile reconfigures the GPU, so its pods have to move first, at a time that suits the service.

Sharing candidates

Workloads whose peak fits in part of a card

From DCGM history, KubeOn takes each GPU workload's peak GPU memory and compute. Workloads that hold a whole GPU but whose peak memory fits a smaller slice, with headroom, become sharing candidates, grouped by the node that would change.

  • Peak GPU memory, not the average, decides the slice
  • Packed onto as few GPUs as possible
  • Production and non-production treated differently

Multi-Instance GPU

MIG applied in your change window

On GPUs with MIG, such as A100 and H100, the optimizer picks the smallest valid profile for each workload, cordons the node, drains its GPU pods through the Eviction API, sets the GPU Operator's MIG configuration and lets the pods reschedule onto their slices. It waits for the policy's window, and for approval if the policy asks for one.

  • Only profiles valid for that GPU model
  • Hardware isolation of memory and faults
  • PodDisruptionBudgets respected during the drain
Sharing policy
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata:  name: dc1-migspec:  kind: Sharing  mode: Approve                   # each repartition waits for a reviewer  nodeSelector:    node.kubernetes.io/pool: gpu  mig:    allowedProfiles: [1g.5gb, 2g.10gb, 3g.20gb]    headroom: 20%                 # above peak GPU memory  timeSlicing:    namespaces: [ml-dev]          # non-production only    maxReplicas: 4  window: "Sat 02:00-06:00 UTC"

Time-slicing

Time-slicing for notebooks, development and small batch

Time-slicing lets several pods take turns on one GPU on any NVIDIA card, including those without MIG. It has no memory or fault isolation, so the optimizer applies it only in the namespaces a policy lists, with the number of replicas chosen so the workloads' peak memory fits together.

  • Replicas per GPU from peak GPU memory
  • Non-production namespaces by default
  • GPU Operator device plugin config managed for you
Time-slicing config the optimizer applies
# dev-use1 / ml-dev: four notebook pods per A10G, non-production onlyapiVersion: v1kind: ConfigMapmetadata:  name: time-slicing-config  namespace: gpu-operatordata:  a10g: |-    version: v1    sharing:      timeSlicing:        resources:          - name: nvidia.com/gpu            replicas: 4# the optimizer then labels each selected node nvidia.com/device-plugin.config=a10g

Where the GPU savings are

52%

average use of the GPUs in a large Microsoft training cluster

Jeon et al., USENIX ATC 2019
18%

average GPU use on Alibaba machines with 74% of GPUs allocated

Weng et al., USENIX NSDI 2022
$8,433/mo

saved by removing one GPU node after the peak at our sample company

Sample company, not a customer result

KubeOn GPU overview

Choosing a method

MIG, time-slicing or neither

MIGTime-slicingNeither
GPUsData center GPUs with MIG, such as A100 and H100Any NVIDIA GPU with the device pluginAny
IsolationMemory, compute and faults isolated in hardwareNone between the pods sharing a GPUWhole GPU per container
SizesFixed profiles per GPU modelAny number of replicasWhole GPUs
How the optimizer applies itCordon, drain, set the MIG config, rescheduleUpdate the device plugin config, restart the pluginNo change
Used forSmall production inference that needs isolationNotebooks, development and small batchWorkloads that keep a GPU busy

A training job running at 90% on 8 GPUs needs nothing. A service holding one GPU at 6% is a sharing candidate. The measurement comes first.

Sharing your reviewers can trust

MIG profiles

The smallest valid profile for each workload's peak memory.

Time-slicing

Replicas per GPU for development and notebooks.

Change windows

Repartitions only when the policy's window is open.

Isolation stated

Each change says what is and is not isolated.

GPUs freed

Whole GPUs released, priced at the node's rate.

Revert

Restore the previous profile from the audit log.

GPU sharing: your questions answered

KubeOn GPU

Keep exploring

Product

GPU autoscaling

Adds GPU nodes when GPU pods wait, uses spot GPUs with an on-demand fallback, packs pods so empty nodes leave, and scales idle GPU workloads to zero, all within the limits each policy sets.

Product

GPU burst

When a region has no GPUs left, attaches GPU nodes from the regions, clouds and data centers a policy allows, priced with data transfer, and moves the pods back home when capacity returns.

Product

GPU cost and utilization

Splits each GPU node's billed cost by the GPUs pods requested, reports GPUs nobody requested as idle GPU time, and shows how busy the allocated GPUs really are, from NVIDIA DCGM metrics.

Evaluate GPU sharing with your team

Bring your platform and FinOps priorities. We will walk through the data, controls and evidence for your evaluation.

Book a demoRead the guide