Rightsizing requests with p95, headroom and a floor

When a CNCF survey asked why Kubernetes increased their cloud spend, 70% of respondents named overprovisioning (InfoQ, CNCF survey on Kubernetes overspend (March 2024)). Datadog's billing data shows the same thing from the other side: across its customers, 29% of container cost was workload idle, meaning capacity pods requested but did not use (Datadog, State of Cloud Costs 2024).

Requests matter more than usage because of how Kubernetes schedules. A pod is placed on a node only if the node has room for its requests, and that room stays reserved for the pod whether it uses it or not (Kubernetes, Resource management for pods and containers). An inflated request is capacity you pay for and nobody else can use.

Why requests drift up and never come down

• They are copied. A new service starts from the last service's manifest, and its numbers.
• They are set once, under pressure. A launch, an incident or a load test sets a number nobody revisits once things are stable.
• The failure modes are lopsided. A request that is too low can cause throttling or an eviction, which someone notices. A request that is too high costs money every hour, which nobody notices.
• Nobody sees the cost. The bill shows nodes, not requests, so the team that sets the number never sees what it costs. See where Kubernetes spend goes.

Average, maximum or percentile?

Every sizing rule picks a statistic from a window of usage. The choice decides whether the rule is safe, cheap, or neither.

• The average (380m) breaks the busiest hour. Half the samples in a skewed workload sit above it.
• The maximum (1400m) is one sample: a deploy, a cache warm-up, a retry storm. Sizing to it pays for the worst hour every other hour of the week.
• The 95th percentile (620m) ignores the top 5% of samples. With headroom on top, it covers the normal peak without paying for the outlier.

The rule

1. Take 7 days of usage per pod, so weekday and weekend patterns are both in the window.
2. Use the 95th percentile, or the mean if the mean is higher (it can be, for a workload that is busy most of the time).
3. Add 20% headroom.
4. Never go below a floor: 0.025 cores for CPU and 0.25 GB for memory, so tiny services keep room to start.
5. Flag a container only when usage is under 30% of its request, so small differences don't create churn.
6. Report containers using 2% of their request or less as possibly abandoned, not as rightsizing candidates.

cpu    = max(p95_cpu    x 1.2, 0.025 cores)memory = max(p95_memory x 1.2, 0.25 GB)

Memory is not CPU

A container can use spare CPU above its request, up to its limit, and is throttled at the limit; it slows down but keeps running. Memory behaves differently: when a node runs short of memory, pods using more than their request are the first to be evicted, and a container that exceeds its memory limit is killed (Kubernetes, Resource management for pods and containers). Before approving a lower memory request, compare it with the workload's highest memory use, its limit and its restart history.

What an hourly rule cannot see

Ship it as a change, not a side effect

A recommendation in a dashboard rarely gets applied. A change with its evidence, a risk level and the patch already written usually does. The patch for the example above is three lines:

resources:  requests:    cpu: 744m      # was 2000m. p95 620m over 7 days, +20% headroom    memory: 1.1Gi  # was 4Gi. p95 0.9Gi over 7 days, +20% headroom

Then shrink the node group. Lower requests move cost from workload idle into cluster idle; the saving reaches the bill only when nodes scale down or pods pack more tightly. Compare the cluster's bill before and after.