Kubernetes
Finding orphaned volumes without deleting the wrong one
Disks outlive the pods and nodes that created them, and keep billing every month. How volumes get orphaned in Kubernetes, how to tell an orphan from a disk someone still needs, and how to delete safely.
UnitVane Engineering
September 21, 20264 min read
Storage is the quiet part of a Kubernetes bill. Compute shrinks when a cluster scales down; disks don't. AWS's documentation is explicit: volumes that are not deleted when an instance terminates continue to incur charges (AWS, How instance termination works). The same is true of persistent disks on every cloud.
Disks are also where old choices linger. Across Datadog's customers, gp2 volumes still made up 58% of the average organization's EBS spend (Datadog, State of Cloud Costs 2024), even though gp3 is up to 20% cheaper per GB, at $0.08 against $0.10 per GB-month, with a 3,000 IOPS baseline at any size (AWS, Amazon EBS General Purpose volumes).
How volumes get orphaned in Kubernetes
A PersistentVolumeClaim asks for storage; a PersistentVolume represents it; a cloud disk backs it. When the claim is deleted, the volume's reclaim policy decides what happens to the disk. With Delete, the disk is removed. With Retain, the volume becomes Released and the disk stays, so an administrator can recover the data by hand (Kubernetes, Persistent volumes: reclaiming). Retain is the right choice for data you care about, and it is also how disks pile up.
- Retained volumes nobody reclaimed. A StatefulSet is deleted, its claims go with it, and the Retain disks stay Released for months.
- Clusters torn down from outside Kubernetes. A cluster is deleted with its cloud tooling; its dynamically provisioned disks are no longer referenced by anything.
- Namespaces deleted in a hurry. Claims go, volumes with Retain stay.
- Disks that were sized for a peak. Not orphaned, but provisioned at 500 GiB for a dataset that never grew past 60.
Five kinds of forgotten storage
| Finding | Signal | Safe default |
|---|---|---|
| Orphaned volume | Disk still billing 24 hours or more after its PersistentVolume disappeared, from a cluster that is still reporting | Snapshot, wait, delete |
| Released volume | PV in Released, Available or Failed, with its disk still billing | Confirm the data is not needed, then delete or re-bind |
| Over-provisioned volume | At least 24 hourly samples over 14 days with 30% or more of the size unused | Migrate to a smaller disk |
| Released file system volumes | Released PVs on a shared file system | Clean up the directories, then the PVs |
| Possible orphans | Billed for 7 days with no PV and no cluster tag, in non-production accounts | Investigate; never delete automatically |
Why "cluster still reporting" matters
Delete safely: snapshot, wait, delete
A snapshot keeps only the data that is actually written and costs a fraction of the disk. Take one, keep it for a waiting period your data owners agree on, then delete the disk. If someone needed it, restoring from the snapshot is a ticket, not an incident.
# 1. Snapshot the volume and record where it came fromaws ec2 create-snapshot --volume-id vol-0abc123def4567890 \ --description "Orphaned PV pvc-7f3e from cluster prod-use1-a" \ --tag-specifications 'ResourceType=snapshot,Tags=[{Key=reason,Value=orphaned-volume}]'# 2. Wait until the snapshot is completeaws ec2 wait snapshot-completed --snapshot-ids snap-0123456789abcdef0# 3. After the agreed waiting period, delete the volumeaws ec2 delete-volume --volume-id vol-0abc123def4567890Over-provisioned is a migration, not a resize
Most cloud block volumes can grow but not shrink. Reclaiming space from an over-sized volume means creating a smaller one and copying the data, usually in a maintenance window, so size it with real headroom: at least 1.5 times the observed peak, and never below 1 GiB. Where you are migrating anyway, move gp2 volumes to gp3 in the same change.
Who does what
Turn measured waste into a review queue
- 01
Gather history
Confirm billing and the usage window required by each finding.
Diagnostics
- 02
Apply rules
Evaluate request sizing, node capacity, storage and off-hours opportunities.
Savings engine
- 03
Price the change
Estimate monthly savings against the applicable billing data.
KubeOn hub
- 04
Prioritize
Review evidence and risk before taking a proposal into your pipeline.
Platform team
Read-only by default. KubeOn changes a cluster only where you grant write access, feature by feature.
Read the workflowShould we switch every StorageClass to Delete?
No. Retain protects data from an accidental claim deletion, which is worth a lot for databases. Keep Retain where data matters, and review Released volumes regularly instead.
Can we automate the deletion?
Automate the evidence, not the deletion. A disk that looks orphaned can belong to a cluster being rebuilt, a migration in progress or a forensic hold. Let a person approve each one, with the snapshot as the safety net.
What about snapshots that pile up instead?
Tag every snapshot you take with the reason and the source, and give the tag a retention rule. A snapshot with no tag and no owner is the next orphan.
How does KubeOn handle volumes?
It joins every disk on the bill to the PersistentVolume and cluster it belongs to, reports the five findings above with their cost, and writes the snapshot-then-delete or migration steps into a proposal your team reviews. It never deletes anything.