What idle GPUs actually cost

A GPU instance bills by the hour whether it’s working or sitting idle, so what it actually costs you comes down to utilisation: the share of those hours spent on real work. For most early-stage teams that share is lower than they’d guess, and the good news is that most of the fixes take an afternoon rather than a rebuild.

Where idle time hides

Idle GPU spend rarely comes from one big mistake. It builds up in a few ordinary places:

• Dev and staging boxes left running around the clock. A week has 168 hours and you might use 50 of them, so the rest is billed time doing nothing.

• Oversized instances. Hardware picked for the biggest job it might ever face spends most of its life on smaller ones, and you pay full price the whole time.

• Always-on endpoints for spiky traffic. An inference server sized for weekday peaks sits nearly idle every night and weekend, but the billing doesn’t stop.

• Finished experiments nobody shut down. A training run ended weeks ago, the instance is still there, quietly billing under a name nobody recognises. That’s the classic zombie resource.

Measure before you fix

One weekly number tells you where you stand: GPU hours billed against GPU hours busy. You don’t have to work it out by hand. On the instance, nvidia-smi shows current usage and utilisation over time, and most providers show the same metric in their monitoring dashboards under instance metrics. Add it to your weekly review.

A rough rule: below about 40 per cent sustained utilisation, it’s worth changing how the workload runs. Above 70 per cent you’re already running lean, and there’s not much more to save.

Fixes, roughly in order of effort

None of these need a new tool.

• Schedule shutdowns for anything that isn’t production. An instance scheduler, or a one-line cron job that stops dev and staging boxes at night and on weekends, takes an afternoon and is usually the single biggest saving you’ll find. On Linux, a crontab line like 0 20 * * 1-5 sudo shutdown -h now powers the box down at 8pm on weekdays, and most clouds have a built-in scheduler where you just set start and stop hours.

• Terminate finished work, don’t just stop it. A stopped instance can still bill for attached storage, so when an experiment ends, end its resources too.

• Use scale-to-zero for spiky inference. Serverless GPU endpoints spin up on request and stop billing when the traffic stops. The cost is a cold start of a few seconds on the first request after idle, which is fine for internal tools and low-traffic features. If you need consistent sub-second responses, it won’t fit.

• Autoscale production on queue depth. For inference, the pending-request queue is a real-time signal, and processor and memory metrics lag behind it, so scaling on queue depth keeps capacity closer to demand.

• Batch the work that can wait. Embedding jobs, evaluations, and scheduled reports can run in windows on shared hardware instead of each holding its own instance open all day.

• Right-size before you upgrade. Before renewing the big GPU, benchmark whether a smaller one, or a slice of one, meets your latency target. Run a typical workload on the smaller card and time it with time curl or a short script that records latency. Even one batch of your real data will tell you whether the performance holds.

• Use spot capacity for training that can be interrupted. The discounts are steep, and if your job checkpoints regularly you lose little when an instance gets reclaimed.

Keep it visible

Idle spend creeps back as soon as you stop watching, so make utilisation part of your weekly review instead of a one-off audit. Check it alongside cost per request, and GPU spend stops being a surprise on the invoice and becomes a cost you can control.