The short answer: a GPU instance bills by the hour whether it is serving requests or waiting for them. Utilisation, the share of billed hours spent doing useful work, is the number that decides what your hardware actually costs you, and for most early-stage teams it is far lower than anyone assumes.

Where idle time hides

Idle GPU spend rarely comes from one dramatic mistake. It accumulates in four ordinary places:

• Development and staging boxes running around the clock. A week has 168 hours; a working week uses perhaps 50 of them. An instance left up continuously bills the other 118 for nothing.

• Oversized instances. Hardware chosen for the biggest job it might ever run spends most of its life loafing through smaller ones, billing at full rate throughout.

• Always-on endpoints serving spiky traffic. An inference server sized for the weekday peak sits close to zero every night and all weekend, and the meter runs regardless.

• Finished experiments nobody terminated. The classic zombie resource: a training run ended weeks ago and the instance is still there, billing quietly under a name nobody recognises.

Measure before you fix

One weekly number tells you where you stand: GPU hours billed against GPU hours busy. The utilisation metric comes free with the hardware (nvidia-smi on the box, or your provider’s monitoring agent feeding the same figure into its dashboards). Pull it into whatever weekly review you already run.

As a working rule, sustained utilisation below about 40 per cent means a structural change will pay for itself; above 70 per cent, you are running a tight ship and the remaining gains are small.

Fixes, in order of effort

• Schedule shutdowns for everything non-production. An instance scheduler or a small cron job that stops dev and staging boxes at night and weekends is an afternoon of work and typically the single largest saving on this list.

• Terminate, don’t just stop, finished work. A stopped instance can still bill for attached storage. When an experiment ends, end its resources with it.

• Use scale-to-zero for spiky inference. Serverless GPU endpoints spin up on request and stop billing when traffic stops. Latency on the first request is the trade; for internal tools and low-traffic features it is usually a fine one.

• Autoscale production on queue depth. For inference workloads, the pending request queue is a real-time signal; processor and memory metrics lag. Scaling on queue depth keeps capacity close to demand in both directions.

• Batch the work that can wait. Embedding jobs, evaluations, and scheduled reports can run in windows on shared capacity instead of holding their own hardware open all day.

• Right-size and share. Benchmark whether a smaller GPU, or a slice of one, meets your latency target before renewing the big one. Small models often need far less hardware than they were given.

• Use spot capacity for interruptible training. Discounts are steep, and a job that checkpoints regularly loses little when an instance is reclaimed.

Keep it visible

Idle spend returns whenever attention moves elsewhere, so make the utilisation number a standing item rather than a one-off audit. Reviewed weekly next to cost per request, it turns GPU spend from a lump on the invoice into something you steer.