The short answer: AI features cost more than the rest of your stack because inference is billed on every request and grows with usage. Servers, storage, and databases settle into a baseline you can predict month to month; a successful AI feature keeps adding to the bill for as long as people use it.

This post covers where the money actually goes, the one number worth watching, and the habits that keep the bill in proportion to the value. None of it requires new tooling. Most of it is arithmetic, tags, and a weekly look at the right report.

Where does the money go when an AI request comes in?

A single AI request can generate charges in several places at once, and only some of them appear under an obvious heading on the invoice.

• Tokens. If you build on a model API, you pay per token, with input and output priced separately. Long system prompts, large context windows, and verbose responses all raise the count. A retrieval step that stuffs documents into the context can make one query cost fifty times another against the same model.

• GPU hours. If you host models yourself, you pay for the hardware by the hour, whether it is serving requests or sitting idle. Utilisation is the whole game here, and we come back to it below.

• The charges around the model. Responses leaving the cloud incur egress fees. Datasets, checkpoints, and vector databases sit in paid storage. Distributed jobs generate networking charges between zones. These lines rarely say “AI” anywhere, which is why AI spend so often hides inside general compute and storage totals.

• Multiplied calls. Retries, agent loops, and multi-step chains can turn one user action into a dozen model calls. The user sees one answer. The invoice sees twelve requests.

Knowing these categories exist is half the work. The other half is being able to see them, which is what tagging is for.

Inference grows with your success

Inference runs for as long as the feature exists, and it follows your traffic. Training a model, or fine-tuning one, is a project: it has a budget, a duration, and an end date. Inference has none of those.

That difference catches teams out because the training bill arrives first and frames expectations. For products with real production traffic, cumulative inference spend typically passes the training investment within weeks of launch, and it keeps compounding from there. The FinOps Foundation’s guidance on generative AI recommends modelling both training and inference costs by use case before a feature ships, precisely because the second number is the one that scales.

Falling per-token prices do not solve this on their own. Prices per token have dropped sharply year on year, yet total spend keeps rising across the industry, because usage grows faster than unit prices fall. A feature that gets ten times the traffic at half the unit price still doubles your bill.

The practical conclusion: budget for inference as a recurring cost that scales with adoption, and treat the training spend as the smaller, one-off part of the picture.

What is cost per request?

Cost per request is a feature’s total AI spend for a period divided by the number of requests it served. For token-billed models, cost per million tokens does the same job. It is the number that shows whether a rising bill means more customers used the feature or each request quietly became more expensive; total monthly spend on its own cannot tell those two apart.

Getting it takes three steps:

1. Count requests and tokens. Every major model API returns token counts in its responses. Log them. If you self-host, emit request counts as a metric alongside GPU hours.

2. Divide spend by volume. Provider invoice for the feature, divided by requests served, per week.

3. Watch the trend, not the level. A stable cost per request with a rising bill means growth, which is usually good news. A rising cost per request means something changed: a longer prompt, a new retrieval step, a retry loop, a model swap. You want to find out which, in the same week it happened.

This one metric turns the invoice from a verdict into an instrument. It also gives you the input for every pricing and packaging decision the feature will ever face.

Tag your AI spend from the first week

AI charges scatter across compute, storage, API, and networking lines by default. Without tags, they are unrecoverable at the end of the month, and any cost-per-request calculation becomes guesswork.

Three tags are enough to start:

• Model (which model or model family served the request)

• Feature (which product capability it belongs to)

• Team (who owns the decision to change it)

Apply them at job submission or in your request middleware so they flow through automatically. Retrofitting tags onto three months of untagged spend is slow and never fully accurate. Adding them in week one costs an afternoon.

Habits that keep the bill proportionate

These are the operational habits that separate teams whose AI costs track their growth from teams whose costs outrun it. Each one is worth checking against your own setup this week.

• Shut down idle GPUs. Development and staging instances left running overnight and at weekends are among the largest sources of pure waste in AI budgets. Automate the shutdown rather than relying on memory.

• Batch work that can wait. Embedding jobs, evaluations, and report generation rarely need to run the moment they are requested. Batching them raises utilisation and can qualify for cheaper capacity.

• Route by task difficulty. Classification, extraction, and short lookups run well on small, cheap models. Reserve the frontier model for the requests that genuinely need it.

• Cache what repeats. Stable system prompts and frequently referenced documents are strong candidates for prompt caching, which most major APIs now support at a discount.

• Cap retries and loops. Set a hard ceiling on retry counts and agent iteration depth. Unbounded loops are how a bug becomes a five-figure line item.

• Right-size before upgrading. Benchmark whether the smaller GPU, or a quantised model, meets your latency target before reaching for bigger hardware. Often it does.

Set the alert at 80 per cent

Every major provider offers budgets and alerts at no extra cost: AWS Budgets, Microsoft Cost Management, and Google Cloud budget alerts all take minutes to configure. Two settings matter.

Set the alert threshold at 80 per cent of budget rather than 100. An alert at full budget arrives when there is nothing left to do except explain it. At 80 per cent you have a week or two inside the billing cycle to find the cause and act.

And review inference costs weekly. Monthly reviews suit stable infrastructure, where costs drift slowly. Inference moves too fast for that cadence; a regression can double a feature’s cost in days, and a weekly look catches it before it compounds.

One honest caveat: these are alerts, and alerts only. No major provider currently offers a hard spending cap that stops charges at a limit you set. The alert buys you time to respond. It does not respond for you.

Model the growth before it arrives

The final habit is the one that protects your margins rather than your monthly bill. Before an AI feature ships, take your measured cost per request and multiply it out: what does this feature cost at ten times the traffic, and at a hundred times?

That single calculation tells you whether your pricing survives success. A feature costing €0.003 per request under a €50 monthly subscription behaves very differently from one costing €0.14 per request under the same subscription. Founders who run this arithmetic early get to choose their pricing model deliberately. Founders who skip it find out from the invoice.

The same maths supports your infrastructure decisions. Once you know the workload’s shape, you can price it properly: on-demand against committed capacity, API against self-hosted, and one provider’s rates against another’s.

That last comparison is the question we built Clouding around. Our forecasting and what-if scenario testing show what a decision costs before you make it, across AWS, Azure, and Google Cloud, with your own rates and credits applied. If you would rather run the numbers yourself first, everything in this post works with a spreadsheet and your provider’s billing export, and we would genuinely rather you did that than fly without the numbers at all.

Frequently asked questions

Why is my AI bill so high?

Usually because inference scales with usage while the budget was set around training or the infrastructure baseline. Long prompts, unbounded retries, and idle GPUs are the most common specific causes. Calculating cost per request shows which one applies to you.

How do I calculate cost per request?

Divide a feature’s total AI spend for a period by the number of requests it served in that period. Token counts come back in every major model API response, and if you self-host you can log request counts alongside GPU hours. Review the trend weekly rather than the level.

Can I set a hard spending cap with AWS, Azure, or Google Cloud?

No. All three providers offer budgets and alerts at no extra cost, and none of them will stop charges at a limit you set. Setting the alert at 80 per cent of budget leaves time to act inside the billing cycle.

Do falling token prices mean AI costs will sort themselves out?

No. Per-token prices have fallen sharply, yet total spend keeps rising across the industry because usage grows faster than unit prices fall. The bill follows adoption, so the habits in this post matter at any price level.