AI costs on cloud bills are difficult to predict, often based on usage metrics that are impossible to track in advance. As a response, companies are moving workloads to more budget-friendly alternatives instead of defaulting everything to the public cloud. This report lays out the numbers and the new laws tipping towards the customer, and shows what to fix and when (if) to move.
Executive summary (TL;DR)
AI stopped being a research project and became a running cost some time in the last two years. The online chatter is intense. What’s changed is that the invoice now arrives every month and it grows with the product's usage. Because of this, an increasing number of companies are moving parts of their infrastructure, especially AI workloads, back out of the cloud. The analyst firm IDC says around 80% of companies expect to bring some compute or storage back in-house within a year, and about a quarter already did. The trigger is cost, and specifically the parts of the invoice that turned out to be hard to predict.
AI is what tipped it. Inference (running a trained model for live outputs) grows with success, and it is billed by the token, the unit no one can count before the request runs. Fees for moving data out of the cloud often make up 15% to 30% of a standard AI bill, catching teams off guard in many cases. On top of that, GPU capacity is limited and the price can jump without warning, and some data has to stay in a particular country by law. Running AI in the public cloud 24/7 is not the automatic choice for many workloads anymore.
None of that makes cloud the wrong answer, but the days of defaulting to it for everything are over. The most effective strategy now is to evaluate each project on its own merits to find the best place to host it. This report breaks down the costs and provides a guide for choosing the right infrastructure for every job, particularly as moving data becomes simpler and regulations start to tip the balance in favour of the customer.
AI is now the largest operating cost
Training a model is a one off. You rent the hardware, run the job, and the bill stops when the job finishes. Inference is the opposite. It runs every time someone uses your product, so the cost climbs with adoption rather than winding down after launch. Industry analysts put inference at 55% to 80% of enterprise GPU spend in 2026, which is why a successful launch can turn into a huge margin problem a few months later, as rising product adoption directly leads to higher infrastructure costs.
One line deserves special attention: data egress. Providers let you move data in for free and charge you to move it out, usually a few cents per gigabyte. On storage that mostly sits still, that rounds to nothing. On an AI service sending responses to users all day, it grows with every user you add and can reach 15% to 30% of the total. Because it scales with usage rather than with any decision you made, it tends to arrive as a jump nobody can immediately explain.
It creates a catch 22: the growth that shows the product is working is the same thing that pushes the invoice up, so a strong month can be a hard one for the margin, especially when spend climbs faster than the revenue it tracks.
Younger companies, especially AI-first startups, now spend 40% to 50% of revenue on GPU hosting, inference and data, against the 15% to 20% that traditional software companies spend on cloud. Even outside AI-native companies, GPU has grown to roughly 18% of cloud spend, up from about 4% in 2023, according to the FinOps Foundation's 2026 survey. The number is not an engineering detail. It appears in board materials, and it's under the same kind of scrutiny that used to be reserved for headcount.
GPU spend growth:
The GPU build-out behind this is huge and the people accountable for it are under real pressure: in a 2026 survey of 260 finance leaders, two-thirds said their board now ties further AI funding to proof of return, and three-quarters of teams that cannot show that return have held back or frozen AI investment. Deloitte's larger multi-country survey of AI in the enterprise reads the same way, with spend rising and returns still hard to show.
So the person who owns this invoice wants two things: to know what it is really costing, and if the same work could run cheaper somewhere else.
Paying for twenty times the GPU you use
A lot of the spend buys capacity that does nothing. CAST AI’s 2026 analysis, which looked at real usage data from more than 23,000 Kubernetes clusters (the systems that run packaged software workloads), found typical GPU use at 5%. Average CPU use was 8%, down from 10% the year before, and memory sat at 20%. On those numbers, companies are paying for roughly twenty times more GPU capacity than they use. It’s not just about the numbers: a 2026 survey of 1,800 tech leaders found that 97% think they’re wasting cloud budget, and more than half said that over 25% of their total spend is just going to waste.
Average cloud use utilisation:
This doesn't happen because teams are careless, but because these systems are complex to manage. GPU nodes get provisioned for peak demand and then never scaled back. Training jobs finish but the pods (small units of running software) keep running. One workload takes up a whole GPU even when it touches a fraction of the chip. And teams hold on to capacity because letting it go feels riskier than paying for it, given how long it takes to get back. That instinct made sense when compute got cheaper every year. It stopped making sense in 2026. There are ways to claw this back, and the next sections work through them, from how you run the hardware to how you buy it.
The GPU economics are changing
For the first time since cloud services (EC2) became standard in 2006, GPU prices went up rather than down. On 4 January 2026 AWS raised prices for NVIDIA's high-powered AI chip H200 by 15%, which broke a twenty-year run of falling compute costs. An idle GPU that used to be a small line now costs euros an hour to leave running, against cents for an idle CPU core. The hoarding that felt safe before is expensive, and it feeds the shortage that pushes prices up further. Another catch 22, perhaps.
Waste across the wider cloud spend tells the same story. Flexera's 2026 State of the Cloud report, based on 753 cloud decision-makers, found estimated wasted spend climbing to 29%, the first increase in five years after a long drop. The cause was of course AI: resource-heavy instances landing on invoices faster than teams could track them, alongside a scatter of new services with their own pricing. Nearly every team is now on the hook for AI spend, but only about a third report visibility into their AI software, and most say the waste grew over the past year.
None of this is a reason to spend less on AI but to make sure that every project is generating real, and measurable returns.
The savings you already have
Before you look at other providers, look closer at your own. Most teams leave big savings on the table by shopping around before they have fully optimised what they already have. The work splits into two parts: getting more from your hardware and buying it more carefully.
On the hardware side, four levers do most of the work. Request batching groups queries into a single call so the chip carries a full load instead of one request at a time, which lowers the cost per transaction. Prompt caching stores answers to repeated questions so the model does not compute the same reply twice. Quantisation runs a model at lower numerical precision, INT8 or FP16, which cuts the memory it needs and lets it run on smaller hardware with little loss of quality. And spot instances, spare capacity sold at a discount, run at 60% to 90% below standard rates for work that can tolerate interruption, such as background or asynchronous jobs.
Buying capacity involves a timing risk. For example, AWS cut H100 prices by 44% in 2025, leaving those on old contracts paying much more than new customers. To avoid getting locked into yesterday's prices, don't commit to everything at once. Staggering your contracts and keeping some usage on shorter terms lets you adjust as rates change, rather than just renewing the same deal out of habit.
The last piece is managing costs, where standard cloud tools often fall short for AI. A common rule like "switch off anything idle" is dangerous: it causes long startup delays for models, which creates a frustrating wait for users. Looking at average usage also hides the truth, as it ignores the spikes in demand that actually dictate performance. Instead, track demand at the busiest times rather than the average, and set realistic usage targets for GPUs, consider 60% rather than the 30% that works for CPUs. These policies always balance speed with cost. Costs are assigned to specific teams to ensure accountability, and policies are updated as usage patterns change. Saving a little money is pointless if it slows down the experience for users.
All of this is worth doing, and none of it answers the larger question. Every lever here works inside one provider's pricing. None of it can tell you whether that provider, or public cloud at all, is the right home for the workload to begin with.
Three tiers of cloud cost tools
The cloud cost-management market has grown crowded enough that it helps to sort it into three tiers by what the tools actually do.
The first tier covers simple price calculators. Tools like Costimizer let you type in your requirements to compare list prices. These are fine for a rough guess, but they can't see your specific environment, your negotiated rates, or how your costs change over time.
The second tier handles reporting and optimisation. This includes well-known tools like CloudHealth, CloudZero, Vantage, Finout, nOps, and Amnic. These tools are excellent at showing you the bill you already have. They collect your data from different cloud providers and translate costs into business metrics, like cost per customer or per feature. However, they all share a limit: they only focus on the cloud providers you already use. They report on your current setup, but they won't tell you if you should be using a different provider entirely.
A few things to note here. Many of these tools charge a percentage of your total cloud spend, often 1% to 3%, which means their fees grow as your costs grow. Also, older enterprise options like Broadcom’s CloudHealth are expensive and complex, often costing $45,000 a year or more. They are powerful, but clearly built for massive organisations.
The third tier is cross-provider comparison, which is still rare. These tools look at your current environment and pricing, then compare your costs against other providers to see if you could save money elsewhere. They answer the question: "Could I get the same work done cheaper on another cloud?" Note that all three tiers share a common limit: they only compare different public cloud providers. They cannot tell you if you should move off the public cloud entirely, which is a question we will cover next.
The return to private infrastructure
Companies are no longer just planning to move workloads out of the public cloud; they are actively doing it. The trend is widespread, but in AI, the shift is sharper: in one 2026 survey the public-cloud share of production inference fell from 56% to 41% in a single year. This is repatriation: moving data and steady, heavy jobs such as inference to hardware the company controls, usually to escape unpredictable invoices and data-transfer charges.
Public cloud inference:
AI triggered this shift, for two reasons. Continuous inference runs at high, steady utilisation, the one pattern where owning hardware can beat renting it, and it moves a lot of data, which is where egress bites hardest. Where a workload runs flat out around the clock, the sums have started to favour owning. Where it is bursty or unpredictable, renting still wins.
The law is moving the same way. The three major providers waived egress charges for departing customers in 2024, and from 12 January 2027 the EU Data Act removes switching charges across the EU and requires providers to make switching work, through open interfaces and functional equivalence (the new provider must offer the same core functions). The Commission's stated aim is to open the cloud market so customers can switch freely. For companies inside the EU, including in markets such as Sweden and Germany, the cost and friction of leaving are being taken down by regulation.
Cloud cost management is also adapting, to meet user needs, and a real pressure to change. In 2026, the FinOps Foundation updated its mission for the first time in six years, shifting focus from the "value of cloud" to the "value of technology." The practice has changed: instead of just reporting on past spending, the goal is now to shape decisions before money is committed. Proactively deciding where a workload should run, rather than reacting after the fact, is now the core of the role.
Moving infrastructure is not a default solution. Despite the talk of repatriation, public cloud spending grew by 21% in 2025 to $723 billion, indicating that most companies are being selective rather than leaving the cloud entirely. IDC reports that fewer than 10% of companies plan a full exit. In addition, rising hardware and memory costs in 2026, combined with the operational expenses of power, staff, and hardware updates, mean that owning infrastructure is not always the most economical choice. There is a consensus: infrastructure decisions should be made on a per-workload basis, focusing on utilisation and data sensitivity, rather than as a mass retreat from the cloud.
The main obstacle is a lack of clear data. Many teams are still interested in moving, but not all have a concrete understanding of what a specific workload would cost on an alternative platform.
A practical guide to cost control
To make informed decisions you can audit the costs and usage of one representative workload before comparing them against other options. None of this requires a single large decision. It works as a sequence, and most teams can run it over a quarter. This approach works best when finance and engineering teams work together from the start. A small-scale pilot is not expensive and shifts the advantage, allowing negotiations to be built on internal data instead of vendor estimates.
Visibility comes first.
Native billing tools lag, often by a couple of days, which is enough to turn a small overrun into a large one before anyone sees it. Get spend visible faster than that, and tag resources at the point they are created so every cost maps to a team or product. Set a baseline in units the business cares about, such as cost per inference or cloud cost as a share of revenue. Mature teams are moving this way already: the share tracking unit economics has risen to 49% from 40% a year earlier, and across the discipline the top measure of success has shifted from cutting costs to the value the spend delivers. Then set alerts that fire before a budget is breached rather than after. The finance data backs the value of speed here: teams that see spend the same day invest more confidently, and among teams that go over budget, same-day visibility sharply cuts the odds of a serious board consequence.
Then tune what you have.
Turn on the technical levers, batching, caching and quantisation, to lower the cost floor. Move work that can tolerate interruption to spot instances. Right-size against 95th-percentile demand rather than averages, and put latency-aware policies in place so cost savings never come at the expense of the response times your users depend on.
Then look outward.
With switching charges falling away, the next commitment or credit expiry is the moment to ask where each workload belongs: which provider, or whether it should run somewhere other than public cloud at all. Treat the answer as evidence for a decision you are already facing. Sometimes it confirms you are in the right place and gives you a number to negotiate with. Other times it points somewhere cheaper. Either way, the decision runs on data rather than on the deadline, and your board reporting moves from a total-spend figure nobody can act on to a unit-cost figure that shows whether growth is paying for itself.
Making data-driven infrastructure choices
The work inside a single provider is real, and in 2026 everyone's already doing it. Better use of the hardware and careful commitments bring the costs down, and a bit of care over latency keeps the service quick while you do it. Every serious tool helps with some of this. What none of them do is step outside the provider you happen to be on and ask whether it's still the right home for your specific, and unique workload.
That question used to be blocked by exit fees, and was hard to answer even if you got past them, because the data wasn't there. Both are changing at once. The fees are going, and the tools to compare your own setup against the market now cost little enough that a growing company can justify them. Clouding does one part of this: it takes your actual setup, applies the rates, credits and commitments you've actually negotiated, and tells you whether another provider comes in cheaper before your next renewal. It won't move you off the public cloud, and it won't pretend switching is always the answer. It's developed for young, AI-first companies’ wallets, where the enterprise tools start at €40,000 a year, and it hands you the number and lets you decide.
A note on our sources
Many of the studies in this report come from companies that sell cloud cost or infrastructure tools of their own, Broadcom, Flexera and CloudZero among them, and some of what they do overlaps with what we do. Firstly, we would like to thank them for allowing us to read their findings. What we did, however, was to set their research alongside independent analysts like IDC and Gartner. We prefer to hand you the best evidence instead of only the sources that happen to flatter us and our business. If the right tool for your situation turns out to be one of theirs, that is a decision worth making on the numbers.
Sources
Gartner, worldwide IT spending forecast, July 2026: Data-center systems are the fastest-growing spending segment, totaling $822 billion.
EU Data Act: Mandates removal of cloud switching charges and requires functional interoperability starting January 2027.
FinOps Foundation, State of FinOps 2026: Focus shifted from "value of cloud" to "value of technology"; 98% of practitioners now manage AI spend.
HackerNoon and Backblaze / Dimensional Research: Ingress is free, but egress runs a few cents per gigabyte and often reaches 15%–30% of total AI workload cost.
DataCenterDynamics: Major cloud providers waived egress fees for departing customers in 2024 to prepare for regulatory changes.IDC: 80% of enterprises expect to repatriate some compute or storage within a year; roughly 25% have already moved at least one workload.
Gartner, worldwide public cloud spending forecast: Public cloud spending reached ~$723B in 2025 (+21% YoY), indicating that repatriation is selective rather than a mass exit.
Deloitte, State of AI in the Enterprise 2026: Enterprise AI spend is rising, but returns remain difficult to demonstrate.
AWS H100 on-demand price cut: A ~44% price cut in 2025 illustrates the high risk of commitment timing.
Broadcom, Private Cloud Outlook 2026: Public-cloud inference share fell from 56% to 41% YoY; 97% of IT leaders believe some cloud budget is wasted.
Flexera, 2026 State of the Cloud report: Estimated wasted spend rose to 29%; fewer than half of organisations utilise commitment discounts.
Flexera, 2026 State of ITAM report: Only 31% of organisations have visibility into AI software; 59% report increased wasted AI spend.
CAST AI, 2026 State of Kubernetes Optimisation report: Average GPU utilisation 5%; CPU 8%. AWS H200 prices increased by 15% in January 2026.
CloudZero, "Finding the ROI of AI: the finance perspective", June 2026: 66% of boards condition AI funding on ROI; 74% of AI spend is driven outside of finance.
Amnic and AI Business: AI pricing is unpredictable; token counts are opaque, leading to sudden cost spikes.
Industry analyst estimates: Inference accounts for 55%–80% of enterprise GPU spend; AI-first startups spend 40%–50% of revenue on GPU and hosting.
