AI is now a FinOps problem
The FinOps Foundation's State of FinOps 2024 and 2025 reports put the shift in plain numbers: "Managing AI costs" and "Managing commitment-based discounts" have taken the top spots in practitioner priorities, displacing the long-standing "rightsizing EC2" concerns that defined earlier cycles. Flexera's State of the Cloud surveys report the same direction – AI and ML infrastructure is the fastest-growing category of cloud spend for three years running.
That shift breaks a lot of existing FinOps muscle. The tools and dashboards most engineering organisations rely on were designed for a world of steady compute, attributable to services via tags, optimised via instance sizing and commitment discounts. AI workloads violate every one of those assumptions. They are bursty, they are expensive per unit of time, they span GPU, vector DB, inference endpoints, and third-party model APIs, and they are increasingly run by teams who do not own the invoice.
Leak 1 – GPU under-utilisation
The most common and expensive leak is the one that looks cheapest on the dashboard. GPU instances are priced per hour – whether they are running a training job, waiting on a data loader, or sitting idle on a stale Jupyter kernel. Industry reports from Run.AI and Weights & Biases have put median GPU utilisation in enterprise ML clusters at 30-40% across 2023-2025, with many production environments well below that.
A single A100 left running on an AWS EC2 p4d instance idles at roughly US$4/hour. Ten data-science users with "I was going to get back to it" notebooks costs a mid-sized ML team six figures a year before anyone ships a model. The fix is not a moral lecture to engineers – it is policy in code: idle-kernel auto-shutdowns, mandatory TTLs on notebook instances, GPU quotas per team per environment, and schedulers (Kueue, Volcano, Slurm) that pack jobs instead of pinning them.
Leak 2 – Token sprawl and model sprawl
LLM APIs reintroduced variable, per-request pricing to a stack that had mostly grown comfortable with flat-rate compute. Every call to OpenAI, Anthropic, or Google carries an input-token and output-token charge, and the costs compound in non-obvious ways: a RAG pipeline that grew from 4k to 32k context, a chain-of-thought prompt that doubled output length in an A/B test, a retry loop that silently fires three times on transient failures.
Worse, those costs are rarely visible to the team that triggered them. A product manager running an A/B in notebook form, a customer-success agent using an internal assistant, a background job re-indexing a knowledge base – each can spike the same API bill with no attribution. The fix is per-feature token metering, pushed up to the product layer: request-level logging of input/output tokens, tagged by feature and user cohort, rolled up the same way you already roll up cost-per-acquisition or cost-per-request.
Model sprawl compounds the token problem. Every major provider now ships a tiered catalogue (OpenAI's GPT series, Anthropic's Opus/Sonnet/Haiku, Google's Gemini tiers, Meta's Llama variants self-hosted). Teams pick the flagship, never revisit, and pay 5-10× more than necessary for calls a smaller model would serve indistinguishably. Quarterly "model right-sizing" reviews – the AI equivalent of EC2 rightsizing – often recover 30-50% of LLM spend with no user-visible change.
Leak 3 – Commitment and spot strategy misalignment
For self-hosted training and inference, the underlying cloud economics look like any other compute workload – but the commitment strategy that worked for web services breaks on AI. On-demand GPU instances are the most expensive option. Reserved / Savings Plans / Committed Use Discounts offer 30-60% off in exchange for 1- or 3-year commitments. Spot / Preemptible instances offer up to 90% off but can be reclaimed with short notice.
Training workloads are ideal candidates for spot, because they can be checkpointed and resumed. Many teams never set this up; they pay on-demand prices for jobs that could tolerate interruption. Inference workloads are the opposite – they need predictable latency, and spot is usually wrong. A mature AI FinOps practice decomposes the portfolio explicitly: commit for the inference floor, spot for training, and on-demand only for elastic overflow.
The same calculus applies to managed inference endpoints (SageMaker, Vertex AI, Azure ML). Their pricing is convenient – and 2-3× more expensive per GPU-hour than the equivalent raw instance – which is fine if they save engineering time and bad if they sit idle overnight.
A FinOps playbook for AI
A minimum viable FinOps practice for AI looks like this, in order of leverage:
- Tag by team, product, and environment at every layer: compute, storage, vector DB, managed endpoints, third-party model usage. Untagged spend is the attacker's first entry point.
- Meter at the request, not the instance, for LLM and generative workloads. Push input/output tokens, latency, and estimated cost into your observability stack alongside latency and error rate.
- Put policy in code: idle-kernel auto-shutdown, mandatory TTLs, per-team GPU quotas, auto-suspending dev endpoints.
- Run a quarterly model rightsizing. For every LLM-backed feature, re-evaluate the smallest model that passes your eval set. Document the downgrade or the reason to stay.
- Separate commitment strategy by workload shape: commit for inference, spot for training, on-demand for bursts. Treat managed endpoints as optionality, priced accordingly.
- Build "showback" before "chargeback." Give teams a weekly dashboard of their AI spend with attribution; do this for a quarter before you push financial accountability into the P&L.
The governance piece
The best FinOps programs we've seen combine engineering controls with a lightweight governance cadence: a monthly AI-cost review attended by engineering, product, and finance; a standing list of the top 10 cost drivers; and a named owner for each. That is enough structure to keep surprises off the invoice without slowing down the teams actually shipping AI – which, in the end, is the point.
