When companies first started paying serious attention to cloud costs, they found plenty of things that were easy to understand. A virtual machine was still running after a project ended. A storage volume had been detached and forgotten. An oversized instance had been selected two years earlier and never reconsidered. Finding these things didn’t necessarily make them easy to eliminate, but at least the conversation was fairly straightforward.
With AI, figuring out what is actually unused gets harder.
Most discussions about AI cost begin with model usage, and there are good reasons for that. Model charges are visible, relatively easy to measure and, in some cases, large enough to get attention quickly. Teams can look at how many tokens they are consuming, which models are handling which requests and whether a less expensive model can do some of the work. GPU utilization presents another fairly obvious target when an organization is operating its own infrastructure.
The model, however, is only part of what gets built.
A company developing an internal AI application may copy source documents into a separate environment, prepare the data for retrieval and create embeddings and indexes from it. The team will probably keep test data and some history of prompts and responses while it works on the application. Logging and observability become more important once people start using the system because somebody needs enough information to investigate poor or unexpected results. Development, testing and evaluation create their own data, and the application may go through several versions of the way it retrieves and supplies context to the model.
Over time, quite a bit can accumulate around an application that originally looked fairly simple.
The interesting cost problem is not that all of this exists. Most of it exists for a reason. The problem comes later, when somebody has to decide whether those reasons still apply.
Take an old retrieval index. It consumes resources, but knowing that doesn’t tell you much. The underlying documents may have changed enough that the index is no longer useful. On the other hand, the team may need it to reproduce an earlier version of the application or compare old and new results. Evaluation data isn’t much easier. Six months without anyone touching a dataset doesn’t tell you whether it has been forgotten or whether the team will need it the next time an application change produces an unexpected result.
This is different from finding a server that has recorded almost no activity for three months. Lack of activity can be a useful clue when reviewing conventional infrastructure. It is a much weaker clue when the resource is information being retained because somebody expects to need it later.
That makes the economics difficult to read from the bill alone.
The FinOps Foundation now describes AI costs as extending across several technology categories, including public cloud, SaaS, specialized AI services and data-center infrastructure. The supporting costs can include compute, storage, networking and observability. That is closer to what an operating AI application actually looks like, and it also explains why improving the obvious AI expense does not necessarily reduce the overall cost of the application.
A team could do a very good job managing model consumption. It might move routine requests to a less expensive model, reduce unnecessary calls and reserve more capable models for cases where the difference matters. At the same time, the team could be retaining more historical output because it is useful for evaluation, collecting more telemetry to investigate problems and maintaining several generations of retrieval data while the application changes. Taken separately, none of those decisions looks particularly odd.
If the organization measures only model consumption, the cost trend will show one part of that story.
This is where FinOps needs information that it normally cannot get from billing data. Someone has to know why the resource exists.
An engineer may understand the purpose of an old dataset but have no idea what keeping several versions costs. A FinOps practitioner can find the storage expense but isn’t in a position to decide whether the information can be deleted. Security teams may have requirements for certain records, while other information may need to be removed because retaining it creates unnecessary risk. The application owner may know what the business needs without knowing much about the infrastructure and data that accumulated during development.
There isn’t necessarily one owner who can look at the list and decide what stays.
That is not unique to AI, but AI makes it more noticeable because creating another copy of data or another experiment is easy and usually inexpensive at the time. Nobody needs a major budget decision to retain another dataset. The individual amounts may remain unremarkable for quite a while. The cost becomes interesting after the practice has spread across multiple applications and teams.
There is also a natural bias toward keeping things. Rebuilding an index or recreating a clean evaluation dataset takes work, while leaving it alone takes almost none. If nobody knows whether it will be needed again, deletion creates an immediate risk and retention creates a future cost. Organizations are generally better at noticing the first one.
This leaves some fairly mundane questions that are better answered before the bill gets large. Who owns this data? Why are we retaining it? Does a temporary environment have any defined end date? Are several generations of the same resource still useful? If an experiment has been abandoned, what happens to everything that was created around it?
Those questions sound more like housekeeping than AI strategy, but cloud economics has provided plenty of evidence that housekeeping eventually matters.
The complication with AI is that aggressive cleanup isn’t a particularly good answer either. Historical information can have legitimate value. Teams need to evaluate changes, investigate failures and sometimes reproduce earlier behavior. Depending on the application and industry, there may also be requirements governing records and evidence. A policy that treats anything old as waste would be just as crude as keeping everything.
What is missing in many cases is simply a decision about how long the value is expected to last.
That decision is easier to make while people still remember why something was created. A temporary environment can have an expiration date. A dataset can have an owner and a reason for retention. Costs can be associated with the application that created them rather than disappearing into a general infrastructure account. None of this requires a new discipline invented specifically for AI, although applying familiar disciplines may require cooperation among people who haven’t traditionally had much reason to work together.
These questions are easier to ignore when there are three AI experiments than when there are thirty. The FinOps Foundation’s 2026 State of FinOps report found that 98 percent of respondents were managing AI spend, which is a good indication that AI is moving into the ordinary cost-management work of technology organizations rather than remaining a separate experiment.
A model invoice can tell you what the AI did this month. It doesn’t tell you much about the data, indexes, logs, environments and other resources still being kept because of what the AI did six months ago. Some of those things are worth keeping, and some probably aren’t.
Knowing the difference is likely to become a fairly ordinary part of managing AI costs.
