Executives usually picture AI costs as racks of GPUs and long waitlists for silicon. They overlook data storage. That is where the real expense hides.
After a model answers a query, the back-end data remains. Billions of daily prompts, images, and videos require storage. This infrastructure cost is difficult to plan and often ignored in initial AI budgets.
As inference scales, the data footprint grows with it
Traditional AI conversations focus on training. Training is finite. A run starts, uses compute, and ends. Inference is continuous. Every prompt leaves a trace. A training run is a project you can budget and retire. Inference does not retire.
AI-generated video shows the scale of this issue. WD found that an 8-second clip returns to a user as a 3.97MB 720p file. That file leaves more than 500MB of associated data behind. The back-end includes logs, cached context, immediate renders, backups, and safety records. That is a 130x increase over the output file size the user sees. Scale this to an enterprise generating longer, higher resolution video and the back-end footprint grows into the hundreds of gigabytes per clip.
McKinsey projects AI inference demand will grow at a 35% CAGR through 2030. Training grows at 22%. Inference will make up more than 40% of total data center demand by the end of the decade.
A recent WD-sponsored IDC global white paper surveyed 763 IT and business decision makers. Nearly 95% of respondents reported an increase in data volume due to generative AI over the past 12 months. Roughly 61% reported growth of 25% or more.
A senior product manager of a hyperscale cloud service provider said: “You can think of having more storage as an optimization for keeping the GPUs busy. The biggest challenge is to ensure that the GPUs are not idle, waiting for data to show up—hours of GPU idle time becomes very expensive, very fast.” Put in dollar terms: if your storage spend is growing 25% year over year while your compute budget grows in the more predictable cyclical way, storage ends up outpacing the budget line that receives the most attention.
Matching storage to the AI data life cycle
Compute demand rises and falls with training cycles, product launches, and hardware refreshes. The volume of retained data does not fall back down when compute demand slows. Model logs, synthetic outputs, inference results, and more keep accumulating. Organizations are holding onto more of it for longer.
This changes the role of retained data. Information that once moved into an archive and rarely surfaced can now become an input to future AI workloads. As more data is retained, retrieved, and reused, storage must support not only growing capacity, but increasingly dynamic access to it. The same WD-IDC survey found 96% of respondents anticipate needing faster archive retrieval to support inference or retrieval-augmented generation (RAG).
AI workloads require both performance and capacity. Different technologies serve different needs across the data lifecycle. HDDs provide a cost-effective, high-capacity foundation for the vast volumes of data that must be retained and revisited. They offer a significant cost-per-terabyte advantage.
What this means for how AI gets budgeted
For a CFO or CIO, “AI infrastructure spend” may be scoped too narrowly on compute. A model that idles while data loads costs about as much as one idling for lack of a GPU. The storage bottleneck stays hidden until it is already a problem. It surfaces in operating costs and retrieval delays rather than a CapEx line next to the accelerators.
Consider a retailer generating AI product videos for its catalog. Consider an insurer running claims documents through a RAG pipeline for faster underwriting. In both cases, the compute cost is a known, budgeted quantity: a per-generation or per-query rate that finance can forecast. The storage cost is different. It compounds with every video generated and every claim processed. None of that back-end data disappears once the task is finished.
A team that budgets AI as “compute plus a rounding error for storage” may find, in 12 to 18 months, that the rounding error has become a significant fraction of the program’s total cost. This happens right around the time leadership is deciding whether to expand.
Building for economically scalable capacity
Addressing this challenge starts with architecture. Storage strategy tends to be designed for expensive, hot-tier performance (for data accessed instantly and constantly). Most AI data by volume accumulates in the warm, cool, and cold tiers behind it.
Enterprises need economically scalable capacity. Infrastructure must be designed for the full AI data life cycle. This balances performance with capacity, power, reliability, and total cost of ownership as data persists, compounds, and is increasingly reused. This means treating archives that once sat untouched as active assets for RAG and inference. It also means setting retention policies for data’s growing strategic value rather than its original storage cost. Decisions about storage play out across fleets, not individual drives.
Such infrastructure requires storage tiers. Individualized power consumption, reliability, density, and operational efficiency determine whether the system remains economically sustainable as capacity scales.
There are three questions to ask about your organization’s AI infrastructure budget:
- What’s our data retention policy for AI-generated content, and who set(s) it?
- Is our storage procurement tied to training-time performance needs, or to the actual life cycle of the data we’re producing?
- If our AI usage doubled next year, would our storage cost double with it, or worse?
Most organizations cannot answer all three with confidence. That is the gap worth closing before the next budget cycle, not after.
This content was produced by WD. It was not written by MIT Technology Review’s editorial staff.
