How a simple S3 design decision turned into a $7M cost
The hidden tax of 1.5 trillion objects and why your lifecycle policies might be a financial time bomb.
AWS S3 is widely viewed as inexpensive, effectively unbounded object storage. In this case study, that is exactly how it behaved – with the caveat that the data storage decisions our team took cost us close to 40x more than they should have, and a lifecycle policy decision nearly exploded into a $7.2 million bill whilst trying to stop the cost bleed.
The Trillion-Object Blind Spot
This article analyses a production system that accumulated 5.6 PB of data across 1.56 trillion objects in a single bucket. Within one year, monthly storage cost increased from approximately $100k to over $400k, with forecasts exceeding $1M per month just 12 months later.
The root cause was not data volume alone, but architectural fragmentation misaligned with S3’s pricing model. The architecture generated hundreds of small snapshot artifacts per request, causing object count to grow faster than volume. A consolidation experiment showed that by aligning object granularity with S3’s economic structure, equivalent logical data could have been stored at 38x lower monthly cost. This case demonstrates that cost modelling must be treated as a first-class architectural constraint.
Paying for an Index, Getting a Bucket
It’s easy in our day to day to simplify storage costs as purely volumetric – e.g., C≈V, where C = monthly cost and V = stored volume.
For AWS S3, an accurate representation requires considering many more facets. Most of them you can find on the official AWS Pricing calculator – hence I will skip the formula definitions here.
There is a crucial cost factor that most people skip though – and is also omitted in the official AWS cost calculator. Object cardinality. At small scale, object cardinality is negligible relative to volume but at larger scales it becomes a first-order variable.
The main variable that determines whether object count matters is average object size, with the following formula:
s ≈ V / N
Where s = Average object size (GB), V = Total stored volume (GB) and N = total object count.
When average object size falls into the kilobyte range, per-object pricing dominates. The system no longer behaves like bulk storage, instead it’s more akin to a massively distributed index – except you are paying storage-layer economics for index-layer behaviour.
A Real-World Example
In the system analysed, our bucket composition was the following:
- 5.6 petabytes stored
- 1.56 trillion objects
- ~3.5 KB average object size
$400k/monthstorage cost~$50k/monthin request charges
The architecture generated hundreds to thousands of small snapshot artifacts per back end request. Over time, fragmentation compounded. Object count grew faster than volume.
A consolidation experiment showed that the same data could instead have been stored in artifacts averaging ~2.5 MB.
This would have reduced storage cost from $457,000 to $11,900 per month for the same volume of data. This represents a 38× structural reduction.
This reduction was not due to compression or deletion of data. The total logical volume remained constant. Only object granularity changed.

The $7.2 Million Lifecycle “Ghost”
Another (nearly) expensive lesson came from trying to fix our exponentially increasing storage costs. Without permanently losing important insights into our service, the only cost saving alternative we believed to have was resorting to lifecycle transition policies.
At that point in the system’s history, the bucket contained approximately 720 billion objects. We came up with a plan for all objects older than 6 months to transition automatically with Lifecycle Policies from Standard Access to Infrequent Access, then eventually into Glacier.
This solution could have had a seven figure cost, as lifecycle transitions are priced per 1,000 objects. Putting pen to paper, the lifecycle transitions calculation would have been:
720,000,000,000 / 1,000 × $0.01 = $7,200,000
What saved us
Approximately $7.2 million — for a single lifecycle rule on a single bucket. This excludes ongoing storage cost in the new tiers on top of that.
The transition did not execute because most objects were smaller than 128 KB, which do not transition by default. Hence the irony of the same fragmentation pattern that caused excessive steady-state cost also preventing an even larger transition bill.
The Realistic Solutions
With lifecycle transitions proving economically problematic, we were left with three realistic options.
1. One-Time Consolidation Migration
The most promising approach was to address the underlying fragmentation.
Since objects were logically associated with response IDs, we could build a migration script that retrieved all snapshots belonging to a response, aggregated them into a larger artifact, validated the result, and deleted the original granular objects.
This would incur one-time compute and S3 request costs, but drastically reduce object cardinality and make cheaper storage classes economically viable.
Our consolidation experiment had already demonstrated the potential for a 38× reduction in modeled monthly costs. We’d also need to adjust the application to produce consolidated artifacts going forward.
2. Pay the Price and Recover It
Alternatively, we could accept the potential $7.2 million lifecycle transition cost and recover it through lower monthly storage bills.
Our estimates suggested a payback period of approximately two years. That would require around $300,000 in monthly savings to recover the initial expense over 24 months.
While expensive upfront, this could have been financially justified over a longer operating horizon, particularly with storage costs already exceeding $400,000 per month and projected to grow further.
However, transitioning the existing objects would only address the accumulated data. Without overhauling the backend to stop generating hundreds of granular snapshots per request, we’d continue accumulating the same problem. New objects would eventually require their own costly transitions, and over time, we’d find ourselves right back where we started.
3. Delete the Data
The cheapest option financially was also the most destructive: simply delete the historical snapshots.
Unlike lifecycle transitions, S3 DELETE requests and lifecycle expiration actions are free. We could have introduced a retention policy that permanently removed older artifacts, immediately reducing the storage footprint without incurring billions of transition charges.
The trade-off was permanent data loss. Those snapshots contained historical information we wanted to preserve, making deletion a difficult proposition.
And just like the other solutions, we’d still need to reconsider how the backend generated and retained new artifacts to prevent storage costs from growing uncontrollably again.
Our Decision
Ultimately, we decided not to pursue any of the proposed solutions.
There were several reasons behind that decision. First, we had higher-priority engineering work, and overhauling our data persistence architecture was considered too disruptive to justify the immediate investment.
Second, we had already achieved significant cost reductions elsewhere in the system. Despite operating in an organization where cost efficiency was a leadership principle, there was little immediate pressure to pursue further savings.
Finally, management believed that engaging directly with key contacts in AWS’s S3 organization could produce a quicker, less disruptive resolution than undertaking a large-scale migration ourselves.
So the remediation was put aside, and the existing storage architecture remained unchanged.
The experience highlighted something I hadn’t fully appreciated before: even when the engineering problem is understood and the potential savings are substantial, organisational priorities and expectations can outweigh the financial case for fixing it.
The Engineering Takeaway
S3 scaled exactly as designed. Our architecture, however, didn’t scale economically.
This experience reinforced the importance of evaluating object storage against three budgets:
- Volume: How much data are we storing, and what will it cost?
- Cardinality: How many objects are we creating, and how does their size affect pricing?
- Remediation: What will it cost to restructure or migrate this data if the architecture becomes unsustainable?
The last is particularly easy to overlook. A design that’s inexpensive to implement at a million objects can become extraordinarily costly to change at a trillion.
Cost modeling must be treated as a first-class design discipline, not something introduced after the bill becomes a problem.
Originally published on Gabardo Engineering.
