Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data, and How the Lakehouse Made the Slow Option Fast

Cross-posted. This article’s canonical home is iceberglakehouse.com.

By Alex Merced, Head of Developer Relations at Dremio

Here is one of the strangest and most consequential plot twists in the history of data infrastructure: over the past decade, the analytics industry deliberately moved its data onto the slowest storage it could find, and got faster.

Amazon S3 answers a read request in tens to over a hundred milliseconds. A modern block volume answers in one or two, and local NVMe in microseconds. By the only metric storage vendors printed on the box, object storage was a fifty-to-hundred-fold step backward. Yet today the world’s analytical data, the lakehouses, the training sets, the event histories, lives overwhelmingly on object storage, queries against it come back in seconds or less, and nobody serious is moving back. That plot twist is not a paradox. It is a story about which properties of storage actually matter at scale, and about a decade of brilliant engineering, in file formats, table formats, and query engines, that systematically neutralized every weakness the latency number represented.

This article is the full deep dive. What block and object storage actually are, mechanically, without hand-waving. The honest comparison across latency, throughput, scalability, availability, durability, cost, and consistency, with real numbers. The specific wrinkles object storage inflicts on data workloads, and then the heart of the piece: how Apache Parquet, Apache Iceberg, and engines like Dremio overcome each wrinkle, layer by layer, turning eleven nines of cheap durability into interactive analytics. By the end, the plot twist should feel inevitable, and you should be able to reason about any storage decision, and any vendor’s storage claim, from first principles.

Block Storage, Mechanically: The Disk Abstraction

Start with the elder statesman, because everything else is defined against it.

Block storage presents the oldest abstraction in computing: a device that stores fixed-size blocks of bytes, addressable by number, readable and writable in place. It is the disk, virtualized. A block volume knows nothing about files, folders, or meaning. It knows that block 4,000,017 contains these 4,096 bytes, and it will read or overwrite them on demand, fast. Everything humans recognize as files comes from the filesystem, ext4, XFS, NTFS, that the operating system layers on top, translating names and directories into block addresses.

In the cloud, block storage means network-attached virtual disks: AWS EBS, Azure Managed Disks, Google Persistent Disk. A volume attaches to one server, the server formats and mounts it, and applications get the full POSIX contract: open, seek, read, write, append, modify in place, all at low latency. Top-tier volumes like EBS io2 Block Express deliver latencies under one to two milliseconds with hundreds of thousands of IOPS, and local NVMe instance storage goes lower still, into microseconds, at the price of vanishing when the instance does.

The strengths follow from the abstraction. Latency is the headline: sub-millisecond random reads and writes are what databases are built on, which is why every serious transactional system, Postgres, MySQL, SQL Server, SAP HANA with its sub-millisecond hot-tier requirements, runs its data files and write-ahead logs on block storage, no exceptions. In-place modification is native: change one record, rewrite one block. And the semantics are the ones fifty years of software assumes.

The limits are just as structural. A volume attaches to one machine, so sharing requires a database or a network filesystem on top. Capacity scales up, not out: volumes have size ceilings measured in tens of terabytes, and growing past them means sharding across volumes, which is your problem. Durability is good but engineered per volume, with annual failure rates that make backups mandatory rather than paranoid. And the cost per gigabyte runs several times object storage, because you are paying for performance-grade media and reserved capability whether you use it or not. Block storage is a sports car: unmatched at what it does, expensive to park, and the wrong vehicle for hauling freight.

A brief word on the middle sibling, file storage, NFS, SMB, cloud services like EFS: a real filesystem, mountable by many machines at once, with latency and cost between the other two. It earns its place for shared application directories and legacy workloads that demand POSIX across a fleet. For the analytical story this article tells, it plays a supporting role at most, and the main event is the other extreme.

Object Storage, Mechanically: The Warehouse of Sealed Boxes

Object storage threw away the disk abstraction entirely and asked a different question: what is the simplest possible contract for storing unlimited stuff durably?

The answer: a flat keyspace of immutable blobs behind an HTTP API. An object is a key, a string that may look like a path but is just a name, a value, the bytes, up to five terabytes, and metadata. The operations are few and blunt: PUT an object, GET an object or a byte range of it, DELETE it, LIST keys by prefix. There are no directories, no seek-and-modify, no append. To change one byte of an object, you upload a replacement object. That is the whole contract, and its bluntness is the point, because every simplification buys scale.

Behind the API sits the engineering that made S3 and its kin, Azure Blob, Google Cloud Storage, and the S3-compatible world of MinIO, Ceph, R2, Wasabi and company, the largest storage systems ever built. Each object is erasure-coded: split into shards with redundancy math, Reed-Solomon coding, spread across many drives, racks, and availability zones, so that multiple simultaneous hardware failures lose nothing. That is where the famous durability figure comes from: eleven nines, 99.999999999 percent annual durability, a number so extreme it means expected loss of one object per hundred billion per year. Availability is engineered separately and regionally, typically 99.9 to 99.99 percent for standard tiers. And the scale numbers are the ones worth reading twice: S3 alone holds over 500 trillion objects and serves peaks beyond 200 million requests per second. There is no volume to fill, no cluster to grow. The keyspace is effectively infinite, and throughput scales horizontally: one client reading one object gets modest speed, but a thousand clients reading a thousand objects, or a thousand ranges of one huge dataset, aggregate to staggering bandwidth. Cost lands at a fraction of block storage per gigabyte, with tiering to archive classes for a further order of magnitude, though the pricing model has teeth elsewhere: every request is billed, and egress, moving data out to the internet or across clouds, is the line item that has ruined many a budget.

One more property graduated from caveat to strength in recent years, and it matters enormously for what comes later. Object stores were historically eventually consistent: a read after a write might see stale data, which forced analytics tools into elaborate workarounds. On December 1, 2020, S3 shipped strong read-after-write consistency for all operations at no cost, and the serious S3-compatible stores followed. More recently, conditional writes arrived: PUT operations that succeed only if the key does not exist, or only if it still carries an expected version, giving object storage its first honest compare-and-swap primitive. File this fact away. It is the hinge of the transactional story later in this article.

So the trade emerges in full: object storage exchanged latency and mutability for effectively infinite scale, extreme durability, horizontal throughput, and radical cheapness. Fifty to a hundred fifty milliseconds to first byte, versus one or two. No append, no in-place edit, no real directories. Request bills and prefix-level rate limits, on the order of thousands of writes and several thousand reads per second per key prefix, that punish designs treating it like a filesystem. It is a warehouse of sealed boxes: you can store more boxes than you could ever count, retrieve any of them from anywhere, and never worry about losing one, but you cannot edit a box’s contents on the shelf, and walking to the warehouse takes longer than reaching into your desk drawer.

The Scorecard, Side by Side

Compressing the comparison into the dimensions that decide real architectures.

Latency: block wins by two orders of magnitude, sub-two milliseconds against fifty to a hundred fifty, with the newer premium object tiers like S3 Express One Zone splitting the difference at single-digit milliseconds for a price and a single-zone durability trade. Throughput: object wins at scale, because aggregate parallel bandwidth across a distributed fleet beats any single volume’s pipe, and the win grows with parallelism. Scalability: object, without contest, infinite keyspace and horizontal everything against per-volume ceilings and single-attach. Durability: object, eleven nines by erasure-coded design against per-volume failure rates that mandate backup regimes. Availability: comparable when architected properly, with object’s multi-zone replication built in and block’s requiring you to build it. Cost per gigabyte: object by several fold, and by an order of magnitude with tiering, while block charges premium rates and object claws some back through request and egress billing. Semantics: block wins completely, full filesystem behavior against PUT-GET-LIST-DELETE and no in-place anything. Sharing and access: object, HTTP from anywhere with IAM-grade policy, against one-machine attachment.

Read the scorecard and the classical division of labor writes itself, and it remains correct: transactional databases, low-latency serving, boot volumes, and scratch space live on block. Archives, media, backups, and data lakes live on object. AI stacks split the same way instinctively: training corpora on object for scale and throughput, the training runs’ working state on block and NVMe for latency, model artifacts back to object for serving and history.

The interesting question, the one the rest of this article answers, is the workload that refused the division: interactive analytics. It wants the scale, durability, and cost of object storage, its datasets are petabytes, and it wants answers in seconds, which the latency column says object storage cannot give. The lakehouse is the decade-long engineering project that proved the latency column wrong, or more precisely, made it irrelevant. Here is how, layer by layer.

How Analytics Ended Up Here: A Short History of the Migration

Before dissecting the wrinkles, the migration itself deserves a page, because knowing why analytics moved onto object storage explains why nobody is moving back, latency numbers notwithstanding.

The story starts with Hadoop’s HDFS, the first system to make a deep philosophical bet that data should live on cheap commodity storage with the intelligence layered above. HDFS got the economics right and the operations wrong: clusters coupled storage and compute on the same machines, so growing one meant growing both, capacity planning was a permanent argument, and keeping thousands of disks healthy was a career. When the cloud arrived, it offered a clean divorce: object storage as a bottomless, maintenance-free data tier, compute as an elastic fleet rented by the minute, connected by a network fat enough to make the separation workable.

That divorce, storage-compute separation, is the single most important architectural shift of the era, and its logic is worth stating crisply. Coupled systems size for peak: your cluster must be big enough for the Monday morning rush and the month-end close, and it idles at that size all week. Separated systems size independently: storage grows with data, smoothly and cheaply, while compute scales to zero between workloads and to enormous during them. Different teams can bring different engines to the same bytes without copying, which is the multi-engine lakehouse in embryo. And durability becomes the provider’s problem, engineered once at eleven nines, instead of every cluster’s ops team’s problem, engineered repeatedly at whatever nine the budget allowed.

The warehouses proved the model first, Snowflake’s architecture being the famous demonstration that separation plus caching could beat coupled systems at their own performance game, and the open world followed with the harder version: separation without a proprietary engine in the middle. That required the storage tier’s rawness to be tamed by open layers anyone could implement, which is precisely the Parquet-Iceberg-engine stack this article walks. By the time the AI era arrived demanding petabyte corpora with fleet-wide parallel access, the question was settled: the economics and elasticity of the separated model were worth any engineering it took to hide the substrate’s rough edges. The rest of this article is that engineering.

The Wrinkles: What Object Storage Does to Data Workloads

First, name the enemies precisely, because each one gets its own counter. Running analytics naively on object storage hurts in six specific ways.

One: per-request latency. Every GET costs tens of milliseconds before the first byte, so a query whose plan touches ten thousand objects serially spends its life waiting on round trips. Two: listing is slow and expensive. Discovering what files exist by LIST calls over big prefixes takes seconds to minutes and bills per request, and the old Hive-style pattern of listing directories to plan queries dies at scale on this alone. Three: no atomic multi-file operations. A data update that writes fifty files has no way to make them appear simultaneously, so readers can see half-finished states, and two writers can silently trample each other. Four: no append or in-place edit. Immutable objects mean every modification pattern that databases take for granted, append a row, update a record, must be reinvented as write-new-plus-supersede. Five: small files are poison. Millions of tiny objects multiply request counts, listing costs, and per-object overhead, and streaming ingestion produces exactly that unless something intervenes. Six: throttling and economics. Per-prefix rate limits cap naive request storms, every request is a billable event, and egress pricing punishes moving data out.

Every one of these is real, and every one has been systematically defeated. The defeats stack in three layers, the file format, the table format, and the engine, and the elegance of the stack is that each layer’s design assumes exactly the object-storage physics above.

Layer One: Parquet, the File Format Built for Ranged Reads

Apache Parquet’s design, which I covered at depth in my Parquet state-of article, reads like a direct response to wrinkles one and six: make each file self-describing and internally addressable, so an engine can extract exactly what it needs in a handful of large, cheap requests.

The mechanics: a Parquet file packs data in columnar layout, row groups horizontally, column chunks within them, compressed and encoded pages within those, and ends with a footer that maps it all, schema, byte offsets of every column chunk, and min-max statistics per column per row group. Against object storage, that structure converts into a precise access protocol. The engine issues one ranged GET for the footer, tiny, at a known position at the file’s tail. From the footer it learns, without touching anything else, which row groups could possibly satisfy the query’s filters, statistics pruning, and where the needed columns’ bytes physically live. Then it issues a few large ranged GETs for exactly those byte spans, the four columns the query touches out of two hundred, in the two row groups out of forty that survive pruning, and nothing else ever crosses the network.

Count the wins against the wrinkle list. Requests per file drop from many to a handful, and each is a large sequential read, the access pattern object storage serves best, which converts the latency problem into a throughput problem, and throughput is object storage’s strength. Bytes transferred drop by orders of magnitude through column projection and statistics pruning, attacking the request bill and the egress meter together. Compression, columnar layouts compress beautifully, shrinks everything further. One well-formed Parquet file of a few hundred megabytes replaces what row formats would spread across thousands of requests. The file format alone does not make analytics on object storage fast. It makes fast possible, by ensuring that when the upper layers decide what to read, the reading itself is a few big, cheap, parallelizable range requests.

Layer Two: Iceberg, the Table Format That Deleted the Filesystem

Apache Iceberg’s founding insight, the one this whole series orbits, is aimed squarely at wrinkles two, three, four, and five: stop pretending object storage is a filesystem, and carry everything a filesystem would have told you in metadata instead.

Wrinkle two, listing, dies first and most completely. An Iceberg table’s metadata tree, manifests listing data files with their statistics, manifest lists collecting manifests per snapshot, records every file the table contains, with per-file column bounds and counts. Query planning never calls LIST. The engine reads a few small metadata files, prunes entire data files whose statistics cannot match the query, and emerges with an exact reading list, all before touching a byte of data. The pattern that took minutes of listing against big prefixes takes milliseconds of metadata reads, and the statistics pruning routinely eliminates the vast majority of the data before layer one’s ranged reads even begin.

Wrinkle three, atomicity, falls to the snapshot-and-pointer design. A write, however many files it produces, becomes visible in one atomic step: the commit of a new metadata pointer, historically through a catalog performing the compare-and-swap, and increasingly with object storage’s new conditional writes as the primitive underneath, an If-None-Match PUT that succeeds for exactly one contender. Readers see the table before the commit or after it, never between, and concurrent writers conflict cleanly through optimistic retry instead of trampling. This is the moment the sealed-box warehouse acquired transactions, and it is worth pausing on how neat the trick is: immutability, the wrinkle, becomes the mechanism, since old snapshots remain intact and valid forever, which is also where time travel comes from, for free.

Wrinkle four, no append or update, is answered by the merge-on-read machinery I dissected in my deletion vectors article: changes land as new files plus compact records of what they supersede, deletion vectors marking dead rows in bitmap form, and readers assemble current truth from immutable pieces. Row-level change on unchangeable objects, with v3’s vectors making the read-side cost flat and small.

And wrinkle five, small files, is managed rather than eliminated, through the maintenance discipline the format makes possible: compaction rewrites accumulations of small files into well-sized ones, guided by the metadata’s complete knowledge of the file population, and snapshot expiration reclaims the superseded. The format did not repeal streaming’s tendency to mint small files. It made the cleanup a routine, metadata-driven background job instead of an archaeology project.

The compressed claim for layer two: Iceberg carries, in metadata, everything analytics needs that object storage refused to provide, the inventory, the statistics, the transactions, the history, and it stores that metadata as objects too, so the whole system inherits the durability and scale of the substrate it fixes.

An Arithmetic Interlude: Why Small Files Deserve Their Reputation

Wrinkle five gets invoked so often that its actual arithmetic rarely appears, so let me run the numbers once, because they turn a vague warning into a visceral one, and they explain why compaction is a budget line, not a nicety.

Take one terabyte of table data and store it two ways: as 4,000 well-formed Parquet files of 256 megabytes, or as 4 million small files of 256 kilobytes, the kind a too-eager streaming pipeline mints in a busy month. Now run one full-scan query against each, naively.

Requests first. The healthy layout needs on the order of one footer read plus a handful of ranged reads per file: call it 20,000 GETs. The small-file layout needs a footer and a data read per file at minimum: 8 million GETs, four hundred times more. At typical request pricing of roughly forty cents per million GETs, the healthy scan’s request bill is under a penny, and the small-file scan’s is over three dollars, per query, before any compute. Run that dashboard hourly and the request line alone compounds into real money, and this is the cheap dimension.

Latency is the expensive one. Requests overlap under parallelism, but coordination does not: 8 million requests must be planned, issued, tracked, and retried across the fleet, per-prefix rate limits of a few thousand requests per second start throttling exactly this pattern, and footer-to-data ratios collapse, since a 256-kilobyte file spends as many round trips on metadata as on content. Engines report the same shape universally: scan time degrades not linearly but brutally as file counts explode, because the fixed per-object costs, request setup, footer decode, task scheduling, stop amortizing. And the metadata layer above suffers in sympathy: Iceberg manifests tracking 4 million files are themselves a thousand times heavier to read, write, and prune than manifests tracking 4,000.

Now the repair math, which is the encouraging half. Compaction rewrites the 4 million files into 4,000 once, at a one-time compute cost roughly comparable to a single full scan, and every subsequent query inherits the four-hundred-fold request reduction forever. That trade, pay one scan’s worth of work once to fix every future scan, is among the best returns in data engineering, which is why every serious platform automates it and why I flag unmanaged small files as the number one silent killer of lakehouse performance in the field. The lesson generalizes into a rule of thumb worth memorizing: on object storage, file count is a performance dimension as real as data size, target data files in the low hundreds of megabytes, and treat compaction lag as an alert-worthy health metric, because the arithmetic above is running against you every hour it grows.

Layer Three: The Engine, Where Milliseconds Go to Die

Formats make efficient access possible. Engines make it fast, and this layer is where I get to talk about what my colleagues actually build, so the house flag flies here, alongside credit to every engine team solving the same physics.

Parallelism is the first weapon. Object storage’s latency is per request, and its throughput is per fleet, so the engine’s job is to never wait on one request when it could wait on hundreds concurrently. A modern engine planning a scan splits it across executors, and each executor issues deep pipelines of asynchronous ranged reads, prefetching ahead of the compute. A thousand concurrent 8-megabyte range requests turn a 100-millisecond-latency store into a firehose, because the latencies overlap and the aggregate bandwidth is the fleet’s, not the socket’s. This single technique is most of why the latency scorecard stopped mattering: nobody serious reads object storage serially.

Caching is the second. The same hot data gets read across queries all day, and paying object storage’s latency and request bill repeatedly for it is waste. Dremio’s Columnar Cloud Cache, C3, attacks exactly this: transparently caching frequently accessed data on the compute fleet’s NVMe, so repeat reads come from microsecond-latency local media instead of the network, with a claimed elimination of up to 90 percent of I/O costs on the workloads it serves. The general pattern, an NVMe tier between object storage and compute, appears across the serious engines, and it is block storage’s revenge in the nicest form: the fast, expensive medium serving as a small, hot cache in front of the vast, cheap one, each doing what it does best.

Materialization is the third. Some computations are too heavy to repeat regardless of I/O: the aggregation over a year of events behind a popular dashboard. Reflections, Dremio’s transparently substituted materializations, precompute such results into optimized structures and let the planner rewrite matching queries against them silently, with acceleration up to 100x on fitting workloads and, increasingly, autonomous selection of what to materialize based on observed query patterns. Materialization converts recurring query-time cost into scheduled refresh cost, which against object storage’s request-billed economics is as much a cost feature as a speed feature.

Metadata and memory round out the arsenal. Engines cache Iceberg metadata and Parquet footers aggressively, so planning stops re-reading what has not changed. And everything decoded lands in Apache Arrow’s columnar memory format, Dremio’s native representation since inception, so bytes flow from ranged read through decompression into vectorized execution without row-by-row conversion, the copy-tax story of my Arrow article completing the pipeline’s last mile.

Stack the three layers and trace one query to feel the sum. A dashboard asks for last week’s revenue by region over a two-petabyte events table. Iceberg metadata, cached, prunes two petabytes to the forty data files whose date bounds overlap the week: milliseconds, no LIST, no data touched. Parquet footers, cached, prune within files and locate the three needed columns’ byte ranges. Executors issue a few hundred parallel ranged GETs, most served from C3’s NVMe because this table is hot, the remainder overlapping their latencies against S3. Arrow-native execution aggregates the columns vectorized, and if a Reflection covers the pattern, the whole scan was skipped for a precomputed result. The user sees eight hundred milliseconds. The storage bill sees a few hundred requests against a store costing pennies per gigabyte-month with eleven nines of durability. That is the plot twist, mechanically resolved: the slow storage was never made fast. It was made irrelevant to feel, by three layers that each refused to pay its costs more than once.

The New Storage Era: The Substrate Is Evolving Too

The story is not static, and the past two years brought the substrate itself into motion, in ways worth tracking.

The providers are climbing toward the workload. S3 Express One Zone offers single-digit-millisecond object storage for hot data, trading multi-zone durability for speed, a purpose-built tier for exactly the cache-and-hot-path patterns above. Conditional writes, as noted, gave the ecosystem a native compare-and-swap, simplifying the transactional machinery table formats had to build around the storage. Managed table offerings like S3 Tables bake Iceberg awareness, and its maintenance, into the storage service itself. The direction is unmistakable: object storage learning the lakehouse’s needs natively, the substrate rising to meet the layers built atop it.

The market is also diversifying underneath the S3 API. Zero-egress providers like Cloudflare R2 attack the pricing model’s sharpest tooth, flat-rate providers and the self-hosted world of MinIO and Ceph serve sovereignty and cost cases, and the compatibility question has become the buyer’s real diligence item: lakehouse workloads exercise the API’s hard corners, ranged reads at massive concurrency, multipart uploads, conditional writes, strong consistency, and an implementation that covers 80 percent of the surface works in demos and breaks in production. Verify the corners, not the checkbox.

And the pressure from above continues: Iceberg v4’s single-file commit work aims to shrink metadata write amplification against exactly this substrate, Parquet’s footer redesign targets the read-side metadata cost, and every improvement in those specs, covered across this series, is ultimately a further negotiation with object storage physics. The layers and the substrate are co-evolving, which is the healthiest possible description of an infrastructure stack.

Choosing: The Practical Placement Guide

The decision rules I actually give, compressed.

Put on block storage: transactional databases and their logs, always, low-latency serving stores, search and vector indexes with strict latency budgets, and compute-local scratch. The sub-two-millisecond, modify-in-place contract is what these workloads are, and no cleverness above cheaper storage replaces it.

Put on object storage: the lakehouse, whole, tables, metadata, history, all analytical data at rest, ML training corpora, artifacts, backups, archives, and anything measured in tens of terabytes and up with shared, parallel access patterns. Choose it by default for anything analytical, and let the three layers carry the performance.

Between them, build the gradient rather than picking a side: NVMe and block as cache tiers in front of object, premium object tiers for the genuinely hot, standard tiers for the working set, archive tiers for the cold, with lifecycle policies moving data down the gradient by measured access, not by guess. The mature architecture is not block versus object. It is a temperature system, with each medium holding the data whose access pattern it was built for, and the engine hiding the whole gradient behind one query surface.

And two economic rules that outrank the technical ones: measure request patterns before and after format adoption, because a lakehouse done right cuts the request bill as dramatically as the latency, and treat egress as an architectural constraint, keeping compute next to storage and moving results, not data, across the expensive boundaries.

Evaluating an Object Store for Lakehouse Duty: The Checklist

Since S3-compatible has become a spectrum rather than a guarantee, and since lakehouse workloads exercise the API’s hardest corners, here is the diligence list I give teams choosing a store, cloud or self-hosted, for Iceberg duty.

Verify strong read-after-write consistency, in writing, for the specific service and configuration, because the compatible world still varies and the lakehouse assumes it. Verify conditional writes, the If-None-Match and If-Match semantics, because the transactional machinery increasingly builds on them, and their absence forces catalog-only commit paths. Test ranged GET behavior under real concurrency, thousands of simultaneous range requests from a distributed fleet, because this is the lakehouse’s daily access pattern and where partial implementations buckle. Confirm multipart upload completeness, including the part-size boundaries, since every large Parquet write depends on it. Probe the rate-limit and prefix-scaling story at your projected request volumes, and ask specifically how limits scale as key cardinality grows, because compaction and metadata traffic concentrate on prefixes in ways demo workloads never reveal. Price the full model, storage plus requests plus egress, against your measured access patterns rather than your storage size, since two stores identical per gigabyte can differ several fold on a request-heavy or egress-heavy workload. And run one honest end-to-end test before committing: a real table, a real streaming ingest, a real compaction, a real concurrent query load, because an object store that implements most of the S3 API works in demos and breaks in production, and the corner it is missing will be the one your busiest table finds at 2 a.m.

None of this is exotic. All of it is an afternoon of engineering that prevents a quarter of migration regret, and the vendors who welcome the checklist are, reliably, the ones who pass it.

Questions I Hear Most Often

If object storage is so slow per request, why does my lakehouse feel fast? Because nothing in a well-built lakehouse pays that latency serially or repeatedly: metadata pruning eliminates most reads before they happen, the surviving reads are large parallel ranges whose latencies overlap, caches serve the hot set from NVMe, and materializations skip whole scans. The per-request number is real and almost never on the critical path. When a lakehouse does feel slow, the diagnosis usually finds one of the layers missing: small files un-compacted, metadata unpruned, no cache tier, and the fix is the layer, not the storage.

Why not just run the lakehouse on fast block storage? Cost, sharing, and durability, in that order. Petabytes on premium block media costs several fold more for capability the layered stack does not need, block’s one-machine attachment fights the many-engine, many-consumer model that is the lakehouse’s point, and you would inherit the backup and replication engineering that object storage’s eleven nines make unnecessary. The economics only work one way: cheap durable substrate, thin fast cache.

Is eventual consistency still a lakehouse concern? On the major clouds, no: S3 has been strongly consistent since late 2020, and the other hyperscalers match. It survives as a diligence question for S3-compatible and self-hosted stores, where guarantees vary, and as an architecture appreciation point: Iceberg’s snapshot design was born in the eventually consistent era and is why lakehouses never depended on listing or filesystem tricks in the first place, which is also why they run so well on every store since.

What about the durability fine print, eleven nines but 99.9-something availability? Read the two numbers as different promises: durability says your data will not be lost, availability says whether you can reach it this minute. Multi-zone standard tiers make brief unavailability a retry-and-degrade engineering problem, single-zone premium tiers trade real durability scope for speed and belong behind replication or in cache roles, and cross-region strategies remain your job for the disasters beyond a region. The lakehouse pattern helps here too: everything is objects, so replication tooling covers data and metadata alike.

Does S3 Express One Zone change the architecture? It sharpens one layer rather than replacing any: a purpose-built hot tier with object semantics, single-digit milliseconds, and single-zone scope, ideal for cache-like and working-set roles where the engine’s NVMe tier wants a bigger, shared sibling. The three-layer logic is unchanged, formats to minimize reads, engines to parallelize and cache them, and Express simply gives the gradient one more temperature to place correctly.

How do the table format wars interact with the storage story? Less than the marketing implies, because every serious table format, Iceberg, Delta, Hudi, is a variation on the same physics answers: metadata instead of listing, snapshots instead of in-place mutation, compaction against small files. The storage-level reasoning in this article applies across them. The differences that matter live in the layers I cover elsewhere, governance, ecosystem breadth, spec evolution, and openness, which is where my Iceberg conviction comes from, not from storage mechanics.

Closing Thoughts

The block-versus-object question was never really a versus. It was a division of labor waiting to be understood: block storage as the fast, faithful disk for the workloads that are disks’ native tongue, and object storage as something genuinely new, durability and scale as a utility, priced like a commodity, addressable from anywhere. The decade’s real achievement was refusing to accept that the new utility’s latency defined its use: Parquet made reading surgical, Iceberg made the file system unnecessary and transactions possible, and the engines, with parallel I/O, NVMe caching, materialization, and Arrow-native execution, made the remaining costs vanish from human perception. The slowest storage became the foundation of the fastest analytics, because the stack above it was designed, layer by honest layer, around exactly what it is.

That design-around-the-truth-of-the-substrate discipline is, I would argue, the deepest lesson the lakehouse era has to teach, and it is why I keep writing this series: every layer I cover, formats, tables, catalogs, semantics, engines, is a chapter of the same story, open standards absorbing hard physics so that everything above them gets to be simple.

If you want that whole story with the depth an article cannot hold, that is what my books are for. I co-authored Apache Iceberg: The Definitive Guide and Apache Polaris: The Definitive Guide for O’Reilly, with further titles on lakehouse architecture, data engineering, and agentic analytics.

Browse the full collection of my books on data and AI at books.alexmerced.com.