Back to blog

Why I'm Not Convinced About Data Lakehouse Architecture

Benjamin Wootton
2025-12-05
6 min read
Featured image for Why I'm Not Convinced About Data Lakehouse Architecture

I've built 3 data lakehouses and I'm not convinced about this architecture at all. I'm not sure if I'm doing it wrong but I don't think I want to try again.

Unifying the best of data warehouse and data lake sounds compelling, and I was genuinely keen on the idea each time I started. Open formats, cheap storage, one copy of the data and a choice of engines on top. Who wouldn't want that? But all of my projects seemed to suffer from the following challenges, and they were fairly consistent from one project to the next.

To be clear, this is just my experience. Your mileage may vary, and there are certainly teams out there making it work well.

You End Up Rewriting a Data Warehouse

The marketing materials make it sound easy, but you are inevitably thinking about how to layout data on object stores, maintenance jobs, concurrent access, transactions and/or reading the Iceberg spec. Low level non differentiating stuff.

This is engineering effort that could be spent on business logic rather than reinventing storage layer primitives.

On each project we started out with the intention of just landing some files and querying them. Before long we were talking about compaction, small files, partitioning schemes, snapshot expiry and what happens when two jobs try to write to the same table. These are all problems that a database has already solved for you, and solved well. With a lakehouse you either solve them yourself or you take on another vendor product to do it for you, which brings you back to paying for a data warehouse in all but name.

The other thing I noticed is that the skills are hard to find. People who are strong on the business problem aren't usually interested in file layouts, and the people who are interested in them are expensive and tend to end up as a bottleneck for everyone else.

More Expensive Than You Think

You save by using object storage but you end up writing ingestion logic, transformation jobs etc which are expensive to run (e.g., Databricks or EMR). You also pay for billions of S3 access calls. And then the engineering time costs more than license fees you are trying to avoid.

The storage line item looks great on the architecture slide, because object storage really is cheap. The problem is that storage is rarely the biggest part of the bill. The money goes on the compute needed to get data in, shape it and read it back out again, and on the people who build and look after all of that. Those costs are spread over lots of places, so they are harder to see and it takes a while before anyone adds them up.

The total cost of ownership often exceeds what you'd pay for a managed data warehouse when you factor in:

  • Compute costs for transformation engines
  • API call costs for object storage
  • Engineering time for custom tooling
  • Operational overhead

Not Fast Enough for Real-Time Use Cases

You think you don't have a real time use case, but you inevitably want to run an app or something latency sensitive and the lakehouse soon turns out to be too slow for anything fast and interactive. You are then totally snookered and have to implement caching layers which make everything more complex.

This adds another layer of complexity to your architecture and creates cache invalidation challenges.

I've seen this play out a few times. The first use case is a set of batch reports and everything is fine. Then someone wants a dashboard that people can click around in, or to put some numbers into a customer facing application. Queries that take several seconds are fine for an analyst running a report, but they aren't fine for an application user who expects a page to load straight away. Once more than a handful of people are querying at the same time it gets worse again.

The usual answer is to copy the data you need into something faster, which is often a database. At that point you are running a lakehouse and a database, and you have to keep them in sync.

Ecosystem Tooling is Poor Compared to a Data Warehouse

Reporting, data quality, machine learning etc. Everything is made awkward vs SQL data warehouses.

The maturity of tooling around traditional data warehouses is hard to beat. Years of ecosystem development means better integrations, better debugging tools, and better operational workflows.

When something goes wrong in a warehouse, there is usually a query log, an explain plan and a well worn path to finding out why. In a lakehouse you can be digging through Spark logs and object store metadata trying to work out why a job was slow or why a table has ended up in a strange state. BI tools, data quality frameworks and orchestration all work best when they are pointed at something that looks like a database, and it's often a bit of extra work to get them talking to files on a bucket.

My Conclusion

The technology is cool for data geeks and the idea of open standards based data is compelling, but with how well data warehouses can work with unstructured data and with how well they can export/interoperate with things like Iceberg or Parquet, I honestly think I'll just build around a data warehouse from now on unless I'm operating at truly massive scale.

Modern data warehouses have evolved to handle many of the use cases that lakehouses were designed for, while maintaining the simplicity and performance that made them successful in the first place.

I'd still keep an eye on the open table formats. Being able to read and write Iceberg gives you a useful escape route and a way to share data with other teams and tools, and I think that is where the real value is. I just don't think it needs to be the foundation that everything else is built on. Pick a database that is fast and simple to run, and let it talk to the lake where it makes sense.

Portrait of Benjamin Wootton

Written by

Benjamin Wootton

Founder & CTO, New Cadence

We are an independent ClickHouse consultancy. We help businesses deploy ClickHouse open source and ClickHouse Cloud, build solutions on top of ClickHouse for real-time analytics, observability and AI, and resolve performance and reliability issues with their existing deployments.

Connect on LinkedIn
End of article Share