What if dbt and SQLMesh were weapons
The Data Transformation Arms Race
It's been 10 months since my last article comparing the two major players in the transformation layer space. Both have made significant strides in delivering new features, and coincidentally, both recently acquired companies! Their shared goal? Becoming the undisputed leader of the transformation layer.
In this quick dive, I'll share my current perspective on both platforms and hopefully spark your interest to try one (or why not both?).
To make this comparison more entertaining, I'm borrowing from that classic meme format: if programming languages were weapons
Let's apply this to our transformation tools!
dbt
I portrayed dbt as an AK47 because it has a some common characteristics:
Battle-tested by companies of all sizes, it's robust enough to handle most of your use cases. Sure, it occasionally falls short, but there's a whole arsenal of tricks and hacks to help you overcome any obstacle.
It's achieved widespread adoption – every data engineer and analyst worth their salt knows about it. It's not just popular; it's the current "golden standard."
The learning curve is gentle, with an abundance of tutorials and documentation to get you started. You won't need special training to become proficient.
Like a well-maintained AK-47 with its variety of attachments, dbt is highly extensible. Whether you need better observability (think scope) or utilities to speed up your projects (quick-aim mods), the package ecosystem has more options than you'd expect!
SQLMesh
For SQLMesh, I would picture it as a futuristic sniper because of its interesting perks:
It's learned from the design flaws of its predecessors, incorporating those lessons into its core architecture.
It's sleek and powerful, packed with features and potential. Thanks to advanced internals like sophisticated caching and dialect transpilation, it hits targets fast and precise.
It's not as straightforward as dbt – you'll need some weapon handling experience to unlock its full potential.
Early adopters report impressive results, though it's too soon to declare it superior to the trusty AK-47 in all scenarios.
While still operating in a niche, its performance suggests it could become mainstream soon!
The State of Play in 2025
dbt has spent 2024 reinforcing its leadership position as perhaps the most widely adopted data engineering tool in the market. They've heavily invested in their cloud offering, capturing significant market share beyond just tech companies. Their focus has shifted toward GenAI tools, multi-project management, and semantic layer offerings. While core development took a slight backseat, it still received meaningful updates like unit testing and microbatch materialization.
Choosing dbt in 2025 is like choosing Oracle in the 2000s – it's the safe bet that no one will question. And just like Oracle's journey, being the market leader comes with both benefits and challenges. dbt's widespread adoption means you'll find a solution to almost any problem in their extensive community forums, and hiring dbt-skilled developers is relatively straightforward. However, this dominance also means that dbt faces the classic innovator's dilemma: with a large user base and numerous integrations, any significant changes need to maintain backward compatibility. The recent acquisition of SDF Labs might be just the breath of fresh air needed to address some of dbt's key challenges. Is dbt 2.0 on its way?
On the other side of the ring, Tobiko Data (SQLMesh's parent company) hasn't been resting on their laurels. With sometimes multiple releases per day, they've been squashing bugs and shipping features at an impressive pace. Multi-engine support within the same project and dlt sink import are just a few examples. Their cloud offering is actively evolving, promising best-in-class scheduling and granular debugging capabilities. The recent acqui-hire of the Quary team signals their commitment to staying at the forefront of innovation.
Picking SQLMesh in 2025 is akin to choosing PostgreSQL in the 2010s – it's the choice for teams that prioritize flexibility, innovation, and staying ahead of the curve. While it may not match dbt's widespread adoption yet, its rapid evolution and ambitious feature set make it a favorite among forward-thinking developers. If you're betting on a fast-moving, developer-focused tool, SQLMesh might be your ace in the hole!
Both platforms are undoubtedly strong competitors, and it will be interesting to see how the battle unfolds in the next two years. I hope for a healthy competition that drives innovation, advances open-source contributions, and delivers cutting-edge tools—much like what we're witnessing with Databricks vs. Snowflake or Confluent vs. Redpanda in the data engineering space.
So this year, what would you try?
About Me & Current Projects
I'm currently leveraging dbt Core in production at Teads, where we use it with BigQuery to supercharge our data operations. As part of this journey, I've developed and maintain a dbt package that helps with BigQuery monitoring – you can check it out at dbt-bigquery-monitoring.
I actively contribute to both dbt and SQLMesh when time permits, and I can vouch for the excellence of both platforms! They each bring unique strengths to the table.
This week, Teads (where I'm working for over a decade) and Outbrain officially completed their merger, which was first announced last August. I'm thrilled about the opportunity to combine the strengths of both adtech platforms to build a cutting-edge, full-funnel advertising solution—leveraging AI and data across an omnichannel inventory for the best outcomes. While this might sound very corporate, I can assure you that the company fosters a strong engineering culture and prioritizes a premium customer experience.
Looking ahead to 2025, I'm working on something exciting: a dbt adapter for Deltastream, a cloud streaming SQL engine. If you're interested in combining the power of dbt (or SQLMesh) with real-time analytics capabilities, I'd love to chat about it.
I was recently received by Bolaji Oyejide on his data Careers Show. It was a great opportunity to reflect on my journey and the work I’m doing. Get ready to endure my French accent! 😁
Want to connect? I'll be attending the Group By conference hosted by Tobiko Data in San Francisco on March 20th. If you're planning to be there, let's meet up and talk data engineering! If you can’t feel free to catch me on this Substack, Medium or on LinkedIn.
Great analogies ;)