dbt dominates the transformation on SQL workload but is it still going to be the king in 2024 and beyond? What does it take to move from dbt to SQLMesh?
What's the "bigquery sql compute"? Do you mean scheduled queries?
Of course, you can do it without sqlmesh/dbt, you can stitch some shell or python scripts to do it.
You just don't get everything it brings.
A view != a scheduled query and materialized views are the closest thing to it but still not the same.
"U can test ur queries in bigquery fastly in the console" seems like a great software engineering practice 😏 (it doesn't work over time if you don't just do POCs)
Views won't protect you if you change a field in upstream table while sqlmesh can by evaluating the SQL.
Yes you can throw yet another tool like sqllineage but it won't be integrated with terraform so it's not helpful.
I don't get your point: yes, nothing forces you to use sqlmesh nor dbt. projects on data engineering didn't start with their creation but they helped a lot improving developer productivity and applying best software engineering practices.
Terraform is not the same kind of tool as sqlmesh.
Sure if your usage of BigQuery can be summed up to few views, you can even skip Terraform and use the BigQuery UI to create them.
Terraform won't give you proper CI on the SQL, it won't let you factor SQL with macros, you can't split your models, you're limited to views, there's no testing, it won't let you link model together nor show related lineage, ....
Terraform doesn't serve the same purpose though, yes both can create views on BigQuery
What's the "bigquery sql compute"? Do you mean scheduled queries?
Of course, you can do it without sqlmesh/dbt, you can stitch some shell or python scripts to do it.
You just don't get everything it brings.
A view != a scheduled query and materialized views are the closest thing to it but still not the same.
"U can test ur queries in bigquery fastly in the console" seems like a great software engineering practice 😏 (it doesn't work over time if you don't just do POCs)
Views won't protect you if you change a field in upstream table while sqlmesh can by evaluating the SQL.
Yes you can throw yet another tool like sqllineage but it won't be integrated with terraform so it's not helpful.
I don't get your point: yes, nothing forces you to use sqlmesh nor dbt. projects on data engineering didn't start with their creation but they helped a lot improving developer productivity and applying best software engineering practices.
Terraform is not the same kind of tool as sqlmesh.
Sure if your usage of BigQuery can be summed up to few views, you can even skip Terraform and use the BigQuery UI to create them.
Terraform won't give you proper CI on the SQL, it won't let you factor SQL with macros, you can't split your models, you're limited to views, there's no testing, it won't let you link model together nor show related lineage, ....
Terraform doesn't serve the same purpose though, yes both can create views on BigQuery