About

Most Databricks content falls into two camps. Vendor docs tell you what a feature does. Hype posts tell you it will change everything.

Neither tells you what happens when you actually run it in production, on a Tuesday, with a stakeholder asking why the numbers moved.

That’s what The Delta Log is for.

What you’ll find here

Every week I write about building real data platforms on Databricks:

  • Migrations: moving off Snowflake, Synapse and Airflow without a big-bang cutover, and why most migrations stall halfway.
  • Lakeflow pipelines: declarative pipelines, connectors, medallion layers and the trade-offs nobody puts in the demo.
  • Governance: Unity Catalog, access control and data quality that people actually trust.
  • Platform engineering: Terraform, Asset Bundles and CI/CD for teams that want to stop clicking around in the UI.
  • Cost and performance: clustering, compaction and serverless, plus what each of them does to your DBU bill.

Most posts come with a small, runnable notebook. You can import it, break things on purpose, and see the result for yourself. They’re kept cheap to run, so learning doesn’t cost you much compute.

Who writes it

I’m Sanjay, a Senior Data Engineer at Mantel Group in Sydney. I build Databricks and cloud data platforms for clients across industries like fintech and manufacturing.

Before that, I spent two and a half years at REA Group (realestate.com.au), building streaming and batch pipelines on AWS. I’ve been doing data engineering for about seven years, and I studied data science at Monash.

I also run a small SaaS company. That keeps me honest about the other side of data work: someone always pays the bill, and someone always has to make a decision with the numbers.

Who it’s for

If you’re a data engineer, analytics engineer or data lead who runs on Databricks, or is about to, this is written for you. I’ll assume you know what a Delta table is. I won’t assume the docs told you the whole story.

Get it in your inbox

New posts land every week, free. Subscribe here, and if something I wrote matched (or contradicted) what you’ve seen in production, hit reply. I read every one.

Opinions here are my own and don’t represent my employer or its clients.