Databricks

7 items tagged with Databricks

← All tags

  • Keep a PySpark Test Flood Out of Your Agent's Context Window

    7 min read

    A PySpark suite where one fixture breaks floods a coding agent with hundreds of lines of Spark stack traces. Here's the wrapper I built so my agent reads a ~70-line digest instead of 797 raw lines, plus what it doesn't do.

  • Quarantine Bad Rows in Lakeflow SDP Without Breaking Your Clean Table

    10 min read

    A tested inverse-expectations quarantine pattern for Lakeflow SDP: route bad rows to their own dead-letter table while clean rows flow through, plus the NULL trap that silently breaks the split and the companion agent skill that adapts it to your data.

  • The SQL Warehouse Cost Trap Behind Short Databricks Alert Jobs

    8 min read

    A pile of sub-minute alert queries was costing me about half the workspace bill. The fix was not faster queries: it was fewer SQL warehouse startups. Five levers that worked.

  • Two Guardrails for Letting LLM Agents Query Your Databricks Tables

    6 min read

    Hand `databricks experimental aitools tools query` to an LLM agent raw and two things go wrong: it can write, and JSON output bloats your context window. Here's the open-source wrapper I built to fix both.

  • Recover a Lakeflow SDP Pipeline From a Recreated Source Table

    6 min read

    When an upstream Delta table is dropped and recreated, your Lakeflow SDP pipeline breaks with DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE. Here's how to reset checkpoint selection and recover without a full refresh.

  • Your Next DABs Project, Configured in Minutes

    4 min read

    A reusable Declarative Automation Bundles (previously Databricks Asset Bundles, still DABs) template that generates multi-environment projects with medallion architecture, schema-per-user isolation, CI/CD pipelines, and parameterized configs. One command, your choices, a working bundle.

  • Declarative Automation Bundles That Scale with Your Team

    25 min read

    How to configure Declarative Automation Bundles (DABs) for multi-environment team workflows: schema-per-user isolation, environment targets (user/stage/prod), parameterized variables, CI/CD pipelines, and a reusable template you can run today.