databricks-pack

Fix the deploy-time foot-guns of Databricks Asset Bundles (DAB) and the infrastructure operations around them: the bundle-bind gap for UC catalogs and external locations, the "unexpected EOF reading terraform.tfstate" redeploy failure, the schema-GRANT-ordering bug that fails the first deploy, customer-managed-key (CMK) rotation that requires draining the whole workspace, and the PrivateLink cost leak where S3/STS/Kinesis still traverse the NAT. Includes a PreToolUse hook that backs up and validates the bundle's terraform state before every deploy, and a PostToolUse hook that recognises the transient GRANT-ordering failure and recommends a single retry. Use when a databricks bundle deploy fails, before rotating a CMK, when a UC resource cannot be bound to a bundle, or when auditing PrivateLink networking cost. Trigger with "bundle deploy failed", "unexpected EOF terraform.tfstate", "bundle bind external location", "User does not have CREATE TABLE on Schema", "rotate CMK", "privatelink still using NAT".

Diagnose broken or unexplained Databricks compute — slow cold starts, failed cluster launches, Photon paying its premium without the speedup, DBR-upgrade landmines, and spot-interruption shuffle aborts — by correlating a cluster's live event stream across API surfaces. Use when a Databricks cluster won't start, died mid-run, is randomly slow to start, when planning a Databricks Runtime upgrade, or when a job keeps failing on spot loss. Trigger with "databricks cluster won't start", "cluster failed", "why is my cluster slow", "NPIP_TUNNEL_SETUP_FAILURE", "databricks runtime upgrade", "photon not helping".

Hunt down Databricks cost leaks — wasted DBUs, idle clusters, oversized SQL warehouses, and untagged runaway spend — and produce a FinOps cost report. Use when a user asks why their Databricks bill is high, wants to find cost leaks / wasted DBUs / idle clusters, or needs a FinOps cost report. Trigger with "databricks cost", "why is my databricks bill", "find wasted spend", "cost leak".

Guard production Databricks data pipelines — Delta Lake, Liquid Clustering, Structured Streaming, Auto Loader, and DLT — against the twelve foot-guns that fire at scale: OPTIMIZE/auto-compaction conflicts, Liquid-Clustering merge conflicts, VACUUM breaking a streaming checkpoint, RocksDB OOM, Auto Loader schema-evolution stops, and DLT refresh data loss. Includes a PreToolUse hook that blocks DROP/CREATE-OR-REPLACE/VACUUM on a table with active streaming consumers. Use when a Delta MERGE/OPTIMIZE fails with a concurrency exception, a stream breaks after VACUUM or a table replace, an Auto Loader stream stops on a new column, a DLT refresh drops data, or before running a destructive op on a streamed-from table. Trigger with "ConcurrentAppendException", "ConcurrentDeleteDeleteException", "DELTA_FILE_NOT_FOUND", "streaming checkpoint broke", "vacuum broke my stream", "autoloader UnknownFieldException", "dlt full refresh".

Pilot a Databricks Hive Metastore (HMS) → Unity Catalog (UC) migration before the deadline — audit every table's migratability, produce a dependency-ordered migration plan, trace UC's two-level permission model, and pick an environment isolation pattern. Use when a user asks how to migrate off the Hive metastore, move to Unity Catalog, which tables are UC-ready, why a table won't migrate, why a user lacks access after migration, or how to isolate dev/test/prod under UC. Trigger with "migrate to unity catalog", "hive metastore migration", "uc migration", "which tables can migrate", "unity catalog permission".

3 hooks across 2 events