Automate migration of Databricks (Unity Catalog/HMS) notebooks, jobs, schedules, and catalog DDL to Oracle AIDP using natural language. Two-pass execution: dependency resolution then cell-by-cell migration with verification and retry on a live cluster.
Based on adoption, maintenance, documentation, and repository signals. Not a security audit or endorsement.
Pre-migration data-availability scan. Reads a manifest, probes every spark.read.* / saveAsTable target on the AIDP cluster, reports OK / MISSING / EMPTY.
Guided catalog migration flow. Extract Databricks Unity Catalog / HMS metadata, preview the 18-rule DDL rewrite, then batched replay on AIDP. Asks before destructive replay step.
Guided full-job migration flow. Walks the user from "I want to migrate this Databricks job" through DAG → data check → migrate → status, asking before each phase.
Parse + summarize the JOB_REPORT.md produced by a prior migration run. Surfaces cell pass/fail/fix counts per task, lists the failing cells, and recommends a next-step skill (fixup-cell, resume, or all-clear).
Use this agent after aidp-migrate-job completes to review a migrated .ipynb for correctness (NOT just "did it run"). Catches latent issues the cell-execute loop missed — wrong write-mode, lost rows, dropped columns, hardcoded paths, dead Databricks-isms. Outputs a structured review report.
Use this agent when the user has pointed at a single Databricks notebook (.ipynb / .py / .scala) and wants to know what it does, what it depends on, what risks the migrator will hit, and which gotchas (from references/gotchas.md) will apply. Run before manually authoring an entry in a migration manifest, OR as a one-off "should we migrate this?" assessment. Returns a structured report (markdown), does not modify the notebook.
YAML-driven acceptance contract for batch and streaming pipelines that need convergence verification, not just "no exceptions". Declares PASS only after K consecutive zero-result windows (consecutive-zero-window convergence). Use for streaming jobs, slowly-draining backfills, or any pipeline whose success is "the pending queue is empty".
Configure the s3:// → oci:// bucket / namespace mapping the migrator uses to rewrite paths during notebook + catalog migration. Use when (a) the user has external tables / files at s3:// paths that need to land on OCI Object Storage, OR (b) check_data_availability reports "S3 bucket X not found in OCI bucket mapping", OR (c) DDL rewriter logs a missing-bucket warning.
Build a migration manifest (the execution DAG) from a Databricks workspace path or workflow ID. Walks %run dependency chains, captures dbutils.notebook.run invocations, and emits reports/<job>_manifest.json — the input every other execute-skill consumes. Use when the user wants to see what would migrate, or before invoking aidp-migrate-job for the first time on a new workload.
Pre-migration data-availability scan. Reads every notebook in a migration manifest, extracts every spark.read.table / spark.read.parquet / saveAsTable reference, and probes whether each target schema/table/path exists on the AIDP cluster BEFORE you spend Pass-2 cluster time. Use after aidp-build-dag and before aidp-migrate-job, especially the first time you migrate against a target environment.
Targeted rewind of a migrated notebook. Re-executes cells from history index N onwards (or a specific cell range) through the execute+verify+fix loop, with a 'why' reason injected so Claude knows what to fix. Use when aidp-migrate-job left a notebook at RESULT=PARTIAL or the user identifies a specific cell that is wrong post-migration.
Uses power tools
Uses Bash, Write, or Edit tools
This repository contains a curated collection of sample notebooks demonstrating how to build data pipelines, run machine learning workloads, and integrate AI capabilities using Oracle AI Data Platform (AIDP) Workbench — a unified, governed workspace for data engineering, ML, and AI development powered by Apache Spark.
Oracle AI Data Platform Workbench is a unified, governed workspace for building, managing, and deploying AI and data-driven solutions. It brings together notebooks, agent development, orchestration, and catalog management in a single collaborative platform — empowering teams to explore data, fine-tune models, and operationalize AI with trust and speed.
Learn more about AIDP Workbench →
oracle-aidp-samples/
├── getting-started/ # Foundational notebooks for new users
│ ├── Delta_Lake/ # Delta Lake feature walkthroughs
│ └── migration/ # Migrating workloads to AIDP
├── data-engineering/
│ ├── ingestion/ # Connectors and data loading patterns
│ └── transformation/ # Pipeline architectures and table formats
│ ├── liquid-clustering/
│ ├── medallion-lake/
│ ├── scd/
│ └── streaming/
├── ai/
│ ├── agent-flows/ # Agent orchestration and scheduling
│ └── ml-datascience/ # ML, LLM, and AI service integrations
└── shared-utils/ # Reusable utilities and data generators
Foundational examples to help you get up and running on AIDP Workbench.
| Notebook | Description |
|---|---|
| Access ALH Data | Write and query data in Oracle Autonomous AI Lakehouse (ALH) using PySpark insertInto and SQL INSERT statements with external catalogs. |
| Access Object Storage Data | Read and write data from OCI Object Storage using direct access, external volumes, and external tables. |
| Analyse Data Using PySpark | PySpark fundamentals: catalog and schema setup, table creation, data insertion, schema exploration, and matplotlib visualizations. |
| Analyse Data Using SQL | Core SQL operations on AIDP including DataFrame creation, transformations, aggregations, and simple visualizations. |
| ALH External Catalog MERGE | End-to-end MERGE workflow into an ALH table via an AIDP external catalog: insert/update/delete with merge keys and OOS-staging skip optimization. |
| Notebook | Description |
|---|---|
| Use Delta Lake Table | Comprehensive guide covering Delta table operations: updates, merges, time travel, liquid clustering, and vacuuming. |
| Delta Change Data Feed | Capture row-level changes (inserts, updates, deletes) from Delta tables for CDC, incremental processing, and streaming pipelines. |
| Handle Schema Evolution | Add and evolve columns in Delta tables without rewriting existing data, leveraging automatic schema evolution. |
| Delta UniForm Tables | Create Delta UniForm tables that automatically synchronize Iceberg metadata for cross-format interoperability. |
| Notebook | Description |
|---|---|
| Migrate Files from Databricks to AIDP | Recursively export notebooks and files from a Databricks workspace to AIDP using the databricks-sdk library. |
| Download from Git to AIDP | Download notebooks and files from a Git repository as a ZIP archive and extract them directly into an AIDP workspace volume. |
Patterns for connecting to and loading data from a wide range of sources.
Own this plugin?
Verify ownership to unlock analytics, metadata editing, and a verified badge. GitHub access is read-only (username + org membership).
Sign in to claimOwn this plugin?
Verify ownership to unlock analytics, metadata editing, and a verified badge. GitHub access is read-only (username + org membership).
Sign in to claim'%20stop-opacity%3D'0.16'%2F%3E%3Cstop%20offset%3D'1'%20stop-color%3D'rgb(200%2C90%2C60)'%20stop-opacity%3D'0.03'%2F%3E%3C%2FlinearGradient%3E%3C%2Fdefs%3E%3Crect%20width%3D'320'%20height%3D'200'%20fill%3D'url(%23g)'%2F%3E%3Ccircle%20cx%3D'250'%20cy%3D'56'%20r%3D'92'%20fill%3D'rgb(200%2C90%2C60)'%20fill-opacity%3D'0.06'%2F%3E%3Ccircle%20cx%3D'64'%20cy%3D'172'%20r%3D'58'%20fill%3D'rgb(200%2C90%2C60)'%20fill-opacity%3D'0.05'%2F%3E%3C%2Fsvg%3E)
claude plugin install oracle-ai-data-platform-workbench-databricks-migrator@claude-plugins-officialOperate the entire Oracle AI Data Platform (AIDP) Workbench in natural language — a 37-skill agent (not a single-engine orchestrator). Discovers your catalog into a grounding cache (FK/join hints + value dictionaries), turns plain English into accurate Spark SQL, and runs the full lakehouse SQL lifecycle (CREATE/INSERT/UPDATE/DELETE/MERGE/OPTIMIZE/VACUUM/DESCRIBE HISTORY/time-travel on the Spark/Delta lakehouse). Ingests files, profiles data and sets quality rules, authors and repairs cron pipelines, provisions clusters (Compute/AI Compute), and debugs via the Spark UI. Governs the platform (roles + per-resource permissions, credential store, Delta Sharing, audit logs, user settings; plus native Git, bundles, and MLOps/MLflow in Preview) and ships AI — Agent Flows across 13 node types with guardrails (content moderation, prompt-attack prevention, PII detection), Knowledge Base RAG, high-code LangGraph/aidputils agents, and reusable Tools. A semantic model + verified-query repository are matched before free generation for accuracy. Signature differentiators: LLM-in-SQL via ai_generate('openai.gpt-5.4', '<prompt>') and cross-source federation in one Spark session. Runs via the official Oracle aidp CLI with an oci raw-request REST fallback — under either api_key or oci-session-token auth. Additive to your Oracle stack.
oracle-samplesOracle AI Data Platform Workbench Spark connectors for Claude Code. 25 connector skills covering Oracle, OCI, SaaS, JDBC, object storage, streaming, REST, Excel, and multi-cloud data sources, including dedicated Azure SQL, NetSuite, and read-only AIDP Snowflake connector guidance.
oracle-samplesOperate the entire Oracle AI Data Platform (AIDP) Workbench in natural language — a 37-skill agent (not a single-engine orchestrator). Discovers your catalog into a grounding cache (FK/join hints + value dictionaries), turns plain English into accurate Spark SQL, and runs the full lakehouse SQL lifecycle (CREATE/INSERT/UPDATE/DELETE/MERGE/OPTIMIZE/VACUUM/DESCRIBE HISTORY/time-travel on the Spark/Delta lakehouse). Ingests files, profiles data and sets quality rules, authors and repairs cron pipelines, provisions clusters (Compute/AI Compute), and debugs via the Spark UI. Governs the platform (roles + per-resource permissions, credential store, Delta Sharing, audit logs, user settings; plus native Git, bundles, and MLOps/MLflow in Preview) and ships AI — Agent Flows across 13 node types with guardrails (content moderation, prompt-attack prevention, PII detection), Knowledge Base RAG, high-code LangGraph/aidputils agents, and reusable Tools. A semantic model + verified-query repository are matched before free generation for accuracy. Signature differentiators: LLM-in-SQL via ai_generate('openai.gpt-5.4', '<prompt>') and cross-source federation in one Spark session. Runs via the official Oracle aidp CLI with an oci raw-request REST fallback — under either api_key or oci-session-token auth. Additive to your Oracle stack.
oracle-samplesDatabricks development toolkit with skills for data engineering, ML, and AI agents plus MCP tools for direct Databricks operations
databricks-solutionsSkills for migrating dbt projects — moving from dbt Core to the Fusion engine or across data platforms.
dbt-labs5 live-detection Databricks skills — cost-leak-hunter, cluster-forensics, uc-migration-pilot, streaming-guardian, bundle-medic — backed by the databricks-workspace-mcp server.
jeremylongshoreData engineering plugin - warehouse exploration, pipeline authoring, Airflow integration
astronomerDatabricks skills for CLI, Apps, Unity Catalog, Model Serving, Declarative Automation Bundles (DABs), and more.
databricks