Stats
Actions
Tags
Help us improve
Share bugs, ideas, or general feedback.
From adf-master
Guides 2025 Azure Data Factory integration with Databricks Job activities, migration from legacy Notebook/JAR, serverless linked services, ServiceNow V2 connector, Snowflake connectors, and MFA enforcement.
npx claudepluginhub josiahsiegel/claude-plugin-marketplace --plugin adf-masterHow this skill is triggered — by the user, by Claude, or both
Slash command
/adf-master:databricks-2025The summary Claude sees in its skill listing — used to decide when to auto-load this skill
**CRITICAL UPDATE (2025):** The Databricks Job activity is now the **ONLY recommended method** for orchestrating Databricks in ADF. Microsoft strongly recommends migrating from legacy Notebook, Python, and JAR activities.
Guides on Azure Data Factory validation rules including activity nesting limitations, ForEach restrictions, pipeline validation, linked service authentication, resource limits, Set Variable rules, and Data Flow constraints.
Port Databricks notebooks and jobs to Microsoft Fabric: dbutils→notebookutils, Unity Catalog 3-level→Lakehouse 2-level, DBFS→OneLake, Jobs→Spark Job Definitions, Photon→Native Execution Engine.
Migrates Databricks workloads from classic compute to serverless compute. Scans code for compatibility issues like RDDs and DBFS, provides Spark Connect fixes, and guides migration for notebooks and jobs.
Share bugs, ideas, or general feedback.
CRITICAL UPDATE (2025): The Databricks Job activity is now the ONLY recommended method for orchestrating Databricks in ADF. Microsoft strongly recommends migrating from legacy Notebook, Python, and JAR activities.
DatabricksJob (NOT DatabricksSparkJob or DatabricksNotebook)jobParameters (NOT parameters)"authentication": "MSI") recommended| Feature | Notebook Activity (Legacy) | Job Activity (2025) |
|---|---|---|
| Compute | Must configure cluster in linked service | Serverless by default |
| Workflow tasks | Single notebook | Multi-task DAGs (notebook, Python, SQL, DLT) |
| Retry | ADF-level only | Job-level + task-level |
| Repair runs | Not supported | Rerun failed tasks only |
| Git integration | Limited | Full Databricks Git support + DABs |
| Lineage | None | Built-in data lineage |
| If/Else logic | Must use ADF control flow | Native If/Else task types |
For complete JSON examples of Job activity, linked service, and pipeline configurations, see references/databricks-job-examples.md.
ServiceNow V1 connector is at End of Support. Migrate to V2 immediately.
| Feature | V1 | V2 |
|---|---|---|
| Linked service type | ServiceNow | ServiceNowV2 |
| Source type | ServiceNowSource | ServiceNowV2Source |
| Query builder | Custom | Aligns with ServiceNow condition builder |
| Performance | Standard | Enhanced extraction |
| OData support | No | Yes |
Migration steps: Update linked service type to ServiceNowV2, update source type to ServiceNowV2Source, test queries in ServiceNow UI condition builder, adjust timeouts.
Improved performance with 2025 SSL enhancements: enableSsl: true, sslMode: "Require".
Improved performance with KeyPair authentication support and Key Vault secret integration.
New managed identity support for Azure Table Storage and Azure Files connectors (system-assigned and user-assigned).
Spark 3.3 now powers Mapping Data Flows with 30% faster processing, Adaptive Query Execution (AQE), dynamic partition pruning, improved caching, and better column statistics.
Git integration now supports on-premises Azure DevOps Server 2022 via the hostName property.
For complete JSON examples of all connectors, see references/connector-examples.md.
| Scenario | Recommendation |
|---|---|
| Single ADF, simple setup | System-assigned |
| Multiple data factories | User-assigned (shared identity) |
| Complex multi-environment | User-assigned |
| Granular permission control | User-assigned |
| Identity lifecycle independence | User-assigned |
Use ADF's centralized Credentials feature to consolidate Microsoft Entra ID-based credentials across multiple linked services.
Azure MFA is mandatory for all interactive user logins. Impact on ADF:
| Resource | Source Role | Sink Role |
|---|---|---|
| Storage Blob | Storage Blob Data Reader | Storage Blob Data Contributor |
| SQL Database | db_datareader | db_datareader + db_datawriter |
| Key Vault | Get secrets only | Get secrets only |
For complete managed identity JSON examples, see references/connector-examples.md.
Use Databricks Job Activity (MANDATORY) -- Stop using Notebook, Python, JAR activities. Define workflows in Databricks workspace with serverless compute.
Managed Identity Authentication (MANDATORY) -- Use managed identities for ALL Azure resources. Leverage Credentials feature for consolidation. MFA-compliant since October 2025.
Monitor Job Execution -- Track Databricks Job run IDs from ADF output, log parameters for auditability, set up alerts for failures, leverage built-in lineage.
Optimize Spark 3.3 (Data Flows) -- Enable AQE, use 4-8 partitions per core, broadcast joins for small dimensions, dynamic partition pruning.
references/databricks-job-examples.md - Complete JSON for Job activity, linked services, pipeline, and Databricks workspace job definitionreferences/connector-examples.md - Complete JSON for ServiceNow V2, PostgreSQL, Snowflake, Azure Storage MI, Mapping Data Flows, and Azure DevOps Server