From adf-master
Guides 2025 Azure Data Factory integration with Databricks Job activities, migration from legacy Notebook/JAR, serverless linked services, ServiceNow V2 connector, Snowflake connectors, and MFA enforcement.
npx claudepluginhub josiahsiegel/claude-plugin-marketplace --plugin adf-masterThis skill uses the workspace's default tool permissions.
**CRITICAL UPDATE (2025):** The Databricks Job activity is now the **ONLY recommended method** for orchestrating Databricks in ADF. Microsoft strongly recommends migrating from legacy Notebook, Python, and JAR activities.
Generates design tokens/docs from CSS/Tailwind/styled-components codebases, audits visual consistency across 10 dimensions, detects AI slop in UI.
Records polished WebM UI demo videos of web apps using Playwright with cursor overlay, natural pacing, and three-phase scripting. Activates for demo, walkthrough, screen recording, or tutorial requests.
Delivers idiomatic Kotlin patterns for null safety, immutability, sealed classes, coroutines, Flows, extensions, DSL builders, and Gradle DSL. Use when writing, reviewing, refactoring, or designing Kotlin code.
CRITICAL UPDATE (2025): The Databricks Job activity is now the ONLY recommended method for orchestrating Databricks in ADF. Microsoft strongly recommends migrating from legacy Notebook, Python, and JAR activities.
DatabricksJob (NOT DatabricksSparkJob or DatabricksNotebook)jobParameters (NOT parameters)"authentication": "MSI") recommended| Feature | Notebook Activity (Legacy) | Job Activity (2025) |
|---|---|---|
| Compute | Must configure cluster in linked service | Serverless by default |
| Workflow tasks | Single notebook | Multi-task DAGs (notebook, Python, SQL, DLT) |
| Retry | ADF-level only | Job-level + task-level |
| Repair runs | Not supported | Rerun failed tasks only |
| Git integration | Limited | Full Databricks Git support + DABs |
| Lineage | None | Built-in data lineage |
| If/Else logic | Must use ADF control flow | Native If/Else task types |
For complete JSON examples of Job activity, linked service, and pipeline configurations, see references/databricks-job-examples.md.
ServiceNow V1 connector is at End of Support. Migrate to V2 immediately.
| Feature | V1 | V2 |
|---|---|---|
| Linked service type | ServiceNow | ServiceNowV2 |
| Source type | ServiceNowSource | ServiceNowV2Source |
| Query builder | Custom | Aligns with ServiceNow condition builder |
| Performance | Standard | Enhanced extraction |
| OData support | No | Yes |
Migration steps: Update linked service type to ServiceNowV2, update source type to ServiceNowV2Source, test queries in ServiceNow UI condition builder, adjust timeouts.
Improved performance with 2025 SSL enhancements: enableSsl: true, sslMode: "Require".
Improved performance with KeyPair authentication support and Key Vault secret integration.
New managed identity support for Azure Table Storage and Azure Files connectors (system-assigned and user-assigned).
Spark 3.3 now powers Mapping Data Flows with 30% faster processing, Adaptive Query Execution (AQE), dynamic partition pruning, improved caching, and better column statistics.
Git integration now supports on-premises Azure DevOps Server 2022 via the hostName property.
For complete JSON examples of all connectors, see references/connector-examples.md.
| Scenario | Recommendation |
|---|---|
| Single ADF, simple setup | System-assigned |
| Multiple data factories | User-assigned (shared identity) |
| Complex multi-environment | User-assigned |
| Granular permission control | User-assigned |
| Identity lifecycle independence | User-assigned |
Use ADF's centralized Credentials feature to consolidate Microsoft Entra ID-based credentials across multiple linked services.
Azure MFA is mandatory for all interactive user logins. Impact on ADF:
| Resource | Source Role | Sink Role |
|---|---|---|
| Storage Blob | Storage Blob Data Reader | Storage Blob Data Contributor |
| SQL Database | db_datareader | db_datareader + db_datawriter |
| Key Vault | Get secrets only | Get secrets only |
For complete managed identity JSON examples, see references/connector-examples.md.
Use Databricks Job Activity (MANDATORY) -- Stop using Notebook, Python, JAR activities. Define workflows in Databricks workspace with serverless compute.
Managed Identity Authentication (MANDATORY) -- Use managed identities for ALL Azure resources. Leverage Credentials feature for consolidation. MFA-compliant since October 2025.
Monitor Job Execution -- Track Databricks Job run IDs from ADF output, log parameters for auditability, set up alerts for failures, leverage built-in lineage.
Optimize Spark 3.3 (Data Flows) -- Enable AQE, use 4-8 partitions per core, broadcast joins for small dimensions, dynamic partition pruning.
references/databricks-job-examples.md - Complete JSON for Job activity, linked services, pipeline, and Databricks workspace job definitionreferences/connector-examples.md - Complete JSON for ServiceNow V2, PostgreSQL, Snowflake, Azure Storage MI, Mapping Data Flows, and Azure DevOps Server