From adf-master
Guides Azure Data Factory integration with Microsoft Fabric using Lakehouse and Warehouse connectors, OneLake shortcuts, and cross-workspace data copying.
npx claudepluginhub josiahsiegel/claude-plugin-marketplace --plugin adf-masterThis skill uses the workspace's default tool permissions.
Microsoft Fabric is a unified SaaS analytics platform combining Power BI, Azure Synapse Analytics, and Azure Data Factory capabilities. ADF provides native connectors for Fabric Lakehouse and Fabric Warehouse, enabling seamless data movement between ADF and Fabric workspaces.
Generates design tokens/docs from CSS/Tailwind/styled-components codebases, audits visual consistency across 10 dimensions, detects AI slop in UI.
Records polished WebM UI demo videos of web apps using Playwright with cursor overlay, natural pacing, and three-phase scripting. Activates for demo, walkthrough, screen recording, or tutorial requests.
Delivers idiomatic Kotlin patterns for null safety, immutability, sealed classes, coroutines, Flows, extensions, DSL builders, and Gradle DSL. Use when writing, reviewing, refactoring, or designing Kotlin code.
Microsoft Fabric is a unified SaaS analytics platform combining Power BI, Azure Synapse Analytics, and Azure Data Factory capabilities. ADF provides native connectors for Fabric Lakehouse and Fabric Warehouse, enabling seamless data movement between ADF and Fabric workspaces.
The Fabric Lakehouse connector enables read and write operations to Microsoft Fabric Lakehouse for tables and files.
| Activity | Supported |
|---|---|
| Copy Activity (source and sink) | Yes |
| Lookup Activity | Yes |
| Get Metadata Activity | Yes |
| Delete Activity | Yes |
LakehouseworkspaceId and artifactIdLakehouseTableSink (tables), LakehouseFileSink (files)LakehouseTableSourceappend or overwriteFinding Workspace and Artifact IDs:
https://app.powerbi.com/groups/<workspaceId>/...For complete linked service, dataset, and copy activity JSON examples, see references/lakehouse-examples.md.
The Fabric Warehouse connector provides T-SQL based data warehousing capabilities within the Fabric ecosystem.
| Activity | Supported |
|---|---|
| Copy Activity (source and sink) | Yes |
| Lookup Activity | Yes |
| Get Metadata Activity | Yes |
| Script Activity | Yes |
| Stored Procedure Activity | Yes |
Warehouseendpoint, warehouseWarehouseSinkinsert or upsertautoCreate (creates table if missing)For complete linked service, copy activity, stored procedure, and script activity JSON examples, see references/warehouse-examples.md.
ADF supports three integration patterns with OneLake:
| Pattern | Description | Key Benefit |
|---|---|---|
| ADLS Gen2 Shortcuts | Reference ADLS data via OneLake shortcuts (zero-copy) | No data duplication |
| Incremental Load | Watermark-based incremental copy to Lakehouse | Efficient updates |
| Cross-Platform Invoke | Use InvokePipeline activity to call Fabric pipelines | Hybrid orchestration |
OneLake Shortcuts are the preferred approach when data already exists in ADLS Gen2 -- they provide instant zero-copy access without data movement. Use ADF Copy Activity only when data transformation or format conversion is needed.
For complete pipeline JSON examples for all three patterns, see references/onelake-patterns.md.
For Fabric Lakehouse:
For Fabric Warehouse:
CREATE USER [your-adf-name] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER [your-adf-name];
ALTER ROLE db_datawriter ADD MEMBER [your-adf-name];
App Registration Setup:
Use Managed Identity -- System-assigned for single ADF, user-assigned for multiple. Avoid service principal keys when possible. Store any secrets in Key Vault.
Enable Staging for Large Loads -- Use staging with compression for data volumes > 1 GB, complex transformations, or Fabric Warehouse loads.
Leverage OneLake Shortcuts -- Use ADLS Gen2 -> OneLake Shortcut -> Direct Access instead of ADLS Gen2 -> Copy Activity -> Lakehouse. No data movement, instant availability, reduced costs.
Monitor Fabric Capacity Units (CU) -- Track CU consumption per pipeline run, peak usage, and throttling. Optimize with incremental loads, off-peak scheduling, and right-sized parallelism.
Use Table Option AutoCreate -- Set tableOption: "autoCreate" on WarehouseSink for automatic schema management and faster development.
Implement Error Handling -- Configure retry policies on Copy activities and add WebActivity-based failure logging with dependencyConditions: ["Failed"].
| Issue | Error Message | Solution |
|---|---|---|
| Permission Denied | "User does not have permission to access Fabric workspace" | Add ADF managed identity as Contributor; for Warehouse, create SQL user; allow 5 min propagation |
| Endpoint Not Found | "Unable to connect to endpoint" | Verify workspaceId/artifactId; check workspace URL; ensure Lakehouse/Warehouse is not paused |
| Schema Mismatch | "Column types do not match" | Use tableOption: "autoCreate" or explicit column mappings in translator |
| Performance Degradation | Slow copy performance | Enable staging, increase parallelCopies (4-8), increase DIUs (8-32), check CU throttling |
references/lakehouse-examples.md - Complete linked service, dataset, copy activity, and lookup JSON examplesreferences/warehouse-examples.md - Complete linked service, copy activity, stored procedure, and script activity JSON examplesreferences/onelake-patterns.md - Pipeline patterns for shortcuts, incremental loads, and cross-platform Invoke Pipeline