Comprehensive Azure Data Factory knowledge base with official documentation sources, CI/CD methods, deployment patterns, and troubleshooting resources
/plugin marketplace add JosiahSiegel/claude-plugin-marketplace/plugin install adf-master@claude-plugin-marketplaceThis skill inherits all available tools. When active, it can use any tool Claude has access to.
MANDATORY: Always Use Backslashes on Windows for File Paths
When using Edit or Write tools on Windows, you MUST use backslashes (\) in file paths, NOT forward slashes (/).
Examples:
D:/repos/project/file.tsxD:\repos\project\file.tsxThis applies to:
NEVER create new documentation files unless explicitly requested by the user.
This skill provides comprehensive reference information about Azure Data Factory, including official documentation sources, CI/CD deployment methods, and troubleshooting resources. Use this to access detailed ADF knowledge on-demand.
@microsoft/azure-data-factory-utilities requires Node.js 20.xBreaking Change:- Older Node.js versions (14.x, 16.x, 18.x) may cause package incompatibility errors- Update CI/CD pipelines to use Node.js 20.x or compatible versionsGitHub Actions:yaml- name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '20.x'Azure DevOps:yaml- task: UseNode@1 inputs: version: '20.x'Status: Available only for existing customers as of early 2025 Retirement Date: Not yet announced, but feature is officially deprecated Impact: New customers cannot provision Apache Airflow in Azure Data Factory
Official Deprecation Notice:
Migration Path:
Why Deprecated:
Action Required:
Reference:
ADF Mounting in Fabric:
Cross-Workspace Pipeline Orchestration:
Variable Libraries:
Connector Enhancements:
CRITICAL: As of 2025, npm package @microsoft/azure-data-factory-utilities requires Node.js 20.x
Breaking Change:
GitHub Actions:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20.x'
Azure DevOps:
- task: UseNode@1
inputs:
version: '20.x'
Main Documentation Hub:
Introduction to Azure Data Factory:
Library ID: /websites/learn_microsoft_en-us_azure_data-factory
How to Access:
Use Context7 MCP tool to fetch latest documentation:
mcp__context7__get-library-docs:
- context7CompatibleLibraryID: /websites/learn_microsoft_en-us_azure_data-factory
- topic: "CI/CD continuous integration deployment pipelines ARM templates"
- tokens: 8000
npm Package: @microsoft/azure-data-factory-utilities
Key Features:
package.json Configuration:
{
"scripts": {
"build": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index",
"build-preview": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index --preview"
},
"dependencies": {
"@microsoft/azure-data-factory-utilities": "^1.0.3"
}
}
Commands:
# Validate resources
npm run build validate <rootFolder> <factoryId>
# Generate ARM templates
npm run build export <rootFolder> <factoryId> [outputFolder]
# Preview mode (only stop/start modified triggers)
npm run build-preview export <rootFolder> <factoryId> [outputFolder]
Official Documentation:
Method: Git integration + Publish button
Process:
adf_publish branchadf_publish branchWhen to Use:
Limitations:
Migration Path: Modern approach recommended for new implementations
Primary Command: New-AzResourceGroupDeployment
Syntax:
New-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>" `
-Mode Incremental `
-Verbose
Validation:
Test-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>"
What-If Analysis:
New-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>" `
-WhatIf
Primary Command: az deployment group create
Syntax:
az deployment group create \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name> \
--mode Incremental
Validation:
az deployment group validate \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name>
What-If Analysis:
az deployment group what-if \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name>
Key Improvement in Ver2:
Download Command:
# Linux/macOS/Git Bash
curl -o PrePostDeploymentScript.Ver2.ps1 https://raw.githubusercontent.com/Azure/Azure-DataFactory/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1
# PowerShell
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/Azure/Azure-DataFactory/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1" -OutFile "PrePostDeploymentScript.Ver2.ps1"
Pre-Deployment (Stop Triggers):
./PrePostDeploymentScript.Ver2.ps1 `
-armTemplate "<path-to-ARMTemplateForFactory.json>" `
-ResourceGroupName "<resource-group-name>" `
-DataFactoryName "<factory-name>" `
-predeployment $true `
-deleteDeployment $false
Post-Deployment (Start Triggers & Cleanup):
./PrePostDeploymentScript.Ver2.ps1 `
-armTemplate "<path-to-ARMTemplateForFactory.json>" `
-ResourceGroupName "<resource-group-name>" `
-DataFactoryName "<factory-name>" `
-predeployment $false `
-deleteDeployment $true
Version: PowerShell Core (7.0+) recommended
pwsh: true in AzurePowerShell@5 taskpwsh command, not powershellModules Required:
Official Documentation:
Medium Article (Recent 2025):
Microsoft Community Hub:
Community Blog (February 2025):
Essential Actions:
actions/checkout@v4 - Checkout repositoryactions/setup-node@v4 - Setup Node.jsactions/upload-artifact@v4 - Publish ARM templatesactions/download-artifact@v4 - Download ARM templates in deploy workflowazure/login@v2 - Authenticate to Azureazure/arm-deploy@v2 - Deploy ARM templatesazure/powershell@v2 - Run PrePostDeploymentScriptService Principal (JSON credentials):
{
"clientId": "<GUID>",
"clientSecret": "<STRING>",
"subscriptionId": "<GUID>",
"tenantId": "<GUID>"
}
Store in GitHub secret: AZURE_CREDENTIALS
Workload Identity Federation (More secure):
Microsoft Learn:
Community Guides:
Towards Data Science:
Build Pipeline Tasks:
UseNode@1 - Install Node.jsNpm@1 - Install packages, run build commandsPublishPipelineArtifact@1 - Publish ARM templatesRelease Pipeline Tasks:
DownloadPipelineArtifact@2 - Download ARM templatesAzurePowerShell@5 - Run PrePostDeploymentScriptAzureResourceManagerTemplateDeployment@3 - Deploy ARM templatePermissions Needed:
Configuration:
URL: https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide Last Updated: January 2025
Common Issues Covered:
Enable Diagnostic Settings:
Azure Portal → Data Factory → Diagnostic settings → Add diagnostic setting
Send to: Log Analytics workspace
Logs to Enable:
- PipelineRuns
- TriggerRuns
- ActivityRuns
- SandboxPipelineRuns
- SandboxActivityRuns
Kusto Queries for Troubleshooting:
// Failed pipeline runs in last 24 hours
ADFPipelineRun
| where Status == "Failed"
| where TimeGenerated > ago(24h)
| project TimeGenerated, PipelineName, RunId, Status, ErrorMessage, Parameters
| order by TimeGenerated desc
// Failed CI/CD deployments
ADFActivityRun
| where ActivityType == "ExecutePipeline"
| where Status == "Failed"
| where TimeGenerated > ago(7d)
| project TimeGenerated, PipelineName, ActivityName, ErrorCode, ErrorMessage
| order by TimeGenerated desc
// Performance analysis
ADFActivityRun
| where TimeGenerated > ago(7d)
| extend DurationMinutes = datetime_diff('minute', End, Start)
| summarize AvgDuration = avg(DurationMinutes) by ActivityType, ActivityName
| where AvgDuration > 10
| order by AvgDuration desc
Error: "Template parameters are not valid"
Error: "Updating property type is not supported"
Error: "Operation timed out"
Error: "Authentication failed"
Recommended Folder Layout:
repository-root/
├── adf-resources/ # ADF JSON files (if using npm approach)
│ ├── dataset/
│ ├── pipeline/
│ ├── trigger/
│ ├── linkedService/
│ └── integrationRuntime/
├── .github/
│ └── workflows/ # GitHub Actions workflows
│ ├── adf-build.yml
│ └── adf-deploy.yml
├── azure-pipelines/ # Azure DevOps pipelines
│ ├── build.yml
│ └── release.yml
├── parameters/ # Environment-specific parameters
│ ├── ARMTemplateParametersForFactory.dev.json
│ ├── ARMTemplateParametersForFactory.test.json
│ └── ARMTemplateParametersForFactory.prod.json
├── package.json # npm configuration
└── README.md
Only Configure Git on Development ADF:
Rationale: Prevents accidental manual changes in higher environments
Environment Flow:
Dev (Git) → Build → Test → Approval → Production
↓
ARM Templates
Parameter Management:
Set up alerts for:
Recommended Tools:
Official Azure Data Factory Samples:
Community Examples:
Microsoft Q&A:
Stack Overflow:
azure-data-factoryAzure Status:
Situations requiring current documentation:
How to Fetch:
This knowledge base should be your starting point for all Azure Data Factory questions. Always verify critical information with the latest official documentation when making production decisions.
This skill should be used when the user asks to "create an agent", "add an agent", "write a subagent", "agent frontmatter", "when to use description", "agent examples", "agent tools", "agent colors", "autonomous agent", or needs guidance on agent structure, system prompts, triggering conditions, or agent development best practices for Claude Code plugins.
This skill should be used when the user asks to "create a slash command", "add a command", "write a custom command", "define command arguments", "use command frontmatter", "organize commands", "create command with file references", "interactive command", "use AskUserQuestion in command", or needs guidance on slash command structure, YAML frontmatter fields, dynamic arguments, bash execution in commands, user interaction patterns, or command development best practices for Claude Code.
This skill should be used when the user asks to "create a hook", "add a PreToolUse/PostToolUse/Stop hook", "validate tool use", "implement prompt-based hooks", "use ${CLAUDE_PLUGIN_ROOT}", "set up event-driven automation", "block dangerous commands", or mentions hook events (PreToolUse, PostToolUse, Stop, SubagentStop, SessionStart, SessionEnd, UserPromptSubmit, PreCompact, Notification). Provides comprehensive guidance for creating and implementing Claude Code plugin hooks with focus on advanced prompt-based hooks API.