Comprehensive Azure Data Factory knowledge base with official documentation sources, CI/CD methods, deployment patterns, and troubleshooting resources
Provides comprehensive Azure Data Factory knowledge for CI/CD, deployment, and troubleshooting.
npx claudepluginhub josiahsiegel/claude-plugin-marketplaceThis skill inherits all available tools. When active, it can use any tool Claude has access to.
references/activity-types.mdreferences/datasets.mdreferences/expression-functions.mdreferences/linked-services.mdreferences/triggers.mdMANDATORY: Always Use Backslashes on Windows for File Paths
When using Edit or Write tools on Windows, you MUST use backslashes (\) in file paths, NOT forward slashes (/).
Examples:
D:/repos/project/file.tsxD:\repos\project\file.tsxThis applies to:
NEVER create new documentation files unless explicitly requested by the user.
This skill provides comprehensive reference information about Azure Data Factory, including official documentation sources, CI/CD deployment methods, and troubleshooting resources. Use this to access detailed ADF knowledge on-demand.
@microsoft/azure-data-factory-utilities requires Node.js 20.xBreaking Change:- Older Node.js versions (14.x, 16.x, 18.x) may cause package incompatibility errors- Update CI/CD pipelines to use Node.js 20.x or compatible versionsGitHub Actions:yaml- name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '20.x'Azure DevOps:yaml- task: UseNode@1 inputs: version: '20.x'Status: Available only for existing customers as of early 2025 Retirement Date: Not yet announced, but feature is officially deprecated Impact: New customers cannot provision Apache Airflow in Azure Data Factory
Official Deprecation Notice:
Migration Path:
Why Deprecated:
Action Required:
Reference:
ADF Mounting in Fabric:
Cross-Workspace Pipeline Orchestration:
Variable Libraries:
Connector Enhancements:
CRITICAL: As of 2025, npm package @microsoft/azure-data-factory-utilities requires Node.js 20.x
Breaking Change:
GitHub Actions:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20.x'
Azure DevOps:
- task: UseNode@1
inputs:
version: '20.x'
Main Documentation Hub:
Introduction to Azure Data Factory:
Library ID: /websites/learn_microsoft_en-us_azure_data-factory
How to Access:
Use Context7 MCP tool to fetch latest documentation:
mcp__context7__get-library-docs:
- context7CompatibleLibraryID: /websites/learn_microsoft_en-us_azure_data-factory
- topic: "CI/CD continuous integration deployment pipelines ARM templates"
- tokens: 8000
npm Package: @microsoft/azure-data-factory-utilities
Key Features:
package.json Configuration:
{
"scripts": {
"build": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index",
"build-preview": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index --preview"
},
"dependencies": {
"@microsoft/azure-data-factory-utilities": "^1.0.3"
}
}
Commands:
# Validate resources
npm run build validate <rootFolder> <factoryId>
# Generate ARM templates
npm run build export <rootFolder> <factoryId> [outputFolder]
# Preview mode (only stop/start modified triggers)
npm run build-preview export <rootFolder> <factoryId> [outputFolder]
Official Documentation:
Method: Git integration + Publish button
Process:
adf_publish branchadf_publish branchWhen to Use:
Limitations:
Migration Path: Modern approach recommended for new implementations
Primary Command: New-AzResourceGroupDeployment
Syntax:
New-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>" `
-Mode Incremental `
-Verbose
Validation:
Test-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>"
What-If Analysis:
New-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>" `
-WhatIf
Primary Command: az deployment group create
Syntax:
az deployment group create \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name> \
--mode Incremental
Validation:
az deployment group validate \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name>
What-If Analysis:
az deployment group what-if \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name>
Key Improvement in Ver2:
Download Command:
# Linux/macOS/Git Bash
curl -o PrePostDeploymentScript.Ver2.ps1 https://raw.githubusercontent.com/Azure/Azure-DataFactory/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1
# PowerShell
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/Azure/Azure-DataFactory/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1" -OutFile "PrePostDeploymentScript.Ver2.ps1"
Pre-Deployment (Stop Triggers):
./PrePostDeploymentScript.Ver2.ps1 `
-armTemplate "<path-to-ARMTemplateForFactory.json>" `
-ResourceGroupName "<resource-group-name>" `
-DataFactoryName "<factory-name>" `
-predeployment $true `
-deleteDeployment $false
Post-Deployment (Start Triggers & Cleanup):
./PrePostDeploymentScript.Ver2.ps1 `
-armTemplate "<path-to-ARMTemplateForFactory.json>" `
-ResourceGroupName "<resource-group-name>" `
-DataFactoryName "<factory-name>" `
-predeployment $false `
-deleteDeployment $true
Version: PowerShell Core (7.0+) recommended
pwsh: true in AzurePowerShell@5 taskpwsh command, not powershellModules Required:
Official Documentation:
Medium Article (Recent 2025):
Microsoft Community Hub:
Community Blog (February 2025):
Essential Actions:
actions/checkout@v4 - Checkout repositoryactions/setup-node@v4 - Setup Node.jsactions/upload-artifact@v4 - Publish ARM templatesactions/download-artifact@v4 - Download ARM templates in deploy workflowazure/login@v2 - Authenticate to Azureazure/arm-deploy@v2 - Deploy ARM templatesazure/powershell@v2 - Run PrePostDeploymentScriptService Principal (JSON credentials):
{
"clientId": "<GUID>",
"clientSecret": "<STRING>",
"subscriptionId": "<GUID>",
"tenantId": "<GUID>"
}
Store in GitHub secret: AZURE_CREDENTIALS
Workload Identity Federation (More secure):
Microsoft Learn:
Community Guides:
Towards Data Science:
Build Pipeline Tasks:
UseNode@1 - Install Node.jsNpm@1 - Install packages, run build commandsPublishPipelineArtifact@1 - Publish ARM templatesRelease Pipeline Tasks:
DownloadPipelineArtifact@2 - Download ARM templatesAzurePowerShell@5 - Run PrePostDeploymentScriptAzureResourceManagerTemplateDeployment@3 - Deploy ARM templatePermissions Needed:
Configuration:
URL: https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide Last Updated: January 2025
Common Issues Covered:
Enable Diagnostic Settings:
Azure Portal ā Data Factory ā Diagnostic settings ā Add diagnostic setting
Send to: Log Analytics workspace
Logs to Enable:
- PipelineRuns
- TriggerRuns
- ActivityRuns
- SandboxPipelineRuns
- SandboxActivityRuns
Kusto Queries for Troubleshooting:
// Failed pipeline runs in last 24 hours
ADFPipelineRun
| where Status == "Failed"
| where TimeGenerated > ago(24h)
| project TimeGenerated, PipelineName, RunId, Status, ErrorMessage, Parameters
| order by TimeGenerated desc
// Failed CI/CD deployments
ADFActivityRun
| where ActivityType == "ExecutePipeline"
| where Status == "Failed"
| where TimeGenerated > ago(7d)
| project TimeGenerated, PipelineName, ActivityName, ErrorCode, ErrorMessage
| order by TimeGenerated desc
// Performance analysis
ADFActivityRun
| where TimeGenerated > ago(7d)
| extend DurationMinutes = datetime_diff('minute', End, Start)
| summarize AvgDuration = avg(DurationMinutes) by ActivityType, ActivityName
| where AvgDuration > 10
| order by AvgDuration desc
Error: "Template parameters are not valid"
Error: "Updating property type is not supported"
Error: "Operation timed out"
Error: "Authentication failed"
Recommended Folder Layout:
repository-root/
āāā adf-resources/ # ADF JSON files (if using npm approach)
ā āāā dataset/
ā āāā pipeline/
ā āāā trigger/
ā āāā linkedService/
ā āāā integrationRuntime/
āāā .github/
ā āāā workflows/ # GitHub Actions workflows
ā āāā adf-build.yml
ā āāā adf-deploy.yml
āāā azure-pipelines/ # Azure DevOps pipelines
ā āāā build.yml
ā āāā release.yml
āāā parameters/ # Environment-specific parameters
ā āāā ARMTemplateParametersForFactory.dev.json
ā āāā ARMTemplateParametersForFactory.test.json
ā āāā ARMTemplateParametersForFactory.prod.json
āāā package.json # npm configuration
āāā README.md
Only Configure Git on Development ADF:
Rationale: Prevents accidental manual changes in higher environments
Environment Flow:
Dev (Git) ā Build ā Test ā Approval ā Production
ā
ARM Templates
Parameter Management:
Set up alerts for:
Recommended Tools:
Official Azure Data Factory Samples:
Community Examples:
Microsoft Q&A:
Stack Overflow:
azure-data-factoryAzure Status:
Situations requiring current documentation:
How to Fetch:
This knowledge base should be your starting point for all Azure Data Factory questions. Always verify critical information with the latest official documentation when making production decisions.
For detailed JSON schemas and complete reference materials, see:
references/activity-types.md - Complete JSON schemas for all activity types (Copy, ForEach, IfCondition, Switch, Until, Lookup, ExecutePipeline, WebActivity, DatabricksJob, SetVariable, AppendVariable, Wait, Fail, GetMetadata)references/expression-functions.md - Complete reference for all ADF expression functions (string, collection, logical, conversion, math, date/time, pipeline/activity references)references/linked-services.md - Complete JSON configurations for all connector types (Blob Storage, ADLS Gen2, Azure SQL, Synapse, Fabric Lakehouse/Warehouse, Databricks, Key Vault, REST, SFTP, Snowflake, PostgreSQL)references/triggers.md - Complete JSON schemas for schedule, tumbling window, and event triggersreferences/datasets.md - Complete JSON schemas for all dataset types with parameterization patternsActivates when the user asks about AI prompts, needs prompt templates, wants to search for prompts, or mentions prompts.chat. Use for discovering, retrieving, and improving prompts.
Search, retrieve, and install Agent Skills from the prompts.chat registry using MCP tools. Use when the user asks to find skills, browse skill catalogs, install a skill for Claude, or extend Claude's capabilities with reusable AI agent components.
Expert guidance for Next.js Cache Components and Partial Prerendering (PPR). **PROACTIVE ACTIVATION**: Use this skill automatically when working in Next.js projects that have `cacheComponents: true` in their next.config.ts/next.config.js. When this config is detected, proactively apply Cache Components patterns and best practices to all React Server Component implementations. **DETECTION**: At the start of a session in a Next.js project, check for `cacheComponents: true` in next.config. If enabled, this skill's patterns should guide all component authoring, data fetching, and caching decisions. **USE CASES**: Implementing 'use cache' directive, configuring cache lifetimes with cacheLife(), tagging cached data with cacheTag(), invalidating caches with updateTag()/revalidateTag(), optimizing static vs dynamic content boundaries, debugging cache issues, and reviewing Cache Component implementations.