fabric-pipelines

Fabric Pipelines

When to Use

• Orchestrating data movement and transformation workflows in Microsoft Fabric
• Implementing control flow logic (loops, conditions, error handling) in data pipelines
• Scheduling and triggering pipeline runs (time-based, event-based)
• Moving data with Copy Data activity from 100+ sources to Fabric destinations
• Deciding between Fabric Pipelines and Notebooks for data orchestration
• Preparing for Microsoft Fabric Data Engineer Associate (DP-700) exam

Core Jobs

1. Pipeline Structure

• Pipeline = orchestration workflow; not a compute engine (Spark, SQL run separately)
• Hierarchy: Pipeline → Stages (parallel groups) → Activities (individual tasks)
• YAML-compatible definition; visual canvas authoring in Fabric UI
• Fabric Pipelines ≈ Azure Data Factory pipelines — same core concepts, Fabric-native

2. Activity Types

Activity	Purpose
Copy Data	Move data from source to sink; 100+ connectors
Dataflow	Execute Dataflow Gen2 transformation
Notebook	Run a Spark Notebook
Stored Procedure	Execute SQL stored procedure in Warehouse/SQL
Get Metadata	Retrieve file/folder metadata (exists, size, count)
ForEach	Iterate over array; execute child activities per item
If Condition	Branch execution based on boolean expression
Until	Loop until condition is true (with timeout)
Delete	Delete files from Lakehouse Files or storage
Wait	Pause pipeline for specified duration

3. Copy Data Activity

• Primary activity for data movement (not transformation)
• Source → optional column mapping → sink
• Supports: file formats (CSV, JSON, Parquet, Avro, ORC), databases, APIs
• Common pattern: ADLS Gen2 → Lakehouse Files/ (bronze ingestion)
• Schema mapping: auto-detect or explicit column-to-column mapping
• Fault tolerance: skip incompatible rows; log errors to storage

4. Parameters and Variables

Concept	Scope	Set by
Pipeline parameters	Pipeline-wide input	Trigger at runtime, parent pipeline, or manual
Pipeline variables	Pipeline-wide mutable	Set Variable activity within pipeline

• Parameters are immutable during run; variables can change (e.g., counter in ForEach loop)
• ForEach passes @item() to child activities; access parameter with @pipeline().parameters.myParam

5. Triggers

Trigger Type	When it fires
Schedule	Fixed cron schedule (e.g., daily at 2 AM UTC)
Tumbling Window	Fixed non-overlapping time windows; backfill-capable
Storage Event	File arrives in OneLake/ADLS (event-driven ingestion)
Custom Event	Azure Event Grid event matches filter
Manual	On-demand via UI or API

• Tumbling Window = best for incremental load patterns (each window = one time slice)
• Schedule trigger = simple recurring; no window concept; all runs independent

6. Error Handling and Control Flow

• Each activity has: On Success, On Failure, On Completion, On Skip paths
• Retry policy: set max retry count and retry interval per activity
• Failure path: connect activities to handle errors (e.g., send Teams notification on failure)
• Global pipeline timeout: default 7 days; set shorter for critical pipelines
• Monitoring: all run history in Monitoring Hub with per-activity status and error details

Key Concepts

• Copy Data activity — data movement only; no transformation logic; use Dataflow/Notebook for transforms
• ForEach — iterates array parameter; sequential or parallel (set parallelism degree)
• Tumbling Window trigger — non-overlapping fixed intervals; each window has its own run; good for incremental
• Monitoring Hub — central view of all pipeline/activity runs; filter by status, date, pipeline name
• Linked service — connection definition for external data sources (reusable across activities)
• Dataset — pointer to specific data within a linked service (table, file path, query)

Checklist

• Copy Data activity used for movement; Dataflow/Notebook used for transformation?
• Pipeline parameters defined for dynamic values (table names, date ranges)?
• ForEach activity configured with appropriate parallelism (avoid too many concurrent connections)?
• Failure paths connected with notification or logging activity?
• Retry policy set on activities that may have transient failures (network, source availability)?
• Tumbling Window trigger used for incremental loads requiring backfill capability?
• Pipeline timeout set appropriately (not left at default 7 days for short workflows)?

Output Format

• 🔴 Critical — using Copy Data activity for data transformation (it only moves data; no transform logic)
• 🔴 Critical — no failure path on critical activities; failures silently ignored
• 🟡 Warning — hardcoded values in pipeline activities instead of parameters (reduces reusability)
• 🟡 Warning — ForEach with sequential execution on large arrays (causes slow pipeline runs)
• 🟢 Suggestion — use Tumbling Window trigger for incremental load to enable backfill on failure

Exam Tips

• Copy Data activity = data movement, not transformation — for transformation, use Dataflow Gen2 activity or Notebook activity
• ForEach = iterate over array parameter — use sequential for rate-limited sources; parallel for independent items (set batch count)
• Pipeline parameters vs variables — parameters are read-only inputs set at trigger time; variables are mutable within pipeline execution
• Tumbling Window trigger — fixed non-overlapping intervals (e.g., hourly); each window runs independently; enables backfill for missed runs
• Fabric Pipelines ≈ Azure Data Factory — same activity types and concepts; exam may test ADF knowledge in Fabric context
• Monitor pipelines in Monitoring Hub — check activity run details for input/output and error messages; not in workspace item list