datajunction-repo

DataJunction Repo-Backed Workflow

YAML-and-git authoring for DJ. Use this skill when working in a repo-backed namespace — node definitions are YAML files, changes go through PRs, CI/CD deploys.

For modeling decisions (whether to make something a metric, dim, or transform; how to decompose a query), see datajunction-semantic-model. This skill assumes the modeling decisions are made and you're translating them into YAML.

Overview

DataJunction supports repo-backed namespaces where node definitions are stored as YAML files in a git repository. This enables:

• ✅ Version control for your semantic layer
• ✅ Pull request review workflows for changes
• ✅ Branch-based development (feature branches, environments)
• ✅ Declarative configuration (infrastructure as code)
• ✅ Audit trail of all changes
• ✅ Team collaboration with code review

---

Repository Structure

There is no required layout under nodes/. DJ walks the tree recursively (rglob("nodes/**/*.yaml")) and takes the namespace from each YAML's name: field. The folder path is purely organizational. Files can be flat, nested by type, nested by domain, or any mix — DJ doesn't care, and reorganizing later is a free git mv.

That said, you should still choose a layout for your repo so contributors know where to add things. A few common patterns:

By node type (good when most contributors are adding metrics across one domain):

nodes/
  sources/transactions.yaml
  dimensions/user.yaml
  metrics/revenue.yaml
  transforms/clean_transactions.yaml
cubes/revenue_cube.yaml

By domain (good when multiple teams/areas share one repo):

nodes/
  billing/
    sources/payments.yaml
    transforms/payment_events_clean.yaml
    metrics/monthly_revenue.yaml
  user_events/
    transforms/sessions.yaml
    metrics/dau.yaml

Flat (small projects, <30 nodes):

nodes/
  transactions.yaml
  user.yaml
  revenue.yaml

Deciding on a layout

For an existing repo: look at the current nodes/ tree and match it. Consistency within a repo matters more than any specific scheme. Don't introduce a new style alongside an established one.

For a new repo: ask the user a few questions before committing to a layout:

• How many contributors / teams will use this repo?
• Roughly how many nodes do you expect (10? 100? 1000?)
• Are most nodes going to be metrics on top of a few shared transforms, or many independent fact/dim chains?

Defaults from those answers:

• 1 team + <30 nodes → flat is fine
• 1 team + larger → by node type
• Multiple teams → by domain, with optional type subdirs inside each domain folder when a domain itself grows past ~30 nodes

The right call almost never matters as much as it feels like it does — folder layout has zero effect on deployed namespaces, so optimize for "where would a new contributor look for X" and accept that you'll reorganize as the repo grows.

---

Branch-Based Development

Understanding Branch-Based Namespaces

When you create a branch in a repo-backed namespace, DJ creates a corresponding namespace that points to that branch.

Naming convention:

{namespace}.{branch_name}

Examples:

• finance.main → points to main branch of finance repo
• finance.feature-new-metrics → points to feature-new-metrics branch
• finance.staging → points to staging branch

Creating a Feature Branch

Option A: Via DJ API

curl -b ~/.dj/cookies.txt -X POST $DJ_URL/namespaces/finance/branches \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "feature-new-metrics",
    "from_branch": "main"
  }'

# This creates:
# 1. Git branch: feature-new-metrics (from main)
# 2. DJ namespace: finance.feature-new-metrics

Option B: Manually in Git

cd dj-finance/
git checkout -b feature-new-metrics
git push origin feature-new-metrics

# DJ may auto-discover branches

---

YAML Node Definitions

All node types can be defined as YAML files in the repository.

Column-Level Configuration

Column types are auto-inferred from queries

• DJ analyzes your SQL and automatically determines column types
• No need to manually specify types in the columns section

Use CAST() to control specific types

query: |
  SELECT
    CAST(user_id AS bigint) AS user_id,
    CAST(revenue AS decimal(18,2)) AS revenue,
    transaction_date
  FROM finance.transactions

Column attributes

• primary_key — Marks column as part of the primary key
• dimension — Marks column as available for grouping (transforms/facts only)

Optional column fields

• display_name — Human-readable name for UI display

  columns:
    - name: ad_account_id
      display_name: "Ad Account ID"
      attributes:
        - dimension
  

Important constraints:

• ❌ SELECT * is NOT supported — always explicitly list columns
• ✅ Column order in YAML should match SELECT order
• ✅ Use descriptive display names for better UX

Source Node YAML

# nodes/sources/transactions.yaml
name: finance.transactions
description: Raw transaction data from payment system
type: source
catalog: prod_catalog
schema_: finance
table: transactions_table

columns:
  - name: transaction_id
    display_name: "Transaction ID"
    attributes:
      - primary_key
  - name: user_id
    display_name: "User ID"
  - name: amount_usd
    display_name: "Amount (USD)"
  - name: transaction_date
    display_name: "Transaction Date"
  - name: status
    display_name: "Status"

dimension_links:
  - type: join
    dimension_node: common.dimensions.users
    join_type: left
    join_on: finance.transactions.user_id = common.dimensions.users.user_id

  - type: join
    dimension_node: common.dimensions.date
    join_type: left
    join_on: finance.transactions.transaction_date = common.dimensions.date.dateint

mode: published

Notes:

• Column types are auto-inferred from the source table schema
• display_name provides human-readable labels for UI
• join_type can be left, right, or inner (defaults to left)

Dimension Node YAML

# nodes/dimensions/user.yaml
name: finance.user
description: User dimension with attributes
type: dimension
query: |
  SELECT
    user_id,
    username,
    email,
    country_code,
    signup_date,
    tier
  FROM finance.users

primary_key:
  - user_id

columns:
  - name: user_id
    display_name: "User ID"
    attributes:
      - primary_key
  - name: username
    display_name: "Username"
  - name: email
    display_name: "Email Address"
  - name: country_code
    display_name: "Country"
  - name: signup_date
    display_name: "Signup Date"
  - name: tier
    display_name: "User Tier"

dimension_links:
  - type: join
    dimension_node: common.dimensions.date
    join_type: left
    join_on: finance.user.signup_date = common.dimensions.date.dateint

  - type: join
    dimension_node: common.dimensions.country
    join_type: left
    join_on: finance.user.country_code = common.dimensions.country.country_code

mode: published

Notes:

• Dimension node columns don't need dimension attribute (all are dimensions by nature)
• display_name improves readability in query builders and dashboards

Metric Node YAML

# nodes/metrics/revenue.yaml
name: finance.total_revenue
description: Total revenue from completed transactions
type: metric
query: |
  SELECT
    SUM(
      CASE
        WHEN status = 'completed' AND refund_flag = false
        THEN amount_usd
        ELSE 0
      END
    ) AS total_revenue
  FROM finance.transactions

required_dimensions:
  - common.dimensions.date.dateint

metric_metadata:
  direction: higher_is_better
  unit: dollar

owners:
  - [email protected]
  - [email protected]

mode: published

Important metric rules (see datajunction-semantic-model for full discussion):

• ✅ ALWAYS specify owners — critical for governance
• ❌ No WHERE clauses in metric queries (use CASE WHEN instead)
• ✅ Include required_dimensions for time-based metrics
• ✅ Add metric_metadata for direction and unit

Transform Node YAML

# nodes/transforms/clean_transactions.yaml
name: finance.clean_transactions
description: Cleaned transaction data with standardized status
type: transform
primary_key:
  - transaction_id
query: |
  SELECT
    transaction_id,
    user_id,
    amount_usd,
    transaction_date,
    CASE
      WHEN status IN ('complete', 'completed', 'success') THEN 'completed'
      WHEN status IN ('fail', 'failed', 'error') THEN 'failed'
      ELSE status
    END AS status_clean,
    refund_flag
  FROM finance.transactions

columns:
  - name: transaction_id
    display_name: "Transaction ID"
    attributes:
      - primary_key
  - name: user_id
    display_name: "User ID"
    attributes:
      - dimension
  - name: amount_usd
    display_name: "Amount (USD)"
  - name: transaction_date
    display_name: "Transaction Date"
    attributes:
      - dimension
  - name: status_clean
    display_name: "Status"
    attributes:
      - dimension
  - name: refund_flag
    display_name: "Refund Flag"
    attributes:
      - dimension

dimension_links:
  - type: join
    dimension_node: common.dimensions.users
    join_type: left
    join_on: finance.clean_transactions.user_id = common.dimensions.users.user_id

  - type: join
    dimension_node: common.dimensions.date
    join_type: left
    join_on: finance.clean_transactions.transaction_date = common.dimensions.date.dateint

mode: published

Notes:

• primary_key field lists the primary key column(s)
• dimension attribute marks columns available for grouping in metrics
• Column types are auto-inferred from the query
• Columns without dimension attribute are typically measures/facts

Cube YAML

# cubes/revenue_cube.yaml
name: finance.revenue_cube
description: Pre-computed revenue metrics by date and region
metrics:
  - finance.total_revenue
  - finance.avg_transaction_value

dimensions:
  - common.dimensions.date.dateint
  - common.dimensions.date.month
  - common.dimensions.users.country_code

mode: published

When to use cubes:

• Frequently queried metric combinations
• Pre-compute for performance (materialization)
• Dashboard metric sets

Critical: All metrics in a cube MUST share ALL dimensions in the cube. Use get_common_dimensions MCP tool (see datajunction-query) to check first.

---

Temporal Partitions on Cubes

Temporal partitions enable automatic partition filtering for performance optimization. When configured, DJ automatically adds partition filters to SQL queries, dramatically improving query performance on large datasets.

How Partitions Work

A partition is always declared on a column. When that column is a dimension attribute on a cube, DJ uses it as the partition boundary and pushes the filter down to all upstream nodes that link to that same dimension.

Partition field format:

partition:
  type: temporal        # or: categorical
  granularity: day      # second, minute, hour, day, week, month, quarter, year
  format: yyyyMMdd      # Java/Spark date format (e.g. yyyyMMdd → 20240101, yyyy-MM-dd → 2024-01-01)

Declaring a Partition on a Cube

In a cube, declare the partition in the columns: section using the full dimension attribute path as the column name:

# cubes/revenue_cube.yaml
name: ${prefix}revenue_cube
node_type: cube
metrics:
  - ${prefix}total_revenue
  - ${prefix}order_count

dimensions:
  - common.dimensions.date.dateint
  - common.dimensions.geo.country_code

columns:
  - name: common.dimensions.date.dateint   # ← must match exactly the entry in dimensions
    display_name: Date
    attributes:
      - primary_key
    partition:
      type: temporal
      granularity: day
      format: yyyyMMdd

How Partition Filter Pushdown Works

Once a cube column has a partition spec, DJ:

1. Generates SQL with ${dj_logical_timestamp} template variables when include_temporal_filters=True
2. Pushes those filters down to all upstream nodes that link to the same dimension
3. Reduces data scanned by limiting to relevant partitions

For filter pushdown to work, upstream nodes (sources, transforms) must have a dimension link to the same dimension:

# transforms/orders.yaml
dimension_links:
  - type: join
    dimension_node: common.dimensions.date
    join_type: left
    join_on: ${prefix}orders.order_date = common.dimensions.date.dateint
    # ↑ DJ traces this link and pushes WHERE order_date >= X AND order_date <= Y

• ✅ Upstream node links to common.dimensions.date on order_date → DJ pushes WHERE order_date >= X AND order_date <= Y
• ❌ Upstream node has no link to common.dimensions.date → no filter pushed, full table scan

Best Practices for Temporal Partitions

1. Declare the partition on the cube's columns: block

   • Use the full dimension attribute path as the column name (must match exactly what's in dimensions:)
   • Without a partition: declared on a cube column, DJ cannot enable partition filtering for that cube

2. Ensure consistent dimension links across all nodes

   • All upstream sources/transforms must link to the same dimension that carries the partition
   • Use the same join key everywhere (e.g., always dateint, not mixing dateint and date_str)

3. Use appropriate granularity and format

   • granularity: day with format: yyyyMMdd — for integer date partitions like 20240101
   • granularity: day with format: yyyy-MM-dd — for string date partitions like 2024-01-01
   • granularity: month, quarter, year — for coarser partitioning

4. Verify partition filtering is working

   • Use build_metric_sql (datajunction-query skill) with include_temporal_filters=True
   • Check generated SQL includes partition filters on upstream tables
   • If filters are missing, check that upstream nodes have dimension links pointing to the partitioned dimension column

5. Match the physical partition scheme of your warehouse

   • The format must match how partition values are actually stored in the table
   • Align granularity with how data is physically partitioned in storage

Example: Complete Temporal Partition Setup

The partition is declared on the cube's dateint column. DJ pushes the filter down to orders because it has a dimension link to common.dimensions.date.

Step 1: Transform with date dimension link

# transforms/orders.yaml
name: ${prefix}orders
node_type: transform
columns:
  - name: order_date
  - name: product_id
  - name: order_count
  - name: total_revenue

dimension_links:
  - type: join
    dimension_node: common.dimensions.time.date
    join_type: left
    join_on: ${prefix}orders.order_date = common.dimensions.time.date.dateint

query: |
  SELECT product_id, order_date, COUNT(*) AS order_count, SUM(amount_usd) AS total_revenue
  FROM source.prod.orders_f
  GROUP BY product_id, order_date

Step 2: Metrics

# metrics/total_orders.yaml
name: ${prefix}total_orders
node_type: metric
query: SELECT SUM(order_count) FROM ${prefix}orders

Step 3: Cube — declare the partition on the external dimension attribute

# cubes/orders_cube.yaml
name: ${prefix}orders_cube
node_type: cube
metrics:
  - ${prefix}total_orders
dimensions:
  - common.dimensions.time.date.dateint
  - ${prefix}orders.product_id

columns:
  - name: common.dimensions.time.date.dateint   # ← full attribute path, matches dimensions entry
    display_name: Date
    attributes:
      - primary_key
    partition:
      type: temporal
      granularity: day
      format: yyyyMMdd

Result: Queries with include_temporal_filters=True push WHERE order_date >= X AND order_date <= Y to the orders transform.

---

Complete Workflow Example

Scenario: Add a new metric to the finance namespace.

Step 1: Check if finance is repo-backed

Use MCP tool (datajunction-query):

get_node_details(name="finance.total_revenue")
# Check the output for git repository info

Or via REST API:

curl -b ~/.dj/cookies.txt -X GET $DJ_URL/namespaces/finance/git

Step 2: Create feature branch

# Via DJ API
curl -b ~/.dj/cookies.txt -X POST $DJ_URL/namespaces/finance/branches \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "feature-add-churn-metric",
    "from_branch": "main"
  }'
# Creates namespace: finance.feature-add-churn-metric

Step 3: Clone repo and check out branch

git clone https://github.com/myorg/dj-finance.git
cd dj-finance
git checkout feature-add-churn-metric

Step 4: Create metric YAML file

cat > nodes/metrics/churn_rate.yaml <<'EOF'
name: finance.churn_rate
description: Monthly user churn rate
type: metric
query: |
  SELECT
    CAST(SUM(CASE WHEN churned = true THEN 1 ELSE 0 END) AS DOUBLE) /
    NULLIF(COUNT(DISTINCT user_id), 0) AS churn_rate
  FROM finance.user_activity

required_dimensions:
  - common.dimensions.date.month

metric_metadata:
  direction: lower_is_better
  unit: unitless

owners:
  - [email protected]

mode: published
EOF

Step 5: Commit and push

git add nodes/metrics/churn_rate.yaml
git commit -m "Add monthly churn rate metric"
git push origin feature-add-churn-metric

Step 6: DJ syncs automatically

Use MCP tool to verify:

get_node_details(name="finance.churn_rate")

Step 7: Test in branch namespace

build_metric_sql(
  metrics=["finance.churn_rate"],
  dimensions=["common.dimensions.date.month"],
  filters=["common.dimensions.date.year = 2024"]
)

Step 8: Create PR for review

gh pr create \
  --title "Add monthly churn rate metric" \
  --body "Adds a new metric to track monthly user churn rate" \
  --base main

Step 9: Get review, merge PR

gh pr merge --squash

Step 10: Changes sync to production — metric now available in finance.main namespace.

---

Best Practices for Repo Authoring

Column

• ✅ Always explicitly list columns (SELECT * not supported)
• ✅ Add display_name for better UX in query builders
• ✅ Mark primary key columns with primary_key attribute
• ✅ Mark groupable columns with dimension attribute (transforms/facts)
• ✅ Use CAST() in query if you need specific types
• ✅ Column order in YAML should match SELECT order

Cube

• ✅ Use get_common_dimensions MCP tool (datajunction-query) to check compatibility first
• ✅ Only use shared dimensions
• ✅ Use for frequently queried combinations
• ✅ Always set temporal partitions on cubes for performance
• ✅ Ensure all upstream nodes link to the same date dimension
• ✅ Match granularity to physical partition scheme in data warehouse
• ✅ Verify partition filters in generated SQL

Workflow

1. Always create feature branches — never commit directly to default branch
2. Use descriptive branch names — feature-add-revenue-metrics not fix-stuff
3. Write clear commit messages — explain the "why" not just the "what"
4. Keep PRs focused — one logical change per PR
5. Test in branch namespace — validate metrics work before merging
6. Use draft mode first — set mode: draft while developing, published when ready
7. Document in YAML — use description fields thoroughly
8. Assign ownership — ideally teams rather than individuals for continuity