flowx

flowx

ADF to Databricks Lakeflow Jobs translator, delivered as agent skills.

flowx converts Azure Data Factory (ADF) pipeline definitions into Databricks Lakeflow Jobs packaged as Declarative Automation Bundles (DABs). It deterministically translates known activity types and falls back to agentic (LLM-assisted) translation for complex or rare types. flowx runs as a set of agent skills usable from Databricks Genie Code, Claude Code, or any tool that supports the Agent Skills standard.

Architecture

                         flowx Pipeline
                         ==================

  ADF JSON (UC Volumes / Workspace)
        |
        v
  +------------------+
  |  1. DISCOVER     |    Parse ADF ARM/JSON exports
  |  adf_loader.py   | -> Typed AST -> metadata/inventory.json
  +------------------+
        |
        v
  +------------------+
  |  2. CONVERT      |    Registry dispatch + topological sort
  |  engine.py       | -> Pipeline IR (deterministic + agentic gaps)
  +------------------+
        |
        v
  +------------------+
  |  3. PACKAGE      |    IR -> DAB YAML + notebooks + setup scripts
  |  dab_writer.py   | -> Deployable DABs project
  +------------------+
        |
        v
  databricks bundle validate / deploy

The phases are exposed two ways: as skills the agent runs directly (via a Python virtual environment locally), and as a single MCP tool hosted on a Databricks App (for Genie Code). See Running flowx as an MCP server.

Installation

flowx installs in one of two shapes depending on where your agent runs. Full, step-by-step instructions for both are in the installation docs; the summary:

Databricks Genie Code

The phases run as an MCP server (a Databricks App). Clone the repo into /Workspace/Shared (so the app's service principal can read the source), copy skills/ into your skills folder, then run the setup skill:

@flowx-setup

On Databricks, flowx-setup deploys the mcp-flowx app for you. You can also deploy it directly by running the app/deploy_app.py notebook (SDK-based, works on serverless), or app/deploy.sh from a workspace web terminal. Then add the app under Genie Code Settings → MCP Servers → Add Server → Custom MCP server.

Claude Code (and other local agent harnesses)

flowx is a Claude Code plugin distributed through its marketplace:

/plugin marketplace add databricks-solutions/flowx
/plugin install flowx@flowx

Run /reload-plugins, then set up the local runtime once:

/flowx:flowx-setup

This provisions a Python virtual environment (via scripts/bootstrap.sh) and writes a .migration-venv marker the phase skills read. No uv is required for plugin users.

Usage

Run the end-to-end migration:

/flowx:flowx-migrate

Or run individual phases:

/flowx:flowx-discover    # Parse ADF JSON, produce inventory + complexity report
/flowx:flowx-convert     # Deterministic + agentic translation
/flowx:flowx-package     # Generate DABs project

(In Genie Code, invoke the same skills with the @ prefix, e.g. @flowx-migrate.)

Setup

/flowx:flowx-setup keys off the DATABRICKS_RUNTIME_VERSION environment variable (the same signal the rest of the plugin uses to detect Databricks) and prepares one of two execution paths:

• Local / Claude Code (virtual environment). The phases run from the plugin's CLI. Setup runs scripts/bootstrap.sh, which creates a .venv, installs requirements.txt, and writes the resolved interpreter path to a .migration-venv marker file that the phase skills read. Optionally, a local (stdio) MCP server can be registered to drive the phases through MCP tools instead of the CLI.

• Databricks Genie Code (MCP server, no virtual environment). The phases run as a single flowx MCP tool hosted on a Databricks App. Setup runs app/deploy.sh, which stages a self-contained bundle, syncs it to /Workspace/Shared/mcp-flowx, and deploys the mcp-flowx app. You then grant app/data access and register the app under Genie Code Settings → MCP Servers. No venv is created on this path.

Run setup once before any other flowx skill, or again whenever the environment is missing.

Supported ADF Activity Types

Deterministic (16 types)