Apache Hamilton — portable & expressive
data transformation DAGs
Disclaimer
Apache Hamilton is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.
Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Apache Hamilton (incubating) is a lightweight Python library for directed acyclic graphs (DAGs) of data transformations. Your DAG is portable; it runs anywhere Python runs, whether it's a script, notebook, Airflow pipeline, FastAPI server, etc. Your DAG is expressive; Apache Hamilton has extensive features to define and modify the execution of a DAG (e.g., data validation, experiment tracking, remote execution).
To create a DAG, write regular Python functions that specify their dependencies with their parameters. As shown below, it results in readable code that can always be visualized. Apache Hamilton loads that definition and automatically builds the DAG for you!
Functions
B() and C() refer to function A via their parameters
Apache Hamilton brings modularity and structure to any Python application moving data: ETL pipelines, ML workflows, LLM applications, RAG systems, BI dashboards, and the Apache Hamilton UI allows you to automatically visualize, catalog, and monitor execution.
Apache Hamilton is great for DAGs, but if you need loops or conditional logic to create an LLM agent or a simulation, take a look at our sister library Burr 🤖 .
Installation
Apache Hamilton supports Python 3.8+. We include the optional visualization dependency to display our Apache Hamilton DAG. For visualizations, Graphviz needs to be installed on your system separately.
pip install "sf-hamilton[visualization]"
To use the Apache Hamilton UI, install the ui and sdk dependencies.
pip install "sf-hamilton[ui,sdk]"
To try Apache Hamilton in the browser, visit www.tryhamilton.dev