mps

personal data pipeline and intelligence layer. digests github, tangled.org, and bluesky activity, scores items, generates LLM-curated briefings, and maintains phi's long-term memory. self-hosted across two machines joined by Tailscale: a hetzner VM (k3s) runs the prefect control plane and serves the public edge, while all flow execution runs on a home Linux box that polls outbound — the home box does the compute, the VM serves the bytes.

hub · grafana

                        where it runs
  ─────────────────────────────────────────────
   home box ("heavypad")                       hetzner VM (k3s, EU)
   i9 · 24c · 64GB · 1TB SSD + 2TB NVMe        the public edge + control plane
   ──────────────────────────────────         ────────────────────────────────
   • home-pool worker runs ALL flow            • prefect server (control plane)
     execution — outbound poll, no ingress     • hub.waow.tech + grafana (public, TLS)
   • owns the analytics DuckDB +               • serves bytes, not compute
     llm-spend.jsonl (the source of truth)

   home box ──► rsync hub.duckdb + llm-spend.jsonl every 3m, over tailscale ──► VM
   so the edge serves fresh data WITHOUT shipping heavy pages from home (a page
   from home behind the EU edge meant a US→EU→US round trip; the data is ~4MB).

   principle: home = compute & state factory; VM = public edge & bandwidth.
   relays, the PDS, and user-facing products stay cloud.

                        data sources
  ─────────────────────────────────────────────
  github API        ──┐
  tangled PDS       ──┤
  bluesky likes     ──┼──► ingest (hourly) ──► DuckDB
  phi memory (tpuf) ──┘
                                                  │
                                                  ▼
                                          classify-emails [on ingest ✓]
                                                  │
                                                  ▼
                                          transform (dbt)
                                          [on classify-emails ✓]
                                                  │
                        ┌─────────────────────────┼──────────┐
                        ▼                         ▼          ▼
                      brief                    phi-memory-synthesis    hub UI
                  [on transform ✓]         [on transform ✓]
                        │                         │
                        ▼                         ▼
                  briefing.json              TurboPuffer
                                             (phi-users-*)

                        phi identity flows
  ─────────────────────────────────────────────
  phi-tag-maintenance (daily 8am CT) ──► TurboPuffer
          │
          ▼
  curate  ─────────────────► Semble API
  phi-atlas (daily 8am CT) ─► PDS atlas blob
          │
          ▼
  docket  ─────────────────► PDS docket blob

                        publication flows
  ─────────────────────────────────────────────
  leaflet-atlas   (every 6h)  ──► Cloudflare Pages
  typeahead-index (every 3d)  ──► R2  (prefix-index snapshot, built at home)
  bisk-snapshot   (every 10m) ──► R2  (bisk.social standings snapshot)
  pds-records     (ad hoc)    ──► PDS record maintenance

                        cost tracking + health
  ─────────────────────────────────────────────
  costs       (daily 08:00 UTC) ──► PDS cost snapshot ──► hub.waow.tech
  diagnostics (every 5m)        ──► liveness canary (prints host/python/pid)

see docs/hub.md for the full pipeline breakdown.

cp .env.example .env
# fill in: HCLOUD_TOKEN, POSTGRES_PASSWORD, AUTH_STRING, DOMAIN, LETSENCRYPT_EMAIL
uv sync                # install workspace (mps + root)

just init              # terraform init
just infra             # create the VM
just kubeconfig        # wait for k3s, fetch kubeconfig
just deploy            # cert-manager, prefect server (zig chart), postgres+redis, monitoring, dashboards
just storage           # create analytics hostPath + results PVC, apply kubernetes-pool base job template