infra-costs

infra costs

The costs flow (cron 0 8 * * *, daily 08:00 UTC) collects provider billing, writes an io.zzstoatzz.cost.snapshot PDS record at rkey=YYYY-MM-DD (upserted, so re-runs are idempotent), and surfaces it at hub.waow.tech.

reading a snapshot

Use the pdsx MCP tools. list_records and get_record cover this with structured output and no handle/DID/PDS resolution dance:

list_records(repo="zzstoatzz.io", collection="io.zzstoatzz.cost.snapshot", limit=30)
get_record(repo="zzstoatzz.io", collection="io.zzstoatzz.cost.snapshot", rkey="2026-07-24")

pdsx MCP does full record CRUD — reads and create_record/update_record/ delete_record. It is not read-only. Only blob upload, batch JSONL operations, and permissioned spaces need the pdsx CLI.

Raw HTTP only as a fallback when the MCP isn't available. Records are public, so no auth is needed:

DID=$(curl -s "https://api.bsky.app/xrpc/com.atproto.identity.resolveHandle?handle=zzstoatzz.io" | jq -r .did)
curl -s "https://pds.zzstoatzz.io/xrpc/com.atproto.repo.listRecords?repo=$DID&collection=io.zzstoatzz.cost.snapshot&limit=30" \
  | jq -r '.records | sort_by(.uri) | .[] | [(.uri|split("/")|last), (.value.total/100)] | @tsv'

Record fields are total, currency, byProvider, byProject, lineItems, periodStart, periodEnd, generatedAt. There is no items field — jq '.value.items' silently yields nothing and looks like zero cost.

comparing spend across dates — read this first

Snapshots are a measurement, and the measurement has changed. Diff lineItems by service, not byProvider/byProject totals, or a coverage change will masquerade as a spend change.

Known discontinuity: on 2026-07-20 the Fly connector changed shape (7a212e8). Before, one item per app (~26 items) with machines and volumes bundled and snapshots not billed at all. After, a app:compute / app:volumes / app:snapshots split (~55 items) with snapshot storage newly counted — roughly $11/mo of previously invisible cost appeared. Any before/after window spanning that date understates real savings.

Worked example: deleting the typeahead builder removed a $15/mo volume, yet the typeahead project total moved only ~$2.40, because newly-counted snapshots absorbed the difference.

project attribution

There is no provider-side project tag. Attribution is by resource name pattern, longest match wins, in packages/mps/src/mps/costs/projects.py (_RESOURCE_PATTERNS). So relay.waow → relays while relay-api-staging → plyr.fm. Adding a resource whose name doesn't match a pattern lands it in misc or unattributed — add the pattern and a project_for assertion in tests/test_costs.py.

fail-closed is the invariant

A connector that raises is collected into failed_providers, and the flow then refuses to publish rather than shipping a snapshot with a silent hole (flows/costs.py). Never let a connector swallow an error and return partial results — an omitted resource must never be counted as $0.

Hetzner specifics: Cloud API tokens are project-scoped (no account-wide token), so the flow loads a hetzner-tokens Secret block of {label: token} and merges all projects, deduping by server id. There are currently two project tokens covering four Cloud servers — relay-eval, relay, zlay, prefect-server — and that is the entire Hetzner footprint.

Hetzner Robot (the legacy dedicated/auction product) is deliberately out of scope: there are no dedicated servers on this account. hetzner-edge on the tailnet is not a separate machine — it is prefect-server under a different tailnet hostname, same public IP. Don't build a Robot connector without first confirming a dedicated server actually exists; one was written and reverted for exactly this reason.

Fly cleanup

fly volumes destroy <id> is broken without -a — its app-lookup GraphQL query fails with Field 'node' doesn't exist on type 'Queries'. This is Fly-side, not a stale client. Always pass the app:

fly volumes destroy -y -a <app> <vol_id> [<vol_id>...]

fly volumes snapshots list hits the same broken query with no -a escape. Use the Machines REST API instead:

TOK=$(fly auth token)
curl -s -H "Authorization: Bearer $TOK" \
  "https://api.machines.dev/v1/apps/<app>/volumes/<vol_id>/snapshots" | jq .
curl -s -H "Authorization: Bearer $TOK" \
  "https://api.machines.dev/v1/apps/<app>/volumes/<vol_id>" \
  | jq '{size_gb,snapshot_retention,auto_backup_enabled,blocks,blocks_free}'

Destroyed volumes linger in a pending-destroy state and still appear via the REST API after fly volumes list stops showing them. The connector ignores them (3efa153), so they are not double-counted.

snapshots are not cruft

Every volume defaults to auto_backup_enabled: true, snapshot_retention: 5 — a daily snapshot at ~00:02 UTC with a rolling 5-day window. The count is a bounded steady state, not accumulation, and snapshots are never read except on restore. Deleting them individually is pointless; they regenerate. The only real lever is retention:

fly volumes update <vol_id> --snapshot-retention <days> -a <app>

Retention changes apply to future aging-out, so cost falls over the following days rather than immediately.

Judge retention by whether the data is authoritative or derived. typeahead-search holds a derived index — typeahead-index rebuilds it every 3 days and publishes to R2, explicitly "no ingress, no SLA" — so it is set to 2 days. pds-zzstoatzz-io is authoritative PDS data and stays at 5. Do not lower retention account-wide.

related

Running the flow by hand is [[adhoc-runs]]; the Secret blocks it needs are injected per [[deployments]].