aidp-resume-migration

aidp-resume-migration — pick up where you left off

The migrator is designed to be resumable. Each successful Pass-1 dep migration is cached in two places:

• Module-level _migration_cache dict in job_migrate.py (in-memory, per-run).
• On-cluster os.path.exists() probe against the output path (persistent across runs).

Pass-2 task notebooks are similarly skipped if their final .ipynb already exists at <output-base>/<job>/notebooks/....

When to use

• A prior aidp-migrate-job run was killed (Ctrl-C, cluster restart, network drop, timeout).
• The user manually edited a specific dep / task notebook and wants to resume without re-doing the rest.
• The user added a new task to the manifest and wants to migrate only the new task.

Default behavior (resume is automatic)

python3 ${CLAUDE_PLUGIN_ROOT}/engine/scripts/job_migrate.py \
  --manifest reports/<MyJob>_manifest.json \
  --cluster <CLUSTER_ID> \
  --aidp-base <AIDP_BASE> \
  --datalake-ocid <DATALAKE_OCID> \
  --workspace-id <WORKSPACE_UUID> \
  --output-base <output-workspace-path> \
  --oci-profile <profile>

The default flag is --skip-migrated (ON). Every notebook already at the output path is silently skipped. So a plain re-invoke after a crash IS the resume — no special flag needed.

Resuming from a specific task

If the failure was task-specific and you want to start there (skipping all earlier tasks):

python3 ${CLAUDE_PLUGIN_ROOT}/engine/scripts/job_migrate.py \
  --manifest reports/<MyJob>_manifest.json \
  --start-task <substring_of_task_key>
  # + the rest of the standard args

--start-task is a substring match on task_key. Skip every task whose key sorts before the matched task.

Resuming a single task (skip everything else)

python3 ${CLAUDE_PLUGIN_ROOT}/engine/scripts/job_migrate.py \
  --manifest reports/<MyJob>_manifest.json \
  --only-tasks "<task_key_1>,<task_key_2>"
  # + the rest

Useful when you've manually fixed one dep and want to re-run only the tasks that consume it.

Force re-migration of an already-migrated notebook

You typically DON'T want this — it burns Claude tokens. But if you must:

python3 ${CLAUDE_PLUGIN_ROOT}/engine/scripts/job_migrate.py \
  --manifest reports/<MyJob>_manifest.json \
  --no-skip-migrated
  # + the rest

Warning: this overwrites any manual edits the user applied to the previously-migrated .ipynb at the output path. If the user has made manual fixes, REFUSE to use this flag without explicit "yes, overwrite my edits".

When the cache lies

The cache can be wrong in these scenarios — clear it manually if you see them:

Scenario	Fix
User edited the SOURCE Databricks notebook after the prior migration. The migrator still skips it (output already exists).	Delete the corresponding output .ipynb from AIDP workspace + re-run.
User modified the manifest to add a new dep, but a previously-migrated task still uses the OLD dep path.	Delete the output .ipynb for that task + re-run.
The output .ipynb was created but is empty / corrupt (cluster died mid-write).	Delete the bad file + re-run.

There's no --clear-cache flag — the cache IS the filesystem state of <output-base>.

Cluster state when resuming

Before invoking, verify the cluster is still Active (might have been auto-stopped for idle):

# (use the bootstrap skill's check)

If Stopped, start it via AIDP console. The first cell after resume will pay a connection-setup cost (~10-30s), then steady-state resumes.

How to know what's already done

# List all migrated notebooks for this job
oci os object list ... (or use AIDP workspace listing)
ls reports/<job-name>/notebooks/   # local mirror after a successful run

Or just tail the log and look for [SKIP] already migrated: lines on resume.

After this

• Same as aidp-migrate-job — read JOB_REPORT.md, route failed cells to aidp-fixup-cell, etc.