log-fetcher

Log Fetcher

Fetch available logs for further investigations. Currently

 [Python]  Fetch logs via SSH

Quick Start

cd /path/to/incident-investigator
pip install -r requirements.txt

python -m scripts.cli new --incident-id "inc-001"
python -m scripts.cli run --incident-id "inc-001" --all
python -m scripts.cli status --incident-id "inc-001"

Prerequisites

We assume that the user has correctly set up the SSH profile to the Ansible server.

---

Fetch Ansible Logs via SSH

Input: User provides incident context (time window, job IDs, or investigation scope) Output: Raw log files in local directory (default: ~/etl-logs) Scripts: scripts/fetch_logs_ssh.py or scripts/fetch_logs_by_job.py

Prerequisites

Ensure SSH access is configured:

ssh ${REMOTE_HOST}  # Should connect without password prompt

Option A: Fetch by Time/Mode (fetch_logs_ssh.py)

Use this when you want to fetch recent logs or a range of logs based on time.

1. Determine the scope:

   • For recent failures: Use --mode processed --order desc --limit 10-50
   • For specific time window: Use --start-time and/or --end-time to filter by creation date
   • For all recent activity: Use --mode all (includes both processed and ignored)

2. Run the fetch script:

   # Fetch recent logs with limit
   python -m scripts.fetch_logs_ssh \
     --mode processed \
     --order desc \
     --limit 20 \
     --local-dir .incidents/<incident-id>/raw_logs
   
   # Fetch logs within a specific time range
   python -m scripts.fetch_logs_ssh \
     --mode processed \
     --start-time "2025-12-09 08:00:00" \
     --end-time "2025-12-10 17:00:00" \
     --local-dir .incidents/<incident-id>/raw_logs
   
   # Fetch logs from a specific day
   python -m scripts.fetch_logs_ssh \
     --mode all \
     --start-time "2025-12-10" \
     --end-time "2025-12-10" \
     --local-dir .incidents/<incident-id>/raw_logs
   

3. Parameters:

   • --mode: Select log type (processed, ignored, or all)
   • --order: Sort by time (desc = newest first, asc = oldest first)
   • --limit: Number of files to fetch (omit for all files)
   • --start-time: Filter logs created on or after this time (format: YYYY-MM-DD [HH:MM[:SS]])
   • --end-time: Filter logs created on or before this time (format: YYYY-MM-DD [HH:MM[:SS]])
   • --local-dir: Where to store logs (default: ~/etl-logs)

4. Time Filtering Examples:

   • Full timestamp: "2025-12-10 14:30:45"
   • Minute precision: "2025-12-10 14:30"
   • Day only: "2025-12-10"
   • Combine with other filters: --start-time "2025-12-10 00:00" --limit 10 --order desc

Option B: Fetch by Job Number (fetch_logs_by_job.py)

Use this when you know specific job numbers to investigate.

1. Run the fetch script with job numbers:

   python -m scripts.fetch_logs_by_job \
     job_1234567 job_1234568 job_1234569 \
     --local-dir .incidents/<incident-id>/raw_logs
   

   Or without the 'job_' prefix:

   python -m scripts.fetch_logs_by_job \
     1234567 1234568 1234569 \
     --local-dir .incidents/<incident-id>/raw_logs
   

2. Parameters:

   • job_numbers: One or more job identifiers (with or without 'job_' prefix)
   • --local-dir: Where to store logs (default: ~/etl-logs)

3. What it fetches:

   • All transform statuses for specified jobs (*.transform-processed, *.transform-ignored, etc.)
   • Automatically finds matching files on the remote server

Verify the Fetch

• Check that files were transferred successfully
• Note the job IDs from filenames (e.g., job_1234567.json.gz.transform-processed)
• Confirm the time range matches the incident window (for Option A)

Example Output (Option A)

[INFO] Remote host: ${REMOTE_HOST}
[INFO] Remote dir : ${REMOTE_DIR}
[INFO] Local dir  : ${LOCAL_DIR}
[INFO] Mode       : processed
[INFO] Order      : desc
[INFO] Limit      : 10
[INFO] Start time : 2025-12-09 08:00:00
[INFO] End time   : 2025-12-10 17:00:00
[INFO] Remote cmd : cd ${REMOTE_DIR} && find . -maxdepth 1 -type f -name '*.transform-processed' -newermt '2025-12-09 08:00:00' ! -newermt '2025-12-10 17:00:00' -printf '%T@ %f\n' | sort -rn | cut -d' ' -f2- | head -n 10
[INFO] Running rsync: rsync -avz --progress --files-from=- ${REMOTE_HOST}:${REMOTE_DIR}/ ...
Transfer starting: 10 files
job_1234567.json.gz.transform-processed
         456170 100%    2.99MB/s   00:00:00 (xfer#1, to-check=0/10)
...
sent 637 bytes  received 1457000 bytes  14576370000 bytes/sec
total size is 1470251  speedup is 1.01

Example Output (Option B)

[INFO] Remote host: ${REMOTE_HOST}
[INFO] Remote dir : ${REMOTE_DIR}
[INFO] Local dir  : ${LOCAL_DIR}
[INFO] Job numbers: job_1234567, job_1234568
[INFO] Finding files matching job patterns...
[INFO] Found 2 file(s):
  - job_1234567.json.gz.transform-processed
  - job_1234568.json.gz.transform-ignored
[INFO] Running rsync...
job_1234567.json.gz.transform-processed
         456170 100%    2.99MB/s   00:00:00
job_1234568.json.gz.transform-ignored
         321450 100%    2.45MB/s   00:00:00
[SUCCESS] Files transferred to ${LOCAL_DIR}

Next Step

Once logs are fetched, proceed to extract errors:

python -m scripts.cli run --incident-id "<id>" --step 2a