redhat-contribution-report

Red Hat Open Source Contribution Evaluation

Evaluate contributions to one or more open source projects by:

1. Building a contributor roster from LDAP org traversal, GitHub org members, or both
2. Writing the employee roster to a JSON file for sub-agent consumption
3. Centralizing GitHub username resolution via a dedicated sub-agent (LDAP mode)
4. Dispatching 5 parallel KPI sub-agents per project (one per KPI)
5. Generating a consolidated markdown report from checkpoint files
6. Running a final audit to validate report accuracy against checkpoints and live data

Quick Start

LDAP only (Red Hat manager email):

/redhat-contribution-report shuels@redhat.com kubeflow/kubeflow kserve/kserve

GitHub org only:

/redhat-contribution-report kserve kserve/kserve kubeflow/kubeflow

Both (LDAP + GitHub org):

/redhat-contribution-report shuels@redhat.com kserve kubeflow/kubeflow kserve/kserve

Input Format

$ARGUMENTS = [manager_email] [github_org] <project1> [project2] [project3] ...

Arguments are classified by pattern — no flags needed:

• Contains @ → manager_email (LDAP org leader email)
• Contains / → project (GitHub owner/repo)
• Otherwise → github_org (validated via gh api orgs/{name})

At least one of manager_email or github_org must be present, plus at least one project.

• manager_email (optional): Email address of the org leader to scope the LDAP search (e.g., shuels@redhat.com)
• github_org (optional): GitHub organization name (e.g., kserve). Members become the contributor roster.
• projects (required, one or more): GitHub repositories in owner/repo format. If only a project name is given (e.g., kubeflow), attempt to resolve it via gh search repos

Three modes:

Mode	Arguments	Roster Source
LDAP only	email project...	Red Hat LDAP org tree
GitHub org only	org project...	GitHub org member list
Both	email org project...	LDAP + GitHub org (merged, deduplicated)

Evaluation Workflow

Execute these phases sequentially. Do not skip phases.

Phase 1: Input Parsing & Prerequisites

1. Parse $ARGUMENTS by classifying each argument:

   • Contains @ → manager_email
   • Contains / → add to projects list
   • Otherwise → candidate github_org (validate below)

   At least one of manager_email or github_org must be present, plus at least one project.

2. Validate candidate github_org (if present):

   gh api orgs/{github_org} --jq '.login'
   

   If 404, this is not a valid org — treat it as a bare project name and attempt repo search instead.

3. If any project lacks an owner/ prefix, resolve it:

   gh search repos "{project_name}" --limit 5 --json fullName,description,stargazersCount
   

   Select the most likely match and confirm with the user.

4. Determine the mode and print a summary:

   • manager_email only → LDAP only mode
   • github_org only → GitHub org only mode
   • Both → LDAP + GitHub org mode

   Print: Mode: {mode} | Manager: {email or "none"} | Org: {org or "none"} | Projects: {list}

5. Run prerequisite checks:

   GitHub CLI authentication (always required):

   gh auth status
   

   If not authenticated, stop and tell the user to run gh auth login.

   Kerberos ticket (only if manager_email is present):

   klist
   

   If no valid TGT, stop and tell the user to run kinit.

   LDAP connectivity (only if manager_email is present):

   ldapsearch -LLL -Y GSSAPI -H ldap://ldap.corp.redhat.com -b ou=users,dc=redhat,dc=com '(mail=MANAGER_EMAIL)' uid cn 2>&1 | head -5
   

   If this fails, warn the user. Refer to references/LDAP-GUIDE.md for the fallback strategy.

   Validate each project exists:

   gh repo view OWNER/REPO --json name,owner,url
   

   If a project is not found, remove it from the list and warn the user.

   Create output directory:

   mkdir -p reports/
   

Phase 2: LDAP Organization Enumeration

Conditional: Skip this phase entirely if no manager_email was provided. When skipped, create the initial roster JSON:

mkdir -p reports/tmp

Then use the Write tool to save an empty roster to reports/tmp/employee-roster.json:

{
  "generated_at": "YYYY-MM-DDTHH:MM:SSZ",
  "manager": null,
  "roster_source": "github-org",
  "total_employees": 0,
  "resolved_count": 0,
  "resolution_coverage_pct": 0.0,
  "employees": []
}

Then skip directly to Phase 2.5.

---

When manager_email is present, proceed with LDAP enumeration:

Refer to references/LDAP-GUIDE.md for detailed LDAP query patterns and attribute documentation.

All LDAP queries MUST use GSSAPI authentication (-Y GSSAPI). Never use simple auth (-x).

1. Find the manager's LDAP entry:

   ldapsearch -LLL -Y GSSAPI -H ldap://ldap.corp.redhat.com \
     -b ou=users,dc=redhat,dc=com \
     '(mail=MANAGER_EMAIL)' \
     uid cn mail title rhatSocialURL
   

   Record the manager's uid.

2. Discover available GitHub-related attributes: Run a broad attribute query on the manager's entry to discover any GitHub-specific fields:

   ldapsearch -LLL -Y GSSAPI -H ldap://ldap.corp.redhat.com \
     -b ou=users,dc=redhat,dc=com \
     '(mail=MANAGER_EMAIL)' '*' '+' 2>/dev/null | grep -i -E 'github|social|git'
   

   Note any additional attributes found beyond rhatSocialURL.

3. Recursively find all reports (BFS traversal):

   Initialize a queue with the manager's uid. For each uid in the queue:

   ldapsearch -LLL -Y GSSAPI -H ldap://ldap.corp.redhat.com \
     -b ou=users,dc=redhat,dc=com \
     '(manager=uid=CURRENT_UID,ou=users,dc=redhat,dc=com)' \
     uid cn mail title rhatSocialURL
   

   • Deduplicate by uid — if an employee is already in the roster, skip (avoids duplicates from dotted-line reporting or circular references)
   • Add each new result to the employee roster with a depth field tracking the BFS level (manager = 0, direct reports = 1, etc.)
   • Add each result's uid to the queue for further traversal
   • Continue until the queue is empty (no more reports found at any level)

4. Build the employee roster: For each employee, create an entry with:

   • name (from cn)
   • uid (from uid)
   • email (from mail)
   • title (from title)
   • github_username (parsed from rhatSocialURL or other discovered attribute, or null)
   • github_resolution_method (ldap if resolved, null if not)

5. Report roster statistics:

   • Total employees found
   • Total with GitHub usernames resolved
   • Coverage percentage
   • If coverage < 70%, warn that metrics may undercount Red Hat involvement

   If the org exceeds 500 employees, warn the user that this is a very large scope and ask if they want to continue or narrow the search.

6. Write the employee roster to a JSON file for sub-agent consumption:

   mkdir -p reports/tmp
   

   Then use the Write tool to save the roster to reports/tmp/employee-roster.json with this schema:

   {
     "generated_at": "YYYY-MM-DDTHH:MM:SSZ",
     "manager": {"name": "...", "uid": "...", "email": "..."},
     "roster_source": "ldap",
     "total_employees": 125,
     "resolved_count": 40,
     "resolution_coverage_pct": 32.0,
     "employees": [
       {
         "name": "Jane Doe",
         "uid": "jdoe",
         "email": "jdoe@redhat.com",
         "title": "Senior Software Engineer",
         "github_username": "janedoe",
         "github_resolution_method": "ldap",
         "github_resolution_tier": 1,
         "depth": 2,
         "source": "ldap"
       }
     ]
   }
   

   • Set github_resolution_tier to 1 for LDAP-resolved usernames, null for unresolved
   • Set github_username to null for employees without LDAP resolution
   • Set roster_source to "ldap" (will become "both" if Phase 2.5 merges GitHub org members)
   • Set source to "ldap" on each employee entry
   • This file is the single source of truth for the roster — sub-agents reference it by path and never receive the roster inline

Phase 2.5: GitHub Organization Roster

Conditional: Skip this phase if no github_org was provided.

Run the GitHub org roster script to fetch org members and build (or merge into) the roster:

python3 {assets_dir}/github-org-roster.py \
  --org {github_org} \
  --output reports/tmp/employee-roster.json \
  {--merge if Phase 2 already wrote the file (i.e., manager_email was present)}

Use --merge when both manager_email and github_org are present (LDAP + GitHub org mode). The script will:

• Fetch all org members via gh api orgs/{org}/members --paginate
• Enrich each member with profile data via gh api users/{login}
• In merge mode: match by github_username (case-insensitive), mark matches as source: "both", append new members as source: "github-org", keep LDAP-only entries as source: "ldap"
• In standalone mode: create a fresh roster with roster_source: "github-org", manager: null
• Update total_employees, resolved_count, resolution_coverage_pct

Report the roster size and source mode to the user after this phase completes.

Phase 3: GitHub Username Resolution Summary

Review the roster JSON (reports/tmp/employee-roster.json):

• Employees with GitHub usernames from LDAP are marked as Tier 1 (High confidence) (github_resolution_tier: 1)
• Employees from GitHub org membership also have Tier 1 (github_resolution_tier: 1) with github_resolution_method: "github-org"
• Employees without GitHub usernames (github_username: null) will be resolved by the centralized Username Resolution Agent in Phase 3.5

Report the current resolution state to the user, adapted for the active mode:

• LDAP only: Total employees, resolved count, coverage %, note Phase 3.5 resolution
• GitHub org only: Total members, 100% username coverage, no Phase 3.5 needed
• Both: Total employees (LDAP + org), overlap count, combined coverage %

Phase 3.5: Centralized Username Resolution

Conditional: Skip this phase if all employees already have GitHub usernames resolved (e.g., GitHub-org-only mode where 100% have usernames). Check the roster: if zero employees have github_username: null, skip to Phase 4.

When active, launch a single dedicated sub-agent to resolve GitHub usernames for employees who lack resolved usernames. In combined mode, only LDAP-sourced employees without usernames need resolution. This runs once before KPI evaluation, so all KPI agents benefit from the same resolved roster.

Read the Username Resolution Agent prompt template from references/RESEARCH-PROMPTS.md.

Prepare the prompt by substituting:

• {roster_path} with reports/tmp/employee-roster.json
• {project_list} with a comma-separated list of all target projects (e.g., kubeflow/kubeflow, kserve/kserve)
• {workdir} with reports/tmp
• {assets_dir} with the absolute path to redhat-contribution-report/skills/redhat-contribution-report/assets

Launch the sub-agent using Task with subagent_type: general-purpose and max_turns: 8.

Wait for this agent to complete before proceeding to Phase 4. The agent runs a batch Python script that:

1. Searches git history across ALL target projects for @redhat.com emails
2. Confirms matches via gh search commits --author-email
3. For remaining unresolved (if <20), tries gh search users with strict acceptance criteria
4. Updates reports/tmp/employee-roster.json in place with resolutions
5. The agent then writes a resolution log to reports/tmp/username-resolutions.md

After the agent completes, report the updated resolution coverage to the user.

Phase 4: Parallel Per-Project Research

Read the 5 KPI prompt templates from references/RESEARCH-PROMPTS.md.

Read the scoring rubric from assets/scoring-rubric.json.

Create working directories for each project's intermediate files:

mkdir -p reports/tmp/{owner}-{repo}/

Run this for every project before dispatching sub-agents. These directories hold raw API output and checkpoint files.

Compute the evaluation window cutoff date (6 months ago from today):

cutoff_date=$(date -d '6 months ago' +%Y-%m-%d)

For each KPI prompt template, prepare the prompt by substituting:

• {owner} and {repo} with the project's owner and repository name
• {workdir} with the working directory path: reports/tmp/{owner}-{repo}
• {cutoff_date} with the computed 6-month-ago date in YYYY-MM-DD format
• {roster_path} with reports/tmp/employee-roster.json
• {assets_dir} with the absolute path to redhat-contribution-report/skills/redhat-contribution-report/assets

Do NOT substitute {employee_roster} or embed the roster inline. Sub-agents access the roster file via {roster_path} inside python3 scripts. The roster is never loaded into agent conversation context.

Launch 5 Task sub-agents per project, ALL IN PARALLEL in a single message. Use subagent_type: general-purpose and max_turns: 8 for all KPIs. For N projects, this means 5N Task calls in a single message.

Agent	KPI	Focus
KPI 1	PR/Commit Contributions	PRs, commits, code contributions authored or co-authored by roster employees
KPI 2	Release Management	Release managers who are roster employees
KPI 3	Maintainer/Reviewer/Approver Roles	Roster employees in OWNERS, CODEOWNERS, MAINTAINERS, or similar governance files
KPI 4	Roadmap Influence	Enhancement proposals, roadmap features, or design docs led by roster employees
KPI 5	Leadership Roles	TAC, steering committee, advisory board, or other governance body positions held by roster employees

All KPI agents use max_turns: 8. Heavy computation (workflow detection, PR verification, governance file scanning) is handled by standalone Python scripts in {assets_dir}, keeping agent turns minimal.

Each agent writes its results to a checkpoint file in {workdir}/ and returns only a 1-line status message. This keeps orchestrator context minimal.

Refer to references/DATA-SOURCES.md for the specific gh CLI commands each sub-agent should use.

Phase 5: Result Collection & Merge

Collect the output from all sub-agents. There are 5 sub-agents per project (5N total for N projects), each returning a 1-line status message. Detailed results are in checkpoint files.

Step 1: Read the updated roster from reports/tmp/employee-roster.json (updated by the Phase 3.5 Username Resolution Agent).

Step 2: Read the username resolution log from reports/tmp/username-resolutions.md.

Step 3: For each project, read the 5 KPI checkpoint files from reports/tmp/{owner}-{repo}/:

• kpi1-pr-contributions.md — KPI 1 results
• kpi2-release-management.md — KPI 2 results
• kpi3-maintainership.md — KPI 3 results
• kpi4-roadmap-influence.md — KPI 4 results
• kpi5-leadership.md — KPI 5 results

Step 4: Handle missing checkpoints. If a checkpoint file does not exist (agent failed or was rate-limited), mark that KPI as "Not evaluated" with score 1 and confidence "Not Found". Note which KPIs were missing in the Data Quality section.

Step 5: Build per-project Employee Contribution Maps using python3 to scan checkpoint files:

python3 -c "
import json, re, os
roster = json.load(open('reports/tmp/employee-roster.json'))
workdir = 'reports/tmp/{owner}-{repo}'
kpi_files = ['kpi1-pr-contributions.md','kpi2-release-management.md','kpi3-maintainership.md','kpi4-roadmap-influence.md','kpi5-leadership.md']
gh_users = {e['github_username'].lower(): e['name'] for e in roster['employees'] if e.get('github_username')}
for i, f in enumerate(kpi_files, 1):
    path = os.path.join(workdir, f)
    if os.path.exists(path):
        content = open(path).read()
        found = [u for u in gh_users if u in content.lower()]
        for u in found:
            print(f'{gh_users[u]} | @{u} | KPI {i}')
    else:
        print(f'KPI {i}: checkpoint missing')
"

§5.0 KPI 1 Spot-Check (Non-Standard Workflows)

For each project, check if a kpi1-metadata.json file exists in the working directory:

python3 -c "
import json, os, subprocess

workdir = 'reports/tmp/{owner}-{repo}'
meta_path = os.path.join(workdir, 'kpi1-metadata.json')
if not os.path.exists(meta_path):
    print('No kpi1-metadata.json found — skipping spot-check')
    exit(0)

metadata = json.load(open(meta_path))
print(f'Workflow type: {metadata[\"workflow_type\"]}')
print(f'RH verified total (metadata): {metadata[\"rh_verified_total\"]}')

# Read the checkpoint file to extract the reported RH PR count
checkpoint_path = os.path.join(workdir, 'kpi1-pr-contributions.md')
if os.path.exists(checkpoint_path):
    content = open(checkpoint_path).read()
    print(f'Checkpoint file exists: yes')
else:
    print('WARNING: kpi1-pr-contributions.md checkpoint missing')

if metadata['workflow_type'] == 'non-standard':
    print(f'Non-standard workflow detected — cross-validation active')
    print(f'  Merged: {metadata[\"rh_merged_count\"]}, Landed: {metadata[\"rh_landed_count\"]}')
    print(f'  Total verified: {metadata[\"rh_verified_total\"]} ({metadata[\"rh_pct\"]}%)')
"

If the workflow type is non-standard, spot-check 2-3 individual PRs from the top RH contributor by verifying their merge status:

gh pr view {pr_number} --repo {owner}/{repo} --json state,mergedAt,closedAt,number

If the checkpoint file's reported RH PR count diverges from metadata.rh_verified_total, flag it for manual review and add a note to the Data Quality section of the report.

§5.1 KPI Result Aggregation

• Keep per-project KPI results separate — do not average or merge scores across projects.
• Verify that each sub-agent's assigned score matches the rubric thresholds in assets/scoring-rubric.json against the sub-agent's own reported data. If a score appears inconsistent with the data (e.g., score of 4 but data shows < 10% PR contribution), adjust to match the rubric and note the correction.

§5.2 Coverage Verification

After collecting all results:

• Read the final resolution coverage from reports/tmp/employee-roster.json (resolution_coverage_pct field).
• If coverage is below 70%, ensure the undercount caveat appears in the final report.
• Note the resolution coverage and method breakdown in the Data Quality section.

Phase 6: Report Generation

Read the report template from references/REPORT-TEMPLATE.md.

Generate the final report by:

1. Computing today's date:

   date +%Y-%m-%d
   

2. Assembling the report following the template structure:

   • Executive Summary with overall scores table
   • Employee Roster with coverage statistics and unresolved employees table
   • Per-Project Sections — one for each project, each containing:
     • Employee contribution table (name, GitHub username, roles, KPIs)
     • All 5 KPI sections with scores, findings, evidence, and confidence
     • Project score summary table
   • Cross-Project Comparison table and cross-project employee presence
   • Data Quality & Methodology notes
   • Sources list

3. Applying scores using the rubric from assets/scoring-rubric.json

4. Writing the report:

   # File path: reports/YYYY-MM-DD-redhat-contribution-eval.md
   

   Use the Write tool to save the report.

Phase 7: Final Audit

Run a final validation of the generated report against checkpoint files, the scoring rubric, and live GitHub data.

1. Read the Auditor Agent prompt template from references/RESEARCH-PROMPTS.md.

2. Prepare the prompt by substituting:

   • {report_path} with the report file path from Phase 6 (e.g., reports/YYYY-MM-DD-redhat-contribution-eval.md)
   • {roster_path} with reports/tmp/employee-roster.json
   • {workdir} with reports/tmp
   • {assets_dir} with the absolute path to redhat-contribution-report/skills/redhat-contribution-report/assets
   • {projects} with a comma-separated list of all target projects (e.g., kubeflow/kubeflow,kserve/kserve)

3. Launch a single Task sub-agent with subagent_type: general-purpose and max_turns: 10.

4. Wait for the agent to complete. Report the audit results to the user:

   • If PASS: note that all checks passed
   • If PASS WITH WARNINGS: list the warnings
   • If DISCREPANCIES FOUND: list each discrepancy with expected vs. actual values

Cleanup

1. Clean up intermediate files:

   rm -rf reports/tmp/
   

2. Inform the user of the report location and summarize key findings.

Error Handling

• Kerberos/LDAP failure: Warn user. Offer email-only fallback (search git history for @redhat.com). All metrics marked reduced confidence. No org scoping possible. Only applies when manager_email is present.
• GitHub org not found (404): Treat the argument as a bare project name and attempt repo search.
• GitHub org access denied (403): Warn user that private org membership requires org membership or admin token.
• gh rate limited (403): Reduce sample sizes by 50%. Note in Data Quality section. If still limited, report partial data.
• Project not found: Skip the project. Note in the report.
• Governance files not found: Mark KPIs 3 and 5 as low confidence. Use web search as fallback.
• Org exceeds 500 employees (LDAP or GitHub): Warn user. Suggest narrowing scope. Proceed only with confirmation.
• Coverage below 70%: Add warning banner to all contribution metrics in the report.

Reference Files

• references/LDAP-GUIDE.md — LDAP connection, attributes, traversal algorithm, and fallback strategies
• references/DATA-SOURCES.md — All gh CLI commands organized by KPI with parsing guidance
• references/REPORT-TEMPLATE.md — Complete markdown template for the output report
• references/RESEARCH-PROMPTS.md — Sub-agent prompt template with variable substitution instructions
• assets/scoring-rubric.json — Machine-readable scoring thresholds for all 5 KPIs (1-5 scale)