devtu-optimize-skills

Optimizing ToolUniverse Skills

Best practices for high-quality research skills with evidence grading and source attribution.

Tool Quality Standards

1. Error messages must be actionable — tell the user what went wrong AND what to do
2. Schema must match API reality — run python3 -m tooluniverse.cli run <Tool> '<json>' to verify
3. Coverage transparency — state what data is NOT included
4. Input validation before API calls — don't silently send invalid values
5. Cross-tool routing — name the correct tool when query is out-of-scope
6. No silent parameter dropping — if a parameter is ignored, say so

Core Principles (13 Patterns)

Full details: references/optimization-patterns.md

#	Pattern	Key Idea
1	Tool Interface Verification	get_tool_info() before first call; maintain corrections table
2	Foundation Data Layer	Query aggregator (Open Targets, PubChem) FIRST
3	Versioned Identifiers	Capture both ENSG00000123456 and .12 version
4	Disambiguation First	Resolve IDs, detect collisions, build negative filters
5	Report-Only Output	Narrative in report; methodology in appendix only if asked
6	Evidence Grading	T1 (mechanistic) → T2 (functional) → T3 (association) → T4 (mention)
7	Quantified Completeness	Numeric minimums per section (>=20 PPIs, top 10 tissues)
8	Mandatory Checklist	All sections exist, even if "Limited evidence"
9	Aggregated Data Gaps	Single section consolidating all missing data
10	Query Strategy	High-precision seeds → citation expansion → collision-filtered broad
11	Tool Failure Handling	Primary → Fallback 1 → Fallback 2 → document unavailable
12	Scalable Output	Narrative report + JSON/CSV bibliography
13	Synthesis Sections	Biological model + testable hypotheses, not just paper lists

Optimized Skill Workflow

Phase -1: Tool Verification (check params)
Phase  0: Foundation Data (aggregator query)
Phase  1: Disambiguation (IDs, collisions, baseline)
Phase  2: Specialized Queries (fill gaps)
Phase  3: Report Synthesis (evidence-graded narrative)

Testing Standards

Full details: references/testing-standards.md

Critical rule: NEVER write skill docs without testing all tool calls first.

• 30+ tests per skill, 100% pass rate
• All tests use real data (no placeholders)
• Phase + integration + edge case tests
• SOAP tools (IMGT, SAbDab, TheraSAbDab) need operation parameter
• Distinguish transient errors (retry) from real bugs (fix)
• API docs are often wrong — always verify with actual calls

Pattern 14: Reasoning Frameworks Over Tool Catalogs (CRITICAL)

Skills that just list tools ("call A, then B, then C") score 3-5/10 in usefulness tests. Skills that explain HOW to interpret and combine data score 7-9/10. Every skill MUST include:

14a. Interpretation Tables

Map raw API data to biological/clinical meaning. Don't just retrieve — explain.

Bad (tool catalog)	Good (reasoning framework)
"Get GO terms from MGnify"	GO terms → interpretation table: butyrate genes = barrier integrity, LPS genes = inflammation
"Get DepMap dependency scores"	Score < -0.5 = essential, but pan-essential = bad drug target (toxicity); selective = good target
"Get FAERS counts"	PRR > 5 = strong signal, but signal ≠ causation (channeling bias, notoriety bias)

14b. Synthesis Phases

Every multi-phase skill needs a final phase that answers "so what?" — not just collecting data:

• "What changed and why does it matter?"
• "Is this cause or consequence?"
• "What's the actionable recommendation?"

14c. Honest Limitations

If a tool API can't deliver what the skill promises, say so explicitly. Don't describe aspirational capabilities. Example: "DepMap_get_gene_dependencies returns gene metadata only, NOT per-cell-line CRISPR scores."

Pattern 15: Computational Procedures When Tools Can't Help

Some scientific analyses require computation, not just API queries. When no tool exists for a capability, embed a Python code procedure directly in the skill using packages available in ToolUniverse (pandas, scipy, numpy, statsmodels, biopython, networkx).

When to use computational procedures:

Gap	Procedure	Packages
API doesn't return needed data (e.g., DepMap scores)	Download CSV + pandas analysis	pandas
Statistical testing (differential abundance, enrichment)	scipy.stats + FDR correction	scipy, statsmodels
Sequence analysis (alignment, conservation)	Biopython SeqIO + pairwise alignment	biopython
Chemical similarity (analog search, fingerprints)	RDKit fingerprints + Tanimoto	rdkit (visualization extra)
Network analysis (hub genes, clustering)	NetworkX graph metrics	networkx
Scoring algorithms (ACMG classification, viability scores)	Custom Python functions	built-in
Dose feasibility (Cmax vs IC50 comparison)	Numerical comparison + PK data	pandas, numpy

Template for computational procedures in skills:

**Computational procedure: [Name]**
[When to use this: explain the gap it fills]

\`\`\`python
# [What this computes]
# Requires: [packages] (included in ToolUniverse dependencies)
import pandas as pd
from scipy.stats import mannwhitneyu

# Input: [describe expected input format]
# Output: [describe output]
# [Full working code with example data]
\`\`\`

[Interpretation guidance for the output]

Key rules for computational procedures:

1. Only use packages in ToolUniverse dependencies (pyproject.toml): pandas, scipy, numpy, networkx, requests, biopython (optional extra)
2. Include example data so the procedure is immediately testable
3. Explain the output — a code block without interpretation is useless
4. Note when external data download is needed (e.g., DepMap CSV from depmap.org)

Pattern 15b: Download-and-Process for Datasets Without REST APIs

Many critical scientific datasets have NO REST API but provide bulk download files. Skills should include concrete download-and-process instructions when this is the only path to essential data.

Template for download-and-process procedures:

**Step 1: Download data files**
- URL: [exact download page URL]
- Files needed: [filename] (~[size]) — [what it contains]
- Registration: [required/not required]
- Update frequency: [quarterly/annually/etc.]

**Step 2: Process with Python**
[Working code with pandas/scipy that loads the CSV and produces the analysis]

**Step 3: Interpret results**
[Table mapping output values to biological/clinical meaning]

**When files are not available**: [Fallback strategy using API tools]

Known download-only datasets that skills reference:

Dataset	Download URL	Files	Used By
DepMap CRISPR	depmap.org/portal/download/all/	CRISPRGeneEffect.csv (~300MB), Model.csv (~2MB)	functional-genomics, cell-line-profiling
TCGA clinical	portal.gdc.cancer.gov	Clinical + mutation TSVs	cancer-genomics-tcga
GTEx expression	gtexportal.org/home/downloads	GTEx_Analysis_v8_Annotations.csv	expression-data-retrieval
ClinGen gene-disease	clinicalgenome.org/docs/	gene_curation_list.tsv	variant-interpretation
gnomAD constraint	gnomad.broadinstitute.org/downloads	constraint metrics TSV	functional-genomics

Critical rule: Always include a fallback for when the download is unavailable (user may not have registration, file may be too large, etc.). The fallback should use available API tools even if they provide less complete data.

Common Anti-Patterns

Anti-Pattern	Fix
"Search Log" reports	Keep methodology internal; report findings only
Missing disambiguation	Add collision detection; build negative filters
No evidence grading	Apply T1-T4 grades; label each claim
Empty sections omitted	Include with "None identified"
No synthesis	Add biological model + hypotheses
Silent failures	Document in Data Gaps; implement fallbacks
Wrong tool parameters	Verify via get_tool_info() before calling
GTEx returns nothing	Try versioned ID ENSG*.version
No foundation layer	Query aggregator first
Untested tool calls	Test-driven: test script FIRST
Tool catalog without interpretation	Add interpretation tables explaining what data means
Aspirational capabilities	Be honest when APIs can't deliver; add computational procedure instead
Missing statistical analysis	Add scipy/pandas code procedure for computation the tools can't do

Quick Fixes for User Complaints

Complaint	Fix
"Report too short"	Add Phase 0 foundation + Phase 1 disambiguation
"Too much noise"	Add collision filtering
"Can't tell what's important"	Add T1-T4 evidence tiers
"Missing sections"	Add mandatory checklist with minimums
"Too long/unreadable"	Separate narrative from JSON
"Just a list of papers"	Add synthesis sections
"Tool failed, no data"	Add retry + fallback chains

Skill Template

---
name: [domain]-research
description: [What + when triggers]
---

# [Domain] Research

## Workflow
Phase -1: Tool Verification → Phase 0: Foundation → Phase 1: Disambiguate
→ Phase 2: Search → Phase 3: Report

## Phase -1: Tool Verification
[Parameter corrections table]

## Phase 0: Foundation Data
[Aggregator query]

## Phase 1: Disambiguation
[IDs, collisions, baseline]

## Phase 2: Specialized Queries
[Query strategy, fallbacks]

## Phase 3: Report Synthesis
[Evidence grading, mandatory sections]

## Output Files
- [topic]_report.md, [topic]_bibliography.json

## Quantified Minimums
[Numbers per section]

## Completeness Checklist
[Required sections with checkboxes]

Additional References

• Detailed patterns: references/optimization-patterns.md
• Testing standards: references/testing-standards.md
• Case studies (4 real fixes): references/case-studies.md
• Checklists (review + release): references/checklists.md