Tool Evaluator Skill

Tool Evaluator Skill

Sub-agent skill for evaluating tools during /design-ops:setup. Coordinates MCP availability checks and API capability evaluation.

Purpose

When a user selects tools in the setup flow, this skill:

1. Checks if the tool is already installed and connected
2. Invokes the mcp-discovery skill for unknown tools
3. Returns a unified result with connection type and status

Trigger

Invoked by /design-ops:setup during tool evaluation phase. Should run as a background sub-agent.

---

Input

tools:
  - name: "notion"
    pillar: "operations"
  - name: "figma"
    pillar: "design"
  - name: "substack"
    pillar: "analytics"

Output

Returns a capability matrix for each tool, including discovery metadata:

results:
  - tool: notion
    connection_type: mcp
    mcp_name: "notion"
    status: connected    # connected | available | api_only | unavailable | skipped

    # Discovery metadata (from mcp-discovery skill)
    mcp_source: official          # official | vendor | community | none
    mcp_package: "@notionhq/notion-mcp-server"
    mcp_confidence: high          # high | medium | low

    # Capabilities
    capabilities:
      data_types: [pages, databases, tasks, comments]
      reporting:
        daily: [recent_pages, task_counts, recent_comments]
        weekly: [page_activity, task_completion]
        monthly: [content_growth]

    setup_required: false
    warning: null

  - tool: figma
    connection_type: api
    status: api_only

    # Discovery metadata
    mcp_source: official
    mcp_package: null    # MCP exists but is code-focused
    mcp_confidence: low
    api_docs_url: "https://www.figma.com/developers/api"

    capabilities:
      data_types: [files, versions, comments, users]
      reporting:
        daily: [files_edited, active_users]
        weekly: [design_versions, comment_activity]
        monthly: [project_progress]

    setup_required: true
    setup_instructions: "Generate API token at figma.com/developers"
    note: "Official MCP is code-focused. Use API for reporting."

  - tool: gitlab
    connection_type: mcp
    status: available

    # Discovery metadata
    mcp_source: community
    mcp_package: "mcp-gitlab"
    mcp_confidence: medium

    capabilities:
      data_types: [projects, commits, merge_requests, issues, pipelines]
      reporting:
        daily: [recent_commits, open_mrs]
        weekly: [contributions, mr_activity]

    setup_required: true
    setup_instructions: "claude mcp add gitlab -- npx -y mcp-gitlab"
    warning: "Community package - not officially supported"
    warning_code: community_package

  - tool: instagram
    connection_type: unavailable
    status: unavailable

    mcp_source: none
    mcp_confidence: none

    reason: "API limited to business accounts with complex approval process"
    alternatives: ["Buffer", "Later", "Sprout Social"]

---

Evaluation Process

Step 1: Check if Already Installed

Read ~/.claude/settings.json and check for MCP servers:

cat ~/.claude/settings.json | jq '.mcpServers'

For each tool, check:

1. Is there an MCP with matching name?
2. Is it responding to basic queries?

If installed and responding → status: connected

Step 2: Invoke mcp-discovery Skill

If not installed, delegate to the mcp-discovery skill:

skill: mcp-discovery
input:
  tool: "gitlab"
  pillar: "design"

The discovery skill returns:

• mcp_source: official | vendor | community | none
• mcp_package: npm package name (or null)
• mcp_confidence: high | medium | low
• api_docs_url: API documentation URL
• recommendation: mcp | api | both | unavailable
• warning: any warnings about community packages

Step 3: Determine Status from Discovery

Based on discovery_result.recommendation:

Recommendation	Confidence	Status	Connection Type
mcp	high	available	mcp
mcp	medium	available (with warning)	mcp
mcp	low	api_only suggested	api
api	—	api_only	api
unavailable	—	unavailable	unavailable

Step 4: Build Capability Matrix

For connected/available tools, populate capabilities from known data:

Operations pillar:

• Notion, Google Workspace, Linear, Slack: See known capabilities below

Design pillar:

• GitHub, GitLab, Figma: See known capabilities below

Analytics pillar:

• GA4, Dub.co, Plausible, Substack: See known capabilities below

---

Known Tool Capabilities

Operations Pillar

Notion

• Data types: pages, databases, blocks, comments, users
• Daily: recent_pages, task_counts, recent_comments
• Weekly: page_activity, task_completion

Google Workspace

• Data types: calendar_events, emails, documents
• Daily: todays_events, unread_emails
• Weekly: event_count, email_volume

Linear

• Data types: issues, projects, cycles, teams, labels
• Daily: issues_due, assigned_issues
• Weekly: issues_completed, cycle_progress

Slack

• Data types: messages, channels, reactions
• Daily: unread_counts, mentions
• Weekly: channel_activity

Design Pillar

GitHub

• Data types: repos, commits, prs, issues, actions
• Daily: recent_commits, open_prs
• Weekly: team_contributions, pr_activity

GitLab

• Data types: projects, commits, merge_requests, issues, pipelines
• Daily: recent_commits, open_mrs
• Weekly: contributions, ci_stats

Figma (API only for reporting)

• Data types: files, versions, comments, users
• Daily: files_edited, active_users
• Weekly: design_versions, comment_activity

Analytics Pillar

Google Analytics (GA4)

• Data types: pageviews, sessions, events, goals
• Daily: session_count, top_pages
• Weekly: traffic_trends, source_breakdown

Dub.co

• Data types: links, clicks, referrers, geo
• Daily: click_counts
• Weekly: top_links

Plausible (API only)

• Data types: visitors, pageviews, sources
• Daily: visitor_count, top_pages
• Weekly: traffic_trends

Substack (API only, needs wrapper)

• Data types: subscribers, posts, email_stats
• Daily: new_subscribers
• Weekly: subscriber_growth, post_views

---

Integration with Setup Flow

During Chapter 1 (Operations)

User selects: [Notion, Google Workspace, Slack]
↓
Spawn tool-evaluator sub-agent:
  - Check Notion MCP → Found, connected → status: connected
  - Check Google Workspace MCP → Found, connected → status: connected
  - Check Slack MCP → Not found → invoke mcp-discovery
    → mcp-discovery returns: community, medium confidence, warning
  → Slack status: available (with warning)
↓
Return capability matrix
↓
Main agent presents findings with status indicators

During Chapter 2 (Design)

User selects: [GitHub, Figma]
↓
Spawn tool-evaluator sub-agent:
  - Check GitHub MCP → Found, connected → status: connected
  - Check Figma MCP → Found, but code-focused → invoke mcp-discovery
    → mcp-discovery returns: official but code-focused, api recommended
  → Figma status: api_only (with note)
↓
Return capability matrix
↓
Main agent guides Figma API token setup

During Chapter 3 (Analytics)

User selects: [GA4, Substack, Instagram]
↓
Spawn tool-evaluator sub-agent:
  - Check GA4 MCP → Found, connected → status: connected
  - Check Substack → invoke mcp-discovery
    → Returns: no MCP, API available, needs wrapper
  → Substack status: api_only (with wrapper option)
  - Check Instagram → invoke mcp-discovery
    → Returns: unavailable
  → Instagram status: unavailable
↓
Return capability matrix
↓
Main agent offers Substack wrapper creation, skips Instagram

---

Error Handling

MCP not responding:

tool: notion
connection_type: mcp
status: error
error: "MCP timeout - server not responding"
suggestion: "Restart Claude or check MCP server"

Discovery failed:

tool: unknown_tool
connection_type: unknown
status: error
error: "Discovery could not determine connection method"
suggestion: "Check if tool has public API documentation"

---

Usage Example

When invoked by setup flow:

# Pseudo-code for sub-agent task

def evaluate_tools(tools: list[dict]) -> dict:
    results = []

    for tool_info in tools:
        tool = tool_info['name']

        # Step 1: Check if already installed
        mcp_info = check_installed_mcp(tool)

        if mcp_info.found and mcp_info.connected:
            results.append({
                'tool': tool,
                'connection_type': 'mcp',
                'status': 'connected',
                'mcp_source': 'installed',
                'mcp_confidence': 'high',
                'capabilities': get_capabilities(tool)
            })
        else:
            # Step 2: Invoke mcp-discovery skill
            discovery = invoke_skill('mcp-discovery', tool=tool)

            # Step 3: Determine status from discovery
            if discovery.recommendation == 'mcp':
                if discovery.mcp_confidence == 'high':
                    status = 'available'
                    warning = None
                else:
                    status = 'available'
                    warning = discovery.warning

                results.append({
                    'tool': tool,
                    'connection_type': 'mcp',
                    'status': status,
                    'mcp_source': discovery.mcp_source,
                    'mcp_package': discovery.mcp_package,
                    'mcp_confidence': discovery.mcp_confidence,
                    'capabilities': get_capabilities(tool),
                    'setup_instructions': f"claude mcp add {tool} -- npx -y {discovery.mcp_package}",
                    'warning': warning
                })

            elif discovery.recommendation == 'api':
                results.append({
                    'tool': tool,
                    'connection_type': 'api',
                    'status': 'api_only',
                    'mcp_source': discovery.mcp_source,
                    'api_docs_url': discovery.api_docs_url,
                    'capabilities': get_capabilities(tool),
                    'setup_required': True,
                    'note': discovery.note
                })

            else:
                results.append({
                    'tool': tool,
                    'connection_type': 'unavailable',
                    'status': 'unavailable',
                    'mcp_source': 'none',
                    'reason': discovery.reason,
                    'alternatives': discovery.alternatives
                })

    return {'results': results}

---

References

• skills/mcp-discovery/SKILL.md — Dynamic MCP discovery
• references/tool-registry.md — Reference documentation (not source of truth)
• references/config-schema.md — Config structure for storing results

---

Version: 2.0