Web Research Skill

Combines WebSearch with automatic Qdrant storage to build a searchable knowledge base.

Workflow

1. Check Qdrant first    → qdrant-find for existing knowledge
2. Search if needed      → WebSearch for current information
3. Store valuable finds  → qdrant-store with rich metadata
4. Return synthesized    → Combine stored + new knowledge

Step 1: Check Existing Knowledge

Before searching the web, check if the answer already exists:

Tool: qdrant-find
Query: "<user's question or topic>"

If sufficient information exists with recent harvested_at, use it directly.

Step 2: Web Search

When stored knowledge is insufficient or stale:

Tool: WebSearch
Query: "<refined search query>"

Step 3: Store Results

After getting valuable results, store with rich metadata:

Tool: qdrant-store
Information: |
  # <Topic/Question>

  ## Key Findings
  - Finding 1
  - Finding 2

  ## Details
  <Synthesized information from search results>

  ## Sources
  - [Title](URL)

Metadata:
  # Required fields
  source: "web_search"
  content_type: "text"
  harvested_at: "2025-01-04T10:30:00Z"

  # Search context
  query: "<original search query>"
  urls: ["https://example.com/1", "https://example.com/2"]

  # Classification (for filtering)
  category: "technology"
  subcategory: "databases"
  type: "documentation"

  # Technical context (when applicable)
  language: "python"
  framework: "fastapi"
  version: "0.100+"

  # Quality signals
  confidence: "high"
  freshness: "current"

  # Relationships
  related_topics: ["vector-search", "embeddings", "rag"]
  project: "reflex"

Rich Metadata Schema

Required Fields

Field	Type	Description
source	string	Origin: web_search, api_docs, github, manual
content_type	string	text, code, image, video_transcript
harvested_at	string	ISO 8601 timestamp

Search Context

Field	Type	Description
query	string	Original search query
urls	array	Source URLs (array for proper filtering)
domain	string	Primary domain (e.g., github.com)

Classification (Enables Filtering)

Field	Type	Values
category	string	technology, business, science, design, security, devops
subcategory	string	More specific: databases, frontend, ml, networking
type	string	documentation, tutorial, troubleshooting, reference, comparison, news

Technical Context

Field	Type	Description
language	string	Programming language: python, typescript, rust, go
framework	string	Framework/library: fastapi, react, tokio
version	string	Version constraint: 3.12+, >=2.0, latest
platform	string	linux, macos, windows, docker, kubernetes

Quality Signals

Field	Type	Values
confidence	string	high, medium, low - how reliable is this info
freshness	string	current, recent, dated, historical
depth	string	overview, detailed, comprehensive

Relationships

Field	Type	Description
related_topics	array	Related concepts for discovery
project	string	Associated project name
supersedes	string	ID of entry this replaces
parent_topic	string	Broader topic this belongs to

Image References (URL only, no download)

Field	Type	Description
image_url	string	URL to the image
alt_text	string	Image description
image_type	string	photo, diagram, screenshot, chart, icon

Filtering Examples

Find Python documentation:

qdrant-find with filter:
  category: "technology"
  language: "python"
  type: "documentation"

Find recent troubleshooting:

qdrant-find with filter:
  type: "troubleshooting"
  freshness: "current"

Find project-specific knowledge:

qdrant-find with filter:
  project: "reflex"

When to Store

Always store:

• Technical documentation findings
• API patterns and examples
• Error solutions and workarounds
• Best practices and recommendations
• Tool comparisons and evaluations

Skip storing:

• Simple factual lookups (dates, definitions)
• Ephemeral information (current weather, stock prices)
• Information already in Qdrant with same content

Example: Full Research Flow

User asks: "How do I set up GitHub Actions for Python testing?"

# Step 1: Check existing
qdrant-find: "GitHub Actions Python testing setup"
→ No relevant results

# Step 2: Search
WebSearch: "GitHub Actions Python pytest workflow 2025"
→ Returns results with workflow examples

# Step 3: Store
qdrant-store:
  Information: |
    # GitHub Actions Python Testing Setup

    ## Key Findings
    - Use `actions/setup-python@v5` for Python environment
    - Matrix testing across Python versions: 3.9, 3.10, 3.11, 3.12
    - pytest with coverage using `pytest-cov`

    ## Workflow Template
    ```yaml
    name: Python Tests
    on: [push, pull_request]
    jobs:
      test:
        runs-on: ubuntu-latest
        strategy:
          matrix:
            python-version: ["3.9", "3.10", "3.11", "3.12"]
        steps:
          - uses: actions/checkout@v4
          - uses: actions/setup-python@v5
            with:
              python-version: ${{ matrix.python-version }}
          - run: pip install -e .[test]
          - run: pytest --cov
    ```

    ## Sources
    - [GitHub Actions Python Guide](https://docs.github.com/en/actions/...)

  Metadata:
    source: "web_search"
    content_type: "code"
    harvested_at: "2025-01-04T10:30:00Z"
    query: "GitHub Actions Python pytest workflow 2025"
    urls: ["https://docs.github.com/en/actions/..."]
    domain: "github.com"
    category: "technology"
    subcategory: "ci-cd"
    type: "documentation"
    language: "python"
    framework: "pytest"
    platform: "github-actions"
    confidence: "high"
    freshness: "current"
    depth: "detailed"
    related_topics: ["testing", "ci-cd", "yaml", "github"]

Integration with Other Skills

• research-patterns: Use web-research for external searches
• qdrant-patterns: Follows same metadata conventions
• knowledge-ingestion-patterns: Compatible chunking approach
• github-harvester: Similar metadata schema for GitHub content