openevidence-ci-integration

OpenEvidence CI Integration

Table of Contents

• Overview
• Prerequisites
• Instructions
• Output
• Error Handling
• Examples
• Resources

Overview

Integrate OpenEvidence testing and validation into CI/CD pipelines. Covers GitHub Actions workflows, unit tests with SDK mocks, integration tests against sandbox, and clinical validation test suites.

Prerequisites

• GitHub, GitLab, or other CI platform
• OpenEvidence sandbox API credentials
• Test suite configured (vitest recommended)
• Secret management in CI platform

Instructions

Step 1: Create GitHub Actions Workflow

Set up parallel jobs: lint-and-typecheck, unit-tests (with coverage), integration-tests (sandbox only, gated by branch/label), and clinical-validation (on main).

Step 2: Configure Test Environments

Create separate vitest configs: base (unit, 10s timeout), integration (60s timeout, retry 2, maxConcurrency 1 to avoid rate limits).

Step 3: Write Unit Tests with Mocks

Mock @openevidence/sdk to test clinical query service logic, error handling, and response formatting without API calls.

Step 4: Write Integration Tests

Test against sandbox: verify clinical queries return answers with citations, validate empty queries fail gracefully, check citation fields.

Step 5: Add Clinical Validation

Create known-answer test cases that verify responses contain expected medical keywords (e.g., "metformin" for diabetes, "statin" for cardiovascular) with confidence > 0.7.

Step 6: Configure Secrets

Store OPENEVIDENCE_SANDBOX_API_KEY and OPENEVIDENCE_SANDBOX_ORG_ID as GitHub secrets. Use .env.test for local development.

Output

• CI workflow with lint, unit tests, integration tests, clinical validation
• Vitest configurations (unit + integration)
• Unit tests with mocked SDK
• Integration tests against sandbox
• Clinical validation with known-answer testing
• Coverage reports

Error Handling

CI Issue	Cause	Solution
Integration test timeout	API slow or rate limited	Increase timeout, add retry
Secret not found	Missing GitHub secret	Add secret in repo settings
Flaky tests	Network variability	Add retries, improve assertions
Coverage drop	New code untested	Add tests, adjust thresholds

Examples

Package.json Scripts

{
  "test:unit": "vitest run -c vitest.config.ts",
  "test:integration": "vitest run -c vitest.config.integration.ts",
  "test:clinical-validation": "vitest run tests/clinical-validation"
}

See detailed implementation for advanced patterns.

Resources

• GitHub Actions Documentation
• Vitest Documentation