ATR - Agentic Test Runner
ATR is an AI-powered test runner that automatically analyzes failures and runs browser-based behavior tests using natural language.
Features
- AI Failure Analysis: Run any command and get intelligent analysis when it fails
- Browser Behavior Testing: Write tests in natural language, let AI execute them
- Multiple LLM Backends: Supports Google Gemini API, Vertex AI, and CLI tools (Claude, Gemini)
- CLI Backend Support: Use Claude CLI or Gemini CLI as backends - no API keys needed
- MCP Server: Expose browser tools to any MCP-compatible client
- Cross-Platform: Works on Linux, macOS, and Windows
- Extensible: Tool-based architecture for custom extensions
Quick Start
Install
macOS / Linux:
curl -fsSL https://raw.githubusercontent.com/imyousuf/agentic-test-runner/main/install.sh | sh
Windows (PowerShell):
irm https://raw.githubusercontent.com/imyousuf/agentic-test-runner/main/install.ps1 | iex
From source:
go install github.com/imyousuf/agentic-test-runner/cmd/atr@latest
Or download from Releases.
Configure
# Option 1: Using Claude or Gemini CLI (no API key needed)
# If you have claude or gemini CLI installed, ATR auto-detects them
atr config init # Creates config with detected CLI as default backend
# Option 2: Using Gemini API
export GEMINI_API_KEY="your-api-key"
# Option 3: Using Vertex AI
gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT="your-project"
Run
Analyze a failing command:
atr run --cmd "go test ./..."
When the command fails, ATR's AI agent will:
- Analyze the failure output
- Run diagnostic commands
- Read relevant source files
- Provide actionable recommendations
Run a behavior test:
# Create a test file
cat > login.test.txt << 'EOF'
Test: User can log in
Steps:
1. Navigate to http://localhost:3000/login
2. Enter "user@example.com" in email field
3. Enter "password" in password field
4. Click "Sign In" button
5. Verify URL contains /dashboard
EOF
# Run the test
atr run --behavior login.test.txt
Example Output
Executing: go test ./...
Directory: /path/to/project
--- FAIL: TestUserAuth (0.05s)
auth_test.go:42: expected 200, got 401
✗ Command failed (exit code: 1, duration: 2.3s)
Analyzing failure with AI agent...
Using model: gemini-2.0-flash-exp (gemini-api)
======================================================================
ANALYSIS RESULTS
======================================================================
Status: FAILURE
Summary:
TestUserAuth fails because the auth middleware expects a JWT token,
but the test doesn't provide one in the request headers.
Root Cause:
Line 38 in auth_test.go creates a request without Authorization header.
The auth middleware (middleware/auth.go:15) rejects it with 401.
Recommendations:
1. Add mock JWT token to test request
2. Or bypass auth middleware in test setup
3. Check if middleware was recently added to the route
Files Examined:
- auth_test.go
- middleware/auth.go
- routes/api.go
Documentation
- Getting Started - First steps with ATR
- Installation - All installation methods
- Configuration - Config file and authentication setup
- CLI Reference - All commands and flags
- Behavior Testing - Write browser tests in natural language
- Browser Server - HTTP server for programmatic browser control
- MCP Server - MCP protocol server for CLI tool integration
- Architecture - How ATR works internally
- llms.txt - Quick reference for AI agents
Claude Code Integration
ATR includes Claude Code skills for seamless AI-assisted browser automation. Install the skills to enable natural language control of ATR within Claude Code.
Installation
-
Add the ATR marketplace:
/plugin marketplace add imyousuf/agentic-test-runner -
Install the skills plugin:
/plugin install atr-skills@atr-marketplace
Available Skills
| Skill | Description |
|---|---|
| atr-browser | Control browser via ATR server (navigate, click, fill, screenshot) |
| atr-analyze | Run tests with AI analysis (default for test suites - keeps context clean) |
| atr-behavior | Run natural language browser tests |
Usage Examples
Once installed, Claude Code automatically uses these skills when relevant: