harness-evaluator
Compare agentic coding harnesses on token efficiency, task effectiveness, and time efficiency.
Features
Everything you need to benchmark agentic coding harnesses.
Gateway Proxy
Custom HTTP/SSE proxy intercepts every provider call, capturing token usage, cost, and latency with full request/response logging.
Token Accounting
Input, output, cache-read, cache-write, and reasoning tokens tracked per call. Cost calculated from a per-model pricing table.
Docker Isolation
Each harness runs in its own container with --cap-drop=ALL, non-root user, and network policy enforcement.
Resumability & Budgets
Cell-level resumability skips completed cells on re-run. Budget caps stop the run when spend exceeds the configured limit.
Frozen LLM Judge
Open-ended tasks evaluated by a versioned, immutable-prompt judge with structured rubric and anchor-set calibration for drift detection.
Statistics & Reporting
Mixed-effects modeling, variance decomposition, bootstrap CIs, HTML/JSON/CSV reports, and an interactive FastAPI dashboard.
Supported Harnesses
Eleven adapters across three observability tiers. Five are preinstalled in the default Docker image; the rest require a custom image.
| Harness | Adapter | Observability | Notes |
|---|---|---|---|
| OpenCode | opencode | full | Open-source, system prompt visible |
| Aider | aider | full | Open-source, multi-provider |
| Claude Code | claude-code | partial | Closed, proxy captures traffic |
| Codex | codex | partial | Closed, proxy captures traffic |
| Gemini CLI | gemini | partial | Google, proxy captures traffic |
| Antigravity | antigravity | partial | Google, proxy captures traffic |
| Pi | pi | minimal | May bypass proxy |
| OMP | omp | minimal | May bypass proxy |
| GitHub Copilot | copilot | minimal | GitHub, may bypass proxy |
| Cursor | cursor | minimal | Multi-provider, may bypass proxy |
| Kiro | kiro | minimal | AWS, may bypass proxy |
Quick Start
Up and running in five commands.
Architecture
How the pieces fit together.
- GatewayCustom HTTP/SSE proxy (aiohttp) that intercepts provider calls and captures token usage, cost, and latency.
- OrchestratorBuilds the eval matrix (harness × model × task × repeats), manages budget caps, and handles cell-level resumability.
- RunnerDocker-based isolation, one container per run with capability dropping and non-root execution.
- AdaptersPer-harness integration (Python core + TS shims where needed) with a uniform prepare/run/cleanup interface.
- EvaluatorSWE-bench-style (hidden tests) and open-ended (LLM judge) tracks with partial credit and error classification.
- ReportingCLI reports, static HTML/JSON/CSV, and an interactive web dashboard.
- StatisticsMixed-effects models, variance decomposition, and bootstrap confidence intervals.
Start Evaluating
Install from PyPI, pull the Docker image, and run your first evaluation in minutes — no clone required.