harness-evaluator is configured through YAML files for run configs and task definitions, with pricing tables and environment variables for cost accounting and API access.
Run configuration
Run configs are YAML files passed to harness-evaluator run. See runs/sample-run.yaml and runs/sample-minimal.yaml for examples.
use_host_network: false# Optional. Use --network host instead of the
# default bridge. Default: false.
run_as_user: "1000:1000"# Optional. UID:GID the container runs as.
# Defaults to the invoking user.
Field reference
name
Run identifier. Used as the primary key in the results store and in report filenames. Must match [A-Za-z0-9._-]+.
harnesses
List of harness specifications. Each harness is paired with each model to form the eval matrix.
Field
Type
Required
Description
name
string
Yes
Harness identifier (validated against [A-Za-z0-9._-]+)
adapter
string
Yes
Adapter registry name (e.g. opencode, claude-code)
observability_tier
string
No
full, partial, or minimal (default: partial)
config
dict
No
Harness-specific config passed to the adapter
docker_image
string
No
Per-harness runner image override (see below)
version
string
No
Image tag on the run-level image’s repo (see below)
Choosing a harness version
By default every harness in a run uses the run-level docker_image. To evaluate
a specific harness version, set a per-harness image. Precedence is
docker_image > version > the run-level docker_image:
# `version` shorthand: uses this as the tag on the run-level image's repo,
# i.e. ghcr.io/yorch/harness-evaluator-runner:cc-2.1.0
- name: claude-code-2.1
adapter: claude-code
version: "cc-2.1.0"
Because a harness entry’s name is just an identifier and adapter is
separate, you can put two versions of the same harness in one matrix (as
above) to compare them directly. The resolved image is recorded in each result’s
harness_metadata for reproducibility. You are responsible for building/pushing
the referenced images (see Building a specific harness version).
models
List of model specifications. Each model is paired with each harness.
Field
Type
Required
Description
name
string
Yes
Model identifier (validated against [A-Za-z0-9._-]+)
List of task IDs to run, or ["*"] to run all tasks in the library. Task IDs are resolved against the task library. Note that ["*"] includes any multi_phase tasks — these require at least one model with role: review (and one with role: implementation) or build_matrix() will raise a ValueError. To run only single-phase tasks, list their IDs explicitly (see runs/sample-run.yaml).
task_library_path
Path to a directory containing task YAML files. All *.yaml files in this directory are loaded as the task library. Optional — defaults to the task library bundled inside the installed harness-evaluator package (harness_evaluator/tasks), so an installed harness-evaluator works without a repo checkout. Local repo_url fixtures are resolved relative to this directory.
repeats
Number of repeats per cell (harness × model × task). Default: 5. Each repeat is an independent run with a fresh container and repo checkout.
budget_usd
Maximum total spend in USD. When set, the orchestrator uses a reserve-and-reconcile pattern to prevent overspending. Cells are skipped when the remaining budget is insufficient — except cells that are budget-exempt (see cost_mode below), which are never skipped for lack of budget. Set to null or omit for no cap. See Orchestrator → Budget management for the exact admission-gate predicate and exemption rules.
parallel_runs
Number of parallel container runs. Default: 1 (sequential). With parallel_runs > 1, an asyncio.Semaphore limits concurrent executions.
Warning: Budget reservation is async-safe (single-process asyncio.Lock), not thread-safe. Do not run the orchestrator across multiple processes.
run_as_user
The UID:GID the runner container executes as, e.g. "1000:1000". Defaults to
the user invoking harness-evaluator.
The cell workdir is bind-mounted into the container, so the container user must
be able to write to it — that is why the default follows the invoking user
rather than the image’s built-in user. It also means files the harness creates
are owned by you, which the host-side evaluation step (git diff, hidden tests)
depends on.
Override this only for rootless Docker or userns-remap setups, where the host
UID is not the effective UID inside the container. If you do, make sure the
workdir is writable by that UID.
HOME inside the container is always set to /workspace/.home (a directory in
the mounted workdir). The harnesses need a writable HOME, and it is kept out
of the repo directory so harness state never lands in the evaluated diff.
use_host_network
Run containers with --network host instead of the default bridge. Default:
false.
A fallback for environments where containers cannot reach the gateway via
host.docker.internal. With it on, containers share the host’s network
namespace and reach the gateway on 127.0.0.1, and --add-host is omitted —
Docker rejects the two together.
Note: host networking removes the container’s network namespace
isolation. Prefer the default bridge unless host.docker.internal genuinely
does not work in your environment.
Where hidden tests run
SWE tasks are scored by running the task’s test_commandinside the runner
image, not in the interpreter running harness-evaluator. The task’s
setup_script installs its dependencies there, and an ordinary install of
harness-evaluator has neither those dependencies nor pytest — under uvx the
SWE track previously failed every cell with No module named pytest while the
model’s diff was correct.
Diffing and applying the hidden-test patch remain host-side filesystem work.
SWEEvaluator still runs tests in-process when used directly as a library, so
it needs no Docker outside a run.
Minimal example
name: "minimal-first-run"
description: "Minimal first run: one harness, one model, one task"
harnesses:
- name: opencode
adapter: opencode
observability_tier: full
config:
mode: agent
models:
- name: claude-sonnet-5
provider: anthropic
api_key_env: ANTHROPIC_API_KEY
config:
max_tokens: 16384
tasks:
- "swe-bugfix-001"
task_library_path: "./tasks"
repeats: 1
budget_usd: 5.0
Full sweep example
name: "broad-first-pass"
description: "Broad first pass: 5 harnesses, 2 providers, both task tracks"
harnesses:
- name: opencode
adapter: opencode
observability_tier: full
config:
mode: agent
- name: claude-code
adapter: claude-code
observability_tier: partial
config:
max_turns: 50
- name: codex
adapter: codex
observability_tier: partial
config: {}
- name: pi
adapter: pi
observability_tier: minimal
config: {}
- name: omp
adapter: omp
observability_tier: minimal
config: {}
models:
- name: claude-sonnet-5
provider: anthropic
api_key_env: ANTHROPIC_API_KEY
config:
max_tokens: 16384
- name: gpt-5.6-terra
provider: openai
api_key_env: OPENAI_API_KEY
config:
max_tokens: 16384
tasks:
- "*"
task_library_path: "./tasks"
repeats: 5
budget_usd: 100.0
Multi-phase example
A multi-phase run pairs an implementation model with an adversarial reviewer model. See runs/sample-multi-phase.yaml and tasks/multi-phase-bugfix-001.yaml for complete examples.
name: multi-phase-demo
harnesses:
- name: claude-code
adapter: claude_code
models:
- name: claude-sonnet-5
provider: anthropic
api_key_env: ANTHROPIC_API_KEY
role: implementation
- name: claude-opus-5
provider: anthropic
api_key_env: ANTHROPIC_API_KEY
role: review
tasks:
- multi-phase-bugfix-001
repeats: 1
budget_usd: 10.0
The matrix expands to one cell per implementation × review model pair. For a walkthrough, see the Multi-phase evaluation guide.
Task definitions
Tasks are defined as YAML files in the task library directory. Each file can contain multiple tasks under a tasks: key.
Local git repos: paths with a .git directory → git clone
Local directories: paths without .git → shutil.copytree + git init
Relative paths are resolved against the project root, not the current working directory.
setup_script
Shell script executed inside the container before the harness runs. Written to /workspace/setup.sh and executed with bash /workspace/setup.sh in the repo directory. Used for installing dependencies, setting up databases, etc.
task_prompt
The prompt given to the harness. This is the only instruction the harness receives — it does not see the test patch, expected files, or other evaluation metadata.
For multi_phase tasks, the top-level task_prompt is still required but ignored when phases is non-empty. Each phase uses its own task_prompt from the PhaseSpec.
test_command
Command to run tests. Parsed with shlex.split (no shell=True) to prevent shell injection. Commands requiring shell features (pipes, redirects) should be wrapped in bash -c "...".
test_patch
Hidden test patch applied after the harness runs but before evaluation. Applied via git apply - from stdin. The harness never sees this patch — it only sees the original repo and the task prompt.
expected_files
Files that should be created or modified by the harness. Used by the structural checker in the open-ended track to verify the submission includes the expected deliverables.
timeout_seconds
Per-task timeout in seconds. Applied to both the harness execution and the test command. Default: 600.
metadata
Free-form dictionary for additional task metadata. Stored with the task but not used by the evaluator. Useful for filtering or grouping tasks in analysis.
phases
Ordered list of phase definitions for multi_phase tasks. Empty (or omitted) for swe and open_ended tasks. Each phase is a PhaseSpec:
Field
Type
Required
Description
name
string
Yes
Phase identifier (validated against [A-Za-z0-9._-]+). Must be unique within the task.
model_role
string
No
implementation (default) or review. Determines which model runs this phase.
task_prompt
string
Yes
The prompt given to the harness for this phase.
input
string
No
What to inject from prior phases: none (default), diff, output, or review_feedback.
timeout_seconds
int
No
Per-phase timeout. Default: 600.
Phase input types
Input
Injects into the phase prompt
none
Nothing — the phase runs standalone.
diff
Git diff from the prior implementation phase (captured before commit).
output
Stdout + stderr from the prior phase.
review_feedback
Stdout + stderr from a prior review phase.
Model roles
Role
Used by
Description
implementation
All phases by default
The primary coding model. Assigned via models[].role: implementation in the run config.
review
Phases with model_role: review
The adversarial reviewer model. Assigned via models[].role: review in the run config.
Validation rules
multi_phase tasks must define at least one phase.
At least one phase must have model_role: implementation.
Phase names must be unique within a task.
If any phase has model_role: review, the run config must include at least one model with role: review and one with role: implementation.
SWE task example
tasks:
- id: swe-bugfix-001
name: Fix off-by-one in list pagination function
track: swe
difficulty: easy
description: |
The `get_page` function in src/solution.py has an off-by-one bug.
It calculates the end index as `page_number * page_size - 1` instead
of `page_number * page_size`, causing the last item of each full page
to be dropped.
repo_url: tasks/repos/swe-bugfix-001
setup_script: pip install -r requirements.txt
task_prompt: |-
Fix the off-by-one bug in the `get_page` function in src/solution.py.
The correct end index should be `page_number * page_size`.
Design and implement a token bucket rate limiter with configurable
rate and burst capacity. Include comprehensive tests.
task_prompt: |-
Design and implement a token bucket rate limiter in src/rate_limiter.py.
Requirements:
- Configurable rate (tokens per second) and burst capacity
- `allow(n=1)` method that returns True if n tokens are available
- Tokens refill at the configured rate, up to the burst capacity
- Thread-safe implementation
Add comprehensive tests in tests/test_rate_limiter.py.
test_command: python -m pytest tests/
expected_files:
- src/rate_limiter.py
- tests/test_rate_limiter.py
timeout_seconds: 600
metadata:
design_type: rate_limiter
language: python
Pricing tables
Cost is calculated from per-token pricing tables in src/harness_evaluator/gateway/models.py. Prices are in USD per 1 million tokens.
Default pricing
Anthropic current generation:
Model
Input
Output
Cache read
Cache write
claude-fable-5
$10.00
$50.00
$1.00
$12.50
claude-mythos-5
$10.00
$50.00
$1.00
$12.50
claude-opus-5
$5.00
$25.00
$0.50
$6.25
claude-sonnet-5
$2.00
$10.00
$0.20
$2.50
claude-haiku-4-5-20251001 / claude-haiku-4-5
$1.00
$5.00
$0.10
$1.25
Anthropic previous generation (still available):
Model
Input
Output
Cache read
Cache write
claude-opus-4-5-20251101 / claude-opus-4-5
$5.00
$25.00
$0.50
$6.25
claude-opus-4-8
$5.00
$25.00
$0.50
$6.25
claude-opus-4-7
$5.00
$25.00
$0.50
$6.25
claude-opus-4-6
$5.00
$25.00
$0.50
$6.25
claude-sonnet-4-6
$3.00
$15.00
$0.30
$3.75
claude-sonnet-4-5 / claude-sonnet-4-5-20250929
$3.00
$15.00
$0.30
$3.75
claude-sonnet-4-20250514
$3.00
$15.00
$0.30
$3.75
claude-opus-4-20250514
$15.00
$75.00
$1.50
$18.75
claude-haiku-3-5-20241022
$0.80
$4.00
$0.08
$1.00
OpenAI current generation (GPT-5.6 family):
Model
Input
Output
Cache read
Cache write
gpt-5.6-sol / gpt-5.6
$4.00
$20.00
$0.40
$5.00
gpt-5.6-terra
$2.00
$12.00
$0.20
$2.50
gpt-5.6-luna
$0.20
$1.20
$0.02
$0.25
OpenAI previous generation (still available):
Model
Input
Output
Cache read
gpt-5.5
$5.00
$30.00
$0.50
gpt-5.4
$2.50
$15.00
$0.25
gpt-5.4-mini
$0.75
$4.50
$0.075
gpt-5.4-nano
$0.20
$1.25
$0.02
gpt-5.3-codex
$1.75
$14.00
$0.175
gpt-5
$1.25
$10.00
$0.125
gpt-5-mini
$0.25
$2.00
$0.025
gpt-5-nano
$0.05
$0.30
$0.005
o3
$2.00
$8.00
$0.50
o4-mini
$1.10
$4.00
$0.55
OpenAI legacy (for backward compatibility):
Model
Input
Output
Cache read
gpt-4o
$2.50
$10.00
$1.25
gpt-4o-mini
$0.15
$0.60
$0.075
Google Gemini (direct API; gateway does not yet route Google traffic):
Model
Input
Output
Cache read
gemini-3-pro
$2.00
$12.00
$0.20
gemini-3.1-pro-preview
$2.00
$12.00
$0.20
gemini-3-flash-preview
$0.50
$3.00
$0.05
gemini-3.1-flash-lite
$0.25
$1.50
$0.025
gemini-2.5-pro
$1.25
$10.00
$0.125
gemini-2.5-flash
$0.30
$2.50
$0.03
gemini-2.5-flash-lite
$0.10
$0.40
$0.01
Gemini output pricing includes thinking tokens. Gemini uses hourly
context-caching storage pricing rather than a per-token cache-write cost,
so no cache_write column is listed.
Unknown models
When a model is not in the pricing table, get_pricing_strict() logs a warning and returns a zero-cost PricingTable. This means token usage will not count against the budget — a silent budget bypass. The warning makes this visible:
WARNING: No pricing found for model 'my-custom-model'; cost will be $0
and token usage will NOT count against the budget.
Add the model to DEFAULT_PRICING to fix this.
Adding a new model
Add an entry to DEFAULT_PRICING in src/harness_evaluator/gateway/models.py:
DEFAULT_PRICING: dict[str, PricingTable] = {
# ... existing entries ...
"my-new-model": PricingTable(
input_per_million=5.0,
output_per_million=20.0,
cache_read_per_million=0.50,
cache_write_per_million=6.25,
),
}
Environment variables
Required
Variable
Description
ANTHROPIC_API_KEY
Anthropic API key (for Anthropic models and judge calibration)
OPENAI_API_KEY
OpenAI API key (for OpenAI models)
Set by adapters (inside containers)
Variable
Description
ANTHROPIC_BASE_URL
Gateway proxy URL for Anthropic (with ?trace_id=)
OPENAI_BASE_URL
Gateway proxy URL for OpenAI (with /v1 and ?trace_id=)
ANTHROPIC_API_KEY
Passed through from host
OPENAI_API_KEY
Passed through from host
HARNESS_EVALUATOR_TRACE_ID
Cell trace ID for cost attribution
Allowlisted (passed from host to container)
Variable
Description
PATH
Executable search path
HOME
Home directory
USER
Username
SHELL
Default shell
LANG
Locale
LC_ALL
Locale override
TERM
Terminal type
TMPDIR
Temporary directory
Authentication modes
By default, harness-evaluator authenticates to provider APIs using API keys
(auth_mode: api_key). For harnesses that support subscription-based access
(Claude Code OAuth, Codex with a ChatGPT subscription), you can switch to an
OAuth/subscription auth mode so the harness uses your existing subscription
instead of pay-per-token API billing.
The three auth modes
Mode
Value
Description
API key
api_key
Default. Uses the env var named in api_key_env (e.g. ANTHROPIC_API_KEY).
Claude Code OAuth
claude_oauth
Uses a Claude Code OAuth token or credential file. No API key is sent.
Codex ChatGPT
codex_chatgpt
Uses a Codex/ChatGPT subscription credential file. Routes through the ChatGPT backend.
Model spec fields
Field
Type
Required
Description
auth_mode
string
No
api_key (default), claude_oauth, or codex_chatgpt
credentials_path
string
No
Path to an OAuth credential file on the host (for subscription auth)
cost_mode
string
No
platform (default, pay-per-token) or subscription (budget-exempt; real cost still recorded)
credentials_path
For claude_oauth and codex_chatgpt modes, credentials_path points to the
OAuth credential file on the host. The Docker runner copies the credential
file’s parent directory to a temp directory and mounts it writable into the
container so the harness can refresh expired access tokens. The original
credential files on the host are never modified or mounted directly.
The appropriate config env var is set so the harness finds its tokens:
claude_oauth → mounts to /workspace/.claude, sets CLAUDE_CONFIG_DIR
codex_chatgpt → mounts to /workspace/.codex, sets CODEX_HOME
If the file does not exist, the runner logs a warning and skips the mount (the
harness will likely fail to authenticate).
cost_mode
platform (default): Standard pay-per-token cost accounting. Token usage is
priced against the DEFAULT_PRICING table and counts against budget_usd.
subscription: The harness runs on a flat-rate subscription. The real cost
is still computed and recorded in full — in run_results.total_cost and in
progress.total_cost — but the cell is exempt from the dollar budget:
its reservation is $0.00, it is never skipped for lack of budget, and its
cost is excluded from get_billable_cost (the figure checked against
budget_usd). Use this when running on a ChatGPT or Claude Pro
subscription where you are not billed per token. See
Orchestrator → Budget exemption for the
exact rules.
Security considerations
OAuth credential files contain refresh tokens that grant ongoing access to
your account. Treat them with the same care as API keys:
Store credential files with restrictive permissions (chmod 600).
Never commit credential files to a repository.
The Docker runner copies credentials to a temp directory and mounts that
(not the original) so the harness can refresh tokens without touching your
real credential files. The mount is writable so token refresh works.
A writable mount means the harness process can read the refresh token.
Task YAMLs are trusted input (see Task trust model),
but be aware that a malicious task could exfiltrate OAuth tokens via the
network. This is the same risk as API keys — the container has network
access to the gateway.
Credential mount points (.claude, .codex) are excluded from the git
commit diff (including nested paths) as defense in depth, so tokens never
appear in evaluation diffs.
Example: API key (default)
models:
- name: claude-sonnet-5
provider: anthropic
api_key_env: ANTHROPIC_API_KEY
Example: Claude Code OAuth
models:
- name: claude-sonnet-5
provider: anthropic
api_key_env: ANTHROPIC_API_KEY
auth_mode: claude_oauth
credentials_path: "~/.claude/.credentials.json"
cost_mode: subscription
With claude_oauth, the adapter sets ANTHROPIC_BASE_URL to the gateway
proxy but does not set ANTHROPIC_API_KEY. If the CLAUDE_CODE_OAUTH_TOKEN
environment variable is present on the host, it is passed through to the
container.
Example: Codex ChatGPT subscription
models:
- name: gpt-5.6-terra
provider: openai
api_key_env: OPENAI_API_KEY
auth_mode: codex_chatgpt
credentials_path: "~/.codex/auth.json"
cost_mode: subscription
With codex_chatgpt, the Codex adapter passes chatgpt_base_url (with a
/codex path) via the -c config flag instead of openai_base_url. The
gateway proxy routes /codex/responses and /codex/ paths to the ChatGPT
backend (https://chatgpt.com/backend-api/codex). OPENAI_API_KEY and
OPENAI_BASE_URL are not set.
Identifier validation
All identifiers (run names, harness names, model names) are validated against [A-Za-z0-9._-]+ to prevent path traversal and shell injection. Invalid characters cause a ValueError at config load time.
Docker image configuration
The runner image contains 5 preinstalled harnesses (Claude Code, Codex, OpenCode, Pi, OMP) and their dependencies. The adapter registry also includes Aider, Gemini CLI, Antigravity, Copilot, Cursor, and Kiro — to use these, build a custom image with the harness binary installed (see Building a specific harness version below). You can either pull the pre-built image from GHCR or build it locally. See Docker Runner for details on the image contents.
Available tags: latest, sha-<short-hash> (pinned to a commit), semver tags
like 1.2.3 and 1.2 (published from v* release tags), and main.
The default docker_image is ghcr.io/yorch/harness-evaluator-runner:latest, not
a version-pinned tag. It is deliberately unpinned: the installed harness-evaluator
version may be ahead of the newest published image (dev installs, or the window
between a release-please bump and the image build), version tags only exist for
released versions on GHCR, and latest is always available once the first image
is published.
Reproducibility: because the default floats, a run is not reproducible
across time by default — the same config can pull a different image next month.
Pin the image explicitly when results need to be comparable:
Available build args (defaulting to the verified pinned set): CLAUDE_CODE_VERSION,
CODEX_VERSION, OPENCODE_VERSION, PI_VERSION, OMP_VERSION, BUN_VERSION.
The installed versions are recorded as io.harness-evaluator.* OCI image labels, and the
image name is stored in each run’s metadata, so results trace to exact versions.
Reference the built image via docker_image: in the run config.
Publishing a per-harness-version image
The docker-versions.yml workflow (manual trigger) builds and publishes a
runner image with a single harness version override, tagged as
<harness>-<version> (e.g. claude-code-2.0.0). Trigger it from the GitHub
Actions UI with the harness build-arg name and the version to pin. The
resulting image is pushed to GHCR and can be referenced directly:
Task YAMLs — including test_command, setup_script, and repo_url — are
treated as trusted input. The SWE evaluator and the open-ended structural
checker run a task’s test_command on the host (not inside the container),
and setup_script runs inside the container. Do not load task libraries from
untrusted sources. harness-evaluator still validates task id and repo_commit against a
safe charset and skips symlinked untracked files during diff extraction as
defense in depth, but a hostile task definition can execute arbitrary commands.
Off-by-one bugfix with implement → review → revise phases
A curated run config that uses all 20 single-phase tasks is at runs/task-mix.yaml. The multi-phase task has its own sample config at runs/sample-multi-phase.yaml.
Note: Do not edit tasks/repos/*/ contents directly — they are task fixtures. Change the source and re-init via the runner’s _git_init_fresh.