Open-source evaluation tooling

Controlled experiments for coding-agent harnesses.

Hold the model, tasks, and budget constant while you compare one context, tool, or instruction treatment at a time. Inspect every trial and publish verifiable evidence.

Get started View on GitHub

Python 3.10 through 3.14, Linux, macOS, Windows, Apache-2.0

scaffoldscope | offline demorun evidence

Experiment

offline-quickstart

complete

12 / 12

recorded trials

3 / 3

paired blocks

27,477

recorded tokens

$ scaffoldscope status runs/offline-quickstart-1b212f14
Progress: 12 / 12 (100.0%); remaining: 0
Paired blocks: 3 / 3 (100.0%)
Outcomes: context_overflow=3, resolved=9

$ scaffoldscope check runs/offline-quickstart-1b212f14
PASS: result bundle is complete and internally consistent.

Generated by the bundled offline demo. It validates the workflow, not model performance.

Why controlled ablations

Know which part of the harness changed the result.

Agent evaluations often change the model, prompt, tools, context manager, retry policy, and budget together. That produces a score without clean attribution. ScaffoldScope turns harness choices into declared treatments inside paired task and replicate blocks.

Same model. Same tasks. Same budget. One declared treatment.

Paired by construction

Every task and replicate receives each treatment from the same starting state. Treatment order can be randomized deterministically.

Auditable context

The canonical trajectory stays append-only. Derived model views record the complete assistant/tool bundles they retain or drop.

Verifiable evidence

Traces, patches, reports, evaluator overlays, identities, and checksums ship in a deterministic evidence bundle.

Treatment surface

Declare the treatment. Pin everything else.

Start with four built-in context policies, then vary the tool surface or treatment instructions. Versioned plugins can add policies and providers without escaping experiment identity.

Configuration reference

Built-in context-management policies
Policy	Trigger	Behavior
`none`	Never	Keep full canonical history until typed overflow.
`reactive`	Utilization threshold	Summarize salient history and retain recent atomic bundles.
`periodic`	Fixed turn cadence	Compact on declared boundaries with emergency handling.
`selective`	Context budget	Select atomic bundles with deterministic budgeted scoring.

Invariant: assistant actions and tool observations are retained or dropped as one atomic bundle.

How it works

Raw history and model context are separate.

A policy derives a model-facing view without mutating the canonical trajectory. The agent uses only the declared tools, a fixed evaluator checks the workspace, and each worker writes its own evidence before aggregates are rebuilt.

Config and tasks. Paired plan. Agent and context policy. Evaluator. Report and evidence bundle.

Run the core local pipeline without an API key.

The starter uses a deterministic scripted provider to validate planning, execution, traces, and reporting. It is a workflow test, not a model benchmark.

Operator guide Architecture

$ python -m pip install scaffoldscope

$ scaffoldscope init my-study --name my-study
$ scaffoldscope validate my-study/experiment.json
$ scaffoldscope budget my-study/experiment.json
$ scaffoldscope run my-study/experiment.json

Reports and guardrails

One report. Four evidence layers.

The report keeps performance, resource use, context behavior, and validity diagnostics together without collapsing them into one score.

Read the result schema

Outcomes: Solve rate, governed solve, paired wins/losses/ties, terminal status
Resources: Tokens, configured-price cost, cache activity, model, tool, and wall time
Context: Pressure, compaction exposure, compression, source selection, constraints
Validity: Pair coverage, treatment exposure, drift, duplicates, usage completeness

Scientific guardrails

Experiment-design contract

Harness and protocol failures stay in the intention-to-treat denominator.
Provider-reported usage remains separate from local estimates.
Task-level resampling preserves the paired study structure.
Scripted, small, or incomplete panels stay descriptive.

SWE-bench interoperability

Generate with ScaffoldScope. Grade with the official harness.

Import downloaded task rows, run every treatment and replicate cell, and export a uniquely identified prediction file for each cell. Official SWE-bench grading remains the correctness authority; results return as immutable overlays.

SWE-bench workflow

Project maturity

ScaffoldScope is alpha.

Version 0.3.1 has a tested core evidence contract, but it does not carry a 1.0 stability promise or an adequately powered paid-model headline result. Use it to design and audit experiments today.

v0.3.1 release