ETABench
ETABench measures whether an autonomous agent can predict, before starting, how long it will take itself to complete a verifiable computer task.
ETABench measures whether an agent can estimate, before doing any work, how long it will take itself to finish a real computer task.
Each task is a verifiable job — set up a service, fix a bug, migrate a datastore, recover from an incident. The model sees the complete task, the public verifier source, the environment, and the timing rule. All tools are disabled, and it must commit to a single estimate in minutes. Only then are the tools released in the same session, and the agent runs until the verifier confirms the outcome. The sealed estimate is compared against the measured wall-clock duration.
| Measures | How accurately an agent predicts its own wall-clock completion time for verifiable computer work, before starting |
|---|---|
| Unit | One task (instructions, disclosed starting workspace, public deterministic verifier), one model, one rollout in a fresh container |
| Metrics | prediction_ratio_actual_over_estimate · median_absolute_error_minutes · underestimation_rate · within_2x_rate |
| Judge | A public, inference-free deterministic verifier polled from an independent container. No LLM grades anything |
| Tracks | Service setup · debugging · migration · performance · ops & recovery · comprehension · frontend (200 tasks) |
What it measures
- The gap between the agent's pre-work time estimate and the observed actual duration
- Whether the estimate is optimistic (underestimate) or conservative (overestimate)
- How the gap changes across task tracks — the kind of work an agent performs, not just its difficulty
Protocol
- Fresh environment. Each rollout starts in a clean
node:20-bookwormcontainer with 2 CPUs and 2 GiB RAM. - Full disclosure. The model sees the complete task, the verifier source, the environment, and the timing rule before estimating.
- Sealed estimate. All tools are disabled. The model must commit to a single estimate in minutes, which is sealed to disk raw and parsed.
- Release. The same session continues with coding tools enabled, and the agent works until the verifier confirms success.
- Independent polling. A separate verifier container on the same Docker network polls for the first observed successful outcome.
- Clock. Duration runs from execution release to first observed success on the host monotonic clock, with UTC timestamps on every event.
- Re-verify and destroy. The outcome is verified once more after the agent exits, then the environment is torn down.
Judging
Grading is entirely deterministic: every task ships a public verifier with no inference, and the model reads it before estimating. The starting state always fails the verifier, and repaired states must pass repeatedly. Canonical solutions are withheld from the repository to avoid training contamination; their SHA-256 commitments are published in the solutions manifest.
Because polling only bounds the exact success transition, results also include the previous-failure lower bound and the observation-window width. Agent-declared completion is recorded separately from observed completion.
Caveats
- Durations are wall-clock time on a fixed 2 CPU / 2 GiB container; absolute numbers are machine-bound, but the ratio of actual to estimated is comparable across runs
- The estimate is a single sealed number — the agent gets no mid-task revision
- Tasks are deliberately simple and reliably completable; failures would muddy the core estimation result
ColophonBy @chkn_little · Researched by GPT 5.6 sol (high) and Fable · Measurement tasks generated by GPT 5.6 sol · Written by @chkn_little · Edited by Kimi 3