Based on repository activity, growth velocity and community engagement.
24
Growth2/30
Activity11/25
Popularity3/25
Trust8/20
5
Stars
High
Sentiment
Votes
5
README.md
Benchmarking stealth/ox-alpha
Model to benchmark: stealth/ox-alpha from openrouter
Work Log
0. Summary
ox-alpha resolves 58.4% of DeepSWE (66/113) — pier's own aggregate metric. But 80% of tasks landed ≥90% of
their fail-to-pass tests, and 9.7% of the benchmark was lost to tool-call format
failures rather than reasoning — see Final results.
Self-described as "a reasoning model designed for coding, sustained agentic work,
and production workloads."
Reasoning is mandatory.reasoning.mandatory: true, default_enabled: true,
default_effort: "max", supported efforts ["max", "high", "low"]. It cannot be
disabled — only turned down. This is the main wall-clock risk for a 113-task
agentic sweep; the knob if trajectories run long is
--ak reasoning_effort=<low|high|max> on the mini-swe-agent adapter.
Caveat on confidentiality: stealth models on OpenRouter are free because the
provider logs prompts for evaluation. This run is not confidential.
Slug resolution
bench.sh needs the provider-prefixed form openrouter/stealth/ox-alpha.
Verified it resolves through the real code path rather than assuming:
litellm get_llm_provider -> ('stealth/ox-alpha', 'openrouter', None, None)
pier api key var names -> ['OPENROUTER_API_KEY']
OPENROUTER_API_KEY is present in .env and is forwarded into the sandbox by
pier/agents/installed/mini_swe_agent.py:853.
Live smoke call
A minimal completion through litellm, exactly as mini-swe-agent issues it:
Note: 34 completion tokens to emit 4 characters, and the provider reported
reasoning_tokens: 0 despite reasoning content being present — provider-side
token accounting is unreliable for this model, so don't trust its token
counts for cost or effort analysis.
2. Task set
113 task directories in ../deep-swe/tasks. (An earlier count of 117 was wrong —
ls | wc -l also counted dataset.toml, manifest.json, manifest.schema.json
and a README.md symlink.)
so the per-task Dockerfile (clone repo → pip install -r requirements.txt)
never executes locally. That removes the single biggest source of flakiness in a
local run.
Scoring is fail-to-pass / pass-to-pass; reward is 1 only on a full f2p sweep.
3. Choosing the backend
Why not daytona
Daytona caps sandboxes at 10 GB, but tasks request 20 GB. Running there needs
--override-storage-mb 10240, and pier itself warns this "could disqualify you
from leaderboard submissions." Any daytona number carries that asterisk.
Why docker
Docker does not enforce storage_mb, so it honors the task spec as declared
(cpus=2, memory_mb=8192, storage untouched) with no override and no
warning. It is the more faithful of the two backends here.
Docker probe
(The probe job dirs and the single-task smoke run are not kept in the repo —
only the full 113-task run is. Their findings are recorded below.)
Ran the exact docker analogue of the pre-existing daytona probe, using
the nop agent (zero model calls, zero cost), same task via --sample-seed 0 --n-tasks 1.
nop changes nothing, so reward: 0 and f2p 0/108 are the expected result;
the meaningful signal is p2p 1404/1404 — image pulled, container ran, collect
hook fired, verifier executed. The f2p failures are the genuine
AttributeError: 'Response' object has no attribute 'iter_json', i.e. the
unimplemented feature, not a harness fault.
Timings:
| phase | docker | daytona |
|---|---|---|
| environment_setup | 48.00 s | 1.44 s |
| verifier | 22.54 s | 19.49 s |
| trial total | 81.03 s | 23.24 s |
The 47 s gap is entirely the cold image pull; daytona started from a prewarmed
snapshot. Verifier time — the part that actually scales across 113 tasks — is at
parity.
Disk: a projection that had to be corrected
The probe measured +4.11 GB for the first task image and that extrapolates to
113 × 4.1 ≈ 460 GB, which would have meant pruning between tasks. That is
wrong: it ignores layer sharing. Manifest comparison showed 23 of 27 layers are
common (0.75 GB compressed shared base), with only ~0.03–0.09 GB unique per task.
Measured directly by pulling three more task images:
| image | disk delta | pull time |
|---|---|---|
| 1st (probe) | +4.11 GB | 48 s |
| 2nd | +0.30 GB | 2.7 s |
| 3rd | +0.00 GB | 0 s |
| 4th | +0.13 GB | 2 s |
Real projection for all 113: ≈20–25 GB against 113 GB free. No pruning, no
batching. The 48 s cold-pull cost is likewise one-time, not per-task.
What the nop probe could not test
With nop there are no model calls, so pier never created the egress-proxy
sidecar — no docker-compose-egress-proxy.json, no egress-proxy/. The
OpenRouter-through-proxy path inside a no-network container stayed unverified
until the smoke run below.
4. Harness fixes (bench.sh)
Reviewed the wrapper before trusting it; six issues found and fixed, all verified
against the installed pier source rather than assumed. Committed as 735ba19.
Job-name collisions. pier compares the wholeJobConfig for equality and
raises FileExistsError on any mismatch (pier/job.py:196). Several
config-affecting inputs were absent from the job name — concurrency, storage
override, task path, and any -- <pier args> passthrough — so two
incompatible runs would claim one name and the second would die. Added an
8-char config_hash() suffix over exactly those inputs.
--tasks N was not a random sample. pier only shuffles when a seed is set
(pier/models/job/config.py:135); otherwise --tasks N takes the
alphabetically-first N. Seed now defaults to 0 and is always passed, and it
appears in the job name for full runs too.
status crashed on an empty jobs dir — unmatched glob stayed literal and
exited 1. Now prints (no jobs yet).
Misleading resume docs. Plain resume only reruns trials with no
result.json; an errored trial has one, so retrying errors needs
--drop <ErrorType> (pier/job.py:208).
Daytona credential guard rejected DAYTONA_JWT_TOKEN +
DAYTONA_ORGANIZATION_ID despite its own error message offering them.
DAYTONA_STORAGE_MB='' didn't disable the override — :- treats empty as
unset, so the documented escape hatch silently fell back to 10240. Now -.
Also added .gitignore for .env (live keys) and jobs/ (run output).
5. Smoke run — closing the egress gap
Before committing to 113 tasks, ran a single-task run with the real agent and
real model (--tasks 1 --seed 0, same httpx task as both probes) to exercise the
one path nop could not.
The egress path works. With a real agent, pier created the sidecar that was
absent from the nop probe — egress-proxy/ and docker-compose-egress-proxy.json
— and the squid container came up healthy. Its ACL is exactly right:
acl allowed_domains dstdomain "/tmp/allowed_domains.txt" # openrouter.ai only
http_access allow authenticated allowed_domains
http_access deny all
So the sandbox stays no-network for everything except the model endpoint.
Zero 429s, zero auth failures, zero proxy errors across the whole run.
The harness is fully validated — 1404/1404 p2p means no regressions, and 107
of 108 previously-failing tests now pass, so the agent genuinely implemented the
feature.
reward: 0 despite 107/108 is correct, not a bug. Scoring is all-or-nothing:
reward is 1 only on a complete f2p sweep. The single miss was
tests.test_json_stream.test_iter_json_ndjson_invalid_line_raises_and_closes_streaming_response
assert False, where False = <Response [200 OK]>.is_closed
i.e. ox-alpha implemented iter_json() correctly but failed to close the
response when an invalid NDJSON line raises mid-stream. A real edge-case miss on
the model's part, not a harness fault — which is exactly what a benchmark should
be measuring.
Worth noting for interpretation: this is a strict metric. A model can be one
assertion away from a solve and still score 0. partial in rewards (0.99 here)
carries the finer-grained signal.
False alarm worth recording
An early monitor fired on ProxyError, which looked like the egress proxy
failing. It was not — ProxyError is a class name in httpx's _exceptions.py,
which the agent had just cat'd. Zero occurrences in trial.log. Log-grep
monitors over an agent that reads source code will match the source, not the
runtime.
6. Full run
Launched all 113 tasks:
CONCURRENCY=4 ./bench.sh run openrouter/stealth/ox-alpha -- -r 2
job: stealth-ox-alpha__full-s0__docker__f5abcdff
| setting | value | rationale |
|---|---|---|
| backend | docker | honors task spec; no storage override |
| -n (concurrency) | 4 | 8 cores / 2 CPUs per task. Memory is not the binding constraint — the 8192 MB is a cap and actual use is ~1 GB — but CPU over-subscription risks making timing-sensitive tests flaky and corrupting p2p |
| -r 2 | 2 retries | pier defaults to 0, so a single API blip would error a trial over a multi-hour run. pier's default excludes already cover AgentTimeoutError and the verifier errors, so retries hit only transient infra faults |
| --sample-seed 0 | all 113 | seed affects order only when no --n-tasks is set |
Projected wall clock — initial estimate was wrong. The smoke task took
19 m 20 s, which at concurrency 4 suggested 113 / 4 x 19.3 min ≈ 9 hours.
Measured throughput over the first 12 tasks is ~6.2 tasks/hour, not the
~12.6 that implies: real tasks average closer to 38 min, so the smoke task
was roughly 2x faster than typical. Revised estimate: ≈17 hours.
Considered raising concurrency to recover the difference and rejected it.
n_concurrent_trials is part of JobConfig, so changing it makes pier refuse to
resume — the completed trials would have to be abandoned for a new job. More
importantly, 8 concurrent x 2 CPUs on an 8-core host over-subscribes during the
verification phase, risking flaky timing-sensitive tests and corrupted p2p
numbers. Benchmark validity beats wall clock; staying at 4.
Risk being tracked
The smoke task consumed 3.55 M input tokens for one task (74 steps, context
growing each step). Extrapolated across 113 tasks that is on the order of 400 M
input tokens. The key is on a paid account (is_free_tier: false, no credit
cap), but OpenRouter applies a daily request cap to zero-priced models. If that
is hit mid-sweep it will surface as errored trials; recovery is
once the quota resets. The run is resumable by design, so a cap costs time, not
results.
Incident: disk exhaustion (caught and fixed mid-run)
About 45 min in, free disk had fallen from 113 GB to 89 GB after only ~11 tasks —
a burn rate of ~2.2 GB/task that projects to ~230 GB for the remaining 104,
against 89 GB available. The run would have died around task ~50.
Diagnosis took two wrong turns worth recording:
docker system df reported 88.97 GB of build cache, which looked like the
culprit. It was not — docker builder prune -f reclaimed 0 B, because
that cache is almost entirely shared with image layers (SHARED: true), not
additive. Only 3.3 GB was genuinely reclaimable. The headline number in
docker system df double-counts shared storage.
The per-trial derived images looked like the leak — each trial leaves a
<task>__<trialid>-main:latest at a reported ~5 GB. But reported size also
includes shared layers; deleting one actually reclaims ~620 MB (plus
302 MB for its paired -pier-egress-proxy image).
So the real cost is ~0.9 GB of genuinely unique storage per trial, left behind
after the trial finishes. pier deletes the containers (--delete default) but
not these per-trial images.
Note this also corrects the earlier §3 projection of "≈20–25 GB, no pruning
needed". That figure was measured by pulling four task images sampled
alphabetically, which shared far more layers than a random 113 do — different
repos mean different pip install layers. Pruning is required.
Fix: a janitor loop that removes -main and -pier-egress-proxy images for
finished trials every 4 minutes. It keys off the existence of the trial's
result.json, notdocker ps, so a trial sitting between its agent and
verifier phases (briefly no running container) is never touched. Image names are
lowercased and truncated to 32 chars, so matching is done on the unique trial-id
suffix.
First pass removed 11 images and took free space from 89 GB back to 92 GB.
Disk now trends up rather than down.
So 20 of 22 tasks landed ≥90% of their fail-to-pass tests, and only 2 are
real failures. The headline resolve rate of 68% and the mean partial of ~0.94
are telling substantially different stories about the same runs. Both belong in
the final report.
Note the task set is genuinely multi-language — Go (go-git), TypeScript
(ts-pattern, superjson, kysely, koota, yjs, ofetch), and Python
(httpx, fastapi, cattrs, mashumaro) all appear — so this is not a
Python-only measurement.
Agent effort per task: 51 steps minimum, 101 median, 148 maximum. The 74-step
smoke task was below median, which is why it under-predicted wall clock.
Finding: ox-alpha's tool-call reliability is a measurable weakness
Two trials scored 0 without doing any work — pwntools-tube-multiplexing
(0/73 f2p, 14 steps) and go-genai-streamed-function-args (0/6, 66 steps).
Both ended the same way:
Tool call error: <error> No tool calls found in the response.
Every response MUST include at least one tool call. </error>
... (x3 consecutive)
exit: RepeatedFormatError
ox-alpha intermittently returns a response containing no tool call at all.
mini-swe-agent re-prompts with format guidance, but three consecutive failures
abort the episode.
Measured across the first 28 trials:
| | count | share |
|---|---|---|
| trials with ≥1 no-tool-call response | 13 | 46% |
| ...that recovered and still solved the task | 8 | |
| ...that died with RepeatedFormatError (scored 0) | 2 | 7% |
This is not being "fixed", and that is deliberate. pier launches
mini-swe-agent with its stock mini.yaml, overriding only agent.cost_limit
and model.model_class — the format-error threshold is the unmodified upstream
default. Raising it to rescue those two trials would mean running a
non-standard harness and producing numbers not comparable to anyone else's.
More importantly, the failure is real. An agentic benchmark is partly a
measurement of whether a model can reliably drive a tool loop, and ~46% of
episodes hitting at least one malformed turn is a genuine characteristic of this
model, not a harness artifact. Suppressing it would flatter the model and
misreport the result.
It does mean the headline resolve rate carries a ~7% drag from format failures
rather than reasoning failures. That decomposition belongs in the final numbers,
which is why it is recorded here rather than papered over.
Caveat on recovery:exception_info is null for these trials — pier counts
them as completed with reward 0, not as errors. So ./bench.sh resume --drop
will not retry them. They are final.
Typically only one trial is CPU-bound (saturating its 2-CPU limit) while the
others idle waiting on the model API, and memory is nowhere near its cap. So
there is real headroom — concurrency 6 would fit comfortably on this hardware
and is the recommended setting for a future run.
It was not raised mid-run, because n_concurrent_trials is part of JobConfig
and pier refuses to resume a job whose config changed. Raising it means a new
job that re-runs all 113 from zero, discarding completed work:
| path | time to finish (from the 47-task mark) |
|---|---|
| stay at 4 | 66 remaining ÷ 6.4/hr ≈ 10.3 h |
| restart at 6 | 113 tasks ÷ ~9.6/hr ≈ 11.8 h |
Restarting is strictly worse, before even counting the risk that CPU contention
during verification makes timing-sensitive tests flaky and corrupts p2p. A second
parallel job over the not-yet-started tasks was also considered and rejected —
the main job's task list is fixed at 113, so it would re-run them anyway.
Note on disk variance
Task image sizes vary far more than the early sample suggested: 4.5 GB to
10.8 GB (boa-hierarchical-evaluation-cancellation is the largest,
prometheus-typed-label-sorting 7.76 GB). Free space therefore swings with
which tasks happen to be in flight, not with accumulation — a dip to 86 GB
recovered to 105 GB once the large images were reclaimed. This is why the early
"≈20–25 GB total" projection was wrong in both directions.
Status: COMPLETE
All 113 trials finished. n_running_trials: 0, n_pending_trials: 0.
7. Final results
Job:stealth-ox-alpha__full-s0__docker__f5abcdffWall clock: 20 h 39 m 38 s (2026-08-21 08:12:57 → 2026-08-22 04:52:36)
reward is 1 only when every f2p test passes and no p2p test regresses.
Decomposed:
| bucket | count | share |
|---|---|---|
| solved (reward 1) | 66 | 58.4% |
| ...of which solved at the timeout cutoff | 1 | |
| regression-only — full f2p sweep, but broke a p2p test | 2 | 1.8% |
| near-miss — ≥90% of f2p, incomplete | 22 | 19.5% |
| genuine failure — <90% of f2p | 23 | 20.4% |
80% of tasks (90/113) landed ≥90% of their fail-to-pass tests. Ten of the
near-misses failed on a single test.
The two regression-only cases are worth calling out separately, because they are
not "almost solved" in the same sense — the feature worked, something else broke:
It exceeded the 5400 s agent budget, but the repo state at the moment it was
killed passed every test. So it solved the task and never emitted a clean
termination.
pier itself settles this. Its job-level aggregate reports:
reward: 0.584070796460177 -> x 113 = exactly 66
n_errors: 5 (separate metadata, not deducted)
pier averages reward across all 113 trials and counts this one as a solve,
reporting the error count independently. 66/113 = 58.4% is therefore the
canonical, leaderboard-comparable number and is used as the headline.
On the merits this is also right: the agent was killed at 5400 s, so nothing
after the cutoff contributed. The verifier ran against a repo state produced
entirely within budget and it passed 10/10 f2p and 209/209 p2p. The task was
solved inside the allowed time; only the termination signal was missing.
Penalizing it would score the exit protocol, not the work.
The 65 came from a bug in bench.sh's own status helper, which hit
continue on exception_info before ever reading the reward — undercounting any
errored-but-passing trial. Fixed; ./bench.sh status now reports 58.4% and
agrees with pier.
Failure-mode breakdown
Not all 47 non-solves are the same kind of failure:
| mode | count | what happened |
|---|---|---|
| reasoning / incomplete | 31 | got some or most f2p tests, missed the rest |
| AgentTimeoutError | 5 | exceeded the task's own 5400 s (90 min) agent budget |
| RepeatedFormatError | 11 | episode aborted after 3 consecutive no-tool-call responses |
The 5 timeouts were all long-horizon tasks (165–265 steps): eicrud-keyset-pagination-cursor,
scriggo-method-declarations, opa-template-string-reconstruction,
valibot-recursive-schema-composition, oxvg-structural-selector-preservation.
pier's default exclude_exceptions correctly kept -r 2 from retrying them —
retrying a timeout would hand the model wall clock the benchmark does not grant.
The 11 RepeatedFormatError trials are the most actionable finding. That is
9.7% of the whole benchmark lost to ox-alpha emitting responses containing no
tool call, three times in a row, rather than to any reasoning deficit. Left
unmitigated deliberately — pier runs mini-swe-agent's stock mini.yaml, and
tool-call reliability is a legitimate thing for an agentic benchmark to measure.
But it means the 58.4% headline understates the model's reasoning ability while
correctly reporting its agentic ability.
Effort and cost
| metric | value |
|---|---|
| Agent steps | 12,773 total; min 4, median 104, max 265 |
| Input tokens | 1,094.8 M |
| Cached tokens | 1,052.6 M (96.1% cache hit) |
| Output tokens | 7.0 M |
| Cost | $0 (zero-priced stealth model) |
The 96.1% cache hit rate is what makes a 20-hour agentic sweep tractable; without
prompt caching the input volume would be prohibitive. Note the earlier estimate
of "~400 M input tokens" was low by 2.7x.
Reliability of the harness
Across 20 h 39 m and 113 trials: no rate limiting, no auth failures, no proxy
failures, no OOM, no disk exhaustion, no leaked containers. The only 5 errored
trials were the model exceeding its own task budget. Every one of the 113 tasks
produced a verdict.
Caveats on these numbers
Single attempt per task (n_attempts: 1). Given the intermittent
tool-call failures, per-task outcomes carry real run-to-run variance. A
-k 3 run would separate model stochasticity from task difficulty.
The 95% CI [49.2%, 67.1%] is for generalizing beyond this suite. The
58.4% itself is exact — all 113 tasks ran, nothing was sampled.
Not confidential. Stealth models on OpenRouter are free because the
provider logs prompts.
Reasoning effort was left at the model's default (max), which is also
its most expensive setting in wall clock. A --ak reasoning_effort=low run
would likely trade some accuracy for far fewer timeouts.
Recommendations for a rerun
CONCURRENCY=6 — measured headroom (load 3.90/8, memory ~1 GB/container against
an 8 GB cap) says 4 was conservative.
Keep the disk janitor; it is required, not optional.
Consider -k 3 for tighter per-task estimates.
Ecosystem Role
Standard MoltPulse indexed agent.
Embed Badge
Show off your Pulse Score in your GitHub README to build trust and rank higher.