17.9 percent of an unguarded agent's 'done' was false; Receipts shipped zero.
100 SWE-rebench-V2 tasks held out from development and chosen by a rule fixed in advance. NVIDIA Nemotron 3 Super on Nebius Token Factory wrote every fix; a clean Token Factory Sandbox and plain code checked it; SWE-rebench's own grader graded it again.
Primary sample: 100 held-out tasks, Nemotron 3 Super
Unguarded agent: 'done' that was false12 of 6717.9%, 95% interval 10.6% to 28.7%
Receipts: verified fixes6659 on the first verdict, 7 after one appeal
Receipts: false fixes shipped0checked by SWE-rebench's official grader
Model cost per verified fix$0.41Nemotron tokens at Token Factory's listed prices
Verified fixes require a gate PASS and official resolution. Run verdicts show the gate result; official grades are listed separately.
Token Factory reported sandbox cost over all runs: 63.741 (unit not published).
How to read this: Unguarded agent = the agent's own 'done' on its first try. Receipts = the same agent behind a witness sandbox and a model-free gate, with one appeal. Official = SWE-rebench's grader, run separately.
| Instance | Unguarded: claimed done | Unguarded: official | Receipts outcome | Receipts: official | Run | Receipt |
|---|---|---|---|---|---|---|
| rhayes777__pyautofit-888 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| deshaw__pyflyby-289 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| azure__walinuxagent-638 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| 3yourmind__django-migration-linter-258 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| openmdao__openmdao-3029 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| pyfilesystem__pyfilesystem2-462 | yes | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| pytask-dev__pytask-519 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| niconoe__pyinaturalist-72 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| robotframework__seleniumlibrary-1431 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| palantir__python-language-server-829 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| stac-utils__stac-fastapi-729 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| gitpython-developers__gitpython-1832 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| scitools__cartopy-2307 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| kubernetes-client__python-1933 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| datadog__datadogpy-555 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| callowayproject__bump-my-version-180 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| opensearch-project__opensearch-py-843 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| stac-utils__pystac-client-782 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| sdv-dev__rdt-704 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| pybamm-team__pybamm-4196 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| aspp__pelita-798 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| zopefoundation__btrees-163 | no | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| skypyproject__skypy-573 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| jnothman__upsetplot-253 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| dask__dask-5861 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| qutech__qupulse-797 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| bloomberg__memray-552 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| mondeja__mkdocs-include-markdown-plugin-51 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| more-itertools__more-itertools-892 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| yukinarit__pyserde-430 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| pycqa__pyflakes-335 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| azuread__microsoft-authentication-library-for-python-186 | no | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| natcap__invest-787 | not recorded | not recorded | infrastructure error | not recorded | Run | Receipt |
| igorbenav__fastcrud-69 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| researchobject__ro-crate-py-192 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| neogeny__tatsu-320 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| networkx__networkx-3101 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| neurodatawithoutborders__pynwb-2052 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| vacanza__holidays-2106 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| qiskit-community__ecosystem-159 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| theacodes__nox-466 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| encode__django-rest-framework-8979 | yes | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| samuelcolvin__pydantic-2806 | no | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| pgmpy__pgmpy-878 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| rochacbruno__dynaconf-191 | no | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| mirumee__ariadne-codegen-159 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| ntnu-ihb__pythonfmu-158 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| openmdao__aviary-807 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| kozea__weasyprint-2087 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| pypa__pipx-1350 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| educationaltestingservice__rsmtool-669 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| yt-dlp__yt-dlp-13361 | yes | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| twisted__pydoctor-191 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| hhursev__recipe-scrapers-1230 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| pymor__pymor-810 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| pycqa__docformatter-104 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| caikit__caikit-438 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| crewaiinc__crewai-2552 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| apluslms__a-plus-1199 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| quora__pyanalyze-688 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| sfdo-tooling__cumulusci-3624 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| aws-cloudformation__cfn-python-lint-2006 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| cs-si__eodag-380 | no | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| stac-utils__pystac-561 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| gammapy__gammapy-5342 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| wtforms__wtforms-596 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| codezonediitj__pydatastructs-163 | yes | not resolved | REJECTED | resolved | Run | Receipt |
| python__typing_extensions-157 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| ecmwf__earthkit-data-285 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| s-knibbs__dataclasses-jsonschema-93 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| enthought__traits-1534 | no | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| galileo-galilei__kedro-mlflow-585 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| elastic__curator-750 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| pyg-team__pytorch_geometric-7654 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| pymodbus-dev__pymodbus-2502 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| tqdm__tqdm-49 | no | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| msgpack__msgpack-python-331 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| pythainlp__pythainlp-1056 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| virtuslab__git-machete-179 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| rock-learning__pytransform3d-48 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| opendatacube__odc-geo-121 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| jsonpickle__jsonpickle-501 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| ethereum__web3.py-2023 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| pyfar__pyfar-58 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| robotpy__cxxheaderparser-82 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| pipxproject__pipx-680 | no | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| ericmjl__pyjanitor-728 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| dai-lab__copulas-87 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| mie-lab__trackintel-344 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| cuthbertlab__music21-1775 | not recorded | not recorded | infrastructure error | not recorded | Run | Receipt |
| pints-team__pints-1444 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| qiskit__qiskit-terra-8799 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| datalad__datalad-5603 | yes | resolved | REJECTED | resolved | Run | Receipt |
| systemd__mkosi-3346 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| maas__python-libmaas-219 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| crossgl__crosstl-202 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| chezou__tabula-py-135 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| astropy__ccdproc-867 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| pennersr__django-allauth-3447 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| ungarj__mapchete-685 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| qutip__qutip-2561 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| planetlabs__planet-client-python-81 | yes | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
Exploratory: the first 30 tasks again with Nemotron 3.5 Lightning
Unguarded agent: 'done' that was false3 of 2213.6%, 95% interval 4.7% to 33.3%
Receipts: verified fixes2320 on the first verdict, 3 after one appeal
Receipts: false fixes shipped0checked by SWE-rebench's official grader
Model cost per verified fix$0.09Nemotron tokens at Token Factory's listed prices
Verified fixes require a gate PASS and official resolution. Run verdicts show the gate result; official grades are listed separately.
Token Factory reported sandbox cost over all runs: 22.506 (unit not published).
How to read this: Unguarded agent = the agent's own 'done' on its first try. Receipts = the same agent behind a witness sandbox and a model-free gate, with one appeal. Official = SWE-rebench's grader, run separately.
| Instance | Unguarded: claimed done | Unguarded: official | Receipts outcome | Receipts: official | Run | Receipt |
|---|---|---|---|---|---|---|
| rhayes777__pyautofit-888 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| deshaw__pyflyby-289 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| azure__walinuxagent-638 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| 3yourmind__django-migration-linter-258 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| openmdao__openmdao-3029 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| pyfilesystem__pyfilesystem2-462 | no | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| pytask-dev__pytask-519 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| niconoe__pyinaturalist-72 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| robotframework__seleniumlibrary-1431 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| palantir__python-language-server-829 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| stac-utils__stac-fastapi-729 | no | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| gitpython-developers__gitpython-1832 | no | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| scitools__cartopy-2307 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| kubernetes-client__python-1933 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| datadog__datadogpy-555 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| callowayproject__bump-my-version-180 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| opensearch-project__opensearch-py-843 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| stac-utils__pystac-client-782 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| sdv-dev__rdt-704 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| pybamm-team__pybamm-4196 | no | not resolved | VERIFIED on appeal | resolved | Run | Receipt |
| aspp__pelita-798 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| zopefoundation__btrees-163 | no | not resolved | REJECTED | not resolved | Run | Receipt |
| skypyproject__skypy-573 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| jnothman__upsetplot-253 | yes | not resolved | REJECTED | not resolved | Run | Receipt |
| dask__dask-5861 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| qutech__qupulse-797 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| bloomberg__memray-552 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| mondeja__mkdocs-include-markdown-plugin-51 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| more-itertools__more-itertools-892 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |
| yukinarit__pyserde-430 | yes | resolved | VERIFIED on first verdict | resolved | Run | Receipt |