← NM AI Research · research portfolio
An open-weight deployment gets token-level uncertainty for free, and it is the usual answer to “how would you know the model got it wrong?”. Set the operating point yourself and watch what it catches. The task is numeric extraction from SEC compensation tables, scored against each filer’s own XBRL tags, so a wrong answer is a specific wrong figure rather than a matter of opinion.
Interactive front-end over the frozen v1.0 dataset (Section 5) · canonical record: DOI 10.5281/zenodo.21543579 · code & data · NM AI Research, ORCID 0009-0003-4213-7769 · CC BY 4.0
The threshold is fitted per model on that model’s own correct answers, which is the most favourable case for the detector: it is allowed to calibrate on the answer key. Every rate is shown as a count over its denominator.
| model | accuracy | misattributions | threshold H | caught | catch rate | false alarms | AUC |
|---|
One row per model, one mark per model call, positioned by entropy. The axis is logarithmic because the values span seven orders of magnitude, which is itself the finding below. If entropy detected misattribution, the red marks would sit to the right of the line and the blue ones to the left.
Fit the threshold on the model in the row, then apply that same number to the model in the column, and read off how many of its correct answers get flagged. The outlined diagonal is your chosen budget by construction. Every other cell is what happens if you carry a calibrated threshold across to another model.
What this shows. Top-20 truncated token entropy, the uncertainty signal an open-weight deployment gets without extra work, tested as a detector for misattribution: the model returning a real figure from the wrong row or column. It is at chance where the errors concentrate (AUC 0.490 and 0.500 on the two models carrying 46 and 26 errors), and it separates only where errors are already rare. Move the budget to its maximum and watch what still is not caught on the models you would actually deploy.
Why the convention toggle is here. The aggregate catch rate is convention-dependent, swinging 8% to 20% across percentile methods and pooling because it is dominated by one low-accuracy model. The per-model result does not swing. That is why the paper leads with the per-model figure, and the toggle is exposed so you can reproduce the swing rather than take it on trust. Every aggregate on this page is computed per model and then summed, at the operating point you set, so the tile is a count of what your own threshold would have caught rather than a pooled figure.
What it does not show. This is one signal (entropy) on one task (numeric extraction) against one ground truth (the filer’s own XBRL tags), on nine small open-weight models run locally. It is not a claim that no uncertainty method works, and semantic-entropy and self-consistency methods are discussed and cited in the paper rather than tested here. The two interventions that did raise accuracy, model capacity and majority voting over presentations, are black-box and need no weights, which is the paper’s point.
Denominators. 594 model calls, 9 models over 2 decoding regimes. Counts, not percentages, are the primary display throughout; where an N is small the count makes that visible.
Reproducing this. The page is generated by build.py from rescored.json, both in the repository and on the Zenodo record. Python 3 standard library, no dependencies. The build recomputes the paper’s Section 5 table and refuses to write this page if the two disagree.
Conflict of interest. Drafting was assisted by a closed frontier model made by Anthropic, the comparator class this work argues against. Every empirical claim here comes from local runs on pinned open-weight models and is reproducible from the bundle without the author’s cooperation. Stated in full in Section 12 of the paper.
The paper. This is a front-end to Section 5 of The Model Is a Dependency: Testing the Audit Case for Open Weights in Finance, NM AI Research, version 1.0, July 2026, 10.5281/zenodo.21543579. The paper covers the provenance case, the deployment denominator and the structure-versus-scale result as well; this page takes only the detector test.
NM AI Research · ORCID 0009-0003-4213-7769 · CC BY 4.0 · independent research, not investment advice.