Visual extraction benchmark.
Compare original model responses against the eight corrected ground-truth files. Formatting and layout choices are normalized away. Inspect scores, omissions, numeric mistakes, and ordered text differences below.
Overall average F1
Note: These rankings reflect one-shot requests: each model saw the image once and returned a single response, without cropping or enhancing the image, re-examining it, or checking values against row and column totals. Scores with those steps would likely be higher, especially on dense tables, but this benchmark did not measure that.
Arithmetic mean of text content F1 and exact-number F1, with equal weights.
Text content F1
Exact content matches, balancing missing and extra text. Each input has equal weight.
Exact-number F1
Whole numeric values must match. Commas, leading zeros, and trailing decimal zeros are normalized.
Accuracy by language and numeral input group
English only: inputs 1, 3, 6, 8. English + Asian text with Western numerals: inputs 1, 2, 3, 6, 8 (the English set plus Chinese input 2). Japanese: inputs 4, 5, 7. Each input has equal weight; failed requests score zero. Date and family filters apply; this chart always uses its full groups independently of the Compare input filter.
Accuracy by input
Text content F1. Select a score to inspect that response and reference. Failed requests score zero. 100% exact matches · 98–99% · 90–97% · Below 90%
How scoring works
Content F1 compares exact token counts: Latin words, individual CJK characters, and whole numbers. Reordering paragraphs, table rows, or columns does not reduce this score. Precision measures how much returned content is correct; recall measures how much reference content was recovered. Each of the eight inputs has equal weight, including failures.
Normalization removes Markdown/HTML decoration, case differences, punctuation, whitespace, table borders, and English line-end hyphenation. Unicode width, superscripts, numeric separators/decimal padding, numeric Asian characters in quantity/date contexts, and itinerary clock styles are canonicalized. Traditional and modern CJK characters stay distinct. Unknown markers are errors.
These content scores do not verify table-cell placement or word order. The separate ordered character error rate and normalized diff expose those differences, but are diagnostics and do not affect ranking. CER = minimum character edits / reference characters; it can exceed 100% for very long incorrect output.
Only the eight corrected Markdown files are references. Predictions come from retained original responses, never edited result copies. Reference and candidate hashes, all scores, and comparison details are saved in evaluation.json.
Time taken
CLI timing includes process startup. Only successful attempts are included. Task totals sum individual requests, not concurrent run wall time. Failed-request durations are excluded. Missing measurements are unavailable, not zero.
Recorded tokens
Captured input and output tokens, including failed requests with reported usage.
Unreported failure usage is unavailable. Reasoning counts appear in the table below. They are not added again to token totals. Provider token accounting and agent environments differ.
Reported API cost
Captured provider charges in USD; unreported costs are unavailable. OpenRouter usage accounting.
Configuration details
Select a model to inspect its saved extraction. Unreadable markers count literal [?] and [illegible] occurrences; they are not an accuracy score.
| Configuration | Saved | Overall F1 | Content F1 | Number F1 | CER ↓ | Mean CLI time | Input tokens | Output tokens | Reasoning | Reported USD cost | Unreadable | Response size | Failed requests |
|---|
Astra is from the earlier prompt revision. Claude and Qwen reasoning settings use provider defaults. Agent runs have no captured duration or token measurements. Gemini duration includes its CLI turn; Claude and Qwen reported durations come from the LLM response timer.
Inspect an extraction
Highlights follow the formatting-tolerant content score. Reordered content can match; replacement suggestions use nearby text alignment.
Model extraction
Corrected ground truth
Original image
Select the image to enlarge it.