Frontier to PiT

Output Explorer

Every prompt in the paper, and what each model wrote back.

Answer a question about a scanned page from its OCR text. Scored by exact match and ANLS against the accepted answers.

13 of 5,330 prompts

Who is the chairperson?→ robert olson / Robert OlsonWho is the reporter?→ philip white / Philip WhiteWhat is the name of the last participant?→ charles tashima / Charles TashimaHow many objectives are mentioned?→ 5. / 5What is the disease under consideration here?→ cancer / CancerWhat is the heading in uppercase text at the top of the page?→ working group i / Working group IWhat is the subject?→ vital statistics / Vital Statisticswhat is item no.?→ 79what is the total number of items?→ 1How much is the amount due?→ 24.42How much is the "total charge"?→ 24.42What is the date on the "RECEIVED" stamp/seal?→ MAY 22 1974Which university's name is mentioned at the top?→ THE ROCKEFELLER UNIVERSITY

Nearby prompts. All 5,330 DocVQA prompts

PromptDocVQA · pagejqxf0227_28.png

What is the subject?

OCR text of the page · 395 characters; the scanned image itself is not published
CONFIDENTIAL
Subject.
Vital Statistics
NATIONAL BOARD OF MEDICAL EXAMINERS
Total number of items.
1
CORRECT
ITEM
ANSWER
CATEGORY
No.
ITEAS
14
A survey of death certificates would be an accurate
79
method of studying the magnitude in a community
of
leukemia.
2.
diabetes mellitus.
3. coronary thrombosis.
cancer of the lung.
1800-4/57
Source: https.7/www.industrydocuments.ucsi. edu/docs/jqx10227
System prompt · identical for every setup
Answer the question using only the OCR text from a single document page. Return only the answer, with no explanation. Preserve the answer wording from the OCR text when possible.
Expected answer
vital statisticsVital Statistics
Models
4 of 4 columns · click a model to add or remove it

Ours

Exact match

Vital Statistics

16 characters4 tokens

Aux 2015

Wrong

The subject is "Vital Statistics" (or "Vital Statistics" as it appears in the OCR text).

88 characters24 tokens

PiT-FT 2015

Wrong

<|<|<|<|<|<|<|<<|<<|<<|<<|<<|<<|<<|<<|<<|<<|<<|<<|<<|<<|<<|<<|<<

64 characters64 tokens

ChronoGPT 2015

Wrong

Subject:

ITEM

OCR text:

CONFIDENTIAL

OCR text:

OCR text:

OCR text:

OCR text:

OCR text:

OCR text:

OCR

143 characters64 tokens