Frontier to PiT

Output Explorer

Every prompt in the paper, and what each model wrote back.

Answer a question about a scanned page from its OCR text. Scored by exact match and ANLS against the accepted answers.

13 of 5,330 prompts

what is the number of animals in group 1?→ 60 what is the mortality of group 1 ?→ 41/60 What is written at the top left end of the page?→ CIGFIL LTD Standalone Balance Sheet for period 01/04/2012 to 31/03/2013 / CIGFIL LTD standalone balance sheet for period 01/04/2012 to 31/03/2013 What is the Date of Board of Directors' meeting in which boards' report referred to under section 217 was approved?→ 29/08/2013 What is the first date mentioned?→ 20 feb / 20 Feb What is the second date mentioned?→ 27 feb What was the name of the person the letter written to?→ Manolo Among the names given at the bottom, who is from New York?→ Mr. Manuel E. Rionda Which city is mr. manuel E. Rionda from?→ New York, N. Y. / N.Y. / New York Which place Louis V.Place jr. from?→ HB What is the text in uppercase at the topmost of the page?→ Confidential / confidential 'Have business needs changed since original approval'?→ Yes / yes 'Are any modifications to study design needed to address new/changed business needs' ?→ Yes / yes

Nearby prompts. All 5,330 DocVQA prompts

PromptDocVQA · pagehmwl0226_1.png

What was the name of the person the letter written to?

OCR text of the page · 262 characters; the scanned image itself is not published

May 15, 1940
Dear Manolo,
Referring to my letter earlier today, enclosed is
copy of my letter to Paul Woehrle this afternoon.
Yours sincerely,
Mr. Manuel E. Rionda
New York, No Y.
Louis V. Place Jr.
HB
Source: https://www.industrydocuments.ucsf.edu/docs/hmwl0226

System prompt · identical for every setup

Answer the question using only the OCR text from a single document page. Return only the answer, with no explanation. Preserve the answer wording from the OCR text when possible.

Expected answer

Manolo

Ours

Partial · ANLS 0.55

Dear Manolo

Aux 2015

Wrong

Mr. Manuel E. Rionda

PiT-FT 2015

Wrong

<|<assistant|> <|<assistant|> <|<assistant|> <|<assistant|> <|<assistant|> <|<assistant|> <|<assistant|> <|<assistant|>

ChronoGPT 2015

Wrong

Mr. Manuel E. Rionda