Frontier to PiT

Output Explorer

Every prompt in the paper, and what each model wrote back.

One magic number hidden in filler text; retrieve it.

13 of 100 prompts

#074 · needle at 44% depth#075 · needle at 21% depth#076 · needle at 88% depth#077 · needle at 18% depth#078 · needle at 87% depth#079 · needle at 68% depth#080 · needle at 58% depth#081 · needle at 40% depth#082 · needle at 99% depth#083 · needle at 28% depth#084 · needle at 53% depth#085 · needle at 42% depth#086 · needle at 61% depth

Nearby prompts. All 100 Single needle · 32K prompts

PromptRULER niah_single_1 ·32,768 tokens
A special magic number is hidden within the following text. Make sure to memorize it. I will quiz you about the number afterwards.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
[… 738 lines omitted for display …]
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
One of the special magic numbers for ambitious-reorganisation is: 6171716.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
[… 528 lines omitted for display …]
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
What is the special magic number for ambitious-reorganisation mentioned in the provided text?
Original prompt 32,233 tokens, 1,284 linesShown 18 lines around the needleNeedle at 58% depthHaystacks are built per model’s tokenizer, so the needle differs by model
Expected answer
ChronoGPT3484601Aux 20156171716Ours6171716Ours, no steer.6171716PiT-FT3484601Qwen 27B6171716
Models
4 of 4 columns · click a model to add or remove it

Ours

Found the needle

6171716.

13 characters10 tokens

Aux 2015

Found the needle

6171716.

13 characters10 tokens

PiT-FT 2015

Missed

: $1000000 for the first time, $1000000 for the second time, $1000000 for the third time, $2000000 for the fourth time, $3000000 for the fifth time, $4000000 for the sixth time, $5000000 for the seventh time, $6000000 for the eighth time, $7000000 for the ninth time, $8000000 for the tenth time, and $9000000 for the eleventh time. $9000000 for the twelfth time, $9000000 for the thirteenth time, $9000000 for the fifteenth time, $9000000 for the sixteenth

457 characters128 tokens

ChronoGPT 2015

Found the needle

3484601

8 characters3 tokens