20 Oct 2026 One real question from our 60, sent to Claude Sonnet 5.5 two ways
who plays captain phasma in star wars the force awakens
The twist. None of the 8 snippets named her. The top one said only "Phasma is the commander of the First Order's force of stormtroopers" .
Quick was right anyway , from what the model already knew. Deep cost 15 times as much for the same answer.
eval/results/quick.jsonl + deep.jsonl id nq07 (3.15 s $0.00049; 7.46 s $0.00747) · snippets: eval/set.jsonl nq07
One question from this morning (20 Oct): who plays Captain Phasma. Two ways to answer. Quick reads the short text under each search result, the snippets, and answers. Deep opens pages and searches again. These are the real seconds.
Press. The twist: none of the eight snippets named the actress. Press. Quick got it from memory, in about three seconds. Deep paid fifteen times as much for the same answer.
One of four talks
Four talks, one question: how should agents search?
A Search API Battle Royale Same questions, different sources, measured 20 Oct 2026.
B System 1 vs System 2 search Quick search, deep research, and when to route between them.
C Self improving RAG A research loop that learns from its own mistakes.
D Agent native search platforms What search looks like when the user is an agent.
Four talks share one question; this one is lit. Each stands alone. Together they cover where the facts come from, how hard to search, how a research pipeline learns, and what search built for agents looks like.
Claude Code Build Night Bright Data SF · 20 Oct 2026 · Ray Zahid, Immersive Commons
System 1 vs System 2 search
Quick answers, deep research, and how to choose between them.
eval/results/summary_strict.json (60 questions, both lanes, two routers) · stage/research/s1s2/mined.json
I am Ray. I run Immersive Commons, an AI builder floor in Frontier Tower.
This talk is one question: when your agent needs a fact, should it answer fast from the search results, or slow down and read? We measured it this morning, 20 Oct, on sixty questions. Then we tested tools that pick the right way for each question.
The question of this talk
When should an agent stop and read ?
Deep was right more often. It also cost about 13 times as much. So the real question is: which one, for each question?
eval/results/summary_strict.json · deep 44/60 vs quick 39/60 · $0.942 vs $0.074 for 60 questions
The whole talk in one line. Reading pages helps sometimes. It always costs.
Press. Deep research got five more questions right out of sixty, and cost about thirteen times as much. So the useful question is not which lane wins. It is which way to answer each question, chosen before you pay.
1
Part 1 · Foundations
Fast, slow, and what a search costs
Two ways to think, two ways to search, and how to pick one.
Foundations first, slowly. Where the names System 1 and System 2 come from, what a search actually costs an agent, the difference between a snippet and a page, and what a router is.
1.1 Daniel Kahneman, Thinking, Fast and Slow (2011)
Two ways to think , two ways to search
System 1 · fast
Glance and answer
in people intuitive, automatic, cheap in a search agent read the search snippets, answer at once, no tools risk confident and wrong, with no warning System 2 · slow
Stop and work it out
in people deliberate, effortful, expensive in a search agent open pages, search again, then answer risk pays every time, even when the fast answer was fine People switch without noticing. An agent needs a rule: a router .
Kahneman's two systems. System 1 is the glance: fast, automatic, usually fine. System 2 is when you stop and work it out.
For a search agent, System 1 is answering from the snippets. System 2 is opening the pages. Press. You switch between them without thinking. An agent has to be told when. That rule is the router.
1.2 What one search costs an agent · medians over our 60 questions
A search costs an agent in four ways
1,865 Text to read characters in the quick prompt: question plus 8 snippets. One web page alone is about 16,000.
4.5 s Seconds quick, at the median. Deep: 11.0 s . A user, or the next step of a plan, waits for it.
$0.0006 Dollars quick, per question. Deep: $0.0088 . Small, until it runs a million times.
2 to 13 Rate limits steps per deep answer (43 of 60 took 2). Each page fetch counts against a site's request limit.
Deep costs about 13 times as much per answer. Is it 13 times as right ?
research/s1s2/ext_summary.json foundations (prompt chars, cost medians, deep_turns) + s15_page · summary_strict.json median_s, total_cost_usd ($0.942 / $0.074)
What does a search cost an agent? Four things. Text: the quick prompt is under two thousand characters; one page is about sixteen thousand. Seconds: four and a half versus eleven at the median.
Dollars: fractions of a cent, which add up at scale. And rate limits: a deep answer takes two to thirteen steps, and every fetch counts against somebody's quota.
Press. Add it up: deep costs about thirteen times as much. Is it thirteen times as right? That is the rest of the talk.
1.3 Snippet vs full page · median characters, drawn to scale
A snippet shows a tiny part of the page
One snippetthe lines under a link
154characters
What quick readsquestion + all 8 snippets
1,865characters
One pagethe top Wikipedia article
16,293characters
Snippet "Phasma is the commander of the First Order's force of stormtroopers"
Page The full Captain Phasma article, which names the actress. Quick never saw it.
research/s1s2/ext_summary.json foundations.snippet_chars_median, quick_prompt_chars_median, s15_page.median_page_chars · eval/set.jsonl nq07
To scale. One snippet is about a hundred and fifty characters. All eight together, with the question, under two thousand. One page is sixteen thousand.
Press. Here is the Phasma snippet: it tells you who Phasma is, not who plays her. Press. The page has the name. The quick lane never sees the page.
1.4 Terms
Four terms we use from here on
Router
A small, fast judge that picks the lane.
Reads the question and snippets, returns a score from 0 to 1.
Phasma scored 0.06: go deep.
Threshold
The score where you switch lanes.
At or above it: quick. Below it: deep. We tried 0.3, 0.5, 0.7.
At 0.5, one router sent 72% deep.
AUC
How well a score puts right answers above wrong ones.
Pick one right and one wrong case at random: the chance the right one scores higher.
0.5 is a coin flip. 1.0 is perfect.
Perfect router
Knows in advance if quick will be right.
Quick when quick was right, else deep. Nobody has one: it is the ceiling.
Shows how much routing could win.
content/s1s2.md section 2 · eval/results/route_jev.jsonl nq07 (0.06) · summary_strict.json deep_share 0.717 at 0.5
Four words, one per press. A router is a small judge that picks the lane. The threshold is the cut off score. AUC says how well a score puts the right cases above the wrong ones: point five is a coin flip, one is perfect.
And the perfect router: imagine knowing in advance whether quick will be right. Nobody can. It tells us the size of the prize.
1.5 The system we tested · same search for every lane
Two lanes and a router
Question one of 60 Search 8 results, saved once Router quick enough? score ≥ 0.5
Lane 1 · Quick 8 snippets · 1 step
4.5 s · $0.0012 score < 0.5
Lane 2 · Deep opens pages, searches again, up to 12 steps
11.0 s · $0.0157 Answer graded by a rule
content/s1s2.md section 3 · eval/run_claude.py lines 29 to 45 (lanes) · summary_strict.json median_s, total_cost_usd / 60
The system. A question comes in, one search runs, eight results. Press. The router reads them and gives a score. Press. High score: lane one, quick. Claude reads the snippets and answers, one step, no tools.
Press. Low score: lane two, deep. Claude opens pages and searches again, up to twelve steps. The box is longer because it takes longer. Press. Either way, one rule grades the answer.
1.6 Watch the split · each lane moves at its measured median
The router decides where the money goes
eval/results/summary_strict.json deep_share, total_cost_usd, correct (Hybrid jev @ 0.5; Oracle router) · lane speed = median_s
Each bar is a question. Quick crosses in four and a half seconds, deep in eleven. Our hosted router, Jev, sent seventy two percent deep. Forty four right, seventy nine cents for sixty questions.
Press. The perfect router sends only thirty five percent deep. Forty eight right, thirty two cents. Same lanes. Only the split changed.
2
Part 2 · Method
How we kept the test fair
We kept the questions, search, model and grader the same for every test.
How we measured. The trick is to hold everything still except the one thing you are testing: the lane.
2.1 The run · morning of 20 Oct 2026
Test setup and scoring
60 questions 30 one-fact and 30 two-step questions, each with a known right answer. From Natural Questions and HotpotQA.
1 saved search Wikipedia, 8 results, saved once. Every lane and router saw the same results.
1 model Claude Sonnet 5.5 in both lanes, through claude -p.
2 lanes Quick: snippets only, no tools, 1 step. Deep: opens pages and searches again, up to 12 steps.
2 routers Jev: a hosted service, about 0.1 s. Laya: runs on our own machine, about 3 s. Both switch at 0.5.
v2 grader A rule, not a model: is the right answer in a reply of 40 words or fewer? We fixed it after checking by hand.
eval/set.jsonl · eval/run_claude.py · eval/route.py · eval/grade.py MAX_WORDS · mined.json auc_jev.median_ms 102, auc_laya.median_ms 2973
Sixty questions: half need one fact, half need two steps. One search, saved once, so every system saw the same eight results. One model, Sonnet five point five, in both lanes.
Press. Two lanes, two routers: Jev, a hosted scoring service, and Laya, which we ran on our own machine. Press. And a grader that is a rule, not a model. We fixed it once, after reading its mistakes. That story comes later.
3
Part 3 · Results
What each system got right, and what it cost
Six ways to answer the same 60 questions.
Results. Three numbers for every system: how many it got right, how long it took, what it cost.
3.1 60 questions · routers switch at 0.5
Deep is right more often. A perfect router beats both.
system
right, of 60
median seconds
$ for 60
sent deep
QuickSystem 1 · snippets only
39
4.5 s
$0.07
0%
System 1.5snippets + the top page
39
4.1 s
≤ $1.27
reads 1 page
DeepSystem 2 · opens pages
44
11.0 s
$0.94
100%
Routed · Jevhosted router
44
9.9 s
$0.79
72%
Routed · Layarouter on our own machine
45
10.6 s
$0.58
58%
Routed · selfClaude rates itself, keeps at 0.7+
45
5.6 s
≤ $1.01
37%
Perfect routerthe ceiling, not a real system
48
4.9 s
$0.32
35%
Deep got 5 more right out of 60. A standard test says that could be luck (p = 0.267 ).
eval/results/summary_strict.json rows, tests · System 1.5, self: research/s1s2/ext_summary.json rows (afternoon run; ≤ = upper bound, see limits)
The table. Quick: thirty nine right, four and a half seconds, seven cents. Deep: forty four, eleven seconds, ninety four cents. Reading one page, System 1.5, changed nothing in accuracy.
The two outside routers land next to deep. The highlighted row is the afternoon test: Claude rating its own answer. Forty five right at a five and a half second median. The perfect router would get forty eight. Press. And plainly: quick versus deep, p is point two seven. With sixty questions, that gap could be luck.
3.2 Up and to the left is better · median seconds per answer
Only Claude rating itself moved toward the ceiling
eval/results/summary_strict.json median_s, accuracy (hybrids @ 0.5, Oracle) · research/s1s2/ext_summary.json (System 1.5; self @ 0.7, run one call at a time)
Up is more right, left is faster. Quick and System 1.5 sit low and left. Deep sits high and far right, and both outside routers land right next to it: they send most questions deep.
The line joins the best real trade-offs: System 1.5 and the self-router. Every other system is slower and less right than a point on that line. The dashed ring is the perfect router, a ceiling no real system reaches. The amber dot is the only real router that moved toward it: Claude rating its own answer. It got deep's accuracy at close to quick's speed.
3.4 Three real questions · answers, seconds and $ from the logs
One quick win, one deep win, one double miss
nq07 · quick wins
Who plays Captain Phasma in The Force Awakens?
right answer: Gwendoline Christie
quick Gwendoline Christie 3.15 s
deep Gwendoline Christie 7.46 s
Not in any snippet. Both routers said go deep. Deep paid 15 times as much.
hp17 · only deep right
Which actor and dancer starred in the 1945 film Johnny Angel?
right answer: George Raft
quick Gene Kelly 2.55 s
deep George Raft 10.09 s
The search missed. Quick named the only dancer in the snippets and gave no sign it guessed.
hp27 · both miss
What year was the team Luke Clark plays for founded?
right answer: 1963
quick 1996 7.12 s
deep 1894 19.44 s
Two steps. Both lanes got the first step wrong. Deep spent 7 times as much and was still wrong.
eval/results/quick.jsonl, deep.jsonl, route_jev.jsonl, route_laya.jsonl ids nq07, hp17, hp27 · research/s1s2/mined.json rows
Three real questions. Phasma: quick right from memory, deep paid fifteen times more. Johnny Angel: the search missed, and quick confidently named Gene Kelly, the only dancer it saw. Deep found George Raft.
Luke Clark: a two step question. Find the club, then its founding year. Both went to the wrong club. Deep took nineteen seconds to be wrong a different way.
3.5 All 60 questions, one dot each
Mostly they agree. The router only matters for 13 .
35 both right 9 only deep right 4 only quick right 12 both wrong
For 47 questions the lane does not change the answer, so the cheap lane is the right call. Only the 9 and the 4 reward a good router.
research/s1s2/mined.json counts (both_right 35, only_deep 9, only_quick 4, both_wrong 12)
Every question as a dot. Thirty five both lanes got right. Twelve both got wrong. For those forty seven, sending the question deep buys nothing.
Press. The router only matters for thirteen: nine where only deep was right, four where only quick was. That is a small target, and it is the whole game.
4
Part 4 · The insight
We asked the routers the wrong question
They checked "are the snippets enough?" We needed "will the quick answer be right?"
Why did the routers land next to deep? Because we asked them the wrong question. And the reason is the twist from the first slide.
4.1 Was the answer in the snippets? Was quick right?
Quick is often right when the snippets lack the answer
27 of 39
of quick's right answers had no answer in the snippets . Claude used what it already knew, or worked it out.
quick right
quick wrong
Snippets hold the answer
12easy: answer in view
1
Snippets do not
27 a router that checks snippets sends these deep, for nothing
20
research/s1s2/mined.json crosstab_snippets_vs_quick (12 / 1 / 27 / 20) · counts.quick_right 39
This is the insight. Quick got thirty nine right. In twenty seven of them, the answer was not in the snippets at all. Phasma was one of them. The model already knew.
Press. When the snippets do hold the answer, quick is almost always right: twelve of thirteen. Press. But a router that asks "are the snippets enough" sends all twenty seven of these deep, and pays for nothing.
4.2 Router quality as AUC · 0.5 is a coin flip, 1.0 is perfect
Jev spots good snippets, but not right answers
research/s1s2/mined.json auc_jev (0.793 title+snippet, 0.608 quick_is_right) · auc_laya (0.435, 0.537)
The same thing in one number. Jev answers the question we asked, are the snippets enough, quite well: point seven nine. On the question routing needs, will quick be right, it drops to point six one. Only a little better than a coin flip.
Laya is close to a coin flip on both. Point four four and point five four.
4.3 Afternoon test · quick also gives its confidence, 0 to 1
The best router was Claude itself
AUC on "will the quick answer be right?"
Selfits own confidence
0.73
Layaour machine, ~3 s
0.54
0.5 coin flip
It judges its own answer , including what it knows from memory. The other routers only judged the snippets.At 0.7 or above, keep quick. Below, go deep. 45 of 60 right at a 5.6 s median. Laya's router: 45 at 10.6 s.
Every answer under 0.2 was wrong (3 of 3). But 6 wrong answers said 0.8 or more. Watch overconfidence.
research/s1s2/ext_summary.json auc_quick_will_be_right, rows (self @ 0.7), buckets_self_confidence · results/self.jsonl (re-derived: 45, 5.64 s)
This afternoon we tried the cheapest router there is. Same quick prompt, one extra line: give a confidence from zero to one. Its AUC: point seven three. Better than both outside routers.
Press. It judges its own answer, memory included. Press. Keep the answer at point seven or above, else go deep: forty five right at a five and a half second median. Press. Not clean: six wrong answers were very sure.
4.4 Each router's score per question, stacked by whether quick was right
Jev mixes right and wrong. Claude's self-rating sorts them.
Jev · "are the snippets enough?"
quick was right quick was wrong score buckets 0 to 1, left to right: go deep ← → keep quick
research/s1s2/mined.json buckets_jev · ext_summary.json buckets_self_confidence (self lane, 40 right of 60)
Left: Jev's score for each question. Solid means quick was right, dashed means wrong. Both kinds pile into the go deep bucket: nineteen right answers sent deep for nothing.
Press. Right: Claude's own confidence. Thirty of its forty right answers sit at the top. Everything at the bottom is wrong. Still six sure and wrong at the top.
4.5 System 1.5 · quick plus the top Wikipedia page, first 12,000 characters, no tools
Reading one page did not help
39 of 60 right. The same as quick. It fixed 3 questions and broke 3 others.
The top page held the right answer, in the part we sent, for only 16 of 60 .
It cost about what deep costs: about $1.0 for 60, after adjusting for high afternoon prices. Claude reads the whole page, and every word costs.
Median 4.1 s : fetching a page took only 0.29 s. Speed was fine. The page was often the wrong one.
ext_summary.json rows, s15_vs_quick (fixed nq05 hp08 hp17; broke nq04 hp00 hp03), s15_page · ~$1.0: content/s1s2.md 4h cost caveat ($1.27 raw)
We tried a middle lane: quick, plus the top page pasted in. Thirty nine right, the same as quick. It fixed three and broke three.
Press. The top page only held the answer sixteen times. Press. And it cost about as much as deep. Press. Speed was fine. Reading more of the wrong page does not help. Deep wins by searching again, not by reading longer.
4.6 A short story · a test can have bugs too
One grader rule nearly flipped the result
Grader v1 · replies over 12 words scored wrong
Grader v2 · 40 word cap, after a hand audit
Why it picked a side. Deep explains its answers: median 4 words vs 2. Right but long, like nq21: "1 January 1901 (Edmund Barton was appointed then; he won the first federal election in March 1901)" . Read what your grader marks wrong.
research/s1s2/mined.json cap12_rescore (38 / 37; deep lost nq04 nq08 nq12 nq14 nq19 nq21 nq25), answer_words · eval/grade.py MAX_WORDS
A short story. Our first grader marked any reply over twelve words wrong, so a model could not list five guesses. Press. Under that rule, deep scored below quick. The talk would have said deep research does not help.
Press. A hand audit found right answers with a short aside. We raised the cap to forty. Press. The rule had punished exactly the lane that does careful work. Read your grader's failures before you believe it.
5
Part 5 · Build it tonight
A router in Claude Code
Cheap first pass, a confidence check, escalate only when needed.
So what do you build tonight. About ten lines: answer cheap first, check confidence, and only escalate when the check says so.
5.1 The pattern · cheap first pass, confidence, escalate
Answer fast. Pay for System 2 only when needed .
answer.py def answer(q): hits = search(q) # 1 search call, e.g. Bright Data reply = claude(QUICK + hits + q, tools=[]) # System 1 text, conf = parse(reply) # "CONFIDENCE: 0 to 1" if conf < 0.7: # set from your own tests return deep(q) # System 2: WebFetch, 12 turns return text # later: add a classifier trained on "was quick right"
content/s1s2.md section 7 · research/s1s2/run_ext.py (self lane) · ext_summary.json auc_quick_will_be_right 0.73, rows self @ 0.7 · eval/run_claude.py
Press one: one search. Press two: the quick lane answers from the results, no tools. Press three: it also says how sure it is. That one line was our best router.
Press four: under point seven, hand it to deep research, not one more page. Press five: otherwise ship the cheap answer. Press six: once you know which quick answers were right, train a small model on that and use it too.
5.2 The same pattern as Claude Code subagents
One lane per subagent , the main session routes
Subagent 1
quick-answer
tools none input the question and the search results must end with CONFIDENCE: 0 to 1 Main session
the router
reads the quick answer and its confidence decides ship it, or hand it to deep logs both lanes on a sample, to learn Subagent 2
deep-research
tools WebFetch, WebSearch, the Bright Data MCP budget up to 12 steps returns an answer with the page it came from Not benchmarked tonight with the Bright Data MCP. Measure your own lanes first.
In Claude Code this is three pieces. A quick answer subagent with no tools. A deep research subagent with WebFetch, WebSearch and the Bright Data MCP. And the main session in the middle, choosing.
Press. Honest note: our numbers used Wikipedia search, not Bright Data. Run your own lanes before you trust the split.
5.3 What we would do differently · advice from 60 questions
Ask the model first. Then train on what matters .
Start with the model's own confidence. Free, and the best router we tested: AUC 0.73 vs 0.61 and 0.54.Train on "was quick right" , not "are the snippets enough". 27 of quick's 39 right answers had no answer in the snippets.That data is easy to get. Run both lanes once on 50 to 100 of your own questions and grade them.Escalate to real research , not one more page. Set the cut-off from your budget. Test it on questions you did not tune on.Read your grader's failures by hand before you believe any of it.
research/s1s2/ext_summary.json auc_quick_will_be_right · mined.json crosstab_snippets_vs_quick · System 1.5: ext_summary.json rows
Five things. Start with the model's own confidence: it was our best router and it is free. Then train on "was quick right", with labels you get by running both lanes once.
Escalate to real research, not one more page, at a threshold set from your budget. And read what your grader marks wrong, by hand, before you believe a number. That one nearly cost us this talk.
6.1 Honest limits
One day, 60 questions, 20 Oct 2026. A snapshot .
Only 60 questions. Quick vs deep: p = 0.267. Each router vs deep: p from 0.375 to 1.0. Any of these gaps could be luck.One search backend. Saved Wikipedia search, 8 results, one query. DuckDuckGo blocked us that morning.One model. Claude Sonnet 5.5 in both lanes.Two question sets. Two-step: quick and deep tie, 20 of 30. Deep's whole lead is on one-fact questions: 24 vs 19.The answer key has faults. We flagged 10 of 60 by hand. Without them: quick 78%, deep 84%, p = 0.549.The 0.7 cut-off was picked on the same 60 questions, not tested on new ones. 6 wrong answers were very sure.API prices only , no router or search cost. Afternoon runs cost about 10 times more per call: upper bounds.Timing. Morning lanes ran 3 calls at once, afternoon one at a time. Compare seconds loosely.
summary_strict.json tests, by_source · eval/exclusions.json · summary_clean.json · content/s1s2.md 4h, 6 · ext_summary.json buckets_self_confidence
The limits, plainly. Sixty questions, so none of these gaps is significant. One search backend, Wikipedia, because DuckDuckGo blocked us. One model.
Deep's whole lead comes from single fact questions. Ten answers in our key were faulty; dropping them keeps the same order. The point seven cut-off was picked on the same sixty questions. Afternoon prices ran high, so treat those dollars as upper bounds. Run it on your own questions.
System 1 vs System 2 search Ray Zahid · Immersive Commons
Don't ask if the snippets are enough. Ask if the fast answer will be right .
agent-search-talks.pages.dev/s1s2
One line to take home. Your model knows more than its snippets show. So route on whether the fast answer will be right. The cheapest way to ask is to ask the model itself.
The deck, the numbers and every file behind them are at this link. Thank you.
short cut 01 / 01
Overview · arrows + Enter or click · Esc to close