Search in the age of agents

Same question 50 minutes after Netflix released its earnings

The same model gives different answers when it reads different sources

The answer is only as good as the input.

research/battle/answers_summary.json (fresh: none 4, tavily 15) · results_fresh/*.jsonl id fr05 · content/battle.md §A

Before agents two places to search

Before agents, search had two places

A person read each result and decided if it was correct.

Now sources that an agent can use in the same minute

An agent can search six types of source

We made this list of six types. It is not a standard. Our research pipeline searches six channels at the same time: ~/.claude/orchestration/workflows/research.yaml

Memory facts that the model learned in training

A model answers from its training and from search results

Use search for new events and for facts that the model gets wrong.

research/battle/answers_summary.json (all60 none 40, google 40; fresh none 4) · research/s1s2/mined.json crosstab_snippets_vs_quick (27 of 39)

3.6 Round 3 · only the search results change

On news, search makes the difference

On news, search changed the score from 4 to 15.

research/battle/answers_summary.json all60 + fresh (none, google, exa, tavily) · model claude-sonnet-5-5, no tools · graded by eval/grade.py correct

1.3 Snippet vs full page · median characters, drawn to scale

A snippet shows a tiny part of the page

research/s1s2/ext_summary.json foundations.snippet_chars_median, quick_prompt_chars_median, s15_page.median_page_chars · eval/set.jsonl nq07

What a page shows you the layers that an agent can read

An agent reads a web page in layers

Each lower layer costs more tokens and more time. Each lower layer also contains more answers. When you select a source, you also select the layer that one call gives you.

research/s1s2/ext_summary.json foundations.snippet_chars_median, quick_prompt_chars_median, s15_page.median_page_chars · research/battle/summary_all60.json mean_snippet_chars (medians)

Six measures the range that we measured on 20 Oct 2026

No source is the best on all six measures

battle: TOS_PRICING.md (Serper $1) · summary_all60.json (claude.cost_usd_total, about $30 per 1,000; mean_snippet_chars; answer_in_top5; median_latency_s; overlap) · summary_fresh.json · results/retried.jsonl (DuckDuckGo blocked 28 of 30; Firecrawl 10 a minute)

3.4 Overlap · how many top 5 links two sources share, 60 trivia

Nine sources, but many show the same pages

research/battle/summary_all60.json overlap.<a|b>.mean_jaccard (n = 60 per pair)

3.9 Which sites come back · 10 questions people argue about, top 5 each

Each source favours different kinds of sites

government, universities, international bodiesencyclopediaforum / video / socialreview / affiliateeverything else

research/battle/contested_summary.json per_source.shares (50 results each) · classes from a hand made domain list in contested.py

Demo 1 a classifier selects the source before the search

A fast model labels the type of query

# the same task for Jev (about 0.1 s per call)
POST api.typesafe.ai/v1/systemone
{"state": "The user asked: netflix q3 2026 revenue",
 "questions": {"kind": {"type": "choice",
   "instructions": "What kind of search query is this?",
   "criteria": ["recent news",
     "fact a model would know", "two-step fact",
     "evidence or research", "what people say",
     "papers"]}}}

routing evidence: battle summary_fresh.json, answers_summary.json, summary_all60.json (HotpotQA), contested_summary.json shares, TOS_PRICING.md · Jev median 102 ms: research/s1s2/mined.json auc_jev.median_ms

Routing result same model, same prompt, 75 questions with known answers

Routing gave 2 more correct answers

Routing can also skip search for facts that the model knows. This saves cost and time.

This result assumes a classifier with no errors, on two query types. We tested for one afternoon.

research/battle/answers_summary.json (all60: exa 45, tavily 38; fresh: exa 13, tavily 15)

4.2 What three sources gave us · Sonnet answered once from each of Google, Tavily and Exa

When sources agree, the answer is more often right

Agreement shows your agent when to check again.

research/battle/vote.json all60 + fresh: voting, union_answer_in_results · prices: TOS_PRICING.md ($1 + $7 + $8 per 1,000)

Demo 3 when Google, Tavily and Exa disagree, use deep research

Deep research fixed 1 more of the 13 hard questions

Talk B used the same 60 questions, so we already had a deep research answer for each question. Of the 13 questions, 7 ask for one fact and 6 ask for two facts. The sample is small: 13 questions.

research/meta/disagree_deep.py reads battle/answers/{core,rest}/*.jsonl and eval/results/deep.jsonl, graded by eval/grade.py

Main result ask the model with no search, then search, then compare

When memory and search agree, the answer is almost always correct

If memory and search agree on an old fact, use the answer. If they disagree on news, use the search result.

research/meta/disagree_deep.py over battle/answers/{core,rest,fresh}/{none,exa,tavily,google}.jsonl · "agree" is the text match in vote.py same() · Sonnet 5.5 · talk B: Claude's own confidence, AUC 0.73; Jev 0.61; Laya 0.54

System design lessons from the four talks

Ask memory first. Search one source. Read pages only if they disagree.

def answer(q):
    mem, conf = claude_no_tools(q, ask_confidence=True)      # System 1 from memory
    kind = jev_choice(q, KINDS)                              # select the source
    found, hits = claude_from(search(SOURCE[kind], q, k=5), q)
    if kind == "recent news" or same(mem, found):
        return found, cite(hits)                              # news, or memory and search agree
    pages = read_full(top(hits + search(OTHER_WEB[kind], q), 3))
    return claude_from(pages, q, quotes_must_match=True)      # System 2 only here

We measured each step. We did not run the full loop tonight.

1.2 The same search, twice

Search results for a person, and for an agent

docs.tavily.com/documentation/api-reference/endpoint/search (response example) · left panel is an illustration

2.1 ora.ai, a public grader of agent readiness · its leaderboard

Most of the web is not ready for agents

ora scores four things out of 100: can an agent find you (20), get in (30), use you (40), pay you (10). Only 28 of 5,204 sites score 90 or more. Two score 100.

ora.ai public leaderboard, pulled 2026-10-17 · research/agentnative/chart-data.json · ora.ai/methodology · IC: ora scan 2026-10-20

2.2 Companies that sell search and web data · ora score of each main site

Companies that sell to agents are ahead

This score measures how ready the site is for agents. It does not measure search quality. Scans dated 2026-09-18 to 2026-10-17.

ora.ai public leaderboard, pulled 2026-10-17 · research/agentnative/chart-data.json (search_and_data_vendors, category_medians)

Live test four public APIs, read only, 20 Oct 2026, 13:18 to 13:30 PT

Agents can find most sellers. Use, access and payment need work.

Agent native means that an agent completes each step without a person.

research/agentnative/walk.md (each cell has a source) · Wikipedia has almost none of the new files. Agents can use it today: JSON with no key, errors that name the bad parameter, published limits.

Rate limits what happened during our tests on 20 Oct 2026

Rate limits control which agents can use your data

Each limit must tell the agent what to do next.

battle: summary.json (ddg, openalex), results/retried.jsonl · mediawiki.org/wiki/Wikimedia_APIs/Rate_limits · agentnative: walk/probes.jsonl (Exa 402) · none of the APIs that we tested sends the draft RateLimit headers

1.4 How the library picks what you see: two searches, one list

score(d) = Σ w · 1 / (60 + rank)

content/foundations.md §3 terms 5 and 6 · life/research/cross_search.py (_rrf_weighted, fuse block)

The method How a research system improves

A research system improves in three moves

Why it matters to you: any AI that answers from your documents can miss. These moves work on your library too.

1. 1Notice what is missingSearch outside your library. List what it lacks.demo: 0 of 30 relevant papers were there
2. 2Add itGet the full text, not just the title. Cut it into passages.demo: 6 papers, 195 passages
3. 3Check the answerAsk again. Check every quote. Run a fixed test.demo: 11 of 11 quotes found

Then go round again. Each round, the library keeps what it learned.

rag/demo2/20_gap.json (0 of 30) · demo2/33_chunk.txt (195) · demo2/60_after_answer.json (quote check 11 of 11) · ~/.claude/orchestration/workflows/research.yaml

Move 3 Same questions, asked again · added papers in the top 8

rag/demo2/11_before_retrieval.json · 50_after_retrieval.json · 70_summary.json · 01_subquestions.json (17:45:57) · research.brain_search k = 8, served path

The payoff Same question, prompt and model · it may use only the top 8 passages

The answer, before and after

Before: general advice. After: a measured result, with its cost.

rag/demo2/40_before_answer.json · 60_after_answer.json (claude-opus-5-5, both saved in full; lines shown are excerpts) · scripts/demo2_answer.py (quote check: normalised text match) · 87%: Comparative Evaluation of Advanced Chunking … Clinical Decision Support (2025), p = 0.001

Small and large models each row is a test result

Use small models for yes or no. Use large models to copy quotes.

research/s1s2/mined.json (auc_jev, auc_laya), ext_summary.json (self AUC) · life/research/experiments/exp001, exp002, exp004, exp005 (content/selfrag.md slides 19, 20)

Six lessons from one afternoon of tests

Six lessons for agent search