Four talks, one question: how should agents search?
A
Search API Battle Royale
Same questions, different sources, measured 20 Oct 2026.
B
System 1 vs System 2 search
Quick search, deep research, and when to route between them.
C
Self improving RAG
A research loop that learns from its own mistakes.
D
Agent native search platforms
What search looks like when the user is an agent.
Talk D Claude Code Build Night · Bright Data SF · 2026-10-20
Agent native search platforms
What search looks like when the user is a program, not a person.
Ray Zahid · Immersive Commons
1
Part 1 · Foundations
What an agent needs
1.1 What an agent does with search
An agent is an AI that decides its next step and uses a tool to do it
A person does this once and reads the screen. An agent does it thousands of times, alone. If a step needs a person, the agent gets stuck there.
1.2 The same search, twice
Search results for a person, and for an agent
For a person · illustration
who is leo messi
Sponsored Football shirts, big sale today …
Lionel Messi Facts | Britannicabritannica.com › factsArgentine footballer, born 1987 in Rosario, who …
Lionel Messi · Wikipediaen.wikipedia.org › wikiLionel Andrés Messi is an Argentine professional …
Messi stats and newsexample-sports.comLatest goals, transfer talk and …
More results
We value your privacy Accept all
I'm not a robot
For an agent · Tavily's documented response shape
{"query": "Who is Leo Messi?", "results": [{ "title": "Lionel Messi Facts | Britannica", "url": "https://www.britannica.com/facts/Lionel-Messi",cite it "content": "<snippet>",read it, no scraping "score": 0.81,rank it "published_date": "Tue, 11 Mar 2025 17:00:00 GMT"}],is it fresh? "usage": {"credits": 1},what did it cost? "request_id": "123e4567-…"}report a failure
Each field is written down, and its name never changes.
docs.tavily.com/documentation/api-reference/endpoint/search (response example) · left panel is an illustration
1.3 Six questions an agent asks a platform
Discover
Can the agent find you, and learn what you do?
Small files at fixed web addresses that describe you: llms.txt, an agent card, an API catalog.
Auth
Can it get a key with no password?
A person approves once, with a short code on a phone (device code). An error says where to sign in. The key allows only named actions.
Tools
Can it call you without reading a manual?
Each action lists its inputs in a fixed format. MCP is the standard plug between AI and tools; OpenAPI describes a web API.
Results
Can it read, cite and date the answer?
The same fields every time: url, content, score, date. Plain text (Markdown) on request.
Errors
When it fails, does the error say what to do?
A fixed error code, a line on how to fix it, a request id. If busy, say when to retry.
Pay
Can it pay without a card form?
The server replies with a price a program can read (HTTP 402). x402 is the way to pay it and retry.
1.4 Discover · a mailbox at a fixed address
what an agent can fetch before it knows anything about youGET /llms.txt # Acme Search > Web search API for agents. - [Search](https://acme.dev/docs/search.md)GET /.well-known/agent-card.json {"name":"Acme", "url":"https://api.acme.dev", "skills":[…]}GET /.well-known/api-catalog {"linkset":[{"anchor":"https://api.acme.dev", "service-desc":[{"href":"…/openapi.json"}]}]}
llms.txtA convention, not a standard. Proposed 2024-09-03. It is not clear that the big AI companies read it.
Agent cardA2A v1.0, under the Linux Foundation. check primary
API catalogA list of your APIs. A published standard, RFC 9727.
1.5 Auth · device code (RFC 8628): a person approves on a phone
The agent gets a key. It never holds the password.
If the agent calls with no key, the 401 error says where to sign in:
HTTP/1.1 401
WWW-Authenticate: Bearer resource_metadata="https://mcp.acme.dev/.well-known/oauth-protected-resource"
rfc-editor.org/rfc/rfc8628 (device grant) · RFC 9728 (protected resource metadata) · example host acme.dev is made up
1.6 Errors · real replies to a call with no key, 20 Oct 2026
Two error messages. Only one says what to do next.
Exa · 401
{"error":"Invalid API key. Provide a valid key using 'Authorization: Bearer <key>' or 'x-api-key: <key>'. Create a key at https://dashboard.exa.ai/api-keys", "tag":"INVALID_API_KEY", "requestId":"…"}
fixed error code
INVALID_API_KEY
how to fix it
where to make a key
request id
yes
Tavily · 401
{"detail":{"error":"Unauthorized: missing or invalid API key."}}
fixed error code
none
how to fix it
none
request id
none
A few extra words turn a dead end into a next step. When too busy (429), say when to retry (Retry-After).
research/agentnative/walk.md rows 10, 11 · draft-ietf-httpapi-ratelimit-headers-11 (a draft)
1.7 Pay · HTTP 402, "Payment Required"
The server names a price. The agent pays, then retries.
The real terms, 20 Oct 2026$0.007 in USD Coin, a stablecoin, on the Base network (eip155:8453), x402Version: 2. The reply on the first slide.
Statusx402 was open sourced by Coinbase in May 2025. It moved to an x402 Foundation at the Linux Foundation on 2026-04-02. check primary Adoption figures conflict, so we quote none.
A person needs layers 3 and 4. An agent needs all seven.
2
Part 2 · The web today
How ready is the web?
First, scores for 5,204 sites from one public grader. Then four real APIs we tested on 20 Oct 2026.
2.1 ora.ai, a public grader of agent readiness · its leaderboard
Most of the web is not ready for agents
ora scores four things out of 100: can an agent find you (20), get in (30), use you (40), pay you (10). Only 28 of 5,204 sites score 90 or more. Two score 100.
ora.ai public leaderboard, pulled 2026-10-17 · research/agentnative/chart-data.json · ora.ai/methodology · IC: ora scan 2026-10-20
2.2 Companies that sell search and web data · ora score of each main site
Companies that sell to agents are ahead
all sites, median 42
Agent Tools category, median 59.5
zenrows.com
96
apify.com
96
tavily.com
92
exa.ai
90
parallel.ai
89
brightdata.comevent host
89
firecrawl.dev
84
linkup.so
64
jina.ai
61
perplexity.ai
59
serper.dev
33
This is how ready the site is for agents, not how good the search is. Scans dated 2026-09-18 to 2026-10-17.
ora.ai public leaderboard, pulled 2026-10-17 · research/agentnative/chart-data.json (search_and_data_vendors, category_medians)
2.3 The method · a standard usability test, with a program as the user
We acted as the agent and tried every step it would try.
No account, no key, no payment. We read public pages and called each API with no key to see the errors.
13 checks across the 7 layers. Each result links to its evidence.
Not a quality test. We compared no search results.
One afternoon, one network, about twelve minutes, run by a script.
Five personas the full method uses
Coding agent: "add search to this app"
MCP native agent: "connect from the site alone"
Procurement agent: "get access, no human typing a password"
Crawler or answer engine: "can I read the site at all?"
Evaluating agent: "compare vendors from what is published"
At each step, four questions: does it know what to do, can it find how, does it understand the response, can it tell it made progress?
No device code option. Its 401 error does say where to sign in, as the standard asks.
brightdata.com
Sign in with a browser
Grants authorization_code, refresh_token.
No device code option. Its 401 error does say where to sign in, as the standard asks.
api.exa.ai/search
402: here is the price
No key needed to start. Its MCP server also listed its tools with no key.
1 of 3 vendors lets an agent start alone.
Three ways in for an agent with no person present: device code, a keyless trial, or pay per call.
research/agentnative/walk.md rows 8 and 9 · oauth-authorization-server metadata on each host · walk/probes.jsonl
Finding 3 Wikipedia's search API · keyless
Wikipedia works for agents without the new files
GET en.wikipedia.org/w/rest.php/v1/search/page (no q)
400 {"error":"parameter-validation-failed", "name":"q",
"failureCode":"missingparam",
"messageTranslations":{"en":"The \"q\" parameter must be set."}}
Search works with no key: ?q=agent returns results (200).
Published limits: 10 calls a minute if anonymous, 200 if the caller names itself. It says when to retry.
An OpenAPI spec at /api/rest_v1/?spec. No llms.txt, no agent card, no MCP.
Agent native is mostly about taking the human out of the loop, not about adding files.
research/agentnative/walk.md, Wikipedia rows · mediawiki.org/wiki/Wikimedia_APIs/Rate_limits
Finding 4 Usage limits
Nobody sends rate limit headers yet.
Rate limit headers would tell an agent, in each reply, how many calls it has left. They are a draft standard.
Exa, Tavily and Wikipedia publish limits and say when to retry. None documents the draft headers.
Bright Data names its "too many requests" errors (rate_limit, sr_rate_limit). We found no published numbers.
The cheapest fix needs no standard: put the cost of each call in the reply.
OpenAPI spec 7 points, public API 7. Publishing it turned on more checks: the usability maximum grew from 81 to 150.
2
Be a real developer product.
Developer portal 6, OAuth sign-in 5. Then clear errors, versions, paging, safe retries, limit headers, SDKs and a CLI: 78 usability points.
3
Let an agent join with a human's OK.
Device code: a person approves on a phone. The agent cannot give itself a key.
4
Get listed where agents look.
The grader searches the web for your name and checks public directories (npm, the MCP registry, skills.sh). You cannot fix this with code. It took the longest.
5
The last point: tools inside the web page.
WebMCP lets a web page offer tools to an agent. We put it in the page's HTML so the grader could see it. Worth +1.2, as the grader predicted. WebMCP is a Chrome trial feature. check primary