20 Oct 2026, 13:22 PT We asked our own research brain
How can a RAG system decide when to retrieve and check its own answers?
1 The Power of Noise: Redefining Retrieval for RAG Systems 2024 2 Search Engines in an AI Era: The False Promise of Factual and Verifiable… 2024 3 REAPER: Reasoning based Retrieval Planning for Complex RAG Systems 2024 4 TeaRAG: A Token-Efficient Agentic RAG Framework 2026 5 Retrieval-Augmented Generation for Large Language Models: A Survey 2023
Related papers, but not the answer. 0 of the 4 papers that started this idea were in our library.
By 13:34 PT the same day, it found all four. We had to help it.
stage/research/rag/demo/before_served.json (research.brain_search, k = 8) · rag/scripts/find.py title search · rag/STATUS.md
At 13:22 today (20 Oct) I asked our own research brain this question. The question of this talk, in fact.
Press. These are its top five. All about RAG. None about a system deciding for itself when to retrieve.
Press. Four papers founded this idea: Self-RAG, Corrective RAG, Adaptive-RAG and FLARE. None was in the corpus.
Press. Twelve minutes later it found all four. This talk is how, and why it was not magic.
Evidence: before_served.json, find.py, STATUS.md 13:22 to 13:34.
One of four talks
Four talks, one question: how should agents search?
A Search API Battle Royale Same questions, different sources, measured 20 Oct 2026.
B System 1 vs System 2 search Quick search, deep research, and when to route between them.
C Self improving RAG A research loop that learns from its own mistakes.
D Agent native search platforms What search looks like when the user is an agent.
Four talks share one question; this one is lit. Each stands alone. Together they cover where the facts come from, how hard to search, how a research pipeline learns, and what search built for agents looks like.
Claude Code Build Night Bright Data SF · 2026-10-20 · Ray Zahid, Immersive Commons
How a research pipeline learns from its mistakes
Our research brain is a search engine over 6,927 papers, run on 7 machines. One database died. Our checks caught us being wrong.
stage/research/rag/db_stats3.txt + demo/learn_1_seed.txt · rag/farm_nodes.txt · memory project_pg_host_ic2.md
I am Ray. I run Immersive Commons, an AI builder floor in Frontier Tower.
For the last six months we have run our own research brain. Almost seven thousand papers, seven machines doing the embedding, and one database that died on us.
Everything tonight is measured on that system, most of it this afternoon (20 Oct).
Evidence: db_stats3.txt, farm_nodes.txt.
The question of this talk
How does a research pipeline get better on its own?
1 Look in more than one place, and keep going while you still find new things.diversity and saturation 2 Check every quote against its source. Let two separate reviewers judge each claim.verification 3 Keep what you found. Keep a fixed test, so you see when a change makes it worse.the shelf (saved papers) and the eval (the test)
Short answer: it does not. It gets better when a program checks it , and when it keeps what it learns.
Three answers, one per press. Look widely and do not stop early. Check every quote, and make two reviewers argue. Keep what you found, and keep an exam.
Press. The honest version: it does not get better on its own. The checks make it better.
The surprise of the talk: those exams caught our own wrong conclusions, twice.
Running example Our research brain on 20 Oct 2026
Our research brain: a searchable library, or corpus , of papers
6,927 research papers, title and abstract
1,017 of them with full text
60,200 passages cut from those full texts
40 topics
7 machines processing papers, in the 8 days to 20 Oct
rag/db_stats2.txt (pdf_status extracted, session rows) · db_stats3.txt (59,984 + 216) · demo/learn_3_chunk.txt · corpus_coverage.json (39 + 1 topics) · farm_nodes.txt
Live counts from this afternoon (20 Oct). Almost seven thousand papers. Only about a thousand have full text; the rest are a title and an abstract.
Press. The tool says almost twenty one thousand papers. Fourteen thousand of those are our own session transcripts. That detail comes back later and bites us.
Evidence: db_stats2.txt, db_stats3.txt, learn_3_chunk.txt, corpus_coverage.json.
1
Part 1 · Foundations
From a question to a cited answer
RAG (retrieval augmented generation): the AI looks up sources first, then answers from them.
Before we make anything learn, we need the parts. Five minutes of foundations. Plain words first, then the technical version, then our own numbers.
1.1 The pipeline in plain words
How RAG gets from a question to an answer
Question you ask in words Retrieve 3 search methods, 20 results each Rank merge the lists Read the passages, word for word Answer + cite each claim names its paper not enough? search again
Five steps. You ask. We retrieve a few dozen candidates. We rank them. We read the actual passages, not the titles. We answer, and every claim points at a paper.
Last press: the dashed arrow. Not enough? Search again. That arrow is the whole second half of this talk.
Evidence: content/foundations.md section 2.
1.2 Our real pipeline
Our real system, step by step
INGEST · RUNS BEFORE ANY QUESTION
Seed 40 topics, lists of papers Acquire 13 paper sources, free PDFs Enrich velocity, marks, summaries Embed 7 machines, 768 numbers each Index a database on machine ic2 RETRIEVE AND FUSE · PER QUESTION
Query words + meaning A · passage vector (meaning) B · passage BM25 (keywords) C · paper vector (title + abstract) books: vector, BM25, tree · 0 books on 20 Oct 2026 Merge (RRF) k = 60, top 20 each Boost, one row per paper marks, velocity, summaries · top k SERVE
20 Oct 2026: backup server ic5 answers; main server offline every hit: paper id · title · year · URL · passage
content/foundations.md §2 · life/research/cross_search.py · research.brain.health() base immersivecommons5:8010 at 13:27 PT, 2026-10-20 · rag/tailnet_nodes.txt
The real thing. The top lane runs all the time: seed topics, fetch papers, enrich them, embed them on seven machines, index them in Postgres.
Press. Per question, three retrievers each return twenty. Two read passages, one reads titles and abstracts. Press. Fusion merges them. Press. A server hands back every hit with its source.
Today (20 Oct) the main search server is offline; a backup on ic5 is serving.
1.3 Four terms
Chunk
One passage cut from a paper, short enough to read at a glance.
Retrieval works on chunks; one vector cannot hold a 30 page paper.
1,017 full text papers → 60,200 passages
Embedding
A list of numbers that stands for what a passage means.
A vector made by a model. Closeness is the angle between two vectors (cosine).
768 numbers · our top passage: closeness 0.741 (cosine)
Vector index
A filing system that finds the nearest meanings fast.
It checks only part of the index (HNSW), so it trades a few misses for speed.
default setting: 4 of the true top 20 · tuned: 20 of 20
BM25
Score a passage by how many of your rare words it holds.
Each repeat of a word counts for less (k1 1.2). Long passages are scaled down (b 0.75). Rare words count more.
right passages found in the top 10: our old "keyword" search 0.076 · real BM25 0.457
content/foundations.md §3 · rag/demo_rrf_hybrid.json · life/research/cross_search.py (ef_search comment, docstring) · rag/retrieval_eval_latest.txt
Four words. A chunk is one passage. An embedding is a list of numbers that stands for meaning. A vector index finds nearest meanings fast, and it can quietly miss: with the default setting it found 4 of the true top 20.
BM25 counts your rare words. Our old keyword branch was not BM25 at all, and it scored 0.076. Hold that thought.
1.4 Two kinds of search, one list
score(d) = Σ w · 1 / (60 + rank)
Hybrid search
Ask a meaning finder and a word finder at once, then merge.
Keyword results (BM25) and meaning results (vectors), merged into one list. BM25 catches exact names and codes. Vectors catch the same idea in other words.
we run three paper retrievers per question
Reciprocal rank fusion
Each list gives points for position. Points add up across lists.
It uses only positions, so different kinds of score never need converting. k = 60. Each of our three searches has weight w = 1/3.
first place in one list: 1/61 = 0.0164
content/foundations.md §3 terms 5 and 6 · life/research/cross_search.py (_rrf_weighted, fuse block)
Hybrid search: ask a meaning finder and a word finder at the same time. Each catches what the other misses.
Then merge with reciprocal rank fusion. Each list gives a passage points for its position. First place earns most. Points add up. It only uses ranks, so scores on different scales never need converting.
1.5 Running example · real ranks
"Does combining keyword search with vector search improve retrieval? "
A · passage vectormeaning of the passage
1 RAG for NLP: a survey2 Open-Source Agentic Hybrid RAG Framework3 RAG for NLP: a survey (another passage)4 ViSAR: Adaptive-k Retrieval5 RAG for NLP: a survey (another passage) B · passage BM25keyword match
1 Open-Source Agentic Hybrid RAG Framework2 Improving LLM applications in biomedicine with RAG3 Deep deterministic policy gradient: a review4 Structural Feature Engineering for GEO5 RAG for AI-Generated Content: A Survey C · paper vectortitle + abstract
1 WARP: Multi-Vector Retrieval2 PyLate: Late Interaction Models3 Scaling Retrieval-Augmented Reasoning4 Jina-ColBERT-v25 GaussDB-Vector
fused place · paper list A list B score = sum × 1/3
#1 RAG for LLMs: A Surveyin neither top 5 #7 → 1/67 = 0.01493 #10 → 1/70 = 0.01429 0.00974 code: 0.0097
#2 Policy gradient reviewoff topic #20 → 1/80 = 0.01250 #3 → 1/63 = 0.01587 0.00946 code: 0.0095
#6 Hybrid RAG Frameworkone passage: first in B; A found another not in list #1 → 1/61 = 0.01639 0.00546 code: 0.0055
Good in two lists beats first in one. But lists can agree by accident: an off-topic review came second.
rag/demo_rrf_hybrid.json · demo_rrf_hybrid_run2.txt (RANKTABLE) · rag/scripts/demo_rrf.py · content/foundations.md §1 and §3 term 6
Our running example. Press: each retriever's own top five, from the real system.
Press: the fused winner is in none of these top fives. Press: it was 7th and 10th. Two decent ranks. Press: second place is a reinforcement learning review. 20th on meaning, 3rd on keywords. Off topic, and enough. Press: the paper that was first in a list ends up 6th.
We checked the arithmetic against what the code printed. It matches.
Live Try it: keyword, meaning, merged
KeywordBM25 over your exact words Meaningpretend embedding: 6 concepts, not a real model FusedRRF: 1/(60 + keyword rank) + 1/(60 + meaning rank)
a toy in the page: 8 sentences, no network, no model (spec: content/foundations.md §4)
A toy, running in this page. Eight sentences. The default query: D1 wins all the way through. D2 is on topic but shares no words, so only meaning finds it. D6 is about a maze; it shares the word search, and it lands third fused. That is our policy gradient paper in miniature.
Click X-1000: only keywords can find a product code. Click find paraphrases: keywords find one sentence, meaning finds several.
Fusion does not make a list smart. It rewards agreement, and agreement can be an accident.
1.6 Three terms for checking answers
Grounding
Every claim points at the exact words it came from.
A claim is grounded when its quote appears in the source text. A simple text match checks it, no AI judge.
our workflow will not ship a quote it cannot find
Recall vs precision
Did we find the good stuff? Is what we found any good?
recall@k: how many of the right passages made the top k. precision@k: how many of the top k are right.
served top 5: 4 on topic, 1 not (our reading)
Eval
A fixed exam with known answers, run again after every change.
Fixed questions, known right answers, one score, and a fingerprint (hash) of the grader.
128 real questions mined from our sessions
The rest of this talk measures these three things and acts on the results.
content/foundations.md §3 terms 9 to 11 · ~/.claude/orchestration/gates/research_citation_integrity.py · rag/brain_search_served.json · life/research/retrieval_eval/README.md
Three more words. Grounding: every claim points at its exact words, and a plain string match can check it. Recall and precision: did we find the good stuff, and is what we found any good. Eval: a fixed exam you run again after every change.
Everything after this slide is about measuring those three and acting on the measurement.
2
Part 2 · The loop
A pipeline that checks itself
Nine steps. At six of them, a program can say no.
Now the dashed arrow. Our research workflow is a loop of nine steps. At six of them a program, not a model, decides whether the run may continue.
2.1 The research loop
the library grows: what it finds stays
go again
frame split the question ground what do we have already? research_corpus_floor scan ×6 six places at once research_channel_coverage saturated? still finding new? research_saturation triage drop repeats, pick the best deep read claims + exact quotes experiment run it, if reading cannot settle it research_experiment_reproducible verify ×2 two reviewers, each alone research_verdict_adjudicate synthesize report + findings research_citation_integrity = a program checks this step the agent cannot talk its way past it
~/.claude/orchestration/workflows/research.yaml (phases, gates_on_fail: kick_back_to: scan) · orchestration/gates/research_*.py
Clockwise from the top. Frame splits the question. Ground asks how much we already have. Scan runs six channels at once.
Press. Saturated? If we are still finding new things, go again. Press. Triage, then deep read: every claim with a word for word quote. An experiment when reading cannot settle it.
Press. Two blind verifiers, then the report. Press. And what it found stays on the shelf for the next question. Every shield is a program the agent cannot argue with.
2.2 Search widely, stop when nothing new turns up
Search diversity
Six scouts, six places: our brain, papers, web, code, our notes, the news.
We count results per place, so one silent place cannot hide behind five busy ones.
Real case, round 2: the brain search found 0 sources and said why: "connection timeout ... not a statement that the literature is absent"
Saturation
Stop when new searches mostly bring back what you already have.
Stop when under a quarter of results are new (0.25), or after round 3.
Why: the correction I typed most often was "KEEP RESEARCHING"
Without a count per place, a broken search and an empty one look the same.
orchestration/gates/research_saturation.py (docstring) · research_channel_coverage.py · life/projects/layered-agent-permissions/handoffs/scan__scout-brain.json
Two ideas. Diversity: six scouts in six places, and coverage counted per channel. Saturation: stop when new searches mostly repeat. Under a quarter new, or three rounds.
Where it came from: the most repeated correction in our own sessions was me typing KEEP RESEARCHING. Now a program says it.
And a real case: a scout found zero sources because the server timed out. It said so out loud. That is the difference between dead and empty.
2.3 Two checks no model can argue with
Two separate reviewers A skeptic and a balanced reader judge every claim. Neither sees the other. If they disagree, the claim is dropped. A claim with no verdict stops the run.
research_verdict_adjudicate
Quote check Every quote must appear character for character in the source. Quotes under 12 characters are rejected. Only judged claims enter the report.
research_citation_integrity
"These segments are then embedded using all-MiniLM-L6-v2 model, and the resulting vectors are stored and indexed through the FAISS library , enabling efficient similarity-based retrieval."
found · ships the resulting vectors are stored and indexed through the FAISS library
no match · dropped vectors are kept in a FAISS index for fast similarity search
One real run: 419 claims , all judged by both reviewers. 416 kept, 3 dropped . The quote check failed once, on a "quote" that was a star counter. It passed after the fix.
gates/research_verdict_adjudicate.py · research_citation_integrity.py (_MIN_QUOTE = 12) · passage: rag/brain_search_served.json hit 1; the second quote is written to show a paraphrase · weba-infra-2026-10 verdicts/adjudicated.json, retro.md
Two checks. Press: two blind readers judge every claim; a split does not ship. Press: every quote must appear character for character in the source.
Press: a real passage from our brain, and a quote that is in it. Ships. Press: a paraphrase of the same sentence. True in spirit, not in the source. Dropped. The check is one line of string matching.
Press: on one real run, 419 claims, 3 dropped, and the gate caught one bad quote.
2.4 A real run, end to end · weba-infra-2026-10
One real run on web tools for AI agents. It never ran out of new sources.
brain papers web code internal signal
round 3 · share that was new
0.55
0 0.25 stop line 1
127 new of 232 found. The 3 round limit stopped it, not the stop rule. We did not move the 0.25 line based on one run.
wc -l life/projects/weba-infra-2026-10/scan/round*/<channel>.jsonl · handoffs/scan__scout-*.json (round 3 new/found) · retro.md ("Stopped on round 3 (MAX_ROUNDS)", item 9)
One real run from October 2026: "how has agent ready web infrastructure moved since June". Six channels, three rounds. Press, press, press: 371, 182, 232 sources found.
Press. In round three, 55 percent were still new. Far above the 25 percent stop line. So saturation never fired; the hard cap of three rounds did. The retro wrote that down and refused to tune the threshold on one run.
Rounds one and two new share was not stored, so we do not draw it.
3
Part 3 · The evidence
We taught it one question in twelve minutes
Then our own exams caught us being wrong. Twice.
Back to the question from the start. Everything in this part ran on the live database between 13:22 and 13:34 today (20 Oct). Then two older exams, which caught us out.
3.1 The learning demo · 20 Oct 2026, 13:22 to 13:34 PT
Where the four founding papers rank for our question
Self-RAGarXiv 2310.11511 not on shelf
Adaptive-RAGarXiv 2403.14403 not on shelf
Corrective RAGarXiv 2401.15884 not on shelf
FLAREarXiv 2305.06983 not on shelf
log scale, lower rank is better
rag/demo/learn_1_seed.txt, after1_exact_ranks.txt (paper rank by cosine) · after2_exact_ranks.txt (best passage rank) · before_after_nosession.json (sub-questions, transcripts excluded) · STATUS.md
Four founding papers, one row each. Further left is better. The amber band is the top eight a reader sees.
Press. We added the four papers. One command, and a farm machine embedded them in the same second. Self-RAG lands 43rd. Each retriever only keeps twenty. Nothing changes.
Press. We fetched the PDFs and cut 216 passages. Best passage: 81st. Knowing a paper exists is not knowing what it says, and even knowing what it says is not enough.
Press. Then the frame step: four sub-questions in research language. All four land in the top eight. Press. Get it, read it, ask in its words.
3.2 Same library, two ways to ask · new papers in the top 8
question before after the shelf grew
natural how can a RAG system decide when to retrieve and check its own answers0 0, top 8 unchanged sub-question 1 self-reflective retrieval augmented generation that critiques its own output0 Self-RAG #1 sub-question 2 adaptive retrieval: choosing whether to retrieve based on question complexity0 Adaptive-RAG #2 sub-question 3 evaluating retrieved documents and correcting retrieval when results are wrong0 Corrective RAG #7 sub-question 4 retrieving actively during generation when the model is uncertain0 FLARE #1 , Corrective RAG #5all four together founding papers in the top 80 of 4 4 of 4
Caveat: Claude wrote these questions after it knew which papers we added. In the real workflow, they are written before any search. So it takes both: the right questions and the right papers on the shelf. Our Claude Code chat logs are left out here.
rag/demo/before_after_nosession.json · rag/scripts/beforeafter_nosession.py (BEFORE excludes session + self-improving-rag; AFTER excludes session) · cross-check before_after.json
The detail. The natural question: zero before, zero after. The shelf alone did not fix it.
Press. The four sub-questions, same code, same corpus. Each one pulls its paper into the top eight. Press. Union: zero of four becomes four of four.
Press. The caveat matters. I wrote these questions knowing what was added. So the claim is the right questions plus the right shelf, not the shelf alone.
3.3 Yes or no: is this passage relevant? · exp001 → exp004 → exp005
We wrote that a small model matches Claude. A harder test said no.
share of irrelevant passages correctly rejected 1.00
llama3.2 3Ba small model, on our own machines 0.95
the exam exp001, 80 passages. The wrong ones were about wine chemistry.
exp001 README, line 1 as of 20 Oct 2026: "Superseded in part ... Do not quote the neg-reject or accuracy figures below as operational numbers."
life/research/experiments/exp001_fleet_triage/results/SCORES.md · exp004_nearmiss/README.md + results/llama3.2-3b-fewshot.jsonl vs golden.jsonl · exp005_rule_alignment/README.md
A yes or no job: is this passage relevant? August, exp001: the small model rejects 95 percent of irrelevant passages, Claude all of them. Press. F1 0.82 against 0.79. We wrote: no detectable difference.
Press. Its own limits section said the wrong answers were too easy. Wine chemistry. So exp004 used near misses from other agent topics. The small model fell to 0.67. Press. Same prompt to Claude: 0.94.
Press. We wrote that at the top of the old result instead of deleting it.
3.4 exp002 · copy out claims with their quotes from 40 real passages
A small model invents quotes. A simple text match caught it.
0% Claude 0 of 324 claims made up · 322 word for word, 2 near
19% llama3.2 3B 8 of 43 claims made up · 32 word for word, 3 near
a made-up quote hyperparameter values that we use in each environment is provided in Table 6
A small model (3B) can judge text it is shown. It cannot reliably copy it. So small models get yes or no jobs. Claude gets anything that copies a source.
life/research/experiments/exp002_claim_extraction/README.md (2026-08-16, n = 40, identical prompt hash ab1cde3f8cfe, near = at least 90% overlap)
Second job: pull claims, each with a quote. Same prompt, forty real passages, graded by plain string matching.
Press. Claude: zero fabricated. Press. The small model: 19 percent of its quotes were not in the passage. Press. Here is one. The passage has no such sentence.
Press. That became a routing rule. Small models for yes or no behind a golden set. Claude for anything that must copy a source. And that is why the quote check exists.
3.5 The exam that found a broken retriever
Our "keyword" branch was not keyword search.
It returned nothing for 98 of 128 questions. Real keyword search (BM25) replaced it on 2026-09-12. We have not re-run this test since the rebuild on 2026-10-12.
life/data/retrieval_eval/runs.jsonl row 2026-09-11T22:50Z · rag/retrieval_eval_latest.txt · life/research/cross_search.py docstring · life/research/retrieval_eval/README.md
Third exam. 128 real research questions mined from our own sessions, 24 held out.
Our hybrid had a keyword branch that required every word to match. It found nothing for 98 of 128 questions. Recall 0.076. The code comment now calls it a dense only search wearing a lexical badge. Real BM25 replaced it the next day.
Measured in September. It has not been re-run on the rebuilt corpus.
3.6 The library saved its own test
30 trivia questions, top 5 results each. Amber = a log of our own test.
each column is one question · rows are ranks 1 to 5 grey = a research paper
30/30 questions whose top result was a log of our own test
141/150 results that were our own logs
1/30 answers found once our logs are left out
top hit for "where does a roadrunner live in the desert" Answer the question using only these web search results. Question: where does a roadrunner live in the desert …
stage/research/battle/results/brain.jsonl (session_leak per result) · battle/summary.json → brain · rag/session_leak_db.txt (260 eval rows ingested 08:35 PT, 2026-10-20)
This morning (20 Oct) another talk's eval asked thirty trivia questions to every source, including our corpus. Those eval sessions were ingested into the corpus at 08:35.
Press. Each square is one result. Amber is our own transcript. Press: thirty of thirty top hits. 141 of 150 results. Press: without them, one answer out of thirty.
Press. The top hit for the roadrunner question is the eval prompt itself. It did not know the answer. It remembered being asked. Keep the exam out of the library.
3.7 The shelf · research papers in the live database
The library grew for five months. Then one day it held zero papers.
0
2,000
4,000
6,000
8,000
Apr
May
Jun
Jul
Aug
Sep
Oct
our paper lists, added up (shape only, not exact history)
398 farm launch 4,724 6,181 frozen eval view 0 ~09-30: old DB host dies 6,312 6,927 10-12 rebuilt from seed files FULL TEXT PASSAGES
211,726 Sep 11
~60,200 Oct 20
rag/growth_chart.json · life/data/research/papers/*.jsonl _meta.fetched_at · rag/db_stats3.txt · memory project_pg_host_ic2.md · life/research/retrieval_eval/README.md · points are measured, lines join them
The dashed line is our seed files, the lists of papers we chose. Treat it as shape, not exact history. The dots are what the live database held when we measured.
Press. Four hundred at launch, six thousand by September. Press. Around September 30 the database host died. Zero. The June design notes had listed exactly that single point of failure.
Press. Twelve days later we rebuilt it in one batch from the seed files. Press. The papers came back, because the seed files were the backup. The full text did not.
3.8 The machines that turn papers into numbers · 20 Oct 2026, 13:27 PT
ic1 offline · old DB host, dead since ~09-30
ic2 database + Ollama · 3,365 / 12,138
ic3 offline
ic4 offline
ic5 1,403 / 51,947 · serving search 20 Oct
ic6 offline
ic7 online · no embedding
ic8 offline
ic9 online · no embedding
ic10 offline
ic11 3,572 / 4,595
ic12 offline
ic13 2,398 / 8,644
ic14 offline
ic15 offline
ic16 3,643 / 3,970
ic17 offline
ic18 3,625 / 2,362
ic19 offline
ic20 2,951 / 13,664
sc2 offline · the main search server
Each machine takes a paper or passage nobody else has claimed (FOR UPDATE SKIP LOCKED), turns it into 768 numbers, and saves them.
Papers are cut into ~1,800 character passages that overlap by 200. The PDF downloader has been off since the rebuild.
Broken since the rebuild: the free PDF finder and the table of citation links.
7 of 20
known machines did any work in the 8 days to 20 Oct 2026. We call it a 22 machine farm.
database host, also embeds
papers / passages processed
online, no embedding seen
offline
rag/tailnet_nodes.txt (tailscale status, 2026-10-20 13:27 PT) · rag/farm_nodes.txt (embedded_by, last 8 days) · life/research/embed/ (worker.py, chunk_papers.py, oa_finder.py, edges.py) · rag/db_stats.json
The farm. Every tile is a machine the tailnet knows. Lit tiles embedded something in the last eight days. Amber is the database.
Press. Seven of twenty did any work. We still call it a 22 node farm. The main search server is offline, and ic5 is quietly carrying the load.
Press. The workers are simple: claim a row nobody else has, embed it, write it back. Press. Two ingest tools broke on the rebuilt schema. We found one of them this afternoon (20 Oct), during the demo.
4.1 Honest limits · checked 20 Oct 2026
What is still wrong with it
Most of the shelf cannot be quoted: 1,017 of 6,927 papers have full text.
rag/db_stats2.txt + demo An abstract alone is not enough: Self-RAG ranked 43rd for its own topic until we asked in its words.
slide 3.1 · after1_exact_ranks.txt Merging rewards lists that agree by accident: an off-topic review came second.
content/foundations.md §1 The stop rule has never ended a run. The 3 round limit did.
weba-infra-2026-10 retro.md The library takes in our own chat logs, test logs included, so it can answer from its own test.
battle/results/brain.jsonl · 30 of 30 top hits Only about a third of the machines work, and a backup server is answering searches.
farm_nodes.txt · tailnet_nodes.txt Two intake tools fail on the rebuilt database. Leaving out a topic crashes for every topic except news.
demo/learn_2a_oa.txt · rag/db_stats.json · cross_search.py _scope_key Not re-checked on 20 Oct: the research tool's rate limit, and whether test logs are kept out of the library.
unverified popularity and summary scores now lift results after the merge.
cross_search.py _enrich_paper_rows the title and abstract search now covers every topic.
cross_search.py, paper-meta branch comment
Before you copy any of this, what is still wrong. Most of the shelf is abstracts only. Fusion rewards accidents. Saturation has never stopped a run by itself.
The corpus can learn its own homework. The farm is a third of its size. Two ingest tools broke on the rebuild and we found a search bug during today's demo (20 Oct).
Two things from June are fixed. Everything here is checked today (20 Oct), except the two marked unverified.
4.2 Build it tonight · a self improving research loop
research_loop.py · each scout: a Claude Code subagent or a Bright Data MCP call CHANNELS = ["web" , "papers" , "code" ] # different blind spots subqs = frame(question) # ask in the papers' words shelf = load_jsonl("shelf.jsonl" ); seen = {c["url" ] for c in shelf} for rnd in range(1, 4): # hard cap: 3 rounds found = [s for q in subqs for ch in CHANNELS for s in scout(ch, q)] new = [s for s in found if s["url" ] not in seen] seen |= {s["url" ] for s in found} if found and len(new) / len(found) < 0.25: break # saturated claims = [c for s in shortlist(seen) for c in extract_claims(fetch(s))] claims = [c for c in claims if norm(c["quote" ]) in norm(fetch(c["url" ]))] keep = [c for c in claims if skeptic(c) in OK and balanced(c) in OK] append_jsonl("shelf.jsonl" , keep) # the corpus grows append_jsonl("evals.jsonl" , {"q" : question, "rounds" : rnd, "kept" : len(keep)})
This is the whole idea in fourteen lines. Three channels. Frame the question in the papers' words. Start from the shelf.
Up to three rounds; stop when under a quarter is new. Extract claims with a big model. The quote check is one line and needs no model. Two blind lenses. Then write to the shelf, and log the run.
The evals file is what makes it improve: ten questions with known good sources, re-run after every change. Our full version: research.yaml, six channels, six gates.
Close Self improving RAG
It did not get smarter. It got checked , and it kept what it learned.
slides and codeagent-search-talks.pages.dev/selfrag
One line to take home. Our research brain did not get smarter. It got checked by programs it cannot argue with, and it kept what it learned on the shelf.
Look widely. Check every quote. Keep an exam, and keep the exam out of the library. Thank you.
Press for the link to the slides and code.
short cut 01 / 01
Overview · arrows + Enter or click · Esc to close