Compiler Analogy¶
Summary¶
The core conceptual framing of the LLM Wiki pattern: knowledge processing maps directly to software compilation, where raw sources are source code, the LLM is the compiler, the wiki is the compiled executable, and linting is the test suite.
The Mapping¶
| Compilation Stage | LLM Wiki Equivalent | Explanation |
|---|---|---|
| Source code | Raw articles, papers, transcripts in raw/ |
The raw, unprocessed input material |
| Compiler | LLM processing raw info | Reads sources, creates summaries, links documents, structures knowledge |
| Executable | The compiled wiki pages | What we actually query and run — pre-synthesized, cross-referenced knowledge |
| Test suite | Linting pass | Finds gaps, stale data, broken links, ensures data integrity |
| Runtime | Running queries | Agents search the wiki for information to answer questions |
Why This Analogy Matters¶
Traditional RAG executes source code on every request — re-reads the same papers, re-chunks the same PDFs, re-synthesizes the same answers. Every query pays the full compilation cost.
LLM Wiki compiles knowledge once (and incrementally as sources are added), then queries run against the optimized artifact. The compilation step is expensive, but it pays for itself across every subsequent use.
Karpathy's Token Shift¶
Karpathy noted he's "spending more of my tokens with my agents manipulating knowledge like Markdown and Obsidian instead of manipulating code." He works with knowledge the same way engineers work with code — the compiler analogy makes this parallel explicit.
Origin¶
The analogy was articulated by Cole Medin in his video analyzing Karpathy's pattern, making the abstract concept concrete for engineers familiar with build pipelines.