Building an Observer That Turns Claude Code Sessions into a Knowledge Base

The Problem

I run 10–15 Claude Code sessions a day across multiple projects. Each one produces decisions, architectural patterns, and working context. When the terminal closes, all of it vanishes into a JSONL transcript file — thousands of lines of interleaved tool calls that no one will ever search.

The knowledge compounds only if someone extracts and files it. I was not going to do that manually.

The Underlying System: Karpathy's LLM Knowledge Base

The observer sits on top of an Obsidian vault built on Andrej Karpathy's LLM Knowledge Base architecture. In April 2025, Karpathy described using Claude Code to maintain a knowledge base out of plain markdown files — no vector database, no embeddings, no chunking pipeline. Just an LLM that reads and writes files.

The core idea: raw sources go in, the LLM compiles them into interlinked wiki pages, and the wiki is what you query. The human curates source material. The LLM is the compiler. The wiki is the compiled product.

This works because of three properties that make it better than RAG at personal scale:

1. The index fits in the context window. A master index.md file lists every wiki page with a one-line summary. The LLM reads this first, picks the relevant pages, then reads those. No embedding similarity search, no retrieval pipeline. At 509 pages (~73KB index), the entire table of contents fits in a single context window.

2. Knowledge compounds instead of decaying. When you ask a question and the answer is valuable, the LLM files it back as a new wiki page. Each query makes the wiki richer. Unlike chat, where synthesis evaporates when the window closes, the wiki accumulates.¹

3. Relationships are explicit, not computed. Wiki pages link to each other with [[wiki-links]]. The LLM maintains these as it writes. A Go pattern learned at work links directly to a trading-agent research page. These cross-domain connections are more valuable than any embedding similarity score because the LLM understood the relationship when it created the link.

The gap in Karpathy's architecture is ingestion. He clips articles manually. For coding sessions, the knowledge is trapped in transcript files that nobody will ever manually process. The observer fills that gap — it automates the "raw sources go in" step for every Claude Code session.

The Solution: A Two-Phase Observer

The observer is a background daemon that automates ingestion from Claude Code sessions into the Karpathy-style wiki. It runs in two phases with different cost profiles:

Capture runs at session-end speed. It parses the transcript, extracts .ai/ planning files, and distills a structured summary. Compilation runs on a schedule — it reads the raw session notes, merges them with existing wiki pages, and maintains a 509-page cross-referenced knowledge base.

At typical usage, the system costs about **$0.65/day** (~$20/month).

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                  PHASE 1: CAPTURE                        │
│               (immediate, per-session)                   │
│                                                          │
│   Claude Code Session                                    │
│         │                                                │
│         ├── SessionStart hook                            │
│         │   → inject compressed wiki index into context  │
│         │                                                │
│         ├── PostToolUse hook (Write/Edit to .ai/)        │
│         │   → write .pending.json hint                   │
│         │                                                │
│         └── SessionEnd hook                              │
│             → write .hint.json                           │
│                    │                                     │
│                    ▼                                     │
│   observer.js (daemon, polls every 30s)                  │
│         │                                                │
│         ├── detect ended sessions (snapshot diff)        │
│         ├── find transcript (3-tier lookup)              │
│         ├── parse JSONL → structured data                │
│         ├── filter (skip empty / self-capture)           │
│         ├── copy .ai/ files → projects/ + raw/           │
│         ├── distill with Haiku (4 attempts + backoff)     │
│         ├── write session note → raw/<slug>.md           │
│         └── write heartbeat timestamp                    │
│                                                          │
└──────────────────────┬──────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────┐
│               PHASE 2: COMPILATION                       │
│             (3×/day via launchd)                          │
│                                                          │
│   compile-wiki.sh                                        │
│         │                                                │
│         ├── find unprocessed raw/*.md                    │
│         ├── inject full wiki/index.md into prompt        │
│         ├── Sonnet 5 (up to 30 agentic turns)            │
│         │   ├── create/update wiki/ pages                │
│         │   ├── update wiki/index.md                     │
│         │   └── append to wiki/log.md                    │
│         ├── mark processed                               │
│         └── generate compressed index summary            │
│                                                          │
│   audit-quality.sh (post-compile)                        │
│         ├── Tier 1: structural metrics (no LLM)          │
│         └── Tier 2: Haiku scoring (1×/day)               │
│                                                          │
│   enforcement backstop                                   │
│         └── strip date prefixes from any wiki filenames  │
│                                                          │
└─────────────────────────────────────────────────────────┘

The two-phase design is the key economic trade-off. Capture must be fast and cheap because it runs on every session. Compilation can be slow and expensive because it runs three times a day. Haiku handles the per-session distillation at sub-cent cost. Sonnet handles the knowledge synthesis that requires understanding 509 existing wiki pages.

The Components

The system lives in _system/session-capture/ — 30 files, about 5,765 lines total (including a 64-test suite and four quality-gate scripts).

Core Pipeline

Hooks

Scheduled Tasks

Configuration

Tests

64 tests across 4 modules, using node:test with no external dependencies:

Run with ./test-runner.sh (which cds into the session-capture directory and calls node --test test-*.js).

How Sessions Get Captured

Step 1: Hook signals

When a session ends, the SessionEnd hook writes a .hint.json file in under 100 milliseconds. This file contains the transcript path and working directory — enough for the observer to find and process it.

During the session, every Write or Edit to an .ai/ path fires the PostToolUse hook, which writes a .pending.json hint. The observer copies these files on its next poll cycle, so .ai/ planning artifacts are captured even if the session crashes before ending cleanly.

Step 2: Observer detects the session ended

The observer polls herdr's² session.json every 30 seconds. It diffs the current snapshot against the previous one. When a pane disappears or its session ID changes, that session is marked as ended.

Step 3: Three-tier transcript lookup

Finding the transcript is harder than it sounds. The observer tries three strategies in order:

1. Hint file path — the .hint.json from the SessionEnd hook contains the transcript path directly
2. CWD-derived slug — reconstruct the Claude project directory slug from the working directory
3. Brute-force search — scan all project directories for a matching session ID

The third tier exists because herdr's state can be stale, or the session can end between poll cycles in ways that lose the cwd context.

Step 4: Parse and filter

extract-transcript.js streams the JSONL transcript and extracts:

• Session metadata (ID, project, timestamps, duration)
• .ai/ file writes (from Write/Edit/MultiEdit tool calls)
• Artifact tool calls
• Assistant text blocks
• Cost data

The observer then filters out sessions that are not worth capturing:

• Non-CLI sessions (web, API)
• Empty sessions (no .ai/ files, no artifacts, no substantial text)
• Self-capture sessions (more on this below)

Step 5: Distill with Haiku

distill.js spawns claude -p --model claude-haiku-4-5 with the extracted text and asks for a structured JSON digest:

{
  "title": "Implement webhook retry with exponential backoff",
  "digest": "Added retry logic to the webhook delivery...",
  "decisions": [
    "Used exponential backoff with jitter instead of fixed intervals",
    "Capped retries at 5 attempts over 24 hours"
  ],
  "openThreads": [
    "Dead-letter queue for permanently failed webhooks not yet implemented"
  ]
}

If the call fails — rate limit, network blip, timeout — the system retries up to 4 attempts with exponential backoff (3s → 15s → 60s). Rate-limit errors use a minimum 30-second delay. Only after all 4 attempts fail does it fall back to regex extraction of the session's Recap block^[My Claude Code setup ends every response with a structured 📋 Recap block containing changed files, decisions, and next steps. The regex fallback parses this directly from the transcript.] — a degraded but usable result. This three-tier chain (LLM with retry → Recap regex → degraded note) means sessions are never silently lost.

Step 6: Write the session note

pipeline-core.js writes a markdown file to raw/ with YAML frontmatter:

---
title: "Implement webhook retry with exponential backoff"
domain: work
date: 2026-09-15
tags: [webhooks, reliability, go-backend]
source: session-capture
related: ['[[webhook-delivery-pipeline]]', '[[error-handling-patterns]]']
---

> Added retry logic to webhook delivery with exponential backoff...

## Decisions
- Used exponential backoff with jitter instead of fixed intervals
- Capped retries at 5 attempts over 24 hours

## Open Threads
- Dead-letter queue for permanently failed webhooks not yet implemented

## Session Timeline
| Time | Event |
|------|-------|
| 14:23 | Started implementation |
| 15:47 | Tests passing, PR ready |

Step 7: Mark captured

A <session-id>.captured.json file records the outcome. The observer loads these on startup, so crashes lose no state. Currently 2,096 captured sessions on disk.

The Self-Capture Problem

The hardest edge case in a self-referential system: the observer spawns claude -p for distillation. That subprocess is itself a Claude Code session. If the observer captured its own distillation calls, it would create an infinite loop — each capture triggering another capture.

The solution is a single environment variable: SESSION_CAPTURE_CHILD=1.

Every hook script checks this variable first and exits immediately if set. The distillation subprocess sets it. The wiki compiler sets it. The backfill processor sets it.

# Every hook script starts with this guard
if [ "${SESSION_CAPTURE_CHILD:-}" = "1" ]; then
  exit 0
fi

This is checked across nine files — four hook scripts guard against it, and five scripts set it when spawning claude -p. If any one of them missed it, the system would recurse.

Wiki Compilation

Three times a day, compile-wiki.sh runs via launchd. It:

1. Finds raw session notes not yet in processed.txt
2. Injects the full wiki/index.md (~73KB, 509 pages) into the prompt
3. Runs Sonnet 5 with up to 30 agentic turns
4. The LLM creates new wiki pages, updates existing ones, and maintains cross-references
5. An enforcement backstop strips date prefixes from any wiki filenames the LLM creates
6. Generates a compressed index summary for session injection

The compiler is instructed to update existing pages first. Before creating a new page, it checks the index for an existing page on the same topic. A recent batch processed 74 raw session notes into only 3 new pages and 10 updated pages — the compiler merged related sessions into existing concept pages rather than creating duplicates.

Filename enforcement

Wiki pages use slug.md filenames — concept-oriented, no date prefix. The date lives only in the frontmatter date: field. The compile prompt states this rule, but LLMs do not always follow instructions. So compile-wiki.sh includes a post-compile backstop that renames any date-prefixed file (2026-09-15-some-topic.md → some-topic.md) and fixes all [[wiki-links]] that referenced the old name.

This is a pattern worth noting: prompt says the rule, code enforces it. The LLM is instructed, not trusted.

What the output looks like

The compiler produces genuine knowledge synthesis, not summarization. Here is what a compiled wiki page looks like:

A pattern page has: what it is, why this shape (not the obvious one), the rejected alternative, a code example, when to reach for it, and source links back to the raw session notes.

A work page covering multiple sessions has: merged decisions (with PR numbers and commit hashes), technologies used, open threads, and a session timeline table at the bottom.

A concept page has: definition, related concepts via [[wiki-links]], key sources, and a "what's still unknown" section.

Quality audit

After each compilation, audit-quality.sh runs a two-tier quality check:

Tier 2 samples up to 5 notes and scores them on a 1–5 rubric. This closes the feedback loop — quality is measurable, not subjective.

Quality Gates

A wiki compiled by an LLM needs verification by an LLM. The system runs four post-compile quality gates, each independent and incrementally checkpointed so unchanged pages are not re-checked.

Backlink audit

audit-backlinks.sh scans each compiled page for unlinked mentions of wiki slugs and converts them to [[wiki-links]]. A Perl script skips frontmatter, fenced code, inline code, and existing links to avoid false positives. Only the current batch's changed pages are processed by default.

Contradiction detection

audit-contradictions.sh is a three-phase lint (598 lines):

1. Graph phase (no LLM): builds a link graph from related: frontmatter and body [[wiki-links]], then clusters pages by connected component
2. Detection phase (Haiku): sends each cluster to Haiku for contradiction detection — direct conflicts, temporal inconsistencies, and tension between claims
3. Report phase: aggregates findings by severity into report.md and contradiction-audit.jsonl

Clusters are content-hashed so unchanged clusters are skipped on re-runs.

Source fidelity

audit-source-fidelity.sh verifies that wiki claims match their cited raw sources (663 lines):

1. Structural phase (no LLM): parses ## Sources sections, checks that [[source-*]] pages and raw/ file paths exist. Flags broken references.
2. Content phase (Haiku): spot-checks whether key claims in wiki pages are supported by the cited sources. Flags unsupported claims, distorted claims, and attribution errors.
3. Report phase: aggregates into source-fidelity-audit.jsonl

Compile metrics

compile-metrics.sh tracks proxy metrics for quality regression:

One JSON line per compile run, appended to compile-metrics.jsonl.

State Machine

Each session transitions through five states, all persisted as JSON files on disk:

The observer holds two in-memory structures: a previousSnapshot map (herdr panes → session data) and a capturedSessions set (session IDs). The set rehydrates from .captured.json files on startup, so a daemon crash loses no state.

Cost Breakdown

Transcript parsing, file I/O, and the observer daemon itself are all local — no LLM cost.

The two-phase split is what keeps this affordable. If the compiler ran on every session instead of batching 3×/day, the Sonnet cost alone would be ~$2.25/day.

What We Fixed After the Audit

I ran a formal architecture review on the system, which identified seven weaknesses. Five have been addressed in four sprints:

What remains open

Separate polling from processing. The observer's main loop blocks during distillation (up to 120s). If five sessions end simultaneously, they queue sequentially. A work queue with configurable concurrency would fix this. In practice, sessions rarely end simultaneously — the 30s poll interval spreads them out — so this has not caused problems yet.

Race window between observer and compiler. The observer writes to raw/. The compiler reads raw/. A concurrent write and read on the same file is theoretically possible, though APFS atomic writes make it unlikely for the small files involved.

Key Design Decisions

Running It

The system runs via three launchd plists on macOS:

# Observer daemon — runs continuously, restarts on crash
com.duncanleung.session-capture-observer
  → node _system/session-capture/observer.js
  → KeepAlive + RunAtLoad

# Wiki compiler — runs 3×/day
com.duncanleung.wiki-compile
  → _system/session-capture/compile-wiki.sh
  → StartCalendarInterval: 08:00, 13:00, 20:00

# Watchdog — checks observer health every 5 min
com.duncanleung.session-capture-watchdog
  → _system/session-capture/watchdog.sh
  → StartInterval: 300

The observer needs:

• Node.js (tested on v22)
• jq (for shell config loading)
• Herdr (terminal workspace manager with session.json)
• Claude CLI (claude -p for distillation and compilation)
• An Obsidian vault (or any markdown-file knowledge base)

The Result

After several months of running, the system has captured 2,096 sessions into a 509-page cross-referenced wiki. The wiki covers coding patterns, work decisions, research notes, and project context — all searchable through Obsidian's native search and linked through [[wiki-links]].

The most valuable output is not individual session notes. It is the compiled pages — the ones where the compiler merged five sessions about the same topic into a single concept page with decisions, patterns, rejected alternatives, and open threads. That is the knowledge that would have been lost entirely without automation.

Every coding session now compounds into the next one.