> ## Documentation Index > Fetch the complete documentation index at: https://warden-re.io/llms.txt > Use this file to discover all available pages before exploring further. # Knowledge base schema > Table-by-table reference for the WARDEN SQLite knowledge base: stable identities, provenance ranking, the may_overwrite economy, and the kb-text export format. The WARDEN knowledge base (KB) is a single portable **SQLite file** (one per project), opened with `KnowledgeBase("project.db")`. The schema is applied (idempotently via `CREATE TABLE IF NOT EXISTS`) on every open from `src/warden/kb/schema.sql`, loaded at runtime through `importlib.resources`. No migration runner is required. Two SQLite PRAGMAs are set at open time: `PRAGMA journal_mode = WAL` (concurrent readers alongside the single writer, which matters when `warden mcp` and a local agent run simultaneously) and `PRAGMA foreign_keys = ON` (version-scoped tables cascade-delete when a `module_versions` row is removed). Status: alpha. The schema is functional and exercised end-to-end by `warden demo`. Additional columns (DWARF source info, wasm2c harness references) are tracked in the [roadmap](/project/roadmap). ## The linchpin design choice > Annotations are keyed on a function's **stable content identity**, not on a per-version row-ID. The `functions` table is the **per-version appearance log**: it records that function index 42 appeared in version `v1` with a given `stable_id`. The `symbols` table is the **durable annotation layer**: it holds names, types, summaries, and evidence keyed on that same `stable_id`. When a vendor ships `v2.wasm` and every table index shifts, WARDEN re-ingests and re-fingerprints. A function that was index 42 in `v1` might become index 51 in `v2`, but its `stable_id` (a composite content hash) stays the same. The annotation in `symbols` is still there, automatically. That is the carry-over mechanism that makes RE incremental. See [core concepts](/concepts) for how `stable_id` is computed from structural, semantic, fuzzy, and exact fingerprints. *** ## Table reference ### `meta` Lightweight key/value store for project-level metadata. Accessed via `KnowledgeBase.get_meta()` and `set_meta()`. | Column | Type | Constraints | Notes | | ------- | ---- | ----------- | -------------------------------------------------------------------- | | `key` | TEXT | PRIMARY KEY | Arbitrary metadata key. | | `value` | TEXT | NOT NULL | Current keys: `schema_version` (set to `"1"` on init) and `project`. | *** ### `module_versions` One row per ingested `.wasm` file, representing a point in the target's version history. The `id` here is the foreign key anchor for `functions`, `thread_model`, and `diffs`. | Column | Type | Constraints | Notes | | -------------------- | ------- | -------------------------- | ---------------------------------------------------------------------------------------------------- | | `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Surrogate; referenced throughout. | | `label` | TEXT | NOT NULL UNIQUE | Human-readable name, e.g. `"v1"`. Set via `--label` on ingest. | | `wasm_path` | TEXT | | Recorded path to the original `.wasm`. | | `glue_path` | TEXT | | Path to the accompanying Emscripten JS glue, if any. | | `wasm_sha256` | TEXT | NOT NULL | SHA-256 of the raw bytes; guards against re-ingesting a changed file under the same label. | | `emscripten_version` | TEXT | | Inferred from Oracle matches or the JS glue (e.g. `"3.1.55"`). | | `inferred_flags` | TEXT | | JSON object: Oracle/glue-inferred build flags (`-O2`, `-pthread`, etc.). | | `glue_info` | TEXT | | JSON: parsed `GlueInfo` from the Emscripten JS glue (export map, dynCall signatures, pthread shape). | | `num_functions` | INTEGER | DEFAULT 0 | Total functions (imported + defined). | | `num_imported` | INTEGER | DEFAULT 0 | Imported function count. | | `shared_memory` | INTEGER | DEFAULT 0 | `1` if the module uses shared memory (pthreads). | | `ingested_at` | TEXT | DEFAULT CURRENT\_TIMESTAMP | UTC timestamp of ingest. | | `notes` | TEXT | | Free-text annotation set via `--notes` on ingest. | *** ### `functions` The per-version appearance log. Each row records one function as it appeared in one version, with all fingerprinting data needed for identity matching and Oracle lookup. This table does **not** hold human annotations. Those live in `symbols`. | Column | Type | Constraints | Notes | | ------------------- | ------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------- | | `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Referenced by `oracle_matches`. | | `version_id` | INTEGER | NOT NULL FK → `module_versions.id` ON DELETE CASCADE | Which version this appearance belongs to. | | `func_index` | INTEGER | NOT NULL | WebAssembly function index within this version. | | `stable_id` | TEXT | NOT NULL | **The carry-over key.** Content identity hash. See `idx_functions_stable`. | | `exact_hash` | TEXT | NOT NULL | Hash of the raw instruction bytes; changes on any edit. | | `structural_hash` | TEXT | NOT NULL | CFG-shape hash; robust to constant and immediate changes. See `idx_functions_struct`. | | `minhash` | TEXT | NOT NULL | JSON int array (MinHash signature for fuzzy/Jaccard similarity). | | `histogram` | TEXT | NOT NULL | JSON `{opcode_class: count}` (opcode-class frequency histogram). | | `call_targets` | TEXT | NOT NULL | JSON list of import names this function calls. | | `local_calls` | INTEGER | DEFAULT 0 | Number of calls to other locally-defined functions. | | `type_signature` | TEXT | | WebAssembly type signature (always present for defined functions). | | `instruction_count` | INTEGER | DEFAULT 0 | Total instructions in the function body. | | `body_size` | INTEGER | DEFAULT 0 | Raw byte size of the function body. | | `is_import` | INTEGER | DEFAULT 0 | `1` for imported functions (no body; just a type stub). | | `raw_name` | TEXT | | Name as found in the name section, export table, or import descriptor, if any. | **Unique constraint:** `(version_id, func_index)`. One row per function per version. **Indices:** `idx_functions_stable` on `stable_id`; `idx_functions_version` on `version_id`; `idx_functions_struct` on `structural_hash`. *** ### `symbols` The durable annotation layer. **Keyed on `stable_id`** so that an annotation created against `v1` is automatically present when the same function appears in `v2`, `v3`, and beyond. All writes go through `KnowledgeBase.upsert_symbol()`, which enforces the provenance/confidence economy (see below). | Column | Type | Constraints | Notes | | ---------------- | ------- | ----------------------------- | ------------------------------------------------------------------------------------------------- | | `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | | | `stable_id` | TEXT | NOT NULL | Content identity. Indexed (`idx_symbols_stable`). | | `kind` | TEXT | NOT NULL DEFAULT `'function'` | `'function'` \| `'global'` \| `'struct'` \| `'type'`. | | `name` | TEXT | | Recovered or assigned name. | | `type_signature` | TEXT | | Inferred or confirmed type. | | `summary` | TEXT | | Free-text description of what the function does. | | `provenance` | TEXT | NOT NULL | Who produced this annotation. See [Provenance ranking](#provenance-ranking). | | `confidence` | REAL | NOT NULL DEFAULT 0.0 | `0.0`–`1.0`. Drives the write-economy gate. | | `evidence` | TEXT | | JSON list of `{kind, detail}` objects (the evidence trail). | | `source_ref` | TEXT | | Upstream source URL or file/line (e.g. a musl commit URL). Set for Oracle-confirmed runtime code. | | `locked` | INTEGER | DEFAULT 0 | `1` means human-verified; immutable to any automated write. | | `created_at` | TEXT | DEFAULT CURRENT\_TIMESTAMP | | | `updated_at` | TEXT | DEFAULT CURRENT\_TIMESTAMP | Updated on every accepted write. | **Unique constraint:** `(stable_id, kind)`. One annotation record per identity per kind. Never insert directly into `symbols`. Always go through `upsert_symbol()`. A direct INSERT bypasses the economy gate and can silently clobber a human-verified, locked annotation. *** ### `structs` Recovered memory-region and struct layouts, stored as first-class structured data rather than free-text comments. | Column | Type | Constraints | Notes | | ------------ | ------- | -------------------------- | ------------------------------------------------------------ | | `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | | | `name` | TEXT | NOT NULL UNIQUE | Struct name, e.g. `"pthread_mutex_t"`. | | `layout` | TEXT | NOT NULL | JSON list of `{offset, size, type, name}` field descriptors. | | `provenance` | TEXT | NOT NULL | Typically `"agent"` or `"oracle"`. | | `confidence` | REAL | NOT NULL DEFAULT 0.0 | | | `notes` | TEXT | | Free-text notes. | | `updated_at` | TEXT | DEFAULT CURRENT\_TIMESTAMP | | Written via `KnowledgeBase.upsert_struct()`, which uses `ON CONFLICT(name) DO UPDATE`. *** ### `thread_model` The concurrency spine: lock-to-guarded-data relationships, atomic sites, TLS references, and worker entry points. Version-scoped because the threading model may change between releases. | Column | Type | Constraints | Notes | | -------------- | ------- | ------------------------------------------- | ------------------------------------------------------------ | | `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | | | `version_id` | INTEGER | FK → `module_versions.id` ON DELETE CASCADE | | | `kind` | TEXT | NOT NULL | `'lock'` \| `'atomic'` \| `'tls'` \| `'worker-entry'`. | | `site` | TEXT | | Function `stable_id` or address where the site was observed. | | `guarded_data` | TEXT | | Named memory region this lock protects (for `kind='lock'`). | | `detail` | TEXT | | Additional free-text detail. | | `provenance` | TEXT | NOT NULL DEFAULT `'agent'` | | | `confidence` | REAL | NOT NULL DEFAULT 0.0 | | Written via `KnowledgeBase.add_thread_fact()`; retrieved via `thread_facts(version_id)`. *** ### `oracle_matches` Records of individual Oracle identifications: a function fingerprint matched to a known upstream runtime or libc function. References `functions.id` (not `stable_id`), so each match is version-specific: the same logical function may match at a different `func_index` in each version, generating a separate row. | Column | Type | Constraints | Notes | | -------------------- | ------- | ---------------------------------------------- | ---------------------------------------------------------------------------------- | | `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | | | `function_id` | INTEGER | NOT NULL FK → `functions.id` ON DELETE CASCADE | Which per-version function row was matched. | | `matched_name` | TEXT | NOT NULL | The real upstream name, e.g. `"__stdio_write"`. | | `library` | TEXT | | Source library: `musl` \| `emscripten` \| `libc++` \| `compiler-rt` \| `dlmalloc`. | | `emscripten_version` | TEXT | | Emscripten version range where this signature was observed. | | `opt_level` | TEXT | | Optimization level of the corpus artifact, e.g. `"-O2"`. | | `score` | REAL | NOT NULL | Match score (0.0–1.0). | | `source_ref` | TEXT | | Upstream source URL or file/line from the corpus artifact. | **Unique constraint:** `(function_id, matched_name)`. An Oracle match is a separate record from the `symbols` annotation it triggers. `warden oracle identify` writes both: the match record here, and a `symbols` row (provenance `"oracle"`) via `upsert_symbol`. *** ### `diffs` Stored cross-version diff reports (the semantic changelogs produced by `warden diff`). One row per ordered pair of versions. The `report` column holds the full JSON classification (unchanged, structurally-equivalent, fuzzy-matched, added, removed) and carry-over counts. | Column | Type | Constraints | Notes | | ----------------- | ------- | ------------------------------------------- | -------------------------------------------------------------------------- | | `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | | | `from_version_id` | INTEGER | FK → `module_versions.id` ON DELETE CASCADE | Earlier version. | | `to_version_id` | INTEGER | FK → `module_versions.id` ON DELETE CASCADE | Later version. | | `report` | TEXT | NOT NULL | JSON diff report (function classifications, carry-over counts, changelog). | | `created_at` | TEXT | DEFAULT CURRENT\_TIMESTAMP | | **Unique constraint:** `(from_version_id, to_version_id)`. *** ### `audit_log` Append-only trail of every symbol write attempt, whether accepted or rejected. Rows are never modified, only inserted. Used to answer "who changed this name and when?" and "why was this agent proposal rejected?". | Column | Type | Constraints | Notes | | ------------ | ------- | -------------------------- | --------------------------------------------------------------------- | | `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Monotonically increasing. | | `stable_id` | TEXT | | The identity that was affected. May be `NULL` for non-symbol actions. | | `action` | TEXT | NOT NULL | `'created'` \| `'updated'` \| `'rejected'`. | | `actor` | TEXT | NOT NULL | Provenance of the writer (`'human'`, `'oracle'`, `'agent'`, etc.). | | `detail` | TEXT | | Human-readable reason string returned by `may_overwrite()`. | | `created_at` | TEXT | DEFAULT CURRENT\_TIMESTAMP | UTC timestamp. | Retrieve recent entries with `KnowledgeBase.audit_log(limit=100)`. *** ## Deep analysis tables: per-function analysis, renames, and live events Schema version 2 adds three tables for the per-function deep agent engine. They are created with `CREATE TABLE IF NOT EXISTS`, so an existing project upgrades in place on the next open. ### `function_analysis` One row per function (keyed on `stable_id`, like `symbols`) holding the distilled per-function understanding that flows up to a parent agent. Raw WASM is re-derived from the module on demand and is never stored here. | Column | Type | Notes | | --------------- | ---- | --------------------------------------------------------------------- | | `stable_id` | TEXT | UNIQUE. The identity the analysis attaches to. | | `raw_c` | TEXT | The lifter's decompiled C (generic variable names). | | `clean_c` | TEXT | The agent's cleaned C with variables renamed. | | `understanding` | TEXT | A short paragraph on what the function does. | | `rename_map` | TEXT | JSON `{generic_name: meaningful_name}`. | | `status` | TEXT | `'open'` or `'closed'`. A closed function is reused, not re-analyzed. | | `model_used` | TEXT | Which backend or model produced it. | | `confidence` | REAL | Calibrated `[0, 1]`. | Read with `KnowledgeBase.get_analysis(stable_id)` or `analyses_for_version(version_id)`; write with `upsert_analysis(...)`, which is partial (a `None` field keeps the existing value). ### `rename_history` Append-only log of every name, variable, and summary change, so the whole session is reversible. | Column | Type | Notes | | ------------------------- | ------- | ------------------------------------------------ | | `stable_id` | TEXT | The function affected. | | `kind` | TEXT | `'symbol-name'`, `'variable'`, or `'summary'`. | | `target` | TEXT | The variable for `kind='variable'`, else `NULL`. | | `old_value` / `new_value` | TEXT | Before and after. | | `actor` | TEXT | Who made the change. | | `reverted` | INTEGER | `1` once undone. | `add_rename(...)` records a change; `revert_rename(rename_id)` applies the inverse; `rename_history_for(stable_id)` lists the trail. ### `agent_events` The live activity feed the UI polls so you can watch the per-function agents work in real time. | Column | Type | Notes | | -------------------------- | -------------- | ---------------------------------------------------------------------------------------- | | `version_id` | INTEGER | The module version being analyzed. | | `stable_id` / `func_index` | TEXT / INTEGER | Which function. | | `phase` | TEXT | `queued`, `analyzing`, `proposed`, `written`, `closed`, `reused`, `skipped`, or `error`. | | `detail` | TEXT | A short message (often the proposed name). | `record_event(...)` appends an event; `events_since(after_id)` returns newer events for polling. *** ## Provenance ranking Every symbol write carries a `provenance` label. `PROVENANCE_RANK` in `src/warden/kb/models.py` assigns an integer priority that the `may_overwrite()` gate uses to decide whether a new write may displace an existing annotation. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} PROVENANCE_RANK = { "human": 100, "oracle": 90, "export": 60, "import": 55, "string-xref": 50, "diff-carry": 40, "agent": 30, } ``` Any provenance string not present in this dict defaults to rank `10`. Human is sovereign at the top; raw agent proposals sit at the bottom. The ranking is also shown in the [core concepts](/concepts) table. *** ## The `may_overwrite()` economy `may_overwrite()` in `src/warden/kb/models.py` is the gatekeeper called by every `KnowledgeBase.upsert_symbol()` invocation. It receives the new write's provenance and confidence alongside the existing annotation's provenance, confidence, and lock state, and returns `(allowed: bool, reason: str)`. The rules, exactly as implemented: If `existing_provenance is None` the slot is empty and any write succeeds. Reason: `"new symbol"`. If `new_provenance == "human"` the write wins unconditionally, regardless of what already exists. Reason: `"human override"`. A subsequent `lock_symbol()` call then sets `locked = 1` to protect it. If `existing_locked` is true and the writer is not `"human"`, the write is rejected. Reason: `"existing symbol is locked (human-verified)"`. The `locked` flag is set by `warden set-name` (which locks by default) and by `KnowledgeBase.lock_symbol()`. An agent may only write if one of two conditions holds: * The existing annotation is also `"agent"` and `new_confidence > existing_confidence` (strictly higher). Reason: `"higher-confidence agent write"`. * The existing provenance has a rank strictly below 30 (the agent rank), i.e. an unknown source. Reason: `"outranks existing automated source"`. An agent can never overwrite `oracle`, `export`, `import`, `string-xref`, `diff-carry`, or `human` annotations. The rejection reason names the existing provenance and its confidence. For all other provenances (oracle, export, import, string-xref, diff-carry): * Higher rank wins outright. * At equal rank, `new_confidence >= existing_confidence` is required. * Otherwise the existing annotation takes precedence. Rejected writes are not errors. They return `(False, reason)` and are appended to `audit_log` with `action='rejected'`. This lets you re-run the entire agent crew on every update without clobbering any verified human work or high-confidence Oracle annotations. ### Example economy decisions | New write | Existing | Outcome | | ------------------- | ------------------- | ---------------------------------------- | | `agent`, conf 0.80 | (empty) | Allowed: new symbol | | `agent`, conf 0.80 | `agent`, conf 0.60 | Allowed: higher-confidence agent write | | `agent`, conf 0.80 | `agent`, conf 0.90 | Rejected: lower confidence | | `agent`, conf 0.95 | `oracle`, conf 0.85 | Rejected: agent may not overwrite oracle | | `oracle`, conf 0.90 | `agent`, conf 0.95 | Allowed: oracle outranks agent | | `human`, conf 1.00 | `oracle`, locked | Allowed: human override | | `oracle`, conf 0.90 | `human`, locked | Rejected: symbol is locked | *** ## KnowledgeBase API `KnowledgeBase` (in `src/warden/kb/database.py`) is a pure-stdlib `sqlite3`-backed class that supports the context-manager protocol. Schema is applied idempotently on every open. ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from warden.kb import KnowledgeBase with KnowledgeBase("project.db") as kb: ... ``` ### Representative calls ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} version_id = kb.add_module_version( label="v1", wasm_sha256="abc123...", wasm_path="app_v1.wasm", emscripten_version="3.1.55", num_functions=512, num_imported=40, shared_memory=True, ) v = kb.get_version("v1") # -> ModuleVersion | None vs = kb.versions() # -> list[ModuleVersion], ordered by id latest = kb.latest_version() # -> ModuleVersion | None ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} from warden.kb import Symbol written, reason = kb.upsert_symbol(Symbol( stable_id="a3f9c0b2...", kind="function", name="malloc", provenance="oracle", confidence=0.97, source_ref="https://github.com/emscripten-core/emscripten/blob/main/system/lib/dlmalloc.c#L...", evidence=[{"kind": "oracle-match", "detail": "score=0.97 lib=dlmalloc"}], )) # written=True, reason='new symbol' # written=False, reason='...' if a higher-authority annotation already exists sym = kb.get_symbol("a3f9c0b2...", kind="function") # -> Symbol | None ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} funcs = kb.functions_for_version(version_id, include_imports=False) symbols = kb.symbols_for_stable_ids([f["stable_id"] for f in funcs]) # symbols: dict[stable_id -> Symbol] ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} c = kb.coverage(version_id) print(f"{c.named}/{c.defined} ({c.coverage_pct}%)") print(f" oracle={c.oracle_named} human={c.human_named} agent={c.agent_named}") ``` `coverage_pct` is computed over defined functions only (imports excluded from the denominator). ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} kb.record_oracle_match( function_id=func_row_id, matched_name="__stdio_write", library="musl", emscripten_version="3.1.55", opt_level="-O2", score=0.94, source_ref="https://git.musl-libc.org/...", ) ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} # Lock against any future automated write. kb.lock_symbol("a3f9c0b2...", kind="function") ``` Equivalent to `warden set-name ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} kb.store_diff(from_version_id=1, to_version_id=2, report=report_dict) report = kb.get_diff(1, 2) # -> dict | None ``` ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} entries = kb.audit_log(limit=50) # list of dicts: id, stable_id, action, actor, detail, created_at ``` *** ## The kb-text export format `warden export