> ## Documentation Index
> Fetch the complete documentation index at: https://warden-re.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Ingestion & normalization
> How WARDEN parses a .wasm binary (and optional JS glue) into a structured Module, fingerprints every function, and seeds the knowledge base with names it can recover for free, all in pure Python with no native toolchain.
Ingestion is the mandatory first step before the Oracle, agents, or diff can run.
`warden ingest` reads the raw binary, recovers everything the file reveals for free, and
writes the result into the knowledge base. By the time the command returns, every function
already has a stable identity, and many have a name.
## Overview
A pure-Python WASM binary parser (`ingest/wasm.py`) reads the file section by section
and produces a `Module`: a structured object containing every function, type, import,
export, memory, element segment, data segment, and the name custom section.
If you pass `--glue`, the regex-driven JS glue parser (`ingest/jsglue.py`) extracts the
Emscripten version string, `dynCall_*` signatures, exported symbol names, import bindings,
and pthread / `PROXY_TO_PTHREAD` markers. None of these live inside the `.wasm` itself.
`identity.fingerprint_function()` derives four complementary fingerprints from each
`Function` and writes them into the `functions` table alongside a `stable_id`, the
carry-over key used by every later stage of the loop. See [core concepts](/concepts) for how the
fingerprints work.
Before the Oracle or agents ever run, `project.ingest_into_kb()` mines the name section,
export table, and import table and writes initial symbols into the KB for every function
where a name is recoverable at high confidence.
## The WASM binary parser
`ingest/wasm.py` implements a complete pure-Python reader of the [WebAssembly binary
format](https://webassembly.github.io/spec/core/binary/). No native toolchain (no WABT,
no Binaryen, no system libraries) is required.
### What gets parsed
The parser walks every standard section in a single left-to-right pass:
| Section | What is recovered |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Type | Function type signatures such as `(i32, i32) -> i32`. Free in every module, including stripped ones. |
| Import | All imports, keyed by `module.field`. Imported functions occupy the low function-index slots. |
| Function | Type-index assignments for locally defined functions. |
| Memory | Limits (minimum/maximum pages), shared flag (threads), and memory64 flag. |
| Global | Mutable/immutable globals with their init expressions. |
| Export | All exported names and the function indices they point to. |
| Element | Table initializers; `ref.func` targets are resolved to indices where statically known. |
| Code | Per-function locals and body bytes; each body is immediately passed to the disassembler. |
| Data | Active and passive data segments; active segments carry a constant-expression offset so string extraction can compute absolute addresses. |
| Name (custom) | The `name` custom section (see below). |
| Other custom | Preserved verbatim in `module.custom_sections` for downstream use. |
| DataCount | The data-count section that precedes the code section in bulk-memory builds. It is read and used to validate segment counts, never treated as an error. |
| Tag | The exception-handling tag section. Tag type indices are recorded so handlers in the code section resolve cleanly. |
| Unknown | Any section id the parser does not model is skipped by its declared length and the pass continues. An unfamiliar or future section never aborts ingestion. |
After the pass, export names are attached to their `Function` objects so the
`module.function_name()` resolver can apply the name-section > export > import priority chain.
### Name resolution priority
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def function_name(self, func: Function) -> str | None:
"""Best available name: name section > export > import field."""
if func.index in self.names.function_names:
return self.names.function_names[func.index]
if func.export_names:
return func.export_names[0]
if func.is_import and func.import_field:
return f"{func.import_module}.{func.import_field}"
return None
```
The name section is ground truth when present. Exports are next. Import field names are a
fallback.
### The name section
`ingest/names.py` parses the `name` WebAssembly custom section. Emscripten retains this
section unless you explicitly strip it (e.g. with `wasm-strip`). It is the richest source of
high-fidelity names in a debug or `--profiling-funcs` build.
Three subsections are parsed:
* **Module name** (subsection 0): the module-level name string.
* **Function names** (subsection 1): a map from function index to name. This is the primary
source for `function_names`.
* **Local names** (subsection 2): per-function local-variable names. Useful for the agent
crew when constructing context.
Later subsections (labels, types, tables) are skipped but recorded in
`NameSection.present_subsections`. A malformed subsection does not abort the parse. It is
silently skipped and the rest of the section is still read.
In a production Emscripten build compiled with `-O2` or higher and no `--profiling-funcs`,
the name section is often absent or covers only a handful of exports. The parser handles
both cases gracefully: `NameSection.present` returns `False` and name-section seeds are
not written.
### Full opcode disassembly
`ingest/opcodes.py` contains the instruction decoder. It covers the MVP plus every extension
that Emscripten actually emits:
`i32.extend8_s`, `i64.extend32_s`, and friends.
The `0xFC` prefix family (sub-opcodes 0–7).
`memory.init`, `data.drop`, `memory.copy`, `memory.fill`, `table.*` (also `0xFC`).
`ref.null`, `ref.is_null`, `ref.func`, `return_call`, `return_call_indirect`.
The full `0xFE` prefix family: `atomic.fence`, notify/wait, and all load/store/rmw/cmpxchg
variants. Central to the `-pthread` story.
The `0xFD` prefix family, with per-sub-opcode immediate layouts (memarg, lane byte, 16-byte
shuffles, and no-immediate ops).
`try`, `catch`, `catch_all`, `throw`, `rethrow`, `delegate`, and the newer `try_table`
form. Tag immediates are decoded so handler structure is preserved.
Block, loop, `if`, and `try` headers accept the empty type, a single value type, or a
type-index that names a full multi-value signature. All three encodings are decoded.
Tail calls (`return_call`, `return_call_indirect`) are decoded alongside the reference-type
ops above, so a tail-call-optimized build disassembles with the same fidelity as a plain one.
Each decoded `Instruction` records its `offset` and `size` inside the function body, its
`opcode` (first byte) and `sub_opcode` for prefixed families, a `mnemonic`, an opcode `klass`
(used for the histogram fingerprint), and the parsed immediates.
**The degradation path.** An opcode whose immediate layout the decoder does not model raises
`UnsupportedOpcode`. The code section parser catches that exception and sets
`func.disasm_error` instead of aborting:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
try:
func.instructions = _disassemble_body(code_bytes)
except (UnsupportedOpcode, IndexError, ValueError) as exc:
func.disasm_error = str(exc)
```
A function with `disasm_error` set has `instructions = None` but still has its full raw
`body` bytes. It is still byte-fingerprintable via `exact_hash` and contributes to the KB
with whatever fingerprints can be derived. The KB notes the error so downstream analysis
knows the disassembly is incomplete.
This is the core guarantee: the disassembler never crashes on an unmodeled opcode. Every
decoded instruction records its own `offset` and `size`, so the stream stays in sync byte
for byte with the body. When the decoder reaches something it cannot model it raises
`UnsupportedOpcode` rather than guessing at a length and drifting, and the code section
parser turns that into a per-function `disasm_error`. The rest of the module parses normally
and ingestion always finishes.
If you are targeting a module built with a very recent WASM proposal or a custom toolchain
extension, you may see `disasm_error` on some functions. Those functions are not lost. They
carry an exact body hash and a structural hash derived from the bytes we did decode, but
the opcode-histogram and MinHash fingerprints will be partial. File an issue with the
offending opcodes if you need them modeled.
## The Emscripten JS glue parser
The JS file Emscripten emits alongside the `.wasm` is what `jsglue.py` calls the Rosetta
stone of the module. Facts that do not exist inside the binary at all live only in the glue:
| Fact | How it is extracted |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Emscripten version | Four regex patterns covering `EMSCRIPTEN_VERSION`, `@emscripten/X.Y.Z`, `GENERATED_BY`, and inline package references. |
| `dynCall_*` signatures | All `dynCall_` identifiers found anywhere in the file; the signature suffix (e.g. `viii`, `iji`) describes the indirect-call ABI. |
| Exported symbols | `Module['_foo']` and `wasmExports['bar']` patterns, plus the `asm[...]` form from older glue. |
| Import bindings | Object literal member patterns like `_emscripten_memcpy_js: _emscripten_memcpy_js`. |
| pthread / threading | Presence of `PThread`, `pthread_create`, `_emscripten_proxy`, `PROXY_TO_PTHREAD`, `proxyToMainThread`, `spawnThread`, `worker.js`, `_emscripten_thread_init`. |
| `PROXY_TO_PTHREAD` | Checked explicitly; activates a separate flag on `GlueInfo`. |
| Memory growth | `ALLOW_MEMORY_GROWTH`, `_emscripten_resize_heap`, `growMemory`. |
The parser is intentionally regex-driven and tolerant. Glue shape shifts across Emscripten
versions and optimization levels. A partial, honest `GlueInfo` is preferable to a hard
failure on an unfamiliar layout. If no version string is found, `GlueInfo.notes` records
that explicitly.
Minification is detected via a crude heuristic: if the average line length exceeds 200
characters, `GlueInfo.minified = True`. This signals that symbol extraction may be
incomplete.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from warden.ingest import parse_glue_file
info = parse_glue_file("app.js")
print(info.emscripten_version) # e.g. "3.1.55"
print(info.dyncall_signatures) # e.g. ["ii", "iii", "viii"]
print(info.uses_pthreads) # True / False
print(info.proxy_to_pthread) # True / False
```
The glue is optional. If you only have the `.wasm`, ingestion still works. You lose
the version string, dynCall signatures, and threading-model facts, but the binary is still
fully parsed. Pass `--glue` whenever the glue file is available; it sharpens every
downstream heuristic.
## What ingest seeds for free
`project.ingest_into_kb()` calls `_seed_symbol_for()` for every function immediately after
fingerprinting. The function mines the three tiers of free naming evidence and writes an
initial `Symbol` into the KB if any evidence is present:
| Evidence tier | Provenance | Confidence | Condition |
| ------------- | ---------- | ---------- | ------------------------------------------------------ |
| Name section | `export` | 0.90 | Function index appears in `NameSection.function_names` |
| Export name | `export` | 0.85 | Function has at least one export name |
| Import field | `import` | 0.80 | Function is imported (`is_import = True`) |
These seeds go through `kb.upsert_symbol()`, the same economy-gated write path used by the
Oracle and agents, so they cannot be accidentally overwritten by lower-authority sources
later. A debug build that retains its name section can reach substantial KB coverage before
you run a single Oracle or agent pass.
## Using `warden ingest`
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
warden init # create warden.db
warden ingest app_v1.wasm --glue app_v1.js --label v1
```
Flags:
* ``: path to the `.wasm` file (required).
* `--glue, -g`: optional Emscripten `.js` glue file.
* `--label, -l`: version label; defaults to the filename stem.
* `--notes`: free-text note stored with this version in the KB.
* `--db`: project database path (default `warden.db`, or `WARDEN_DB` env var).
The command prints a summary table:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Ingested 'v1'
─────────────────────────────────
functions 1 847
imported 312
defined 1 535
seeded symbols 427
emscripten 3.1.55
shared memory yes
```
If pthread markers are found in the glue, a warning is printed. The concurrency model is
relevant to several downstream analysis steps.
## Using `ingest_into_kb` from Python
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from warden.kb import KnowledgeBase
from warden.project import ingest_into_kb
kb = KnowledgeBase("warden.db")
result = ingest_into_kb(
kb,
"app_v1.wasm",
label="v1",
glue_path="app_v1.js", # optional
notes="initial analysis",
)
print(result.num_functions) # total functions (imported + defined)
print(result.num_imported) # functions from the import section
print(result.seeded_symbols) # names written for free before Oracle/agents
print(result.emscripten_version) # from glue, or None
print(result.shared_memory) # True if shared-memory flag is set in the binary
kb.close()
```
`IngestResult.glue_info` holds the full `GlueInfo` object if a glue file was parsed.
`IngestResult.notes` carries any parser warnings (e.g. missing version string, empty export
list from a minified glue).
You can also parse a module without touching the KB:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from warden.ingest import parse_file
module = parse_file("app_v1.wasm")
print(module.version) # should be 1 for all current modules
print(len(module.functions)) # all functions, imported and defined
print(module.shared_memory()) # True if threads are in use
print(module.function_name(module.functions[42])) # best available name
# Inspect one function's disassembly
f = module.defined_functions[0]
if f.instructions is not None:
for ins in f.instructions[:5]:
print(ins.mnemonic, ins.immediates)
elif f.disasm_error:
print("partial disassembly:", f.disasm_error)
# f.body still contains the raw bytes
# Extract strings from active data segments
for addr, s in module.strings(minimum_length=4):
print(f"0x{addr:08x} {s!r}")
```
## What comes next
Ingestion does not run the Oracle, agents, or diff. Those are separate commands you layer
on top. The typical sequence after ingest:
Match every defined function against a corpus of labeled Emscripten/musl/libc signatures
to collapse runtime code instantly.
Propose names for the application-specific remainder, gated by the provenance economy.
When a new version ships, ingest it and diff against the previous label.
Emit a C header, pseudocode listing, git-diffable text, or a Ghidra rename script.