> ## Documentation Index
> Fetch the complete documentation index at: https://warden-re.io/llms.txt
> Use this file to discover all available pages before exploring further.
# The Emscripten Oracle
> How WARDEN compiles its own ground truth from the Emscripten toolchain, fingerprints the result into a signature store, and uses it to auto-identify runtime functions in a stripped target, collapsing 40–80% of a typical module to known code.
The Oracle is the stage that typically moves the biggest block of coverage in a single step.
Emscripten, musl, dlmalloc/emmalloc, libc++, and the whole WebAssembly runtime are open source.
WARDEN exploits that: it compiles the exact runtime a target was built with, across the
build-flag matrix that affects codegen, retains the name section, fingerprints every labeled
function into a portable store, and then runs that store against the stripped target. Functions
that match at or above the acceptance threshold get the real upstream name attached immediately,
with provenance `oracle`.
**Alpha status.** The matching engine, signature store, MinHash-LSH index, and the matrix
corpus farm (the `build_matrix.sh` matrix builder, its `manifest.json`, and the
`harvest_directory` harvest step) are all implemented and working, along with the robustness
work covered below: deterministic `identify` tie-breaking, calibrated version-inference
confidence, `SignatureStore.stats()` introspection, and the `evaluate_identification`
precision/recall harness. The seed store that ships with a fresh clone is intentionally empty.
You build your own. The large multi-thousand-signature corpus that would let the Oracle
identify most of a real target still needs the full emsdk matrix run (Docker plus emsdk); the
precision/recall numbers you can produce today are calibration on the labeled modules you have,
not a published accuracy figure. A shared community corpus is on the
[roadmap](/project/roadmap).
***
## Compile your own ground truth
No other FLIRT- or BinDiff-style workflow auto-builds its corpus from the toolchain's own source
across the version/flag matrix. The critical ingredient is **`--profiling-funcs`**.
Without it, an optimized Emscripten build strips the wasm name section. The corpus builder,
`extract_signatures`, looks up each function's name from:
1. `module.names.function_names`: the wasm name section (primary source).
2. `func.export_names[0]`: used if the function is exported and the name section is absent.
If neither yields a name, the function is **silently skipped**. Retaining names is not optional: it is the entire source of ground truth. Use `--profiling-funcs` (or `-g` / `-gsource-map`) on
every corpus build. This flag forces Emscripten to emit the name section even at `-O2` and `-Oz`
where it would otherwise be stripped.
***
## The signature store
### `Signature`
Defined in `src/warden/oracle/signatures.py`. One `Signature` is a labeled fingerprint of a
single function extracted from a corpus build.
| Field | Type | Meaning |
| -------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `str` | The real upstream name, from the wasm name section or export. |
| `library` | `str` | Which library: `musl`, `musl-pthread`, `libc++`, `libc++abi`, `compiler-rt`, `dlmalloc`, `emscripten`, `wasi-libc`, or the `--library` default. |
| `emscripten_version` | `str \| None` | The emsdk version these were built with. |
| `opt_level` | `str \| None` | The `-O` level, e.g. `-O2`. |
| `source_ref` | `str \| None` | Optional upstream source link. |
| `structural_hash` | `str` | Blake2b of the normalized control-flow/opcode skeleton. |
| `exact_hash` | `str` | SHA-256 of the raw function body. |
| `minhash` | `list[int]` | 32-element MinHash over instruction 4-grams (fuzzy matching). |
| `histogram` | `dict[str, int]` | Opcode-class frequency histogram. |
| `call_targets` | `list[str]` | Import call-neighborhood (`module.field` strings). |
| `type_signature` | `str` | WASM type signature string. |
| `instruction_count` | `int` | Total instruction count. |
| `local_calls` | `int` | Count of calls to locally-defined functions. |
`Signature.fingerprint()` reconstructs a `FunctionFingerprint` so the same `similarity()`
function used everywhere in WARDEN can compare a corpus entry against a live target function.
Library classification is automatic, based on name prefixes. `classify_library`
walks an ordered prefix table and returns the first library whose prefix matches the
name. Anything that matches no prefix keeps the fallback `--library` value (default
`musl`). The full table:
| Prefix | Assigned library |
| -------------------------------------------------------------------- | ------------------------------- |
| `emscripten_`, `__em_` | `emscripten` |
| `pthread_`, `__pthread` | `musl-pthread` |
| `_ZNSt`, `_ZSt`, `_ZN` | `libc++` |
| `__cxa_` | `libc++abi` |
| `__muldi3`, `__divdi3`, `__udivdi3`, `__fixunsdfdi`, `__floatunsidf` | `compiler-rt` |
| `dlmalloc`, `dlfree` | `dlmalloc` |
| `__wasi`, `wasi_` | `wasi-libc` |
| (no match) | whatever `--library` you passed |
The `compiler-rt` row covers the soft-float and wide-integer builtins Emscripten
links from compiler-rt. These show up in almost every non-trivial module and used to
land in the `musl` fallback bucket, which made the per-library breakdown misleading.
### `SignatureStore`
A `SignatureStore` is an ordered list of `Signature` objects with JSON persistence. The on-disk
format is plain JSON (portable, git-diffable, and shareable without a database).
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
store = SignatureStore.load("oracle.json") # returns empty store if file is missing
store.add(sig)
store.extend(sigs) # bulk add, deduplicated (see below)
store.save("oracle.json")
len(store) # number of signatures
store.libraries() # dict[library_name, count]
store.stats() # full introspection summary (see below)
```
#### Introspecting a store with `stats()`
`store.stats()` returns one deterministic summary dict for a store, so you can audit
what a corpus actually covers without loading the JSON by hand. It rolls up the counts
that matter when you are deciding whether a corpus is broad enough to identify a given
target:
| Key | Type | Meaning |
| ----------------------- | ---------------- | -------------------------------------------------------------------------------- |
| `total` | `int` | Total number of signatures (same as `len(store)`). |
| `by_library` | `dict[str, int]` | Per-library signature count (same as `libraries()`). |
| `by_emscripten_version` | `dict[str, int]` | Per-version signature count; a signature with no version is counted under `"?"`. |
| `distinct_names` | `int` | How many distinct upstream names are covered. |
Every dict is ordered for stable output: keys are sorted so the same store always prints
the same summary, which keeps it safe to snapshot in a test or diff across runs, and the
result does not depend on the order signatures were added. A store whose
`by_emscripten_version` covers only one version will identify fewer functions in a target
built differently. `stats()` makes that gap visible before you run `identify` and wonder why
coverage is low.
#### Signature dedup on `extend`
The matrix farm compiles the same reference programs across many versions and opt
levels, so the same function body recurs constantly. `extend` deduplicates as it adds:
a new signature is dropped when an existing entry already shares the same identity key.
The key is the tuple of `(name, library, emscripten_version, opt_level, exact_hash)`,
so two builds of `memcpy` that produced a byte-identical body at the same version and
`-O` level collapse to one entry, while a genuinely different `-Oz` body is kept. This
keeps `oracle.json` from ballooning when you re-run the farm or harvest the same
directory twice, and it makes the build idempotent: harvesting the same tree again does
not grow the store.
Dedup happens on `extend` only, the path every corpus and harvest build takes. `add`
is the raw single-signature append and does not deduplicate, so a manual `add` loop
can still create duplicates if you bypass `extend`.
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"version": 1,
"count": 214,
"signatures": [ { "name": "memcpy", "library": "musl", ... }, ... ]
}
```
### The shipped seed store is empty
`src/warden/oracle/seed_signatures.json` ships with `"count": 0` and an empty `signatures`
array. This is deliberate: a pre-built corpus would encode assumptions about which emsdk
versions and flag combinations matter for your target. You build the store that matches your
target and commit it alongside your project.
***
## Identifying functions: `identify` and `ORACLE_THRESHOLD`
`identify` is defined in `src/warden/oracle/match.py`. Given a `KnowledgeBase`, a `version_id`,
and a `SignatureStore`, it:
Imports are excluded. They already have names from the JS glue.
Uses `fingerprint_from_record`. No re-parsing of the original `.wasm` is required.
The `similarity()` function combines four signals:
| Signal | Weight | Method |
| ---------------------------- | ------------------ | ---------------------------- |
| Exact body equality | shortcircuit → 1.0 | SHA-256 match |
| Fuzzy instruction similarity | 0.45 | MinHash Jaccard over 4-grams |
| Opcode-class histogram | 0.25 | cosine similarity |
| Call-neighborhood overlap | 0.20 | import Jaccard |
| Structural skeleton match | +0.10 | hash equality bonus |
The best-scoring signature wins. If `score >= threshold`, it is an Oracle match.
#### Deterministic tie-breaking
Two signatures can score identically against one target function. The matrix farm makes
this common: the same `memcpy` body compiled at the same `-O` level under two different
emscripten versions produces two signatures that are byte-identical apart from their
version label, so both score `1.0`. Picking whichever happened to come first in the store
would make the result depend on insertion order, and the same target could be labeled
`3.1.50` on one run and `3.1.55` on the next.
To keep `identify` deterministic, ties are broken by a fixed, total ordering rather than
by store position. Among signatures that tie on score, the Oracle prefers, in order:
1. the higher `score` (the primary key, already decided above),
2. then the lexicographically smaller `name`,
3. then the lexicographically smaller `source_ref` (a missing `source_ref` is treated as
the empty string, so it sorts first).
Because every field in that chain is part of the signature itself and the comparison is
total, identical inputs always yield the same `OracleMatch` regardless of the order
signatures were added to the store. `identify_indexed` applies the same tie-break over its
candidate pool, so the indexed path returns the same match as the linear scan.
The acceptance threshold is:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
ORACLE_THRESHOLD = 0.82 # src/warden/oracle/match.py
```
When `write=True` (the default), every accepted match is written back to the KB with:
* Provenance `"oracle"` and `confidence = score`.
* A `Symbol` entry with the matched name, type signature, and a summary noting library and version.
* An evidence trail: `{"kind": "oracle", "detail": " @ score="}`.
* A row in `oracle_matches` keyed to the internal function ID.
Human-set names (`provenance = "human"`, `locked = True`) are sovereign. The Oracle cannot
overwrite them, no matter the score.
The `--threshold` flag overrides `ORACLE_THRESHOLD` at the CLI. Lower values catch more functions
at the cost of false positives; higher values are conservative. The default of `0.82` is a
reasonable starting point for functions of moderate size, but you should audit matches after
the first run.
### The `OracleMatch` return type
`identify` returns `list[OracleMatch]`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@dataclass
class OracleMatch:
func_index: int
stable_id: str
matched_name: str
library: str
emscripten_version: str | None
opt_level: str | None
score: float
source_ref: str | None
```
***
## Scaling: the MinHash-LSH index
The default `identify` scan is O(targets × signatures). For a corpus of a few hundred signatures
that is negligible, but once the matrix farm has accumulated thousands of entries across many
versions and opt levels the linear pass becomes a bottleneck. `warden.oracle.index` provides
`SignatureIndex`, a banded MinHash + structural-hash index that reduces each function's search
space to a small set of candidates before any scoring happens.
### How it works
`SignatureIndex.build(store, *, bands=None)` splits each signature's 32-element MinHash into
`bands` equal slices and hashes each slice into a bucket. Two signatures that agree on *any*
whole band land in the same bucket. The probability of that collision grows with their true
Jaccard similarity, so near-matches across `-O` levels still surface. Signatures are also indexed
by their exact `structural_hash`, so structurally identical functions are always found regardless
of their MinHash values.
#### Automatic band selection
`bands` controls the recall/precision tradeoff of the index. Few wide bands cast a wide net
(high recall, larger candidate sets); many narrow bands are selective (smaller candidate sets,
slightly lower recall). The right number depends on how many signatures are in the store, so
you do not have to guess: when you pass `bands=None` (the default), `build` calls
`recommended_bands(len(store))` to pick a value scaled to the store size.
| Store size | `recommended_bands` |
| --------------------------- | ------------------- |
| empty or tiny (`< 64`) | `4` |
| small (`64` to `999`) | `8` |
| medium (`1_000` to `9_999`) | `16` |
| large (`>= 10_000`) | `32` |
The function returns a divisor of 32 (the MinHash width) so the bands partition the signature
evenly with no remainder. As the matrix farm grows the corpus from a few hundred to tens of
thousands of signatures, the index narrows its bands automatically and the candidate sets stay
small. Pass an explicit integer to `bands` to override the heuristic.
`index.candidates(fp)` returns the deduplicated union of every signature sharing a band-bucket
or the structural hash with `fp`. This is a pure dictionary lookup on the hot path.
`identify_indexed` is the LSH-accelerated mirror of `identify`. When the index yields no
candidates for a function it falls back to the full store, so no function is ever silently
dropped. The result is identical to the linear scan at `ORACLE_THRESHOLD = 0.82`.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from warden.oracle.index import SignatureIndex, identify_indexed
from warden.oracle import SignatureStore
from warden.kb import KnowledgeBase
store = SignatureStore.load("oracle.json")
index = SignatureIndex.build(store) # bands=None -> recommended_bands(len(store))
# Query candidates for one fingerprint
candidates = index.candidates(fp) # list[Signature], usually << len(store)
# Full pass: same matches as identify(), faster on large stores
with KnowledgeBase("project.db") as kb:
v = kb.get_version("v1")
matches = identify_indexed(kb, v.id, store, threshold=0.82, write=True)
```
Leave `bands` at its `None` default and let `recommended_bands` scale it to the store. Only
pass an explicit value when you are tuning: raise it to narrow candidates further, lower it to
catch more edge cases at the cost of a larger candidate set.
***
## Version inference
After `identify` returns, call `infer_version` to infer which Emscripten version the target was
built with:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def infer_version(matches: list[OracleMatch]) -> VersionInference:
...
@dataclass
class VersionInference:
emscripten_version: str | None # e.g. "3.1.55"
confidence: float # calibrated, see below
histogram: dict[str, int] # full vote distribution
```
The implementation tallies `emscripten_version` across all matches that carry one. The
version with the highest vote count (the plurality) wins. Ties on vote count are broken by
the smaller version string, so the winner is deterministic when two versions draw. Matches
with no `emscripten_version` are ignored, and the `histogram` reports only the
version-bearing votes.
#### Calibrated confidence
Raw vote share alone overstates certainty. Two matches that both vote `3.1.55` give a vote
share of `1.0`, but two matches is thin evidence, and a target whose runtime spans several
versions can still produce a clear plurality winner that does not deserve full confidence.
The reported `confidence` is therefore the plurality share calibrated down by sample size:
* **Plurality share.** The base figure is the winner's votes over all version-bearing votes.
A clean sweep approaches `1.0`; a split field scores proportionally lower (a 2-2 tie gives
the winner a share of `0.5`).
* **Sample size damping.** A handful of matches cannot pin a version as firmly as hundreds
can, so the share is multiplied by `total / (total + 1)` over the version-bearing match
count. That factor is `0.5` for a single vote and rises toward `1.0` as evidence
accumulates, so a lone match is deliberately modest even though its raw share is `1.0`.
The result stays in `[0.0, 1.0]` and is fully deterministic: identical matches give an
identical confidence, with no clock or random input. When 75% of a few hundred Oracle
matches are tagged `3.1.55`, the calibrated confidence is high; when only two matches agree,
it is held down by the damping factor.
This version pin matters downstream:
* It is stored on the `ModuleVersion` row and shown by `warden versions`.
* The diff engine uses it to classify toolchain churn: if `v1` infers `3.1.50` and `v2` infers
`3.1.61`, functions that changed can be re-examined against the Oracle and reported as
"changed due to Emscripten upgrade" rather than "application change".
* It sharpens dynCall/elem-table convention assumptions for any subsequent JS glue analysis.
The accuracy of the inference is bounded by corpus coverage. A wider flag matrix and more
reference programs produce more matches and a more confident vote.
***
## Measuring identification quality: `evaluate_identification`
A threshold of `0.82` is a starting point, not a proof. Before you trust the Oracle on a
real target you want numbers: how many of its matches are correct, and how many real runtime
functions does it miss. `evaluate_identification` is a precision/recall harness for exactly
that, and it runs against a labeled module without needing emsdk or Docker on the machine.
The idea is to evaluate against ground truth you already have. A corpus build is itself a
labeled module: every function in it carries its real name from the wasm name section. So you
can build a store from one set of labeled modules, run `identify` against a *held-out* labeled
module whose true names are known, and compare the Oracle's answers to those names. Because the
held-out module still has its name section, the harness knows the correct answer for every
function and can score the run.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from warden.oracle import evaluate_identification, SignatureStore
from warden.kb import KnowledgeBase
store = SignatureStore.load("oracle.json")
with KnowledgeBase("heldout.db") as kb:
v = kb.get_version("v1") # an ingested module that still has its true names
report = evaluate_identification(kb, v.id, store, threshold=0.82)
print(report["precision"], report["recall"])
```
It returns a JSON-friendly dict. For each labeled function (one whose `raw_name` survived
ingest from the wasm name section or an export) the harness compares the Oracle's predicted
name against that ground truth and bins the outcome. Functions with no `raw_name` carry no
label, so they contribute nothing to either metric.
| Key | Definition |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `checked` | Labeled functions in the version (those with a `raw_name` ground truth). |
| `true_positive` | Matches whose predicted name equals the ground truth. |
| `false_positive` | Matches that landed on a labeled function but disagreed with the truth. |
| `precision` | `tp / (tp + fp)`: of the matches checked against a label, the fraction correct. Falls back to `0.0` when nothing matched. |
| `recall` | `tp / checked`: of the labeled functions, the fraction the Oracle named correctly. Falls back to `0.0` when nothing was labeled. |
A match on an unlabeled function is ignored: there is no ground truth to confirm or deny it,
so it never inflates the false-positive count.
The whole evaluation is read-only and deterministic: it never writes back to the KB (it scores
the matches `identify` would produce), and the same store, target, and threshold always yield
the same report. That makes it a clean way to sweep the threshold. Run it at several values,
watch precision rise and recall fall as you raise the bar, and pick the operating point that
fits your tolerance for false positives.
This harness measures quality on a held-out labeled module, which you can produce on any
machine. It does not replace a broad corpus. A genuinely confident precision/recall number,
and a corpus large enough to identify most of a real target, still needs the full
multi-thousand-signature emsdk matrix run described below, which requires Docker and emsdk.
Until that run exists, treat these numbers as a calibration tool on the modules you have, not
as a published accuracy figure.
***
## CLI usage
### Step 1: Build a signature store
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
warden oracle build [ ...] \
--out oracle.json \
--emver 3.1.55 \
--opt -O2 \
[--library musl]
```
* ``: one or more `.wasm` files with a name section (`--profiling-funcs` at compile time).
* `--out` / `-o`: the output (or existing) `oracle.json`. If the file already exists, new
signatures are appended; the CLI reports `+N new`.
* `--emver`: the Emscripten version string to tag these signatures with.
* `--opt`: the optimization level (e.g. `-O2`, `-Oz`).
* `--library`: fallback label for names that do not match any known prefix (default `musl`).
After building, the CLI prints the total signature count and the per-library breakdown:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Wrote 214 signatures (+214) to oracle.json
Libraries: dlmalloc=8, emscripten=12, libc++=34, musl=160
```
### Inspect a store before you trust it
`warden oracle inspect` prints what a store covers, so you can audit a corpus without opening
the JSON. It is the CLI surface over `SignatureStore.stats()`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
warden oracle inspect oracle.json
```
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Store: oracle.json
Signatures: 214
Distinct names: 187
Libraries: dlmalloc=8, emscripten=12, libc++=34, musl=160
Versions: 3.1.55=214
```
The `Versions` line maps to `by_emscripten_version` and is the one to read closely. A store
that covers only one version (as above) will identify fewer functions in a target built with a
different toolchain. A broad corpus from the matrix farm shows several versions here. Every
line is sorted, so the same store always prints the same summary.
`warden oracle inspect ` is the CLI surface over `SignatureStore.stats()`. It is being
wired into the CLI separately; `stats()` is the stable library entry point in the meantime.
Both read the same `oracle.json` and report the same counts.
### Step 2: Identify runtime functions in a target
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
warden oracle identify v1 \
--store oracle.json \
[--threshold 0.82] \
[--indexed] \
[--db warden.db]
```
* `v1`: a version label already ingested with `warden ingest`.
* `--store` / `-s`: path to the `oracle.json` built above (default `oracle.json`).
* `--threshold`: override `ORACLE_THRESHOLD`.
* `--indexed`: use the MinHash-LSH index (`identify_indexed`) instead of the linear scan.
Produces identical matches but is significantly faster once the store grows large. The index is
built in memory at the start of each run. No separate build step is required.
The CLI prints a match table and, when `infer_version` succeeds, the toolchain line:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
idx name library score emver
─── ──────────────── ──────── ───── ──────
12 memcpy musl 1.00 3.1.55
14 memset musl 1.00 3.1.55
19 __cxa_throw libc++abi 0.91 3.1.55
...
Inferred Emscripten version: 3.1.55 (confidence 0.91)
```
The confidence on that line is the calibrated value described under
[version inference](#calibrated-confidence), not the raw vote share, so it stays modest when
only a handful of matches agree.
All matches are written into the KB immediately. The match table itself is deterministic: ties
on score are broken by name then source\_ref, so repeated runs print the rows in the same order
with the same labels.
### Step 3: Check coverage
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
warden coverage v1
```
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
defined functions 184
named 112 (60%)
by oracle 98
by human 0
by agent 14
```
***
## The emsdk matrix corpus farm
Building a corpus by hand for each version and flag combination is tedious. The scripts in
`scripts/corpus/` provide a containerized, reproducible alternative. The only host dependencies
are **Docker** and the `warden` CLI.
The farm runs in two stages. First `build_matrix.sh` compiles the reference programs across the
emscripten-version times opt-level matrix and writes a directory of labeled `.wasm` files plus a
`manifest.json` that records how each file was built. Then a single harvest pass turns that whole
directory into one `oracle.json`. Splitting compile from harvest means the expensive Docker
builds happen once and you can re-harvest (after a classifier change, say) without recompiling.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd scripts/corpus
# Stage 1: compile the matrix into a labeled artifact directory + manifest.
./build_matrix.sh \
--versions "3.1.50 3.1.55 3.1.61" \
--opt "-O0 -O2 -Oz" \
--out ./corpus-build
# Stage 2: harvest the whole directory into one signature store.
warden oracle harvest ./corpus-build --out ../../oracle.json
```
### Stage 1: `build_matrix.sh`
For each `(emscripten_version, opt_level)` pair:
`emscripten/emsdk:` is pulled from Docker Hub. Emscripten never touches your host.
Every `.c` file under `scripts/corpus/reference/` is compiled with `emcc`, passing
`--profiling-funcs` and `-sEXPORT_ALL=1`. `--profiling-funcs` forces the name section to be
emitted even at high `-O` levels. Without it, the wasm has no labels and `extract_signatures`
extracts nothing useful.
Each artifact is written to the output directory under a tag built from the version and opt
level (for example `3.1.55_O2/strings.wasm`). The container is discarded; only the compiled
artifacts and the manifest remain. A row describing the build flags is appended to
`manifest.json` next to the `.wasm` files.
The output directory is self-describing. You can commit it, copy it to another machine, or hand
it to a collaborator, and the harvest step reconstructs the same store because every label lives
in the manifest rather than in the shell command that produced it.
### The manifest
`build_matrix.sh` writes one `manifest.json` at the root of the output directory. It is the
contract between the compile stage and the harvest stage: it records, for every artifact, the
labels the harvester needs to tag the extracted signatures (the emscripten version, the opt level,
the extra flags, and the relative path to the `.wasm`). Because the labels travel with the files,
the harvester never has to re-derive them.
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"version": 1,
"entries": [
{
"wasm": "3.1.55_O2/strings.wasm",
"emscripten_version": "3.1.55",
"opt_level": "-O2",
"flags": "",
"source": "strings.c"
},
{
"wasm": "3.1.55_Oz/strings.wasm",
"emscripten_version": "3.1.55",
"opt_level": "-Oz",
"flags": "",
"source": "strings.c"
}
]
}
```
Manifest entry fields:
| Field | Meaning |
| -------------------- | --------------------------------------------------------------------------------------------------- |
| `wasm` | Path to the labeled `.wasm`, relative to the manifest. |
| `emscripten_version` | The emsdk version the image used, passed through to each `Signature`. |
| `opt_level` | The `-O` level, for example `-O2` or `-Oz`. |
| `flags` | Any extra matrix flags for this build (`-pthread`, `-fexceptions`, LTO). Empty for the base matrix. |
| `source` | The reference program the artifact came from, useful when auditing a match. |
### Stage 2: harvesting the directory
A single harvest pass walks the output directory, reads the manifest, fingerprints every named
function in every listed `.wasm`, and folds the results into one store. The library does this with
`harvest_directory`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from warden.oracle import harvest_directory
# Read manifest.json under root, fingerprint every listed .wasm,
# return one deduplicated SignatureStore.
store = harvest_directory("./corpus-build")
store.save("oracle.json")
```
`harvest_directory(root)` loads `/manifest.json`, and for each entry it parses the `.wasm`,
runs `extract_signatures` with the entry's `emscripten_version`, `opt_level`, and `flags`, and
`extend`s them into the store. Because `extend` deduplicates, byte-identical functions that recur
across the matrix collapse to a single signature, so harvesting a large matrix does not produce a
bloated store. The whole pass is deterministic: the same directory always yields the same
`oracle.json`.
```bash CLI theme={"theme":{"light":"github-light","dark":"github-dark"}}
warden oracle harvest ./corpus-build --out oracle.json
```
```python Library theme={"theme":{"light":"github-light","dark":"github-dark"}}
from warden.oracle import harvest_directory
store = harvest_directory("./corpus-build")
store.save("oracle.json")
```
`warden oracle harvest ` is the CLI surface over `harvest_directory`. It is being wired
into the CLI separately; the library function is the stable entry point in the meantime. Both
read the same `manifest.json` and write the same `oracle.json`.
### Why the matrix matters
Emscripten codegen varies across optimization levels in ways that affect fingerprints. A function
compiled at `-O0` has a different structural skeleton than the same function at `-Oz` after
inlining and dead-code elimination. A corpus that covers only one opt level will miss matches in
targets compiled at a different level. The MinHash fuzzy similarity (`ORACLE_THRESHOLD = 0.82`)
tolerates some variation, but breadth of coverage in the corpus is the primary lever for match
rate.
### Reference programs
`scripts/corpus/reference/` currently contains `strings.c`, a program that exercises musl
string functions and the Emscripten runtime. The design calls for more: programs exercising
pthreads, exceptions, different allocator configurations, and ideally portions of the Emscripten
runtime itself. To widen the Oracle's reach, add `.c` files to `reference/` and extend the
`FLAGS` array in `build_matrix.sh` to cover `-pthread`, `-sPROXY_TO_PTHREAD`, `-fexceptions`,
and LTO. Each extra flag set becomes its own matrix entry, with its flags recorded in the
manifest `flags` field, so the harvest step labels those signatures with the build that produced
them.
The corpus accumulates into a single `oracle.json`. Because the format is plain JSON, it can be
committed to your project repository and shared with collaborators. Everyone runs against the
same ground truth without rebuilding.
***
## Using the Oracle as a library
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from warden.oracle import (
SignatureStore,
build_corpus_from_files,
evaluate_identification,
harvest_directory,
identify,
infer_version,
ORACLE_THRESHOLD,
load_seed_store,
)
from warden.kb import KnowledgeBase
# Build a store from a handful of labeled modules by hand
store = build_corpus_from_files(
["runtime_debug.wasm"],
library="musl",
emscripten_version="3.1.55",
opt_level="-O2",
)
store.save("oracle.json")
# Or harvest a whole matrix-farm output directory in one pass
store = harvest_directory("./corpus-build") # reads manifest.json, deduplicates
store.save("oracle.json")
# Audit what the store covers before running it
print(store.stats()) # total, by_library, by_emscripten_version, distinct_names
# Identify against the KB
with KnowledgeBase("project.db") as kb:
v = kb.get_version("v1")
matches = identify(kb, v.id, store, threshold=ORACLE_THRESHOLD)
inferred = infer_version(matches)
print(inferred.emscripten_version, inferred.confidence)
# Measure precision/recall on a held-out labeled module (no emsdk needed)
with KnowledgeBase("heldout.db") as kb:
v = kb.get_version("v1")
report = evaluate_identification(kb, v.id, store, threshold=ORACLE_THRESHOLD)
print(report["precision"], report["recall"])
```
`load_seed_store()` loads `src/warden/oracle/seed_signatures.json` (the empty store that ships
with the package). It is exported from `warden.oracle` for completeness but has no signatures
until you build them.
***
## See also
What happens before the Oracle: parsing, fingerprinting, seeding from exports and the name
section.
The Oracle handles runtime code; the agent crew names what remains.
How the inferred Emscripten version feeds into toolchain-churn suppression during diffing.
Full flag listing for `warden oracle build`, `warden oracle harvest`,
`warden oracle inspect`, and `warden oracle identify`.