ETL and import — schema as primitive, ETL as convenience

The import side is the hardest part of Crosswalker. Everything a user wants to do — query, crosswalk, evidence, version, share — flows through whatever they already have, in whatever shape it arrives, and lands in the Tier 1 schema. Get the import boundary right and the rest of the architecture works. Get it wrong and no amount of cleverness downstream rescues it. For the layer-by-layer map of every convention in the pipeline, see Conventional systems in the stack; for the front-half morph onto Obsidian’s primitives specifically, see Mapping into Obsidian.

This page frames how Crosswalker thinks about that boundary. For the system-wide picture (all six layers, not just import), see the system architecture page.

Artifact and operation boundaries

Per artifact roles and authority, a transform is a bounded change, and every projection on this page is qualified by its target (for example, vault-shape projection or cache projection). Translation is reserved for narrow interlingua/adapter work; compilation is used only when an explicit lowering architecture exists. An ImportRecipe is a machine-evaluable declarative specification, not executable knowledge. SSSOM, STRM, OSCAL, CSV, and Markdown are serializations; ZIP files, folders, and git repositories are carriers; a package requires manifest semantics in addition to a carrier.

The reframe — schema is the primitive, ETL is convenience

Most ingestion tools start with the engine and treat the output schema as a configuration knob. Crosswalker inverts that:

Most ingestion tools

Engine first

The engine is load-bearing; the output schema is a config knob you tweak.

Crosswalker

Schema first

The Tier 1 schema is the load-bearing contract. The engine is optional convenience — anyone who emits valid Tier 1 is a first-class producer.

The Tier 1 schema is the primitive. Crosswalker is, before anything else, an ingestion target — a precise, machine-readable contract that says what canonical Markdown + frontmatter + folder layout + wikilinks looks like. The engine that produces conforming output is convenience. Anyone can produce conforming output any way they like.

You’ve seen this pattern before. HTML and Markdown are the clearest examples: the format is the shared contract everyone agrees on, and a whole ecosystem of tools grew up producing it — nobody owns an “official” generator. The spec is what’s load-bearing; the producers are interchangeable. Tier 1 plays exactly that role for Crosswalker. (Same pattern: JSON Schema, OpenAPI, SBOM formats like SPDX/CycloneDX, CSVW.)

Applied to Crosswalker:

Producers — interchangeable; adopt as many as you like

Bundled ETL engine

convenience

common formats in → Tier 1 out, without leaving Obsidian

External producers

your toolchain

dbt · Python / Polars · scrapers · MCP servers · AI agents

Community packages

roadmap

share reviewable, versioned artifacts through manifest-defined packages

↓

all emit valid Tier 1

Tier 1 schemathe primitive

the load-bearing contract — machine-readable JSON Schema, the single interface every producer targets. Swap the engine; the schema doesn’t move.

This reframe collapses several open architectural questions (build vs buy, DSL choice, in-plugin vs external runtime, recipe location) into one commitment. The engine becomes swappable; the schema does not.

Data input requirements — what Crosswalker can ingest

Crosswalker’s contract is shape-level, not format-level: the engine consumes an iterable of records — an array of objects, each a {column: value} map. It doesn’t have to be tabular — values can be nested objects/arrays, addressable via dotted paths in recipe templates ({family.title}, {a.b.c}). In practice that’s a CSV, an XLSX sheet, or a JSON array of (optionally nested) records. (Native XML / RDF / OWL isn’t v0.1 — pre-convert it to JSON or CSV via the bridge tooling; native parsing is deferred to v0.2+.) That’s the minimum structure Crosswalker can parse — its input floor.

Clearing that floor does not guarantee a correct import. Publisher workbooks can contain banner rows, misleading header offsets, numeric identifiers that collapse, embedded newlines, or hundreds of columns. For supported sources, check the silent-failure checklist before treating a plausible row count as proof.

You can't feed it raw documents — it needs to be a structured format

PDF✗ A PDF — or a web page, a scanned image, a folder of policy .docx files, a raw API blob — has no record structure yet: nothing to iterate, no columns to map to recipe templates. It needs a bridge step first (see extraction & ETL tooling). Crosswalker is deliberately not a document parser or scraper — it’s an ingestion target. Keeping the floor at “structured records” is what keeps the engine small, streaming, and reproducible.

Bridging unstructured → structured is your ETL/extraction tooling’s job — see recommended extraction & ETL tooling for tools by source type. (Mode 1 composes with them directly.) The picture:

Needs a bridge first

PDFDOCXHTMLIMGAPIXMLRDFOWL— red/grey = no records yet · purple = structured but not v0.1-native

↓

bridge it — your ETL / extraction tooling

Bridge tooling your job

ChunkyCSV · JSONaut · Polars / DuckDB · dbt · Docling / unstructured.io · LLM extraction · Excel / Sheets export

↓

⎯⎯ The minimum input — structured records {column: value} · CSV · JSON · XLSX ⎯⎯

↓

Crosswalker takes over

Crosswalker

Mode 1 — bundled engine (rows + recipe)　·　Mode 2 — direct emission (Tier 1 Markdown)

↓

writes

Tier 1 vault

canonical Markdown + frontmatter — the source of truth

Everything above the floor line is your job (or a tool you bring); everything below is Crosswalker’s.

Bridge tooling — by what you're starting from

Most GRC frameworks already clear the floor — NIST / CIS / CRI ship as XLSX / CSV / OLIR, so feed them straight in. For the long tail (messy CSVs, nested JSON, PDFs, scraped sources), see recommended extraction & ETL tooling — the right bridge by source type (ChunkyCSV, JSONaut, Polars / DuckDB / dbt, Docling / unstructured.io / LLM extraction).

The two-mode architecture — bundled projector or direct emission

Updated 2026-05-05 to clarify how external ETL tools (ChunkyCSV, JSONaut, Polars/DuckDB scripts, dbt, etc.) compose with the bundled engine. See the 2026-05-05 two-mode architecture decision log for the full rationale.

The bundled engine has one job: take structured rows + a recipe, and emit Tier 1 Markdown. That’s it. Any producer — including users with no external tooling, ChunkyCSV pipelines, dbt projects, AI agents, MCP servers — chooses between two entry points:

Mode	Entry point	What the producer does	Streaming responsibility
Mode 1 — Bundled projector	Hand structured rows (CSV/JSON/XML/XLSX) + recipe to the bundled engine	Produces a structured input the engine knows how to iterate	Bundled engine streams per-row through render() → write → discard. Producer can also stream upstream (ChunkyCSV → CSV file) — both layers stream.
Mode 2 — Direct emission	Bypass the bundled engine; write Tier 1 Markdown files into the vault	Produces Tier 1 Markdown end-to-end	Producer’s responsibility entirely (typical: per-concept Markdown emission already streams naturally)

Both modes are first-class architectural citizens per the schema-as-primitive commitment. The schema (Tier 1) is the contract; the engine is convenience.

Producer of choice

external or inline · your source data, any shape, any size

↓

chooses an entry point

Mode 1 — Bundled projector

Producer emits structured rows · CSV / JSON / XML / XLSX

raw CSV in the wizard · ChunkyCSV streams a huge XLSX → cleaned CSV · JSONaut normalizes messy JSON · dbt projects SQL → CSV

↳ engine: parse (streaming) → iterate → render() per row → write → discard

Mode 2 — Direct emission

Producer emits Tier 1 Markdown directly

an AI agent writes per-concept .md · a Python script renders YAML + Markdown end-to-end · a pre-built vault bundle from the marketplace

↳ engine: not invoked

↓

both land at

Tier 1 vault

Markdown + YAML frontmatter · conforms to spec/tier1.schema.json · the load-bearing contract

Why both modes matter

Mode 1 lets ChunkyCSV / JSONaut / dbt / Polars scripts compose naturally. They already produce CSV/JSON. Crosswalker just consumes that. The user doesn’t have to teach their existing ETL tools how to emit Markdown — they keep doing what they’re good at, and the bundled engine handles the structured-rows-to-vault projection layer.

Mode 2 lets AI agents and bundled-marketplace publishers emit Tier 1 end-to-end. When an agent extracts an ontology from a corpus, it produces concepts directly — there’s no natural intermediate “rows” representation. Skipping the bundled engine is the right call.

Streaming is at the engine boundary, not before it

The bundled engine is streaming-by-design (per v0.1.4.5 streaming refactor): it accepts rows as an iterator/async-iterator, calls render() per row, writes the file, and discards the row before reading the next one. The full source dataset never exists in RAM at once.

This is why Mode 1 + ChunkyCSV / JSONaut composes well: ChunkyCSV streams from a multi-gigabyte source, emits a smaller-but-still-streaming CSV, the bundled engine streams through that without accumulating, and the resulting Tier 1 vault is materialized on disk file-by-file. End-to-end streaming pipeline.

What “structured” actually means — the Mode 1 input contract

A common follow-up question once the two-mode architecture clicks: how does the bundled engine know an input is “clean enough”? Is there an encoding requirement? A specific JSON shape? An XML schema?

Honest answer: the contract is shape-level, not format-level. The engine consumes an iterable of records. Each record is {columnName: value}. Columns are referenced by name in the recipe’s template strings ({control_id}, {family.title}). That’s the entire contract.

The bundled engine doesn’t validate the input against a “Tier 0.5 schema” because no Tier 0.5 schema exists (2026-05-05 two-mode architecture decision §“No Tier 0.5” confirms this). The engine just calls render(recipe, identityFromRow) on each record and writes the resulting Tier 1 file. Per-row failures (missing template variable, malformed encoding, validation rejection) become per-row errors; the import continues.

Shape requirements

// The engine consumes this:
interface RowSource {
    columns: string[];                              // header — column names referenced by recipe
    rows: Iterable<Row> | AsyncIterable<Row>;       // the records
    rowCount?: number;                              // optional; -1 or omitted if streaming
}

interface Row {
    [columnName: string]: string | number | boolean | null | undefined | object;
    // Nested values OK — recipe templates use dotted access: {obj.field.subfield}
}

Required:

1. Stable columns across records — every row has the same set of keys (missing keys default to undefined, which produces empty values)
2. Recipe-referenced columns are present — if a template uses {control_id}, the source must have a control_id column
3. Values addressable by recipe templates — flat strings are simplest; nested objects work via dotted access; arrays need iterator declarations in the recipe
4. Encoding: UTF-8 — every text format (CSV, JSON, XML, XLSX-as-CSV-export) should arrive UTF-8

That’s it. No strict schema validation upstream of the engine.

Format-by-format guidance for Mode 1 producers

Format	When to use	What “clean enough” looks like
CSV	Tabular, flat, common ETL output, Excel exports, ChunkyCSV pipes	UTF-8; first row is header; consistent delimiter (PapaParse auto-detects ,, ;, \t, |); RFC 4180 quoting for embedded commas/quotes; one logical record per row; no merged cells
JSON (array of records)	Structured records, optionally nested, JSONaut output, dbt JSON exports	UTF-8; top-level array of objects; consistent record shape; nested objects addressable via {a.b.c} templates
JSON (single object with array property)	OSCAL bundles, deeply nested ontologies	UTF-8; recipe declares the iterator path (e.g., source.iterator: $.catalog.controls[*] — JSONata syntax in v0.1+)
XLSX	Excel-native sources (NIST 800-53 catalog as published)	First row is header; one sheet (recipe specifies which); flat cells — no merged cells, no formula references in cell values, no embedded sub-tables
XML / RDF / OWL	RDF/OWL ontologies, ISO XML feeds	Not v0.1. Pre-convert to CSV or JSON via external tooling (the user’s SEACOW / JSONaut handle this). Native XML/RDF parsing deferred to v0.2+.

Is JSON “the way to go” over CSV?

Depends on shape. CSV is fine — and often better — when:

• The source is naturally tabular (compliance frameworks: control_id, family, title, baseline, … fit a CSV grid perfectly)
• You’re already producing CSV via Excel export, ChunkyCSV pipes, or SQL-warehouse export
• Streaming matters (CSV streams trivially line-by-line; JSON streaming requires stream-json or similar)
• File size is moderate-to-large (CSV is denser than JSON for tabular data — no key repetition per record)

JSON is better when:

• Records have nested objects ({ control: { id, title, subcontrols: [...] } })
• One source contains multiple record types (OSCAL bundle = catalog + controls + groups + parameters + back-matter — recipes pick which to iterate)
• Values are naturally arrays (tags, parent CURIEs, related-control lists)
• The source’s authoritative published format is JSON (OSCAL, JSON-LD ontologies, MITRE STIX exports)

For NIST 800-53 r5: CSV is fine (columns: control_id, family, title, baseline, …). For NIST OSCAL JSON catalogs: JSON is native. For ISO 27001 controls: CSV if you have a tabular export; XLSX otherwise. For MITRE ATT&CK: JSON (STIX) is native; CSV exports also exist.

The recipe references columns by name regardless of format. The bundled engine has parsers for each. Format choice is a producer-side decision; the engine adapts.

Why no upfront schema validation?

Two reasons:

1. Adding a schema for the input format would create a Tier 0.5 contract that producers have to satisfy. That violates the schema-as-primitive commitment (Tier 1 is the only contract). External producers would have to learn two schemas instead of one.
2. Per-row error handling is cleaner than upfront rejection. If a 50,000-row CSV has 3 malformed rows, the engine processes 49,997 rows and reports the 3 failures. An upfront schema would either reject the whole file or silently filter — both worse outcomes.

The schema validation that does happen is at the output boundary: every row’s rendered Tier 1 frontmatter is validated against spec/tier1.schema.json before being written (v0.1.4 strict-mode validation). Bad rows produce errors; good rows produce files. The contract is enforced where it matters.

Why ChunkyCSV / JSONaut are the right partners for messy sources

The user’s existing tools fit this gap precisely. They specialize in input cleanup — turning a multi-GB messy XLSX into a streaming UTF-8 CSV with stable columns and proper RFC 4180 quoting. The bundled engine specializes in vault-shape projection — turning structured rows into Tier 1 vault notes per a recipe. Each layer does what it’s good at; the boundary between them is “an iterable of records with stable columns.” That’s the entire contract — and it’s intentionally narrow so external tools don’t need Crosswalker-specific knowledge.

Transform engine depth — where the in-plugin work stops

A natural follow-on question once Mode 1’s input contract is clear: how much of the messy-source-cleanup problem does Crosswalker try to solve in-plugin? Decided 2026-05-05 (see transform-engine-depth and input-formats decision log + Ch 26):

◄ Narrow · engine does lessBroad · engine does more ►

v0.1

Closed 7-filter set

Shipped · baseline

v0.2

Wizard adds basic transforms — rename, trim, regex, split

v0.3

Inline JSONata expression sub-language (already committed, Ch 23 §6 — just wires)

Chosen stopping point

v0.5

Port JSONaut / ChunkyCSV as an opt-in transform engine

v1.0

Full in-plugin transform IDE

Rejected — scope drift, could exceed Crosswalker itself

Phase	Wizard offers	Recipe author authors	External tool handles
v0.1 (shipped)	Column-role config	Closed 7-filter templates	Everything else
v0.2	+ Column rename, value trim, regex extract, simple split	Same templates	Conditional logic, joins, lookups, flat-to-tree, fuzzy matching
v0.3	+ JSONata expression cells for advanced users	+ JSONata 2.x expressions inline ({(baseline = "HIGH") ? "high/" : "mod/"}{control_id}.md)	Multi-source joins, time-series, complex pipelines

Why we don’t port JSONaut / ChunkyCSV: JSONata 2.x is a TS-native, well-maintained library covering ~80% of JSONaut’s declarative-transformation feature set. JSONata is the Ch 23 §6 commitment. ChunkyCSV’s CSV-streaming + multi-GB-cleanup features stay external — that’s literally what ChunkyCSV is for, and forcing it into the plugin would be redundant. Both stay first-class Mode 1 feeders.

Why we don’t build a transform IDE: an in-plugin transform IDE would try to bundle live preview, debugging, profiling, output diff, joins UI, conditional logic UI into one experience. Each piece has a simpler v0.2/v0.3/external alternative (wizard preview, JSONata playground, generation-engine error reports, git diff, JSONata lookup{} syntax). Building the IDE is huge scope drift — could exceed Crosswalker itself.

v0.1 → v0.3+ input format roster

The bundled engine’s input formats and the rationale per format (decided 2026-05-05):

Format	Phase	Why	Stream?
CSV	v0.1 (shipped)	Universal; PapaParse handles encoding + delimiter auto-detect; ChunkyCSV’s natural output	✅ via PapaParse step + v0.1.4.5 streaming refactor
JSONL (newline-delimited JSON)	v0.2	Stream-friendly (line-by-line); native types + nesting; produced by BigQuery, Spark, Databricks, dbt, JSONaut natively	✅ trivial — split on \n, JSON.parse per line
JSON with iterator path	v0.2	OSCAL bundles, deeply-nested ontologies; recipe declares source.iterator: $.catalog.controls[*] (JSONata path)	✅ via stream-json
XLSX	v0.3+ (already partial)	Excel-native sources (NIST 800-53 r5 catalog ships as XLSX)	⚠ sheet-by-sheet via xlsx package
XML / RDF / OWL	v0.3+ if demand justifies	RDF/OWL ontologies, ISO XML feeds	⚠ requires sax-style streaming; defer to v0.3+

JSONL is the v0.2 priority because it’s the genuine midway between “raw user CSV” and “fully-cleaned-input-ready-for-the-bundled-engine”:

Format	Streamable	Schema fidelity	ETL ecosystem	Complexity to integrate
CSV	✅ Trivial	❌ Strings only	✅ Universal	Done
JSONL	✅ Trivial	✅ Native types + nesting	✅ Common (modern data warehouses)	Low (~50 LOC)
Plain JSON array	❌ Whole-array parse	✅ Native	✅ Common	Medium
JSON with iterator	✅ via stream-json	✅ Native + multi-iterator	Limited	Medium
XLSX	⚠ Sheet-by-sheet	⚠ Type-mixed	✅ Excel-native	Medium

JSONL is better than CSV for: nested objects, native types, nullable fields. Better than plain JSON for: streaming, no whole-file parse. ETL ecosystem alignment (modern data warehouses emit JSONL natively). ~50 LOC implementation. Big composability win.

For JSON with iterator path (single object containing an array, like OSCAL): support via recipe source.iterator: $.catalog.controls[*] (JSONata-shaped path expression). Engine uses stream-json to navigate lazily.

What ParsedData is (and isn’t)

The bundled engine’s TypeScript code uses a small interface called ParsedData for the wizard’s preview step:

interface ParsedData {
    columns: string[];
    rows: Row[] | AsyncIterable<Row>;  // streaming-friendly
    rowCount?: number;
}

Concept	What it IS	What it ISN’T
ParsedData	An in-memory TypeScript interface used by the wizard for preview + by the engine as the “structured rows” input shape. May wrap a complete array (small data) OR an async iterator (large data).	A persisted intermediate file format. NOT a tier. NOT something external producers consume or emit. (External producers emit CSV/JSON/XML/XLSX — formats the engine knows how to parse — or Tier 1 Markdown directly.)
Tier 1	The canonical Markdown vault format on disk, conforming to spec/tier1.schema.json. The load-bearing contract.	A serialization-only artifact — it’s the shared vocabulary every producer (bundled engine, external CLI, AI agent) must produce.

ParsedData is just how the engine internally represents “the rows to iterate.” It’s not part of the architectural contract.

How Mode 1 works inside the plugin (v0.1 implementation)

The bundled engine’s per-row pipeline (after Ch 22 + Ch 23 + Ch 24 design phase, implemented in v0.1.2 + v0.1.3 + v0.1.4, with streaming wired in v0.1.4.5):

One row at a time — the engine’s per-row pipeline

Row iterator

streaming source — ChunkyCSV pipe · PapaParse step · AsyncIterator of records

↓

pull one row

Row n

the current record — only this one is resident

↓

scope = the row itself

ConceptIdentity

{curie, scope} — the concept this row becomes

↓

+ recipe

render(Recipe, Identity)

pure function — the single coupling point (Ch 22 §3); deterministic, hashable, replayable

↓

returns

Address

{primary, wikilinkTarget, tags, aliases, frontmatter, body} — body templates are already rendered into ordered regions

↓

+ buildProvenance() + deterministic Markdown assembly

validateTier1Frontmatter()

pre-write gate (v0.1.4) — STRM enforced here

↓

mergeFrontmatter(existing, new)

user_preserve fields survive (Ch 22 §8.4)

↓

write

vault.create() / .modify()

Tier 1 file written to disk

↓

discard row → back for row n+1

Streaming complete

the full dataset never exists in RAM at once

render() is the single coupling point between recipe and vault layout. Pass 1 is vault-independent (deterministic, hashable, replayable). Per Ch 22 synthesis, this purity is what makes canonical-state hashing work.

Canonical recipe versus editable workbench

The workbench does not rebuild an ImportRecipe from a smaller UI model. RecipeDocument.original is the durable preservation authority. StructureMapping / ImportMapping is editable workbench UI/IR derived from that original, and patching changes only workbench-owned regions. Read-only/deferred fields such as output kind, query declarations, graph_edges, and linkStyle survive untouched.

Unsupported states block portable patching instead of disappearing. Current blockers include table-row body output, legacy body transforms, row filters, lookup naming, materialized level notes, non-default missing policies, and link semantics without a canonical recipe surface. See the 2026-07-21 fidelity synthesis.

This closes in-memory workbench fidelity. It does not yet add durable save/open/share UI for user ImportRecipe files; that remains a named UI parity gap.

How sources encode hierarchy

Hierarchy primitives describe the output side: how a vault can express a hierarchy, using folder, file, heading, tag, or wikilink. This section describes the input side: how a source states its hierarchy in the first place. A recipe first has to recognize which encoding the source uses, and only then pick output mechanisms. Most “how do I get the parent links right” questions are encoding questions.

Status last verified: 2026-11-02. Checked against spec/recipe.schema.json (SchemaVer 1.23.0), src/render/template.ts, src/source/nest.ts, src/source/outline.ts, and the shipped recipes/import/*-linked.json.

Encoding (the input lineage form)	What it looks like in the source	Example	How a recipe handles it
The ID carries the path	Each row’s identifier spells out its ancestors	CSF PR.AA-01, ATT&CK T1059.001, 800-53 AC-2(1)	Templates split the ID into levels ({Subcategory|split(.,0)}). Parent links come from the same split, so the hierarchy is derived from what the source asserts
Parent column (adjacency list)	Each row names its parent’s ID	A parent_id or “sub-technique of” column	A parent template on the row, or a join to the parent row. Use it only when the held file actually contains the column
Nested records	Children sit inside their parent in the file	OSCAL catalogs, nested JSON	source.nest walks the levels and turns each one into rows. See schema spec § source.nest
Outline (sparse depth columns)	One column per depth; a row fills only the column for its own depth, and its parent is the nearest row above it at a shallower depth	D3FEND’s tabular export	The engine reads it through source.outline, streaming with a stack bounded by the declared depth. Parent links assert outline containment, not subclass semantics
Multi-value facet cells	One cell lists several values that cut across the hierarchy	ATT&CK tactics on a technique	A list-valued tag template fans out into one tag per value (SchemaVer 1.21.0). Facet notes give each value a hub note
External class hierarchy	The tree lives in a separate ontology file, not the rows	OWL subClassOf	Deferred. Not in v0.1

Two rules follow from this table.

• State the hierarchy once. If levels are already folders with hub notes and parent links, don’t repeat them as nested tags. Use tags only for facets: dimensions that cut across the hierarchy, such as tactics.
• Derive parents from what the source asserts. A dotted ID is an assertion by the publisher. A column the held file lacks is not. For example, the linked ATT&CK default derives a sub-technique’s parent from the ID, not from a “sub-technique of” column.

Variable depth within one encoding (one packed key, ragged depth per row) is the open question in Challenge 42.

Why import is the hardest part

A bullet list of what import has to do correctly, in approximate order of how often it goes wrong:

• Format diversity — CSV, XLSX, JSON, YAML, OSCAL, RDF, MCP server, scraped HTML, OneNote, Notion export. Each has its own tabular-vs-tree shape and its own dirtiness.
• Tabular-to-tree depth crossing — most messy sources are flat tables that encode a tree (parent-id columns, dotted IDs, prefix conventions). Recovering the tree is non-trivial and source-specific.
• Identity stability — the same concept may appear in three sources with three IDs. Tier 1 needs a single canonical identity (CURIE + sha256 CID) per concept; recipe authors decide the canonicalization.
• Polysemic target choice — even after the source is parsed, the recipe author has to choose how it lands: folder hierarchy? heading hierarchy? tag hierarchy? wikilink-graph? Some composition? See hierarchy primitives.
• Depth control — the same source supports different vault shapes depending on how deep the user wants the hierarchy materialized. NIST CSF 2.0 imported “two layers deep” produces one vault; “all the way to subcategories” produces a different one. Both are valid.
• Provenance — Tier 1 needs to record where every concept came from, when, with what version, and (eventually) signed by whom.
• Re-import — version bumps must produce a diff against the prior import, not a silent overwrite or a duplicate vault.
• Round-trip — exporting Tier 1 back to source-shape (or to OSCAL, or to SSSOM) must work without information loss for the round-trip subset.

No off-the-shelf ETL tool does all of these. Tools come close on subsets — dbt nails reproducibility-and-tabular-transform; RML nails schema-mapping-to-graphs; MCP nails external-source-integration. None target Markdown-vault polyhierarchy with provenance and round-trip as a first-class destination.

This is why the import boundary needs first-principles thinking, not a “pick the popular ETL framework and hope” approach.

The four architectural pieces

Crosswalker’s import space breaks into four orthogonal pieces. They compose; users adopt as much or as little as they want. (See the stack above for the one-glance overview — the three producers sitting on the load-bearing schema.)

1. The Tier 1 schema (machine-readable contract)

Single source of truth for what canonical Crosswalker output looks like. Spelled as JSON Schema (or equivalent — CUE, Dhall) so any external tool can validate against it without bespoke parsers. Contains:

• File-naming rules
• Frontmatter shape (required + optional fields, types, allowed values)
• Folder layout conventions (when applicable)
• Wikilink target shape and resolution rules
• Provenance fields (source ref, ingestion timestamp, recipe used, content hash)
• Identity rules (CURIE format, sha256 CID computation)

Anything that emits valid Tier 1 — produced however — is a Crosswalker-conformant import. The schema is the entire interface.

2. The bundled ETL engine (convenience)

A lightweight recipe runtime built into the plugin (or accessible via CLI) that handles the common 80% of imports without leaving Obsidian:

• Common formats out of the box (CSV, XLSX, JSON, YAML)
• Closed primitive vocabulary for transforms
• Recipe authoring through the import wizard UI
• Schema validation on output before write

This is what most users will use most of the time. The engine is not the primitive — it’s just a producer of valid Tier 1, like any other producer. Power users who outgrow it can step into option 3 below without changing how anything downstream of Tier 1 works.

The exact engine implementation (in-plugin TS vs external Python vs hybrid) is open research — see Challenge 23.

3. External producers (custom transforms)

When the bundled engine isn’t enough — messy source, unusual transform, domain-specific cleanup — users run their own toolchain (Python + Polars, dbt, Jupyter notebooks, custom scrapers, MCP servers, AI-agent-driven extraction) and emit Tier 1 directly. They use the schema as the contract.

This path is first-class, not a fallback. Crosswalker explicitly does not try to be the universal ETL engine. It tries to be the universal target.

For the practical version of this path — a complete valid note, the required frontmatter fields, and an AJV snippet to validate your output — see emit valid notes from your own tools.

The user’s own ChunkyCSV and JSONaut are examples of external producers — purpose-built ETL tools that could be configured to emit Tier 1 without being absorbed into Crosswalker’s codebase.

4. Roadmap: community packages (transform once, reuse deliberately)

Comprehensive community sharing is roadmap direction, not a shipped workflow. The goal is to let a maintainer perform a bounded transformation once, publish a reviewable package, and let other users reuse it without repeating source cleanup.

A directory, .zip, or git repository is only a carrier. A Crosswalker package also needs manifest semantics: identity, version, contents, compatibility, integrity, and dependency information. Its contents may include canonical Markdown, an ImportRecipe, and independent mapping serializations, but the carrier does not define their authority.

A future workflow can then support:

• A maintainer publishing a versioned package for a known framework source.
• Users reviewing its manifest and contents before installation.
• Upstream updates producing a new package version and an inspectable diff.
• Import recipes, mapping sets, and framework content remaining independently identifiable.

Registry carrier	Where it lives	Notes
In-repo directory	Subfolder of the Crosswalker repo	Tight CI integration; package manifests validated on merge
Companion git repository	Separate community repository	Independent versioning and contribution flow
Downloadable archive	.zip or equivalent	Portable transport only; still requires the same manifest semantics

The registry and package lifecycle remain deferrable. The artifact boundaries are not: serializations describe content, carriers transport it, and manifests make packages interpretable.

The five-axis recipe selection

Even after a source is parsed, every recipe makes five orthogonal choices about how the source lands as Tier 1:

Axis	Question	Examples
Depth	How many levels of the source hierarchy materialize as vault structure?	NIST CSF 2.0: “two layers deep” vs “all the way to subcategories”
Mechanism	Which of the four hierarchy primitives carry the structure?	Folder, heading, tag, wikilink-graph, or composition
Filter	Which subset of the source is imported?	”Only AC family controls” / “only HIGH-baseline controls” / “all”
Granularity	One file per leaf concept, or one file per group with leaves as headings?	Per-control file vs per-family file
Projection	Which fields of each concept land as frontmatter, body, wikilink, or are dropped?	Control text → body; control ID → frontmatter; baseline → tag

These five axes are why a recipe is a non-trivial, machine-evaluable declarative specification, not a single transform expression and not executable knowledge. They’re also the reason a single source ontology can produce many legitimately different vaults, and why Crosswalker can’t simply auto-import a source without policy choices.

The ~40-primitive transformation catalog

Underneath the recipe, the bundled engine implements a closed set of transformation primitives. Comprehensive but small. Drawn from RML/YARRRML, FNML, JSONata, dbt, and the user’s prior tooling. Grouped by category:

Category	Primitives (illustrative)
Source iteration	iterate-rows, iterate-records, iterate-tree-nodes, iterate-paths, iterate-grouped
Identity / ID synthesis	curie-from-pattern, sha256-cid, uuid-v7, slugify, normalize-id
Field projection	project, rename, drop, default, coerce-type, parse-date
String transforms	trim, lowercase, uppercase, replace, split, join, regex-extract, regex-replace
Tree-from-flat	parent-id-to-tree, dotted-id-to-tree, prefix-to-tree, indent-to-tree
Joins / lookups	inner-join, left-join, lookup-table, fuzzy-match, alias-resolve
Address rendering	folder-path, heading-anchor, tag-path, wikilink-target
Validation / guard	require-field, allowed-values, type-check, schema-validate
Provenance	record-source-ref, record-version, record-timestamp, record-hash

A recipe is a pipeline (or DAG) of these primitives. Composition is what makes the engine complete enough — Macro Tree Transducer theory (per Ch 20 deliverable A) tells us roughly 5–6 algebraic operators are sufficient; the ~40 above unfold those operators into named, recipe-author-friendly forms.

The catalog is not finalized. It is the active design surface.

YARRRML, explained simply

Several of the Ch 20 deliverables recommended YARRRML as the surface DSL for recipes. This is a real choice the user has flagged for further evaluation. Here’s the plain-English version:

YARRRML is a YAML format for writing data-mapping recipes. You write a YAML file that says “this column in my CSV maps to this property in the output, with this transform.” That’s it.

The complicated-looking name comes from “YAML for RDF Mapping Language” — its original use was generating RDF triples. But the structure (sources / mappings / transforms / sinks) is widely useful even if you ignore the RDF heritage and retarget to Markdown notes.

A toy YARRRML-shaped recipe (not real syntax — illustrative):

sources:
  controls: { type: csv, file: NIST_800-53_r5.csv }

mappings:
  control:
    iterator: $.controls.rows
    target:
      file: "Frameworks/NIST 800-53 r5/{{family}}/{{control_id}}.md"
      frontmatter:
        title: "{{control_id}} — {{control_name}}"
        family: "{{family}}"
        baseline: "{{baseline}}"
        tags:
          - "framework/nist-800-53-r5"
          - "framework/nist-800-53-r5/{{family|lowercase}}"
      body: |
        ## Control text
        {{control_text}}

Five things to notice:

1. Sources — declare inputs once, by type and location.
2. Mappings — for each row/record/node, declare the output target.
3. Templates — {{...}} interpolates fields with optional filters (|lowercase).
4. Sinks — file, frontmatter, body, tags. Closed vocabulary; same vocabulary covers any domain.
5. No code — this is data, not Python. An AI agent can read, generate, modify, and validate it.

Why it’s a strong candidate: 12+ years of refinement; mature tooling (Matey editor, parsers); tree-shaped recipes are easier to validate, diff, version, and share than imperative scripts. Anyone — author, agent, validator — can reason about a YAML file mechanically.

Why it’s not a foregone conclusion: YARRRML’s RDF heritage may bring vocabulary baggage; retargeting to Markdown sinks needs a dialect; a custom Crosswalker-native DSL might fit better; or JSON Schema + plain JSONata might be enough without the YARRRML wrapper.

A dedicated explainer page lives at agent-context/agent-tooling/yarrrml-explained (TODO — coming as the Ch 23 research and recipe-DSL design firm up).

How this connects to the rest of Crosswalker

Edge-level concerns (STRM predicates, SSSOM rows, junction notes, ontology diff atoms) operate on concept identity and don’t care which import mechanism produced the vault. The single coupling point is the address-rendering function that turns a concept identity into a wikilink target string given the recipe’s hierarchy-primitive choices. See hierarchy primitives for that separation.

This means the import-side architecture (this page) is strictly orthogonal to the edge-side architecture. A user could import the same NIST 800-53 catalog with three completely different recipes and three completely different vault layouts; the resulting STRM/SSSOM/junction-note structure would be semantically identical.

The same holds for where the vault is produced: a vault built on disk by the cw command line is as operational as a wizard-built one, because the plugin rebuilds its runtime surfaces from the notes; see Producing vaults outside Obsidian.

Where this came from — broader portfolio context

Crosswalker’s stance on import — “schema as primitive, ETL as convenience, marketplace as community escape hatch” — emerged from a portfolio of related tools the user has built over time, all aimed at applied information science with Obsidian as the platform:

• ChunkyCSV — CSV-shaped tabular ETL with tree transducer flavor. Precedent for the “tabular sources need first-class tree-recovery” axis.
• JSONaut — declarative JSON transformation. Precedent for the “external producers should be able to emit Tier 1 without bespoke Crosswalker code” axis.
• SEACOW — meta-framework for knowledge organization in filesystem primitives. Precedent for the “primary folder + parallel tag hierarchy” composition pattern.
• folder-tag-sync — Obsidian plugin that bidirectionally syncs folder hierarchy with tag hierarchy. Precedent for live composition rules between two of the four hierarchy primitives.

Crosswalker is not these tools rebranded — it is the general ingestion target their patterns point at. The marketplace pattern in particular is the answer to the question that ChunkyCSV and JSONaut individually pose: “we keep doing this transform every time someone imports the same source — why?”

Open design questions

Items the architecture has named but not yet committed:

Question	Default lean	Status
Bundle engine implementation language	Pure TypeScript in-plugin (Path A) for v0.1; Hybrid (Path C — opt-in external Python producer) for v0.5+	✅ RESOLVED 2026-05-04 — see Ch 23 synthesis log. Mobile-Obsidian portability + small-OSS contributor pool forced the verdict
Surface DSL flavor	YARRRML-shaped (declarative); v0.1 may ship a simpler subset	Open
Marketplace mechanism	Either in-repo registry or companion repo; both viable	Deferrable
Hardcoded vs declarative dialect spec	Declarative (the dialect IS data even when only one ships)	Provisionally settled
target_schema as JSON Schema (machine-readable) vs prose	Machine-readable	Settled
Recipe schema runtime-agnostic	JSON Schema + AJV; JSONata 2.x as expression sub-language; engine implementations swappable without breaking user files	✅ Settled 2026-05-04 per Ch 23 §4 — the single most important architectural commitment of the design phase
External-producer protocol surface (push-into-Crosswalker via MCP)	Defer until Tier 1 schema lands	Open

See the 2026-05-04 import engine design log for the canonical state of these decisions.

Related

Concept pages:

• Artifact roles and authority — transform, vault-shape/cache projection, recipe, serialization, carrier, and package boundaries
• System model — the illustrated, diagram-first map of the whole pipeline this page is the ingest half of
• Hierarchy primitives — folder, file, heading, tag, wikilink and how recipes compose them via the 5-mechanism grammar
• The problem — hierarchy-vs-graph tension; why ingestion target matters
• What makes Crosswalker unique — Spec / Library / Integrations pillar that this page activates
• Embedded vs server substrates — Tier 2 sidecar projection from Tier 1
• Ontology evolution — re-import semantics + provenance
• Framework data sources — real source bindings and silent-wrong-import gotchas
• Terminology — definitions: Tier 1, recipe, render(), CURIE, ParsedData (Path A only), provenance

Agent context:

• v0.1 schema spec — current Tier 1 contract; the load-bearing primitive this page describes
• Vision — schema-as-primitive is the central architectural commitment
• Tradeoffs — convenience-vs-canonical, bundled-engine-vs-external-producer

Implementation milestones (Path A):

• v0.1.1 — Type system + validation foundation — AJV wires the schema into runtime
• v0.1.2 — render() v1 — the single coupling point ships
• v0.1.3 — Generation engine integration — render() wired through the bundled engine
• v0.1.4 — Junction notes + crosswalk edges — kind discriminator + STRM enforcement at the validation gate
• v0.1.5 — Tier 2 sqlite-wasm sidecar projector — Tier 1 → Tier 2 projection (queries layer)
• v0.1.7 — Exporters — Tier 1 → STRM TSV / OSCAL JSON / SSSOM TSV (round-trip determinism)

Design decisions (synthesis logs):

• 2026-05-05 ETL pipeline clarification — what ParsedData is + isn’t; the three producer paths drawn explicitly
• 2026-05-04 import-engine design log — canonical state of import-engine commitments
• Ch 22 synthesis (target-structure expressivity) — closed 5-mechanism recipe grammar; render() signature
• Ch 23 synthesis (bundle/engine/language) — Path A for v0.1, Path C for v0.5+; runtime-agnostic recipe schema (the most important modularity commitment)
• Ch 24 synthesis (Tier 2 substrate) — sqlite-wasm + sqlite-vec
• Ch 20 synthesis log (formal transformation algebra) — the wargaming setup
• v0.1.3 delivery log
• v0.1.4 delivery log

Research challenges (resolved → archive):

• Ch 21 — Build vs buy ETL engine — meta-question above engine implementation
• Ch 23 — Bundle engine language (archived) — resolved 2026-05-04: Path A for v0.1, Path C for v0.5+

Research deliverables:

• Ch 20 deliverable a (T1TMA primitives)
• Ch 20 deliverable c (RML retargeted) — R2RML lineage of the {var|filter} template grammar
• Ch 22 deliverable (target-structure expressivity)
• Ch 23 deliverable (bundle/engine/language)

Spec files:

• spec/tier1.schema.json — the contract this page describes
• spec/recipe.schema.json — recipe shape Path A consumes