Tagged: Import
Data import pipeline — parsing, transformation, generation
Pages with this tag
Agent tooling — progressive-disclosure space for AI agents helping with imports
1A structured surface for AI agents (Claude, GPT, etc.) that have been pointed at Crosswalker by a user who wants help transforming their data into Tier 1. Header here; body fills as the underlying specs land. Anyone — agent or human — can navigate from this page to the artifacts they need to do the import work.
Prior art
2Existing tools, research, and the history of the Crosswalker project.
Challenge 21: Should Crosswalker build its own import/ETL engine, adopt an existing one, or compose them? — long-term build-vs-buy with critical and adversarial thinking
3Ch 20 confirmed that Crosswalker's import primitive IS an ETL engine (specifically: graph-aware, semantically-constrained, format-diverse). That answered 'what shape should the primitive take.' This challenge asks the strictly upstream question: should Crosswalker BUILD its own ETL engine, ADOPT an existing one wholesale, or COMPOSE — wrap an existing engine and add a thin Tier-1-specific layer? Critical, long-term, opportunity-cost-aware. Evaluate against dbt, dlt, Singer/Meltano, Airbyte, Apache Hop, Kettle/PDI, RMLMapper, Morph-KGC, Apache Beam, Pandas, Arquero, JSONata, Jolt, the user's own ChunkyCSV + JSONaut, and others. Surface the maintenance/governance/single-vendor risks. Recommend a path that's durable for 5–10 years.
Challenge 20: First-principles primitive for the import side (archived)
4First-principles primitive for ImportRecipe. RESOLVED 2026-05-03 by three fresh-agent deliverables + a substantive user/agent dialog. See callout.
Challenge 22: Target-structure expressivity in import recipes (archived)
5RESOLVED 2026-05-04: closed grammar of 5 mechanisms (folder/file/heading/tag/wikilink) × ordered layout × also_emit × graph_edges; render(Recipe, ConceptIdentity) → Address as single coupling point; content addressing computed BEFORE render (target structure is a view); managed/user_preserve frontmatter split; v0.1 ships full schema with folder+file+heading wired. See callout.
Challenge 23: Bundle engine implementation language (archived)
6Bundle engine language — TypeScript in-plugin vs external Python vs hybrid vs Rust/Go-WASM vs JVM. RESOLVED 2026-05-04: Path A (Pure TS in-plugin) for v0.1; Path C (Hybrid: optional external producer) reserved for v0.5+. See callout.
Challenge 25: Two-mode import architecture and streaming
7Should Crosswalker's bundled engine accept a normalized intermediate format produced by external ETL tools, or should external producers emit Tier 1 directly? Resolves how ChunkyCSV / JSONaut / dbt / Polars compose with the bundled engine; resolves the streaming-at-the-right-boundary question. Filed and resolved same session 2026-05-05.
Import primitive formal foundation — Ch 20 synthesis (wargaming setup, not a decision log)
8Three Ch 20 fresh-agent deliverables landed 2026-05-03. Runs A and C converge on a transformation-algebra primitive set (5–6 primitives, RML/YARRRML retargeted to Tier-1, MTT-justified). Run B operates at a complementary boundary-semantics layer (ref/resolve/bind/seal, Backpack-style holes/fills, Nix content addressing). User declined to make Path A vs Path B v0.1 decisions yet — this log is a wargaming setup, not a decision-driver. Centerpiece: 7 concrete worked examples spanning simple-CSV to external-API-protocol scenarios; explicit wargaming questions; reference baseline (Obsidian Importer plugin) of overly-simplified import; the protocol-surface insight that Crosswalker's import side may be a protocol, not just internal Obsidian-plugin logic.
Two-mode import architecture decision — bundled projector + direct emission, both first-class
9Decision: Crosswalker supports two architectural modes for producing Tier 1 vaults — Mode 1 (hand structured rows to bundled engine) and Mode 2 (bypass engine, emit Tier 1 Markdown directly). Both are first-class. ChunkyCSV/JSONaut compose naturally with Mode 1. Streaming is at the engine boundary so huge inputs work. Supersedes earlier 'three producer paths' framing.
Part A of the per-import-folder work closes on a narrowed bar
10Closure record for 2026-09-04. The twenty-third attempt to make the per-import-folder fix safe for a vault that already holds imports meets every check that was actually run: 2,961 unit tests green, build and lint clean, and a full Obsidian walkthrough run with nothing newly failing and nothing gone missing. This is scoped acceptance of Part A, not a shipment. The seven quarantined visual walkthroughs still fail, four large-corpus walkthroughs never ran, and the branch is unmerged. The durable result is the rule the last three attempts supply: a mechanism extended to a new writer states what that writer's subject is made of, and a fact a run carries is a fact a run claims.
Ingestion safety and staged performance
11Decision record for sequencing ingestion work: deliver a correct, usable supported workflow first, then add measured performance capabilities without weakening safety. It separates current evidence from proposals, keeps parser selection open, and preserves v0.1.7 priorities.
A missing field should not fail an import, and a filter you typed should be used
12Two decisions for the ordinary import wizard: a property whose source value is absent is now left off the note instead of failing the row, and the row filter you type is now carried into generation on every path rather than only in the beta workbench. Written as a funnel: the thirty-second version, a worked source row, why it matters, then the exact boundaries that did not move.