Skip to content
Crosswalker
🚧 Early alpha — building the foundation. See the roadmap →

v0.1.4.5 — Iterator-ready engine boundary

Updated

Historical goal

Section titled “Historical goal”

The 2026-05-05 goal was to make the bundled engine streaming-by-design: refactor generateFromRecipe and generateNotes to consume an AsyncIterable<Row> rather than requiring an in-memory ParsedData.rows[] array, then feed that boundary from a CSV API with bounded emitted-row queue backpressure.

That goal came from the 2026-05-05 two-mode architecture decision. The iterator-ready engine boundary shipped. Automatic large-source wizard ingestion did not.

Status

Section titled “Status”

✅ Done (2026-05-05) for the iterator-ready engine boundary. ParsedData.rows: Row[] | AsyncIterable<Row> shipped with backwards compatibility. parseCSVFileStream() exposes async row iteration with a 100-row high-water and 10-row low-water emitted-row queue, and the engine consumes sync or async iterators. That backpressure limits the row queue only; it is not a bound on parser carry, decoded records, the source, or total application memory.

The original completion note recorded 153 passing tests (116 unit + 37 E2E across 8 spec files). Those historical counts were not rerun for this verification.

Status last verified: 2026-09-05 — scoped to the streaming API, engine boundary, wizard call site, and committed E2E source: the iterator and eager-array paths still exist, but the wizard remains eager and the E2E contains no memory measurement.

Current limitation: the wizard’s current >5 MB branch detects streaming mode but calls parseCSVFile(..., { streaming: true }); that function still accumulates every row before returning. tests/e2e/streaming.spec.ts uses 100 synthetic async rows to prove iterator consumption and eager-array compatibility. It does not prove bounded wizard memory, a one-million-row import, or a memory ceiling.

Current delivery posture: a correct, usable eager workflow may precede broader performance work. Advanced streaming remains staged rather than an automatic blocker for v0.1.7, but silent data loss, supported-input crashes, and false completion are never deferrable performance issues. See ingestion safety and staged performance.

Dependencies

Section titled “Dependencies”

Blocks: v0.1.5 — Tier 2 sidecar (sidecar projector also iterates the vault streaming-style; cleanest if engine streaming foundation is in place first)

Original scope (historical)

Section titled “Original scope (historical)”

The checklist below preserves the 2026-05-05 plan. It is not a statement that every item received the proposed scale or memory proof.

In:

  • ParsedData.rows: Row[] | AsyncIterable<Row> — streaming-friendly union; backwards-compatible with existing array consumers
  • parseCSVStream(file) — returns AsyncIterable<Row> directly via PapaParse step callback piped to async generator (no results.data[] accumulation)
  • generateFromRecipe and generateNotes accept either form via for await iteration
  • Historical E2E target — feed a large synthetic CSV (~1M rows or memory-budget-equivalent), confirm RAM does not grow with input size, and confirm Tier 1 output. The committed spec instead uses 100 synthetic rows and has no memory oracle
  • Wizard preview unchanged — still loads first ~50 rows into a small ParsedData for preview UX
  • Wizard detects sources >5 MB with shouldUseStreaming(); binding its generation run to parseCSVFileStream() remains open, as noted under Current limitation

Out:

  • CLI pipe interface (external producer piped into a future Crosswalker CLI) — defer to v0.5+
  • XML / RDF streaming parsers — defer to v0.2+
  • Web Worker offload of parse — Ch 23 §9.5 says workers are unreliable for this; main-thread cooperative-yield is the v0.1 path
  • Programmatic AsyncIterable callers (advanced composition pattern) — exposed via plugin.runImportFromRecipe API; doc page on it deferred to v0.2

Historical task plan

Section titled “Historical task plan”

These unchecked items preserve the original plan; they are not current completion indicators. The Status section above is authoritative about what shipped and what remains open.

  • Update ParsedData type in src/types/config.ts — rows: Row[] | AsyncIterable<Row>
  • New src/import/parsers/csv-parser-stream.ts — parseCSVStream(file): AsyncIterable<Row>; PapaParse step callback writes into a queue; async generator pulls from the queue
  • Refactor generateFromRecipe (in src/generation/generation-engine.ts) — change row loop from for (let i = 0; i < parsedData.rows.length; i++) to for await (const row of asyncRowsOf(rowSource)); helper asyncRowsOf normalizes either array or iterator into AsyncIterable
  • Refactor generateNotes similarly (legacy column-role path)
  • Update runImport and runImportFromRecipe plugin handles to accept the streaming-friendly form
  • Wizard: detect shouldUseStreaming(file); if yes, use parseCSVStream directly without intermediate ParsedData.rows[] accumulation
  • Tests:
    • Unit: streaming parser yields rows in order
    • Unit: engine consumes AsyncIterable and produces same output as it does for an array
    • E2E (tests/e2e/streaming.spec.ts): synthetic 1M-row CSV via streaming path; confirm import succeeds; confirm RAM growth stays bounded; confirm output count matches input row count
    • Existing E2E (full-import-flow, crosswalks, etc.) all still pass — the array-as-iterator path is the degenerate case

Original acceptance goals (historical and unmeasured)

Section titled “Original acceptance goals (historical and unmeasured)”

These were the proposed 2026-05-05 targets. They are not newly passed tests, and the committed streaming E2E does not establish the scale or memory targets below.

  • All 33 existing E2E tests pass unchanged
  • New streaming.spec.ts passes; RAM ceiling confirmed bounded (heap snapshot before/after import on 1M-row CSV stays within ~50 MB)
  • Wizard import-run on >100MB CSV completes without OOM (manual smoke test)
  • No backwards-incompatible API changes — existing callers (passing ParsedData with array rows) keep working without modification

Files to touch

Section titled “Files to touch”
  • src/types/config.ts — ParsedData.rows type union
  • src/import/parsers/csv-parser.ts — keep existing path; add parseCSVStream export
  • src/import/parsers/csv-parser-stream.ts — NEW (or extend csv-parser.ts)
  • src/generation/generation-engine.ts — for await loop; asyncRowsOf() helper
  • src/main.ts — runImport + runImportFromRecipe signature compatibility
  • src/import/import-wizard.ts — wizard import-run uses streaming path for large files
  • tests/csv-parser.test.ts — extend with streaming test cases
  • tests/e2e/streaming.spec.ts — NEW

Open questions

Section titled “Open questions”
  • Should AsyncIterable be the only shape going forward, with arrays auto-promoted? Lean: keep both for zero-friction backwards compat
  • Wizard progress UX when rowCountHint is unknown (true streaming case) — show “N rows processed” without a percentage? Lean yes
  • When the parser fails mid-stream (malformed row 500K of 1M), do we abort the import or continue past the bad row? Lean: configurable; default abort with clear error pointing at row N
Section titled “Related”

Concept pages:

  • ETL and import — two-mode architecture; streaming is at the engine boundary
  • Terminology — ParsedData (now streaming-friendly), AsyncIterable, streaming projector

Agent context:

Design decisions (synthesis logs):

Research challenges:

External producer ecosystem (Mode 1 feeders this enables):

  • Streaming CSV/JSON cleanup tools — natural feeders for row-iterator inputs
  • dbt / Polars / DuckDB — Mode 1 feeders for SQL-shaped sources

Other milestones: