🚧 Early alpha — building the foundation. See the roadmap →
Extraction & ETL tooling — getting your data to the input floor
Crosswalker’s input floor is structured records — an iterable of {column: value} maps (CSV, a JSON array of records, or an XLSX sheet). Crosswalker is an ingestion target, not a document parser or scraper. Getting to the floor from a messy or unstructured source is your ETL/extraction tooling’s job.
This page recommends tools for that bridge step, by what you’re starting from. These are recommendations, not requirements — anything that emits structured records works, and most compliance frameworks already ship structured.
By what you’re starting from
Section titled “By what you’re starting from”| You have… | Reach for | Gets you to | Notes |
|---|---|---|---|
| Huge / messy CSV or XLSX | ChunkyCSV | clean, streaming CSV | Author’s own tool — streams multi-GB sources, cleans headers, handles merged-cell/banner-row mess. Composes with Crosswalker’s Mode 1 streaming. |
| Messy / deeply-nested JSON | JSONaut | normalized JSON records | Author’s own tool — flatten/reshape nested JSON into record arrays addressable by dotted paths. |
| SQL-shaped / tabular transforms | Polars · DuckDB · dbt | CSV / JSON | When you need joins, dedup, type-coercion, or window logic before the floor. DuckDB reads CSV/Parquet/JSON directly; dbt for warehouse-modeled data. |
| PDFs / scanned / HTML docs | Docling · unstructured.io · LLM extraction | tabular records | Layout-aware document → tables/JSON. LLM extraction (structured-output prompting) is the pragmatic path for “pull the control table out of this vendor PDF.” |
| Spreadsheets you maintain | Excel / Google Sheets export to CSV | CSV | The zero-tooling path. Keep curating in the sheet; export when you import. |
| Web pages / APIs | a small script (Python requests + pandas, a scraper) | CSV / JSON | Bespoke per source; the output just has to be a record array. |
| XML / RDF / OWL ontologies | xmltodict · RDFLib · a SPARQL CONSTRUCT → CSV | JSON / CSV records | Structured, but not v0.1-native — convert first. XML is already a tree; RDF/OWL flatten to triples or JSON-LD records. Native parsing deferred to v0.2+. |
| Already structured (most frameworks) | nothing | — | XLSX/CSV/JSON/OLIR clears the floor. Feed it straight in. |
How it hands off to Crosswalker
Section titled “How it hands off to Crosswalker”Once your data is at the floor, Crosswalker takes over via one of two modes:
- Mode 1 — bundled projector: hand the structured rows + a recipe to Crosswalker; it streams each record through
render()and writes Tier 1. Composes naturally with ChunkyCSV/JSONaut (both stream). - Mode 2 — direct emission: your tool (or an AI agent) writes Tier 1 Markdown directly, bypassing the engine.
Related
Section titled “Related”- ETL and import — the input floor, the two-mode architecture, the transformation-primitive catalog
- Conventional systems in the stack — where parsing/extraction sits in the whole pipeline
- Mapping into Obsidian — once you’re past the floor, how data becomes a vault