Tagged: Import
Data import pipeline — parsing, transformation, generation
Pages with this tag
Which workbook bytes made this note — the XLSX source snapshot
13A spreadsheet import now records a fingerprint of the exact bytes it read, and every sheet it reads comes from that one snapshot instead of re-reading a file that may have changed mid-import. Written as a funnel: the thirty-second version, a two-sheet worked example, why it matters for real audit work, what was accepted and rejected, and the memory cost that was accepted deliberately.
Deeper reification, and control over it during import: synthesis
14Eight commits on 2026-09-21 make more of a source export reifiable (a 'maps to another framework' column becomes queryable edge notes) and give the import wizard the controls to decide, column by column, what becomes real. Includes the vault scan that finds framework exports already in a vault, the worksheet suggestion, SchemaVer 1.12.0, and the bounded architect review that fixed one identity-space error before code was written. Unmerged branch work; no real-Obsidian run yet.
MITRE Fight Fraud Framework (F3) added as a recognized source
15The owner asked for MITRE's Fight Fraud Framework (F3) across the knowledge base and wherever else it applies. F3 is now a one-click recognized import with a bundled recipe, and the framework pages list it. Records how F3 reuses ATT&CK ids and leaves open whether a shared id should resolve to the same concept.
The demo now carries the full MITRE F3 catalog and its hierarchy
16Replaces synthetic F3 demo rows with the pinned public v1.2 workbook, real tactic notes and resolved hierarchy links. Existing filters suffice for sub-technique parents; shipped recipes and the F3-to-ATT&CK identity question remain unchanged.
Shared publisher namespaces: qualify actual collisions
17Disjoint catalogs keep publisher identities. Exact ownership claims decide release isolation, existing sets keep their recorded schemes, and later refresh collisions remain safely refused.
Ch 20 deliverable A: T1TMA — Tier-1 Term-Map Algebra (RML retargeted, 6 primitives, MTT-justified, lens-contracted)
18Fresh-agent research deliverable A for Challenge 20. Recommends a YARRRML-shaped DSL retargeted from RDF to Tier-1 Notes. Six primitives: ITERATE, REFERENCE, TEMPLATE, BIND, JOIN, INVERT. Closed Tier-1 slot vocabulary (id, label, body.section, frontmatter.k, links.role, folder, aliases, tags, metadata.sssom-key). JSONata as expression sub-language; CSVW as tabular type profile. Macro Tree Transducer theory as completeness justification; Foster/Pierce lens semantics for the round-trippable subset. ~480 KB total bundle. Concrete NIST 800-53 r5 worked example, v0.1→v0.2 transpiler plan, adversarial cognitive-load check.
Ch 20 deliverable B: Boundary semantics — ref / resolve / bind / seal (Backpack-style holes/fills, Nix content addressing, sheaf-theoretic gluing)
19Fresh-agent research deliverable B for Challenge 20. Operates at a different layer than Runs A and C: not transformation algebra, but boundary semantics. Four primitives: ref (typed reference, Selector + ContextHints), resolve (sandboxed effectful artifact retrieval producing Provenance), bind (artifact + LocalName → VaultDelta), seal (VaultDelta + Manifest → SealedImport). Identity = content-digest; names = presentation; versions = aliases. Append-only Merkle provenance. Object-capability-style sealed manifests. Backpack mixin holes/fills as the type-theoretic model; vaults-as-sheaves over a site of contexts as the categorical model. Three-layer syntax (vault root manifest, import declaration files, materialized notes). Theoretically rigorous; ranges from production-shippable to v3+ aspirational depending on adoption depth.
Ch 20 deliverable C: 5+4 primitive set — RML/YARRRML retargeted, FNML transforms, SSSOM/T filter→action overlay (s-t tgds + MTT + functorial migration justified)
20Fresh-agent research deliverable C for Challenge 20. Recommends a hybrid grounded in RML/YARRRML retargeted from RDF triples to Tier-1 Note tuples. Five import primitives (Source, Term, Map, Join, Function) producing into four output sinks (path, frontmatter, body, wikilink). Theoretical justification: RML's algebraic core matches data-exchange theory's source-to-target tuple-generating dependencies; the 5 primitives correspond to MTT operation classes; bidirectionality as opt-in lens contract on the round-trippable subset. Concrete NIST 800-53 r5 OSCAL JSON example with TypeScript schema and YAML surface. Convergent with Run A on the substantive recommendation; differs in framing (Run A names INVERT as a sixth primitive; Run C makes bidirectionality an annotation flag).
Challenge 23 — Bundle engine implementation language
21Adversarial research deliverable on which language/runtime hosts Crosswalker's bundled ETL engine. Verdict: Path A (Pure TypeScript in-plugin) for v0.1, with Path C (Hybrid: optional external producer) reserved as v0.5+ extension. Rejects Path B (external Python as core), Path D (Rust→WASM), Path E (Go→WASM), Path F (JVM). Reasoning: mobile-Obsidian portability without forks + survival of a niche GRC plugin in a small-contributor OSS world are the two irreversible constraints; every other dimension is tractable engineering. Includes empirical bundle-size figures, path-by-path scoring on 8 dimensions, migration cost analysis, 5-year survival projection, adversarial self-critique, and 9 specific v0.1 commitments.
ETL and import — schema as primitive, ETL as convenience
22Crosswalker's hardest job is taking arbitrary external structure and landing it as canonical Tier 1. The architecture commits to a machine-readable schema as the load-bearing primitive, with a bundled ETL engine, external producers, and a community marketplace as the four orthogonal pieces. This page frames the whole import space.
Ontology lifecycle
23Six recurring Crosswalker capabilities — Acquire, Shape, Connect, Understand, Share, and Maintain — and the artifacts that cooperate across them.
Producing vaults outside Obsidian
24Can a vault be built entirely on disk, by the command line or a script, and still work like one built with the import wizard? Yes. Valid Tier 1 notes are the contract, the plugin derives every runtime surface from those notes, and only per-vault app config has to be recreated per host. This page maps what is identical, what is derived, what is host config, and the invariant that keeps the three paths equal.