Data morphology — the forms structured data takes, and the operations between them

What this page is — a 101 on data morphology

A progressive, intuitive 101 on data morphology: the shared language for how data is shaped and reshaped. The aim is that when you meet a transpose, a pivot, a sum, or an x/y chart, you can name exactly what’s happening — “oh, that’s a reshape, not a transform” — instead of lumping it all into “the table.” It’s deliberately broad and a little verbose; that’s the point. Crosswalker is the application: this is the mental model the whole tool is built and described in, but the model itself is plain data science that applies to a spreadsheet, a SQL table, or an ontology alike.

Where this connects — it grows out of the four-dials log (the seed) and feeds the GRC corpus end-to-end log, where the language gets applied to real audit/compliance views. It names and orders what three concept pages already cover piecemeal — query primitives (transform), view shapes (draw), hierarchy primitives (layout) — and sits under milestone v0.1.6 — Bases query layer. Build companion: the ingestion-tooling log. See also system architecture and terminology.

The name: data morphology

Data morphology — the study of the forms data takes, and the operations that transform one form into another. (“Morphology” is the study of form.) Naming it buys two things: an intuitive mental model and a shared vocabulary — so you can say precisely what shape is this, which operation am I applying, what shape am I rendering, instead of blurring all three into “the table.” None of this is invented here; it’s plain data science (the established grammars it borrows from are spelled out below). What’s useful is holding it as one coherent model — and that model is the language Crosswalker is built and described in.

The mental model: shape, then three knobs

The goal is an intuitive mental model, not an academic chart. Hold this: data has a shape, and then there are three independent knobs. Almost every confusion (“is a pivot a join?”, “transpose vs x/y axes”) is from mixing the knobs.

Shape what you’ve got → turn three knobs: what it says · how it’s laid out · how it looks.

	Plain question	The knob controls…	Everyday words
Shape	what kind of thing is this?	the native form	list · tree · web · sequence
① Transform	what should it say?	the content — which facts are present	keep (filter) · merge (join) · total (sum)
② Reshape	how is it laid out?	the grid layout — same facts, moved	transpose · wide ↔ tall · roll-up / drill-down
③ Draw	how should it look?	the picture — facts → visuals	x / y axes · color · bars vs dots

The one rule that resolves everything: the three knobs are independent.

• Transform changes the facts — a sum removes rows, a join adds columns.
• Reshape changes the layout, never the facts — transpose is the same data, spun.
• Draw changes the picture, never the layout — x/y axes is a visual choice.

One little spreadsheet — coverage by CSF function

function	controls
GV	20
DE	9
PR	15

TRANSFORM

TOTAL

44

total — fewer rows

RESHAPE

↻

GV	DE	PR
20	9	15

transpose — rows ↔ columns

DRAW

chart — a picture

Same table, three knobs, three results — they don’t interfere.

“Is a pivot a join?” No — a pivot turns all three knobs at once (total → spread wide → put on axes), which is why it feels like one big thing. Pull the knobs apart and it’s obvious. Everything below is the rigorous backing for these four moves.

The instinct that started this: a pivot is not a join

“A pivot could also be a left join — is that a thing?” No — and the no is the whole point. Joining combines rows; summing collapses them. They are different families of operation. A pivot is fundamentally a group-and-aggregate (OLAP), not a join. The join is an optional upstream step (to pull in unmapped rows so gaps show). Disentangling that is what this taxonomy is for. (See the four dials log — this extends it from 4 dials into a fuller stack.)

The full stack (the meta-taxonomy)

The shape the data starts in, the three action knobs (transform · reshape · draw), plus the relations inside it and where it physically lives. Each is an independent choice; reading top-to-bottom is the lifecycle of a question.

Layer	The question	Choices	Where it lives today
Shape	how is the data represented?	relation/table · tree · graph · sequence	detailed below
Relations	how do items relate — the kinds of edge?	hierarchical (broader/narrower) · associative (crosswalk) · membership · evidence	terminology, partly
① Transform	what should it say? — change the facts	select · combine · summarize · traverse · compare	query-primitives
② Reshape	how is it laid out? — same facts, moved	transpose · wide ↔ long · roll-up / drill-down	OLAP cube ops · tidyr
③ Draw (view shape)	how is it drawn?	table · list · pivot · graph · hierarchy · timeline	view-shapes ✅
Layout + settings	where do notes live; the knobs	folder/file/heading/tag/wikilink; rowsBy/heatmap/sort	hierarchy-primitives ✅

The Transform knob, in five families (the join-vs-sum split)

Transform — the knob that changes the facts — isn’t one thing; it’s five families, grouped by what each does to a row-set, every one grounded in established theory. (The query-primitives page lists these flat; here they’re grouped.) Note: these are all Transform — Reshape (transpose) and Draw are the other two knobs, not families here.

Family	What it does to rows	Primitives (ours)	Theory it comes from
Select / derive	keep/drop rows + columns; derive new columns	filter · project · bind	relational algebra (σ select, π project)
Combine / relate	widen — stitch two row-sets on a key	join (inner/left/right/full/anti/semi) · set-op (∪ ∩ −)	relational algebra (⋈, ∪, −)
Summarize / reduce	collapse — many rows → one summary per group	aggregate (group-by + count/sum/avg/min/max/density)	OLAP — a GROUP BY extension to classic RA, not RA itself
Traverse	follow edges — walk the graph, transitively	traverse (hop · transitive closure · neighborhood)	Datalog / SPARQL property paths (p*) — beyond RA
Compare	delta between two snapshots	diff	version/ontology-specific

The key distinctions to internalize:

• Combine widens, summarize collapses. A join adds columns (more about each row); an aggregate removes rows (fewer, summarized rows). Opposite directions.
• A pivot = combine (assemble) → summarize (group by row-axis × col-axis, aggregate a measure) → present as a matrix. So a pivot belongs to the summarize family wearing a view shape. “Pivot = left join” was a category error: the left join, if present, is the combine step feeding it.
• Traverse is its own thing. Following broader/maps-to edges transitively (CSF → 800-53 → CRI) is neither a join nor an aggregate — it’s graph reachability (recursive). Classic RA can’t even express it; that’s why Tier 2 uses recursive CTEs.

Data shapes — the form before any operation

Before you turn any knob, the data already has a native shape. There are only a few, and almost everything is one of them (or made of them):

List / table

rows of facts

Tree

nested, one parent

Graph / web

many-to-many · nodes + edges

Sequence

ordered, one after another

Shape	Structure	In Crosswalker
Relation / table	a set of tuples (rows × columns)	a projection of concepts/edges for one question
Tree / hierarchy	mono-parent nesting	a framework’s internal structure (SKOS broader)
Graph	nodes + typed edges, many-to-many	the substrate — concepts are nodes, mappings + hierarchy are edges
Sequence	ordered items — order is the primitive (time is just one way to order)	versions, timelines, audit trail

Graph is the most general of the four — a list is just a graph with no edges; a tree is a graph where every node has exactly one parent. So the deep move is: hold your data in the most general shape that fits (usually a graph), then treat every table, tree, or matrix as a projection of it for one question. That’s the whole game — one shape underneath, many projected views on top. (Crosswalker takes this literally: an ontology is a graph of concepts + typed edges, and every view of it — a coverage matrix, a gap list — is a projection.)

The trivialized taxonomy already exists — borrow it

The relief: this is well-trodden ground in data science, already compressed into small, learnable “grammars.” We don’t invent data morphology — we adopt these and point them at ontology/graph data. This is what should inform how the tool names everything it does at the query / output / projection piece.

Layer	Established grammar	The trivialized verb set	Crosswalker mapping
Shape	Tidy data (Wickham 2014)	rows = observations, cols = variables; long vs wide	data shapes (relation / tree / graph / sequence)
① Transform	Grammar of data manipulation (dplyr; relational algebra, Codd 1970)	one-table: filter · select · mutate · arrange · summarise+group_by; two-table: join (inner/left/right/full/anti/semi) · set-ops (∪ ∩ −)	the five Transform families (select · combine · summarize · traverse · compare)
② Reshape	OLAP cube ops + pivot (tidyr)	transpose · long↔wide (pivot_longer/pivot_wider) · roll-up · drill-down · slice · dice	layout change without changing content
③ Draw (present / visualize)	Grammar of graphics (Wilkinson 1999; ggplot2, Wickham 2010)	data → aesthetics (x, y, color, size) → geometry (bar/point/tile/line) → facets / coords	view shapes (table / pivot / graph / hierarchy / timeline)

The clarifying payoff — the Excel examples live in different layers:

• transpose (swap rows ↔ cols) is reshape — same content, different layout.
• “x / y axes” is draw — mapping a column to a visual position (an aesthetic); the grammar-of-graphics move.
• Excel blurs them into “the pivot table”; data morphology keeps them apart. A pivot table is literally group + aggregate (transform) → long-to-wide (reshape) → x = rows, y = cols, fill = measure (draw).

So one concept page is enough: Shape · Transform · Reshape · Draw, each a one-paragraph borrow from an existing grammar, then “here’s how Crosswalker exposes it.” The five families above unpack Transform; Reshape and Draw are the other two knobs — the moves Excel quietly bundles into “the pivot table.”

Worked examples — each move, before → after

The tables above name the moves; here each is shown, on the same little coverage table — linked from the family rows. (More to come: aggregate · traverse · transpose · set-op · diff.)

Filter

Keep the rows you want — same columns, fewer rows. (σ select in relational algebra · filter() in dplyr · WHERE in SQL.)

BEFORE

GV	20
DE	9
PR	15

→

keep function = DE

AFTER

DE

9

Two rows dropped; the columns (the shape) are unchanged.

Join

Stitch two tables on a shared key — each row gets wider. (⋈ in relational algebra · join() in dplyr · JOIN in SQL. The “combine” move — not a pivot.)

CONTROLS

id	family
AC-2	AC
AU-6	AU

+

EVIDENCE

id	file
AC-2	policy.pdf
AU-6	logs.txt

→

join on id

JOINED

id	family	file
AC-2	AC	policy.pdf
AU-6	AU	logs.txt

The family column came from Controls, the file from Evidence — matched on the shared id. Combine widens.

Aggregate

Collapse many rows into one summary per group — fewer rows. (GROUP BY + a reducer · summarise() in dplyr. The opposite of join: join widens, summarize collapses.)

BEFORE

GV.OC-01	12
GV.OC-02	8
DE.AE-01	5
DE.AE-02	4

→

group by function, sum

AFTER

GV	20
DE	9

Four rows became two — the GV rows summed to 20, the DE rows to 9. Summarize collapses (the opposite of join).

Traverse

Follow edges — and chase them all the way (transitive closure). (SPARQL p* · a recursive CTE. Neither join nor sum: it walks the graph — how CSF reaches CRI through 800-53.)

Two hops (CSF → 800-53 → CRI) collapse into one reachable edge — the dashed arc. Traverse walks the graph; neither join nor sum can.

Transpose

Spin the grid — same facts, rows become columns. (pivot_wider / transpose — an OLAP move. Changes the layout, not the content: the reshape knob, not transform.)

BEFORE

GV	20
DE	9
PR	15

↻

transpose

AFTER

GV	DE	PR
20	9	15

Same six numbers, spun 90° — rows became columns. Reshape changes layout, not facts.

Set-op

Combine by membership — what’s in both, or in one only. (∩ intersection · − difference · ∪ union — set algebra over rows, keyed on identity. A combine move alongside join.)

SET A

AC-2

AU-6

SC-7

∩

SET B

AU-6

SC-7

IR-4

→

A ∩ B

IN BOTH

AU-6

SC-7

The two rows in both sets. A − B would leave AC-2 (only in A); A ∪ B would be all five.

Diff

Compare two snapshots — added, removed, changed. (a version delta — git diff for data. The audit-trail move.)

v1

AC-2	approved
AU-6	approved

→

v2

AC-2	approved
AU-6	deprecated
SC-7	approved

→

diff

DELTA

+ SC-7 added

~ AU-6 approved → deprecated

= AC-2 unchanged

Only the changes survive — + added, ~ changed. Diff is the audit-trail move.

Candidate views for the GRC corpus (the practical payoff)

Each view is a small recipe in the language above — a transform (which facts) → maybe a reshape (layout) → a draw (the view shape). This menu is what to pull from as we test the audit/compliance library:

View	Question it answers	Operation family	View shape	Status
Coverage matrix	where do framework A & B overlap?	summarize (count)	pivot	✅ built
Coverage density	how heavily is each area mapped?	summarize (count, grouped)	table / pivot	✅ built
Gap / orphan	which controls have no coverage?	combine (anti-join)	list / table	next
Overlap	controls in both A and B	combine (set-op ∩)	list	buildable
Only-in-A	in A but not B	combine (set-op −)	list	buildable
Crosswalk chain	A → B → C transitive reach	traverse (closure)	graph / list	closure shipped
Bidirectional lookup	what maps to X (either direction)?	select (filter)	table	✅ built
Confidence-weighted coverage	strong vs weak coverage	summarize (avg/sum of confidence)	pivot heatmap	buildable
Version diff	what changed v1 → v2?	compare (diff)	table	primitive shipped
Framework-pair density	which framework pairs are densest?	summarize (group by pair)	table	buildable

The pattern: most “compliance questions” are one operation family + one shape. Naming the family first tells you which primitive to reach for and which shape fits.

Concept pages to develop (proposal — not building tonight)

1. concepts/data-morphology (new) — the home for this named concept: the data-shape layer (relation / tree / graph / sequence, and the “ontology = graph, views = projections” spine), the operation families, and how the layers compose. The other concept pages become its sub-layers.
2. Extend concepts/query-primitives — group the 8 operations into the 5 Transform families above (select / combine / summarize / traverse / compare) instead of a flat list; add the combine-widens-vs-summarize-collapses framing.
3. Cross-link view-shapes (presentation) + hierarchy-primitives (layout) under the data-morphology umbrella; graduate the four dials log into the layer-composition section.

Concept-page sprawl

Per the user: we may end up with too many concept pages and have to pare them into sections later. Fine for now — capture first, organize once the shape settles. Candidate future grouping: a concepts/foundations/ section (data-shapes, operations, view-shapes, layout) vs concepts/domain/ (the GRC/compliance pages).

Related

• Four dials — layout · join · shape · view-settings — the seed this grows from
• Query primitives (Layer A) · View shapes (Layer B) · Hierarchy primitives (layout)
• Ingestion tooling dev log · System architecture · Terminology