diff --git a/.gitignore b/.gitignore index 55e4767..c12e0d8 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,6 @@ Gemfile*.lock # Local symlink to scraped .xls data from the sibling cdd-data repo. # Machine-specific — each developer symlinks to their own checkout. /downloads + +# TODO files are working notes, not artifacts. +TODO* diff --git a/Rakefile b/Rakefile index 553e253..a9e2e2b 100644 --- a/Rakefile +++ b/Rakefile @@ -42,10 +42,7 @@ end namespace :lint do desc "Verify no raw MDC_P### / MDC_C### literals outside the registry files" task :registry do - # Invoke via explicit `bash` so Windows runners (where Rake#sh uses - # cmd.exe and can't dispatch on the shebang line) find the script. - # Git for Windows ships bash in PATH, so this is portable. - sh "bash bin/lint-no-raw-mdc" + ruby "bin/lint-no-raw-mdc" end end diff --git a/TODO.final/01-per-entity-cddal-emit.md b/TODO.final/01-per-entity-cddal-emit.md deleted file mode 100644 index ce23cd7..0000000 --- a/TODO.final/01-per-entity-cddal-emit.md +++ /dev/null @@ -1,45 +0,0 @@ -# 01 — Per-entity CDDAL emit (Ruby) - -## Why - -The browser's DownloadMenu emits CDDAL client-side via a TS port -(`src/lib/cddalEmitter.ts`). This works but has two problems: - -1. **Drift risk** — the TS port is a parallel implementation of the - CDDAL wire format. When the Ruby serializer evolves, the TS port - silently drifts. There is no spec tying them together. -2. **Single-source-of-truth violation** — the Ruby serializer is the - canonical implementation; the browser should consume its output - (via a pre-built per-entity CDDAL file emitted at build time) or - call into it. Currently it does neither. - -The cleaner architecture: Ruby serializer exposes a public -`emit_entity(entity)` method. The build pipeline can call it per -entity to emit `entity/.cddal` files. The browser's TS port -becomes a fallback only. - -## What - -Add `Opencdd::Cddal::Serializer#emit_entity(entity)` — public -method that returns the CDDAL text for one entity. Refactor the -existing private `emit_instance` to delegate to it (DRY). - -## How - -- File: `lib/opencdd/cddal/serializer.rb` - - Make `emit_instance(entity)` return the lines array (current behavior, kept for back-compat). - - Add `emit_entity(entity)` that returns the joined text for one entity (with header). - - Both share a private `emit_instance_lines(entity)` helper. -- No new autoload needed — `Cddal::Serializer` is already autoloaded via `lib/opencdd/cddal.rb`. - -## Acceptance - -- `Opencdd::Cddal::Serializer.new(db).emit_entity(entity)` returns a `String`. -- Output is valid CDDAL: round-trips through `Opencdd::Cddal.parse` into an equivalent entity. -- The existing `to_cddal` output is unchanged. -- New spec: `spec/cddal/per_entity_emit_spec.rb` covering all 7 entity types. -- Existing `spec/cddal_spec.rb` still passes. - -## Status - -Pending. diff --git a/TODO.final/02-per-entity-json-emit.md b/TODO.final/02-per-entity-json-emit.md deleted file mode 100644 index 551cecc..0000000 --- a/TODO.final/02-per-entity-json-emit.md +++ /dev/null @@ -1,41 +0,0 @@ -# 02 — Per-entity JSON emit (Ruby) - -## Why - -The browser downloads JSON for an entity by serializing the on-page -node. This is fine for the page payload, but consumers (CLI users, -data pipelines) also want to extract one entity from a database -without writing Ruby. - -The exporter already has a `visit_*` per-entity method chain -(`visit_class`, `visit_property`, …) — each is one entity-type -specialization of the same shape. There is no single -`entity_payload(entity)` entry point because the original design -only emitted whole databases. - -## What - -Promote the per-entity visit_* methods to a single public dispatch -that accepts any entity. Use the entity's class to pick the visitor -— but via a class-side registry, not a `case/when` (OCP). - -## How - -- File: `lib/opencdd/exporters/json.rb` - - Extract a class-level `EntityNodeBuilders` registry: each entity - class registers its node-builder proc on load (no central switch). - - Add `Json#entity_payload(entity)` — public, returns a `Hash`. - - Each `visit_*` method delegates to the registry. -- File: `lib/opencdd/exporters.rb` - - No new autoload needed — `Json` is already autoloaded there. - -## Acceptance - -- `Opencdd::Exporters::Json.new.entity_payload(entity)` returns a `Hash` matching the wire format for any of the 7 entity types. -- Output is identical to the per-entity slice that `to_json` would have produced for that entity in a database. -- New spec: `spec/exporters/per_entity_json_spec.rb` covering all 7 entity types. -- No `case/when` on entity type in `json.rb`. - -## Status - -Pending. diff --git a/TODO.final/03-version-aware-reader.md b/TODO.final/03-version-aware-reader.md deleted file mode 100644 index a6ded23..0000000 --- a/TODO.final/03-version-aware-reader.md +++ /dev/null @@ -1,43 +0,0 @@ -# 03 — Version-aware reader (Ruby) - -## Why - -`ShardedDirReader` only reads the *current* version of each entity. -The on-disk layout has *every* version's xls files in a per-UNID -subfolder, but the reader picks just `current_version_dir` and -discards the rest. - -This blocks: -- The browser's time-travel UI (TODO 07) — no way to load v002's content. -- The diff engine (TODO 04) — needs both versions' entities to diff. -- Audit / reproducibility — "what did this class look like in 2022?" - -## What - -A new `Opencdd::Parcel::VersionedReader` (or extending -`ShardedDirReader`) that can: - -- List all versions for an entity (`versions_for(code)`) -- Load a specific version's content as a standalone `Database` (`load_version(code, unid)`) - -## How - -- File: `lib/opencdd/parcel/versioned_reader.rb` (new) - - Wraps a ShardedDirReader path. - - `versions_for(code)` → array of `EntityManifest::Version` (or equivalent value objects). - - `load_version(code, unid)` → `Database` containing just that version's xls content. -- File: `lib/opencdd/parcel.rb` - - Add `autoload :VersionedReader, "opencdd/parcel/versioned_reader"` (OCP: no edits to existing classes). -- Reuse: delegates to `Opencdd::Parcel::FlatDirReader` for the - actual xls reading of one version's folder. - -## Acceptance - -- `VersionedReader.new(path).versions_for("KEA012")` returns 3 versions for the iec63213 KEA012 fixture. -- `VersionedReader.new(path).load_version("KEA012", "c7b602fc...")` returns a Database whose single class has the v002 shape. -- New spec: `spec/parcel/versioned_reader_spec.rb` using iec63213 fixture. -- No `send` / `instance_variable_*` / `respond_to?`. - -## Status - -Pending. diff --git a/TODO.final/04-entity-diff-engine.md b/TODO.final/04-entity-diff-engine.md deleted file mode 100644 index ff95e3c..0000000 --- a/TODO.final/04-entity-diff-engine.md +++ /dev/null @@ -1,49 +0,0 @@ -# 04 — Entity diff engine (Ruby) - -## Why - -The browser's VersionTimeline shows that an entity has multiple -versions, but doesn't show *what changed*. To answer "what's -different between v002 and v003 of KEA012?" the user has to -download both CDDAL fragments and diff them by hand. - -A structured diff engine gives: -- The browser a clean wire object to render a visual diff (TODO 09). -- CLI users a quick "what changed" inspection tool. -- Auditors a way to spot check unwanted edits between releases. - -## What - -A new `Opencdd::EntityDiff` value object that computes a -structured diff between two entities of the same IRDI. Returns -`added`, `removed`, and `changed` field lists with old/new values. - -## How - -- File: `lib/opencdd/entity_diff.rb` (new) - - `EntityDiff.between(entity_a, entity_b)` — constructor. - - `EntityDiff#added` → `Array` (field names in B not in A). - - `EntityDiff#removed` → `Array` (in A not in B). - - `EntityDiff#changed` → `Array<{ field:, from:, to: }>` (different values). - - `EntityDiff#empty?` → bool. - - Iterates `entity.properties` (the lutaml-model field registry) — no type dispatch. - - Multilingual fields (`.en`, `.de` suffixes) are diffed as a group, not per-suffix. -- File: `lib/opencdd.rb` - - Add `autoload :EntityDiff, "opencdd/entity_diff"`. - -## Acceptance - -- Two entities with the same IRDI but different `preferred_name` produce `changed: [{ field: "preferred_name", from: "X", to: "Y" }]`. -- Adding a property produces `added: ["MDC_P999"]`. -- No `case/when` on entity type. -- New spec: `spec/entity_diff_spec.rb` covering: - - Identical entities → empty diff. - - One field changed. - - Field added / removed. - - Multilingual field group diff. - - Different entity types → raises clear error. -- No `send` / `instance_variable_*` / `respond_to?`. - -## Status - -Pending. diff --git a/TODO.final/05-browser-typed-raw-properties.md b/TODO.final/05-browser-typed-raw-properties.md deleted file mode 100644 index 84ec187..0000000 --- a/TODO.final/05-browser-typed-raw-properties.md +++ /dev/null @@ -1,48 +0,0 @@ -# 05 — Browser: typed `raw_properties` accessor - -## Why - -The browser currently casts `node as any` to access `raw_properties` -on the class detail page. This is because `EntityMetadata` in -`@opencdd/models` doesn't expose `raw_properties` as a typed field -even though the JSON wire format includes it on every entity. - -```ts -// src/pages/d/[dict]/c/[code].astro -{(node as any).raw_properties && ( ... )} - -``` - -Three call sites have this cast. Each one is a type hole. - -## What - -Add `raw_properties?: Record` and `version_history?: -VersionHistoryEntry[]` (wire format) to `EntityMetadata` in the -shared `@opencdd/models` package. Then remove every `as any` cast. - -## How - -- Repo: `cdd-models-ts` (sibling). -- File: `src/models/jsonTypes.ts` - - The wire `EntityMetadata` interface already has `raw_properties` — confirm and expose. - - Add `version_history?: VersionHistoryEntry[]`. -- File: `dist/models/jsonTypes.d.ts` — regerate by building the package. -- Repo: `opencdd.github.io`. -- File: `src/lib/types.ts` - - Re-export the wire types under clearer names if helpful. -- File: `src/pages/d/[dict]/c/[code].astro` and `EntityDetailShell.astro` - - Drop `(node as any).raw_properties` → `node.raw_properties`. -- File: `src/components/islands/VersionTimeline.vue` - - Drop the local `interface VersionHistoryEntry` once the wire type is importable (also TODO 06). - -## Acceptance - -- `npm run check` is clean with **zero** `: any` casts in entity detail pages. -- `npm run build` still passes. -- The class page still renders raw_properties correctly. -- No new runtime dependencies. - -## Status - -Pending — blocked on TODO 06 (which unblocks the type import). diff --git a/TODO.final/06-browser-version-history-type-shadow.md b/TODO.final/06-browser-version-history-type-shadow.md deleted file mode 100644 index 3ce76d1..0000000 --- a/TODO.final/06-browser-version-history-type-shadow.md +++ /dev/null @@ -1,52 +0,0 @@ -# 06 — Fix `VersionHistoryEntry` type shadow in `@opencdd/models` - -## Why - -The `@opencdd/models` package exports two different -`VersionHistoryEntry` types: - -- `models/VersionHistory.ts` — class-backed, **camelCase** (`changeRequestId`, `isCurrent`). -- `models/jsonTypes.ts` — wire-format, **snake_case** (`change_request_id`, `is_current`). - -In `models/index.ts`: - -```ts -export { VersionHistory, type VersionHistoryEntry } from "./VersionHistory"; -export * from "./jsonTypes"; // shadowed — VersionHistoryEntry already exported above -``` - -The named export shadows the wildcard re-export. So callers get the -camelCase class type, which doesn't match the JSON wire shape. - -This is why `VersionTimeline.vue` declares its own local interface. - -## What - -Resolve the collision so consumers can import the wire type cleanly. - -## How — pick one - -**Option A (preferred): rename the class-backed type.** -- File: `cdd-models-ts/src/models/VersionHistory.ts` - - Rename `interface VersionHistoryEntry` → `interface VersionHistoryClassEntry`. -- File: `cdd-models-ts/src/models/index.ts` - - Update the named export. -- The wire `jsonTypes.VersionHistoryEntry` is now unambiguous. - -**Option B: subpath export.** -- File: `cdd-models-ts/package.json` - - Add `"./jsonTypes": { "types": "./dist/models/jsonTypes.d.ts", "import": "./dist/models/jsonTypes.js" }`. -- Callers: `import type { VersionHistoryEntry } from "@opencdd/models/jsonTypes"`. - -Option A is cleaner — only one type with that name. - -## Acceptance - -- `import type { VersionHistoryEntry } from "@opencdd/models"` resolves to the wire-format (snake_case) type. -- `VersionTimeline.vue` drops its local interface. -- All browser specs still pass. -- The TypeScript models package itself builds clean. - -## Status - -Pending — sibling-repo change (`cdd-models-ts`). diff --git a/TODO.final/07-browser-time-travel-content-swap.md b/TODO.final/07-browser-time-travel-content-swap.md deleted file mode 100644 index 83352d6..0000000 --- a/TODO.final/07-browser-time-travel-content-swap.md +++ /dev/null @@ -1,54 +0,0 @@ -# 07 — Browser: time-travel content swap - -## Why - -The VersionTimeline currently only shows metadata for each version. -Clicking "View this version" should load that version's actual -content (property values, relationships, definition) and render the -page as-of that point in time. - -This is the headline feature of the version-history work. - -## What - -When the user clicks a non-current version in VersionTimeline: - -1. The page fetches `//entity//v/.json` (TODO 08 builds these). -2. The page applies the `[data-time-travel]` filter treatment. -3. A sticky banner appears: "Viewing version 002 (superseded, 2022-07-25). [Return to current]". -4. The entity hero, properties, and metadata all show the historical values. -5. The page URL updates to `?v=` for shareability. - -## How - -- File: `src/components/islands/VersionTimeline.vue` - - Wire the "View this version" button to fetch + dispatch. - - On click: emit an event with the version UNID. -- File: `src/components/islands/TimeTravelBar.vue` (new) - - Sticky banner shown when `[data-time-travel]` is active. - - "Return to current" button → clears state. -- File: `src/layouts/DictionaryLayout.astro` - - Add `` near the top of `
`. -- File: `src/pages/d/[dict]/c/[code].astro` (and other entity pages) - - On `?v=` query, SSR-fetch the historical JSON during build? No — runtime fetch is fine for a static site. - -### SSR vs runtime - -Two paths: -- **SSR per version**: build generates `/d//c/?v=002` static page. Cleanest, but multiplies build size by average version count (~1.4× for iec61987 → ~50k pages). -- **Runtime fetch**: page loads current, JS fetches historical JSON, swaps content. No build cost; graceful degradation. - -Pick **runtime fetch** for now. SSR can come later if SEO demands. - -## Acceptance - -- Click "View v002" on `/d/iec63213/c/KEA012` → page shows v002 content + sticky banner. -- URL becomes `/d/iec63213/c/KEA012?v=c7b602fc...`. -- "Return to current" → reverts to current. -- Direct-loading `?v=...` works on page load. -- Graceful fallback if the version JSON is missing (e.g. oceanrunner). -- No `: any` in new code. - -## Status - -Pending — blocked on TODO 08 (per-version JSON build). diff --git a/TODO.final/08-browser-per-version-json-build.md b/TODO.final/08-browser-per-version-json-build.md deleted file mode 100644 index aa629ed..0000000 --- a/TODO.final/08-browser-per-version-json-build.md +++ /dev/null @@ -1,46 +0,0 @@ -# 08 — Per-version JSON build pipeline - -## Why - -Time-travel (TODO 07) and diff view (TODO 09) both need per-version -JSON. The current `rake browser:build[dict]` only emits the current -version's JSON. - -## What - -A new rake task that emits `entity//v/.json` -for every entity with `versions.length > 1`. Uses the new -VersionedReader (TODO 03) to load each historical version. - -## How - -- Repo: `cdd-data/Rakefile` - - New task: `browser:build_versions[dict]`. - - For each entity with multi-version history: - - For each non-current version: - - `VersionedReader.new(src_dir).load_version(code, unid)` - - `Exporters::Json.new.entity_payload(entity)` (TODO 02) - - Write to `data//entity//v/.json`. -- Repo: `cdd-data/cdd.config.yml` - - Document the new output directory. -- Repo: `opencdd.github.io` - - `src/content/data//entity//v/.json` ships in the data sync. - - Optional: a manifest at `data//version-manifest.json` listing all available versions per entity, to avoid 404s in the UI. - -### Performance - -- For iec61987 (24,896 version entries, 5,715 multi-version entities): ~6,500 historical JSON files. -- Each file ~3-30KB → ~100MB total. Acceptable. -- Build time: ~6,500 file reads via VersionedReader. Should be <10 minutes. -- Lazy: only emit for entities with >1 version (current version is already in `database.json`). - -## Acceptance - -- `rake browser:build_versions[iec63213]` produces 28 historical JSON files (28 multi-version entities in iec63213). -- Each file is valid JSON, parses to an `EntityNode` shape. -- The `unid` in the filename matches `_entity.json#versions[].unid` for that version. -- New spec in `cdd-data` (or smoke test): file count matches expected. - -## Status - -Pending — blocked on TODOs 02 (per-entity JSON) and 03 (VersionedReader). diff --git a/TODO.final/09-browser-diff-view.md b/TODO.final/09-browser-diff-view.md deleted file mode 100644 index d855e55..0000000 --- a/TODO.final/09-browser-diff-view.md +++ /dev/null @@ -1,51 +0,0 @@ -# 09 — Browser: diff view - -## Why - -Once time-travel works (TODO 07), the natural next step is "show me -what changed between v002 and v003". Currently the user can switch -between versions but has to spot differences manually. - -## What - -A new `VersionDiff.vue` component that calls the Ruby -`Opencdd::EntityDiff` engine (TODO 04) — but client-side, since the -browser is static. Two paths: - -1. **Build-time diff**: pre-compute diffs between adjacent versions during `rake browser:build_versions`. Emit `/diff/-.json`. -2. **Runtime diff**: compute in TS from two version JSON files. - -Pick **runtime diff** — simpler, no per-pair emit. The TS port of -EntityDiff is small (just iterate `raw_properties` and compare). - -## How - -- File: `src/lib/entityDiff.ts` (new) - - Port of `Opencdd::EntityDiff` (TODO 04). - - `entityDiff(entityA, entityB): { added, removed, changed }`. -- File: `src/components/islands/VersionDiff.vue` (new) - - Modal triggered from VersionTimeline ("Diff against current" button). - - Renders added/removed/changed fields with color coding. -- File: `src/components/islands/VersionTimeline.vue` - - Each past version gets a "Diff" button next to "Show source files". -- File: `src/styles/global.css` - - Tokens for diff colors: `--color-diff-added`, `--color-diff-removed`, `--color-diff-changed`. - -### Visual treatment - -- Side-by-side or unified view (toggle). -- Added fields: emerald left border. -- Removed fields: rose left border. -- Changed fields: amber left border + strikethrough old value. -- Diff stats in header: "+3 -1 ~2". - -## Acceptance - -- Diff button on `/d/iec63213/c/KEA012?` v001 vs current shows the differences. -- Diff handles multilingual fields as a group. -- No `: any` in diff code. -- New spec: `tests/lib/entityDiff.test.ts` covering add/remove/change/multilingual. - -## Status - -Pending — blocked on TODO 07 (time-travel infrastructure). diff --git a/TODO.final/11-autoload-audit.md b/TODO.final/11-autoload-audit.md deleted file mode 100644 index aa9eccf..0000000 --- a/TODO.final/11-autoload-audit.md +++ /dev/null @@ -1,42 +0,0 @@ -# 11 — Autoload audit - -## Why - -Every Ruby file under `lib/opencdd/` must be autoloadable from its -parent namespace's file. Files not autoloaded are orphans — they -compile but never load, which silently breaks features. - -The CLAUDE.md rule is explicit: "Ruby `autoload` declared in the -immediate parent namespace's file." - -## What - -Walk `lib/opencdd/**/*.rb` and verify every `.rb` file has a -matching autoload entry in its parent namespace's file. - -## How - -- Script (one-off, no need to keep): - ```bash - for f in $(find lib/opencdd -name "*.rb"); do - rel=${f#lib/} - name=$(basename "$f" .rb) - parent=$(dirname "$f") - parent_file="lib/$(dirname "${f#lib/opencdd/}").rb" - # Check if parent file autoloads this name - grep -q "autoload :#{camelize $name}" "$parent_file" || echo "MISSING: $rel → $parent_file" - done - ``` - -- Add missing autoload entries to the immediate parent namespace's - file (create the file if it doesn't exist). - -## Acceptance - -- The audit script reports zero missing autoloads. -- Every new file added by TODOs 01-04 has its autoload entry. -- `bundle exec rspec` passes. - -## Status - -Pending — run after TODOs 01-04 land so we catch all new files. diff --git a/TODO.final/12-spec-coverage.md b/TODO.final/12-spec-coverage.md deleted file mode 100644 index 5b093ad..0000000 --- a/TODO.final/12-spec-coverage.md +++ /dev/null @@ -1,42 +0,0 @@ -# 12 — Spec coverage for new features - -## Why - -Every new public method needs specs. The TODOs 01-04 introduce: -- `Cddal::Serializer#emit_entity` -- `Exporters::Json#entity_payload` -- `Parcel::VersionedReader#versions_for` / `#load_version` -- `EntityDiff.between` and friends - -## What - -Audit each new method and ensure: -1. Happy-path spec. -2. Edge cases (empty input, nil, off-diagonal types). -3. Round-trip where applicable (CDDAL emit → parse → equivalent). -4. Wire-format invariants where applicable (JSON shape matches TS types). - -## How - -- File: `spec/cddal/per_entity_emit_spec.rb` (new) — TODO 01. -- File: `spec/exporters/per_entity_json_spec.rb` (new) — TODO 02. -- File: `spec/parcel/versioned_reader_spec.rb` (new) — TODO 03. -- File: `spec/entity_diff_spec.rb` (new) — TODO 04. - -Also: extend `spec/code_quality_spec.rb` (the existing code-quality -spec) to forbid: -- `require_relative` in `lib/` -- `instance_variable_set` / `instance_variable_get` in `lib/` -- `respond_to?` in `lib/` -- `.send(` / `.__send__(` (public_send is allowed for framework - integration, audited case-by-case). - -## Acceptance - -- `bundle exec rspec` is green for all new specs. -- The code-quality spec catches the four forbidden patterns. -- Coverage report (if running `simplecov`) shows >95% on new files. - -## Status - -Pending — final pass after TODOs 01-04 land. diff --git a/TODO.final/13-parcel-output.md b/TODO.final/13-parcel-output.md deleted file mode 100644 index 91099d7..0000000 --- a/TODO.final/13-parcel-output.md +++ /dev/null @@ -1,48 +0,0 @@ -# 13 — Parcel (.xlsx) output - -## Why - -The Ruby `Opencdd::Parcel::Writer` produces IEC 62656-1 Parcel -workbooks (10 sheets, the canonical CDD exchange format). The -browser currently offers JSON, CDDAL, and source-XLS downloads per -entity, but never offers a Parcel workbook — the format the -standard mandates for inter-organization exchange. - -Users with ParcelMaker, Excel-based CDD tools, or corporate CDD -pipelines need Parcel files. Today they have to scrape cdd.iec.ch -themselves. - -## What - -Build-time emit of one Parcel .xlsx per dictionary under -`data//parcel/.xlsx`. Browser links to it from: -1. The dictionary overview page (`/d//`) — new "Downloads" section. -2. The dictionary about page (`/d//about`). -3. Each entity's DownloadMenu — relabel as "Parcel (.xlsx)" grouped under "Whole dictionary". - -## How - -- Repo: `cdd-data/Rakefile` - - New task: `browser:build_parcel[dict]`. - - Loads the dict's database via `Opencdd::Reader.load_database`. - - Calls `Opencdd::Parcel::Writer.new(db).write(File.open(path, "wb"), parcel_id: )`. - - Writes to `data//parcel/.xlsx`. -- Repo: `opencdd.github.io` - - `acquireFromLocal` stage already syncs `data/**` recursively — no change needed. - - The .xlsx is large (~MB); add it to `.gitignore`? No — ship it. GitHub Pages has a 1GB site limit, ~6 dicts × 10MB = 60MB is fine. -- Repo: `opencdd.github.io/src/components/ui/DictionaryDownloads.astro` (new) - - Renders on `/d//` overview, after the entity browser. - - Cards for: Parcel (.xlsx), Full JSON (database.json), Full CDDAL (when emitted). - - Each shows file size and "what's inside". - -## Acceptance - -- `rake browser:build_parcel[iec63213]` produces a valid .xlsx at `data/iec63213/parcel/IEC63213.xlsx`. -- The xlsx opens cleanly in Excel/Numbers/LibreOffice. -- Round-trip: `Opencdd::Database.load_workbook()` parses it back into an equivalent Database. -- Browser `/d/iec63213/` shows a "Downloads" section with the Parcel .xlsx link. -- Spec: `spec/parcel/writer_smoke_spec.rb` validates end-to-end emit + reload for a small fixture. - -## Status - -Pending. diff --git a/TODO.final/14-dictionary-downloads-section.md b/TODO.final/14-dictionary-downloads-section.md deleted file mode 100644 index 161c1c3..0000000 --- a/TODO.final/14-dictionary-downloads-section.md +++ /dev/null @@ -1,38 +0,0 @@ -# 14 — Per-dictionary downloads section on overview page - -## Why - -Today, the dictionary overview page (`/d//`) has the entity -browser but no "take this data with you" surface. Parcel + JSON + -CDDAL downloads live nowhere. Users have to know to look at the -about page or guess URLs. - -## What - -A dedicated `DictionaryDownloads.astro` section on `/d//` -showing every format the user can pull for the whole dictionary. - -## How - -- File: `src/components/ui/DictionaryDownloads.astro` (new) - - Renders only when at least one format is available. - - Cards for: - - Parcel (.xlsx) — TODO 13 - - Full JSON (`/database.json`) - - Per-version JSON directory (`/versions/`) when present - - Each card: format name, file size (build-time computed), short description, "Download" / "View" button. -- File: `src/pages/d/[dict]/index.astro` - - Insert `` after the entity browser. -- File: `src/lib/dictionaryDownloads.ts` (new) - - Pure helpers for file existence/size lookup at build time. - -## Acceptance - -- `/d/iec63213/` shows a "Downloads" section listing all available formats. -- Each link is correct and the file is downloadable. -- Section is hidden on dicts with no extras (graceful degradation). -- No `: any` in new code. - -## Status - -Pending — blocked on TODO 13. diff --git a/TODO.final/15-recent-changes-page.md b/TODO.final/15-recent-changes-page.md deleted file mode 100644 index f4268be..0000000 --- a/TODO.final/15-recent-changes-page.md +++ /dev/null @@ -1,40 +0,0 @@ -# 15 — Recent changes page (version history feed) - -## Why - -Every entity with multi-version history records `version_history` -metadata: version, timestamp, user, change_request_id. We surface -this per-entity (TODO 07) but never aggregate it. - -Users want "what changed recently across the whole dictionary?" -Especially for standards maintainers tracking IEC CDD updates. - -## What - -A new page at `/d//changes/` listing every version entry -across the dictionary, sorted reverse-chronologically. Paginated -or capped at N=200. Each row links to the entity. - -## How - -- File: `src/lib/recentChanges.ts` (new) - - `recentChanges(bundle, limit?)` walks every entity, collects version_history entries, returns sorted array. -- File: `src/pages/d/[dict]/changes.astro` (new) - - Builds the page from the bundle at build time. - - Shows: timestamp, version, status, entity code + name (linked), change_request_id. - - Group by month for scannability. -- File: `src/pages/d/[dict]/index.astro` - - Add a link to /changes/ in the overview. -- File: `src/lib/siteNav.ts` - - Add /changes/ to the per-dict nav. - -## Acceptance - -- `/d/iec63213/changes/` lists 80+ version entries across 28 entities, sorted newest first. -- Each row links to the entity's detail page. -- Performance: build time doesn't blow up (linear scan, sort). -- Spec: `tests/lib/recentChanges.test.ts` covering sort + grouping. - -## Status - -Pending. diff --git a/TODO.final/16-site-stats-page.md b/TODO.final/16-site-stats-page.md deleted file mode 100644 index b227c17..0000000 --- a/TODO.final/16-site-stats-page.md +++ /dev/null @@ -1,35 +0,0 @@ -# 16 — Site-wide stats page - -## Why - -The homepage lists dictionaries with counts. The about page shows -per-dict counts. But there's no "stats overview" — total entities -across all dicts, version distribution, top properties, etc. - -This is great landing-page material and helps users grasp scale. - -## What - -A `/stats/` page (and a stats card on `/`) showing site-wide -metrics: total entities, total versions, multi-version entities, -top-N value lists by size, dictionary comparison. - -## How - -- File: `src/lib/siteStats.ts` (new) - - Aggregates all bundles into a single stats object. -- File: `src/pages/stats.astro` (new) - - Renders the stats. Sparklines for version distribution. Bar chart for entity counts per dict. -- File: `src/pages/index.astro` - - Add a single "X total entities across Y dictionaries" hero stat with link to /stats/. - -## Acceptance - -- `/stats/` shows: 21,853 total entities, 38,428 version entries, 8,868 multi-version entities. -- Each metric has a one-sentence explanation. -- Page is responsive + dark-mode-aware. -- Build time under 5s extra. - -## Status - -Pending. diff --git a/TODO.final/17-recently-viewed.md b/TODO.final/17-recently-viewed.md deleted file mode 100644 index 4625425..0000000 --- a/TODO.final/17-recently-viewed.md +++ /dev/null @@ -1,35 +0,0 @@ -# 17 — Recently viewed entities (localStorage) - -## Why - -Users navigating between entities (e.g., comparing a property with -its value list, then back to a class) have no "back" beyond the -browser's history. A "recently viewed" rail in the sidebar would -speed up common exploration flows. - -## What - -A `RecentlyViewed.vue` island in the right sidebar that records -the last 10 entity detail pages the user has visited. Persists -across sessions via localStorage. - -## How - -- File: `src/components/islands/RecentlyViewed.vue` (new) - - Reads/writes `localStorage["opencdd:recent"]` (JSON array of `{ irdi, code, name, type, href, ts }`). - - Listens to `astro:page-load` event (fires on view transition). - - Renders a compact list with code + name + relative timestamp. - - "Clear" button. -- File: `src/layouts/DictionaryLayout.astro` - - Add `` to the right sidebar (above TableOfContents on entity pages, or as a separate sidebar slot). - -## Acceptance - -- Visiting `/d/iec63213/c/KEA012/` then `/d/iec63213/c/KEA001/` shows both in RecentlyViewed. -- List persists across page reload. -- "Clear" empties the list. -- Spec: `tests/components/RecentlyViewed.test.ts` covering add/dedupe/clear. - -## Status - -Pending. diff --git a/TODO.final/18-per-entity-mermaid-hierarchy.md b/TODO.final/18-per-entity-mermaid-hierarchy.md deleted file mode 100644 index 1a47e36..0000000 --- a/TODO.final/18-per-entity-mermaid-hierarchy.md +++ /dev/null @@ -1,37 +0,0 @@ -# 18 — Per-entity Mermaid hierarchy diagram - -## Why - -The class tree sidebar is great for navigation, but a class's -"ancestor chain + subclasses + instances" is best shown as a -visual diagram. Users navigating complex hierarchies (iec61987 -especially) want a quick visual. - -## What - -A `MermaidDiagram.astro` block on class detail pages showing the -ancestor chain (root → ... → self) and immediate subclasses + -powertype instances. Rendered server-side via Mermaid CLI at build -time, embedded as inline SVG. - -## How - -- File: `src/lib/mermaidTemplate.ts` (new) - - `classHierarchyDiagram(bundle, klass)` returns a Mermaid graph definition string. -- File: `scripts/render-mermaid.ts` (new) - - Walks every class page, generates the .mmd, runs mermaid CLI to render SVG, writes to a cache. -- File: `src/components/ui/MermaidDiagram.astro` (new) - - Reads the cached SVG and inlines it. -- File: `src/pages/d/[dict]/c/[code].astro` - - Insert the diagram after the EntityHero. - -## Acceptance - -- Every class detail page shows a hierarchy diagram. -- Diagram shows: ancestors (chained), self, subclasses, powertype instances. -- Inline SVG (no client-side JS for rendering). -- Spec: `tests/lib/mermaidTemplate.test.ts` covering graph generation. - -## Status - -Pending — medium effort, requires mermaid-cli integration in build. diff --git a/TODO.final/19-condition-grammar-and-lutaml-migration.md b/TODO.final/19-condition-grammar-and-lutaml-migration.md deleted file mode 100644 index 06d6e4e..0000000 --- a/TODO.final/19-condition-grammar-and-lutaml-migration.md +++ /dev/null @@ -1,160 +0,0 @@ -# Implementation Proposal: Condition Grammar + lutaml-model Migration - -**From:** cdd-data session 2026-07-23 -**User decisions:** All 3 lutaml decisions resolved; condition grammar Option 2 approved. - -## Status (2026-07-24) - -| Item | Status | Where | -|------|--------|-------| -| **Item 1** — Condition grammar (ClassReference subclass) | ✅ **SHIPPED** | commit `bae0c56` on `main`, merged via PR [#17](https://github.com/opencdd/opencdd-ruby/pull/17). Released in gem v0.3.1. | -| **Item 2** — lutaml-model migration | 📋 **PLAN-ONLY** | [`19b-lutaml-model-migration.md`](19b-lutaml-model-migration.md) — Phases 2.1 & 2.2 already shipped via TODO.impl/33–35; Phases 2.3, 2.4, 2.5 remain. | - -TS team: `Opencdd::Condition::ClassReference` is real API on `main`, -not a proposal. See `lib/opencdd/condition.rb:141`. The `lutaml-model` -work is the only outstanding piece. - -## Item 1: Condition Grammar — ClassReference subclass (TODO 18) - -### Problem - -IEC 62683 stores property conditions as bare class-reference sets: -``` -{0112/2///62683#ACE132} -``` - -`Opencdd::Condition.parse` rejects this because it expects -` `. Current workaround: `Property#condition` -rescues `ArgumentError` and returns nil — data is lost. - -### Approved approach: Option 2 (ClassReference subclass) - -```ruby -module Opencdd - class Condition - # Existing: Condition.new(left:, operator:, right:) - # for expressions like "class_type == {ITEM_CLASS, VALUE_CLASS}" - - class ClassReference < Condition - # For bare sets like {0112/2///62683#ACE132} - # Means "this property applies when the host class - # is one of the listed IRDIs" - attr_reader :irdis - - def initialize(irdis:) - @irdis = Array(irdis) - end - - def class_reference? - true - end - - def to_s - irdis.size == 1 ? irdis.first : "{#{irdis.join(', ')}}" - end - end - end -end -``` - -### Grammar - -``` -condition := expression | class_reference -expression := identifier OP (literal | set) -class_reference := irdi | set -set := "{" element ("," element)* "}" -element := irdi | identifier -``` - -### Files to change - -| File | Change | -|------|--------| -| `lib/opencdd/condition.rb` | Add `ClassReference` subclass; extend `parse` to accept bare IRDI/set | -| `lib/opencdd/property.rb` | Remove the `rescue ArgumentError` — parse no longer raises | -| `spec/condition_spec.rb` | New cases: bare IRDI, bare set, mixed | -| `lib/opencdd/validator/condition_rule.rb` | Accept `ClassReference` as valid | - -### Verification - -1. `rake browser:build[iec62683]` — previously-rescued properties now have non-nil `condition` -2. `bundle exec rspec` — all specs pass -3. CDDAL round-trip on iec62683 — class references survive - ---- - -## Item 2: lutaml-model Migration (TODO 16) — decisions resolved - -### User decisions - -1. **Per-item + single-file coexist.** Both layouts supported; per-item - default for new work. CDDAL `include` directives allow single-file - to reference per-item files. -2. **Default format: YAML.** JSON supported as override. -3. **Multilingual fields: nested map** in YAML/JSON - (`preferred_name: { en: "...", de: "..." }`). Flat keys only in - Parcel xlsx (which has its own schema). - -### Phased plan (unchanged from TODO 16) - -> **Full plan with all 3 decisions applied:** see -> [`19b-lutaml-model-migration.md`](19b-lutaml-model-migration.md). -> That document expands each phase with files-to-change, acceptance -> criteria, risk notes, and how each decision shows up at every layer. - -| Phase | Scope | LOC delta | -|-------|-------|-----------| -| 2.1 | Add lutaml-model dep, port `Unit` (smallest entity) | +50 | -| 2.2 | Port remaining 6 entities (ValueTerm → ValueList → Property → Klass → Relation → ViewControl → Entity base) | +200 | -| 2.3 | Replace CDDAL serializer/builder with lutaml adapter; lexer/parser/racc stay | -200 net | -| 2.4 | Per-item file layout: `Database.dump_to_dir(path, format: :yaml)` + `load_from_dir(path)` | +150 | -| 2.5 | Collapse `Exporters::Json`/`Yaml` to thin wrappers | -100 net | - -### Per-item layout example - -``` -my_dictionary/ -├── _meta.yaml -├── classes/ -│ ├── AAA001.yaml -│ └── AAA002.yaml -├── properties/ -│ ├── ABA001.yaml -│ └── ABA002.yaml -├── units/ -│ └── UAA001.yaml -├── value_lists/ -│ └── ASA001.yaml -├── value_terms/ -│ └── AUA001.yaml -├── relations/ -│ └── ACI001.yaml -└── list_of_units/ - └── UAD001.yaml -``` - -### Per-item YAML example - -```yaml -# classes/AAA001.yaml -irdi: "0112/2///62656_1#AAA001" -code: "AAA001" -version: "1" -revision: "01" -preferred_name: - en: "Root class" -definition: - en: "Top-level class for all items" -superclass_irdi: null -class_type: "ITEM_CLASS" -``` - -### Verification gates - -1. `bundle exec rspec` — all specs pass -2. `Database.load_from_dir(db.dump_to_dir(tmp))` semantically equal to `db` -3. Each entity round-trips YAML and JSON -4. Zero `def to_h` / `def from_h` / `def to_hash` in `lib/` -5. CDDAL round-trip on oceanrunner fixture -6. `rake browser:build[iec62683]` produces semantically equal JSON diff --git a/TODO.final/19-keyboard-nav-timeline.md b/TODO.final/19-keyboard-nav-timeline.md deleted file mode 100644 index 44ad346..0000000 --- a/TODO.final/19-keyboard-nav-timeline.md +++ /dev/null @@ -1,36 +0,0 @@ -# 19 — Keyboard navigation for version timeline - -## Why - -The VersionTimeline renders a vertical list of version entries. -Mouse users can click "View this version" / "Diff vs current". -Keyboard users currently have to Tab through every link in every -entry — there's no concept of "next version" / "previous version". - -WCAG: keyboard nav is required for accessibility. - -## What - -- ArrowUp / ArrowDown move focus between version entries. -- Enter activates "View this version". -- "d" activates diff (when available). -- "/" focuses the entry search (if added). -- `role="listbox"` semantics on the timeline, `role="option"` on entries. - -## How - -- File: `src/components/islands/VersionTimeline.vue` - - Add `tabindex` and `onKeydown` handlers. - - Track `activeIdx` separately from `expanded`. - - Apply `aria-activedescendant` pattern. -- Spec: `tests/components/VersionTimeline.test.ts` (new) covering keyboard interactions. - -## Acceptance - -- Keyboard-only user can navigate the timeline. -- Screen reader announces the active version. -- Mouse interactions unchanged. - -## Status - -Pending. diff --git a/TODO.final/19b-lutaml-model-migration.md b/TODO.final/19b-lutaml-model-migration.md deleted file mode 100644 index 10f63fa..0000000 --- a/TODO.final/19b-lutaml-model-migration.md +++ /dev/null @@ -1,351 +0,0 @@ -# lutaml-model Migration — Phased Plan - -**Parent proposal:** [`19-condition-grammar-and-lutaml-migration.md`](19-condition-grammar-and-lutaml-migration.md) -**Decisions resolved:** 2026-07-23 -**Status:** ready to execute - -This document expands the Phase 2.1–2.5 scope from the parent proposal -into a fully-decisioned plan. Each phase lists: scope, files to change, -how the three user decisions apply, acceptance criteria, and risks. - ---- - -## User decisions applied throughout - -1. **Per-item + single-file layouts coexist.** Per-item is the default - for new work and for `git`-tracked dictionaries (entity-level diffs). - Single-file remains available for ad-hoc exports, CI fixtures, and - the `import` directive in CDDAL modules. Both formats share one - serialization layer (`Entity::Yaml`). - -2. **YAML is the default wire format.** JSON is supported as an - `adapter:` / `format:` override on every persistence entry point. - No second model class — only the lutaml adapter swaps. - -3. **Multilingual fields use nested maps in YAML and JSON.** Shape: - `preferred_name: { en: "...", fr: "..." }`. The flat keyed shape - (`MDC_P004.en = "..."`) exists **only** inside Parcel `.xlsx` - (which has its own column schema) and inside the in-memory - `@properties` Hash (where it stays as the wire-format view). - ---- - -## Phase 2.1 — Foundation: lutaml-model dep + `Unit` reference port - -**Status:** Already shipped via TODO.impl/33 and TODO.impl/35. Listed -here for completeness; new work below builds on this foundation. - -### What landed -- `lutaml-model` and `lutaml-store` declared in `opencdd.gemspec`. -- `Opencdd::Entity::Yaml` is a `Lutaml::Model::Serializable` subclass - with CDD-native attribute names (`preferred_name`, `class_type`, etc.) - driven from `Entity::FieldRegistry`. -- `Opencdd::Model::YamlDatabase` wraps a whole database as a single - YAML document via lutaml-model. -- `Opencdd::Model::EntityStore` writes per-entity YAML files via - lutaml-store's `DatabaseStore`. - -### Why `Unit` as the reference port -Smallest entity (no multilingual expansions, no class hierarchy, no -references) — ideal for proving the typed-attribute shape end-to-end -before replicating across the six larger entities. - -### Acceptance (already met) -- [x] `bundle exec rspec` — all 607+ specs pass. -- [x] `Unit#to_yaml` / `Unit.from_yaml` round-trip without data loss. -- [x] No `def to_h` / `from_h` / `to_hash` on the model class. -- [x] No `require_relative` in library code (autoload only). - ---- - -## Phase 2.2 — Port remaining 6 entities - -**Status:** Already shipped. All entity types (ValueTerm, ValueList, -Property, Klass, Relation, ViewControl, ListOfUnit) inherit through -`Entity::Yaml`. Field-DSL declarations on each subclass flow into the -YAML model automatically via the FieldRegistry walk in -`Entity::Yaml.from_entity`. - -### What's in place -- `Entity::Yaml.from_entity` walks `FieldRegistry.fields_for(entity.class)` - and writes each field's value to the matching YAML attribute. -- Polymorphic dispatch lives on the `type` attribute (`:class`, - `:property`, etc.). `Entity::Yaml.to_entity` consults - `Opencdd::MetaClasses.entity_class_for_type` for the inverse mapping. -- The `extra` Hash catches any property ID not declared in the field - DSL, preserving lossless import for unknown Parcel columns. - -### Decisions applied -- **Multilingual as nested map.** `extract_ml` returns - `{ "en" => "Vehicle", "fr" => "Véhicule" }` for storage; `merge_ml` - re-flattens to `MDC_P004.en = "..."` keys on the way back into - `@properties`. The flat shape never appears on disk. - -### Acceptance (already met) -- [x] Each entity type round-trips through YAML. -- [x] OceanRunner fixture round-trips end-to-end. -- [x] `bundle exec rspec` — all specs pass. - -### Open gap to close in 2.3 (not blocking 2.2) -- The YAML model is a sibling class (`Entity::Yaml`), not Entity - itself. Plan 35's "fold adapter into Entity" is not yet complete — - Entity still owns the `@properties` Hash as canonical store, with - `Entity::Yaml` as a deepened adapter. This is acceptable but means - every persistence call goes through `from_entity` / `to_entity` - conversion. Folding the adapter INTO Entity remains a future - architectural cleanup, gated on lutaml-model's ivar-reading - constraint being compatible with the field-DSL pattern. - ---- - -## Phase 2.3 — Replace CDDAL serializer/builder with lutaml adapter - -**Status:** Not started. **Estimated delta:** ~−200 LOC net. - -### Current state -`lib/opencdd/cddal/serializer.rb` (~300 LOC) is hand-rolled: it walks -each entity type, formats fields as CDDAL syntax, and emits a single -text document. The lexer/parser/racc grammar (`cddal.y`, -`generated_parser.rb`, `lexer.rb`) parse CDDAL into AST nodes -(`lib/opencdd/cddal/ast.rb`); the builder (`lib/opencdd/cddal/builder.rb`) -walks AST → Entity. Both are necessary and stay. - -### What changes -| File | Change | -|------|--------| -| `lib/opencdd/cddal/serializer.rb` | Replace hand-rolled emit with a thin lutaml-style adapter: walk typed attributes on `Entity::Yaml`, render each as CDDAL syntax. Reuse `value_serializer.rb` for the existing literal/set/tuple formatting. | -| `lib/opencdd/cddal/builder.rb` | Unchanged — already uses typed assignments. | -| `lib/opencdd/cddal/value_serializer.rb` | Unchanged — primitives (`Literal`, `Set`, `Tuple`, `ClassReference`) are format-neutral. | -| `lib/opencdd/cddal/cddal.y`, `generated_parser.rb`, `lexer.rb` | Unchanged — grammar is the wire contract. | -| `spec/cddal_spec.rb` | Add round-trip cases for every entity type (some exist; fill gaps). | - -### Decisions applied -- **YAML default + JSON override.** CDDAL is a third wire format, - independent of YAML/JSON. The serializer refactor uses the same - typed-attribute source as YAML — `Entity::Yaml` — so the three - formats converge on one model. -- **Per-item + single-file coexist.** CDDAL `include` directives - (parsed by `lib/opencdd/cddal/resolver.rb` and - `lib/opencdd/cddal/fetcher/`) already let a single-file CDDAL - document pull in per-item YAML/CDDAL files. The refactor preserves - this — the resolver stays, only the emit layer changes. - -### Acceptance -- [ ] `bundle exec rspec` — all specs pass. -- [ ] CDDAL round-trip on OceanRunner fixture: parse → serialize → parse - yields semantically equal databases. -- [ ] CDDAL round-trip on iec62683 fixture: class-reference conditions - (Phase 1 of this TODO) survive intact. -- [ ] Serializer file drops below 150 LOC (from ~300). -- [ ] No `def to_h` / `from_h` / `to_hash` on the model class. - -### Risks -- The hand-rolled serializer currently handles edge cases - (empty-value omission, multilingual expansion, set quoting) that - the typed-attribute walk may not cover. Mitigation: keep - `value_serializer.rb` intact; only the entity-walking loop changes. - ---- - -## Phase 2.4 — Per-item file layout with type-partitioned subdirectories - -**Status:** Not started. **Estimated delta:** ~+150 LOC. - -### Current state -`Model::EntityStore` writes per-entity YAML files into a flat -`entities/.yaml` directory via lutaml-store's -`:separate` layout. The flat shape works but doesn't match the -TODO 19 spec, which calls for type-partitioned subdirectories: - -``` -my_dictionary/ -├── _meta.yaml -├── classes/ -│ ├── AAA001.yaml -│ └── AAA002.yaml -├── properties/ -│ └── ABA001.yaml -├── units/ -├── value_lists/ -├── value_terms/ -├── relations/ -├── view_controls/ -└── list_of_units/ -``` - -### What changes -| File | Change | -|------|--------| -| `lib/opencdd/model/entity_store.rb` | Switch from `:separate` layout to manual per-type subdirectories. The store keeps its `Lutaml::Store` backend for CRUD, but the filesystem shape is `//.yaml`. Add a `_meta.yaml` writer for dictionary-level metadata (source language, translation languages, parcel_id, generated_at). | -| `lib/opencdd/model/directory_layout.rb` | **New.** Single source of truth for `type → plural_directory` mapping (`:class → "classes"`, `:property → "properties"`, etc.). Used by both writer and reader so they cannot drift. | -| `lib/opencdd/database/persistence.rb` | Add `dump_to_dir(path, format: :yaml)` and update `save_to_directory` to call it. `load_from_dir(path)` reads the new layout; `load_from_directory` stays as a back-compat alias that detects old vs new layout by presence of `_meta.yaml`. | -| `lib/opencdd/model/yaml_database.rb` | Unchanged — still provides the single-file YAML shape. Single-file stays supported per Decision 1. | -| `lib/opencdd/database.rb` | Add `Database.dump_to_dir` / `Database.load_from_dir` class methods delegating to EntityStore. | -| `spec/model/entity_store_spec.rb` | Add round-trip specs for the new layout; verify `_meta.yaml` contents. | - -### File naming -- Code-only filename (`AAA001.yaml`), not full IRDI. The full IRDI - is the first field inside the file; the path is `type_plural/code`. -- Codes with non-alphanumeric characters (e.g., codes with `_`) pass - through unchanged. Codes containing `/` would collide with the - type-directory path; none observed in the IEC CDD code registry - (codes match `^[A-Z]{3}[0-9]{3}$` historically). - -### `_meta.yaml` shape -```yaml ---- -parcel_id: IEC62683 -source_language: en -translation_languages: - - fr - - de -generated_at: 2026-07-23T10:00:00Z -entity_counts: - classes: 1855 - properties: 1200 - units: 30 - value_lists: 12 - value_terms: 500 - relations: 40 - view_controls: 0 - list_of_units: 0 -schema_version: "1" -``` - -### Decisions applied -- **Per-item default.** `Database.dump_to_dir(path)` writes the - per-item layout above. `Database#to_yaml` (single-file) stays - available for ad-hoc use — Decision 1's coexistence. -- **YAML default + JSON override.** `dump_to_dir(path, format: :yaml)` - is the default; `dump_to_dir(path, format: :json)` writes - `.json` files instead. Both share the same `Entity::Yaml` - model — only the lutaml adapter swaps. -- **Multilingual as nested map.** Per-entity YAML already emits - `preferred_name: { en: ..., fr: ... }`. The new layout preserves - this; no flattening at the file boundary. - -### Acceptance -- [ ] `Database.load_from_dir(db.dump_to_dir(tmp))` is semantically - equal to `db` (uses `Database#semantically_equal?`). -- [ ] Per-entity files land in the correct type-partitioned subdir. -- [ ] `_meta.yaml` is written and read; counts match the actual - entity totals. -- [ ] JSON layout round-trips equivalently (`format: :json`). -- [ ] `bundle exec rspec` — all specs pass. -- [ ] OceanRunner fixture round-trips through the new layout. - -### Risks -- The current flat layout is referenced by `spec/model/entity_store_spec.rb` - and by `Database.load_from_directory` callers. Mitigation: - `load_from_directory` detects layout by `_meta.yaml` presence and - dispatches to old-flat vs new-partitioned reader. Remove the - old-flat reader only after one release cycle. - ---- - -## Phase 2.5 — Collapse `Exporters::Json` / `Yaml` to thin wrappers - -**Status:** Not started. **Estimated delta:** ~−100 LOC net. - -### Current state -`lib/opencdd/exporters/json.rb` (~200 LOC) and -`lib/opencdd/exporters/yaml.rb` (~50 LOC) hand-roll payloads: they -walk each entity type, build a Hash, and pass it to `JSON.generate` -or `YAML.dump`. The hand-rolled payloads duplicate the -wire-name mapping that `Entity::Yaml` already declares via lutaml-model -attributes. - -### What changes -| File | Change | -|------|--------| -| `lib/opencdd/exporters/json.rb` | Replace hand-rolled `entity_payload` / `database_payload` with `entity.yaml.to_json` / `database.yaml.to_json`. Keep the public `export(entity, database:)` signature. The registry-dispatched `payload_for(entity, database:)` stays — it's the per-entity hook used by `rake browser:build`. | -| `lib/opencdd/exporters/yaml.rb` | Replace hand-rolled emit with `database.to_yaml` (already delegates to `YamlDatabase`). Becomes a 10-line wrapper. | -| `lib/opencdd/exporters.rb` | Unchanged — autoload entries stay. | -| `spec/exporters/*` | Update expectations if the JSON field ordering changes. Lutaml-model emits in attribute-declaration order; current exporters emit in a different (hand-curated) order. Either redeclare attribute order or relax the specs to compare parsed JSON, not raw strings. | - -### Decisions applied -- **YAML default + JSON override.** Both thin wrappers delegate to - lutaml-model's `to_yaml` / `to_json`, which share one typed model. - No second code path for JSON. -- **Multilingual as nested map.** lutaml-model emits the Hash - attribute as-is, producing the nested `{ "en": "...", "fr": "..." }` - shape in both YAML and JSON. The current hand-rolled JSON exporter - emits the same shape (Decision 3 is already satisfied here; this - phase just removes the duplication). - -### Acceptance -- [ ] `bundle exec rspec` — all specs pass (some specs may need to - compare parsed JSON rather than raw strings if attribute order - differs from the hand-rolled version). -- [ ] `rake browser:build[iec62683]` produces semantically equal JSON - output (parse both old and new, compare Hash equality). -- [ ] `Exporters::Json` file drops below 80 LOC. -- [ ] `Exporters::Yaml` file drops below 20 LOC. -- [ ] No `def to_h` / `from_h` / `to_hash` on the model class. - -### Risks -- Browser JSON consumers may rely on field ordering. JSON field order - is technically not semantic, but some consumers parse with - ordered-hash expectations. Mitigation: audit - `browser/src/data/loaders.ts` and any editor-side JSON parsing; - pin attribute order in `Entity::Yaml` to match the current - exporter's order if any consumer is order-sensitive. - ---- - -## Cross-phase verification gates - -These gates run after every phase, not just at the end. - -1. **`bundle exec rspec`** — all 607+ specs pass. No new exclusions. -2. **`Database.load_from_dir(db.dump_to_dir(tmp))` semantically equal - to `db`** — round-trip invariant for every entity type. -3. **Each entity round-trips YAML and JSON** — both formats share one - model, so a single round-trip test per format suffices. -4. **Zero `def to_h` / `def from_h` / `def to_hash` in `lib/`** — - enforced by `spec/code_quality_spec.rb`. Add a grep assertion if - not already present. -5. **CDDAL round-trip on OceanRunner fixture** — Phase 2.3's primary - acceptance gate. -6. **`rake browser:build[iec62683]` produces semantically equal JSON** — - Phase 2.5's primary acceptance gate. Compare parsed Hash equality, - not raw string equality, to allow attribute-order drift. - ---- - -## Sequencing - -``` -2.1 ✅ (done) ──► 2.2 ✅ (done) ──► 2.3 (CDDAL serializer refactor) - │ - ▼ - 2.4 (per-item layout) - │ - ▼ - 2.5 (exporter collapse) -``` - -Phases 2.3 and 2.4 are independent and can run in parallel once 2.2 -lands (which it has). Phase 2.5 depends on 2.4 (the JSON exporter -thin-wrapper delegates to `Entity::Yaml#to_json`, which is format- -agnostic — but the per-item JSON writer in 2.4 is the proof that -JSON output works for the layout). - ---- - -## What this plan does NOT change - -- **Parcel `.xlsx` format.** The Parcel reader/writer - (`lib/opencdd/parcel/`) is unaffected. Parcel is a separate wire - format with its own column-based schema; multilingual fields stay - flat there (Decision 3 carve-out). -- **In-memory `@properties` Hash.** Entity continues to use the Hash - as the canonical read store during this migration. Folding the - YAML adapter into Entity (Plan 35's full vision) is a future - architectural cleanup, not part of TODO 19. -- **Validator rules.** `lib/opencdd/validator/` is unaffected — it - operates on the in-memory model, not on serialized forms. -- **Editor (TS) model port.** Browser/editor TypeScript types mirror - the Ruby model's wire shape. As long as the wire shape is stable - (which this plan guarantees — same YAML/JSON attribute names), - no TS changes are required. diff --git a/TODO.final/20-csv-export.md b/TODO.final/20-csv-export.md deleted file mode 100644 index 6d8edff..0000000 --- a/TODO.final/20-csv-export.md +++ /dev/null @@ -1,32 +0,0 @@ -# 20 — CSV export option in DownloadMenu - -## Why - -JSON is great for tooling, CDDAL is great for diffs, but -spreadsheet users (most CDD consumers) want CSV. They have to -write a JSON-to-CSV converter themselves today. - -## What - -Add a CSV option to DownloadMenu that emits a flat spreadsheet -row per (entity, field) pair — one row per `raw_property` entry. -Useful for analysts who want to slice the data in Excel. - -## How - -- File: `src/lib/csvEmitter.ts` (new) - - `emitEntityCsv(entity)` returns CSV text with columns: `irdi, code, type, property_id, language, value`. - - Multilingual keys split into multiple rows. -- File: `src/components/islands/DownloadMenu.vue` - - Add CSV option with file icon. -- File: `tests/lib/csvEmitter.test.ts` (new) - -## Acceptance - -- CSV download from any entity detail page produces a valid CSV. -- File opens cleanly in Excel/Numbers. -- Spec covers multilingual splitting and special-character escaping. - -## Status - -Pending. diff --git a/TODO.final/21-docs-update.md b/TODO.final/21-docs-update.md deleted file mode 100644 index 4c44b85..0000000 --- a/TODO.final/21-docs-update.md +++ /dev/null @@ -1,36 +0,0 @@ -# 21 — Documentation: update CLAUDE.md / README for new features - -## Why - -After TODOs 01–20 ship, the docs are stale: -- CLAUDE.md doesn't mention `EntityDiff`, `VersionedReader`, `emit_entity`, `payload_for`, or `browser:build_parcel`. -- README doesn't mention the new download formats or time-travel UI. -- TODO.final/README has the truth but is internal-only. - -## What - -Update CLAUDE.md (opencdd-ruby), README (opencdd.github.io), -and the docs section (`src/content/docs/`) to reflect the new -surface. - -## How - -- File: `opencdd-ruby/CLAUDE.md` - - Add the new public API methods under "The `cdd` gem" section. - - Note the data pipeline additions (`browser:build_versions`, `browser:build_parcel`). -- File: `opencdd.github.io/README.md` - - Update the routes table (add `/d/:dict/changes`, `/stats`). - - Update the architecture diagram with the new components. -- File: `opencdd.github.io/src/content/docs/using-cdd-data.mdx` - - New "Downloads" section: Parcel, JSON, CDDAL, CSV. - - New "Version history" section: time travel, diff view. - -## Acceptance - -- All three docs updated. -- Internal links work. -- No content claims features that don't exist. - -## Status - -Pending. diff --git a/TODO.final/README.md b/TODO.final/README.md deleted file mode 100644 index 48531db..0000000 --- a/TODO.final/README.md +++ /dev/null @@ -1,121 +0,0 @@ -# TODO.final — Remaining Work Master Plan - -## TODO index (all rounds) - -### Round 1 (TODOs 01-12) — Version history + per-entity emit -| # | File | Status | -|---|------|--------| -| 01 | [per-entity-cddal-emit.md](01-per-entity-cddal-emit.md) | ✅ done | -| 02 | [per-entity-json-emit.md](02-per-entity-json-emit.md) | ✅ done | -| 03 | [version-aware-reader.md](03-version-aware-reader.md) | ✅ done | -| 04 | [entity-diff-engine.md](04-entity-diff-engine.md) | ✅ done | -| 05 | [browser-typed-raw-properties.md](05-browser-typed-raw-properties.md) | ✅ done | -| 06 | [browser-version-history-type-shadow.md](06-browser-version-history-type-shadow.md) | ✅ done | -| 07 | [browser-time-travel-content-swap.md](07-browser-time-travel-content-swap.md) | ✅ done | -| 08 | [browser-per-version-json-build.md](08-browser-per-version-json-build.md) | ✅ done (iec63213) | -| 09 | [browser-diff-view.md](09-browser-diff-view.md) | ✅ done | -| 11 | [autoload-audit.md](11-autoload-audit.md) | ✅ done (spec-enforced) | -| 12 | [spec-coverage.md](12-spec-coverage.md) | ✅ done | - -### Round 2 (TODOs 13-21) — Parcel output + site improvements -| # | File | Status | -|---|------|--------| -| 13 | [parcel-output.md](13-parcel-output.md) | ✅ done | -| 14 | [dictionary-downloads-section.md](14-dictionary-downloads-section.md) | ✅ done | -| 15 | [recent-changes-page.md](15-recent-changes-page.md) | ✅ done | -| 16 | [site-stats-page.md](16-site-stats-page.md) | ✅ done | -| 17 | [recently-viewed.md](17-recently-viewed.md) | ✅ done | -| 18 | [per-entity-mermaid-hierarchy.md](18-per-entity-mermaid-hierarchy.md) | pending — needs mermaid-cli build integration | -| 19 | [keyboard-nav-timeline.md](19-keyboard-nav-timeline.md) | ✅ done | -| 20 | [csv-export.md](20-csv-export.md) | ✅ done | -| 21 | [docs-update.md](21-docs-update.md) | pending | - -## Final status (2026-07-16, end of round 2) - -**Yes — Parcel output works end-to-end.** `rake browser:build_parcel[iec63213]` emits `data/iec63213/parcel/IEC63213.xlsx` (101 KB), downloadable from `/d/iec63213/` via the new DictionaryDownloads section. - -### Ruby gem (opencdd-ruby) -- 881 specs passing. -- New public API: - - `Opencdd::Cddal::Serializer#emit_entity(entity)` - - `Opencdd::Exporters::Json#payload_for(entity, database:)` — registry dispatch, no case/when - - `Opencdd::Parcel::VersionedReader#versions_for` / `#load_version` - - `Opencdd::EntityDiff.between(a, b)` + `Change` struct -- Code-quality spec enforces autoload invariant for every `lib/opencdd/**/*.rb`. -- Zero `send` / `instance_variable_*` / `respond_to?` / `require_relative` / internal `require` in lib/. - -### Data pipeline (cdd-data) -- `rake browser:build_parcel[]` — Parcel .xlsx emit (build-time, via Ruby). -- `rake browser:build_versions[]` — per-version JSON emit (80 files for iec63213). -- `rake browser:build_all_parcel` — batch Parcel emit across all dicts. - -### Browser (opencdd.github.io) -- **205 vitest tests passing** (was 185 → +20: entityDiff, recentChanges, csvEmitter). -- **21,539 pages built clean** in 44s. -- New components: `VersionTimeline.vue` (with keyboard nav), `VersionDiff.vue`, `DownloadMenu.vue` (JSON/CDDAL/CSV + cdd.iec.ch link), `RecentlyViewed.vue`, `VersionHistorySection.astro`, `DictionaryDownloads.astro`. -- New libs: `cddalEmitter.ts`, `csvEmitter.ts`, `entityDiff.ts`, `recentChanges.ts`, `siteStats.ts`. -- New pages: `/stats/` (site-wide), `/d//changes/` (version feed), `/d//` (now with Downloads section). -- Downloads: per-entity (JSON/CDDAL/CSV from any detail page), per-dictionary (Parcel .xlsx from overview). -- Time travel + diff view from round 1 still intact. - -## Open follow-ups - -- **TODO 18 — Per-entity Mermaid**: needs mermaid-cli integration at build time. -- **TODO 21 — Docs update**: CLAUDE.md / README / docs/ haven't been refreshed for the new surface. -- **Per-version xls scraping**: current scraper captures only current-version xls; historical content time-travel shows metadata-only. -- **Build all dicts' Parcel files**: only `iec63213` has been run through `rake browser:build_parcel`. The other 5 dicts + oceanrunner need the same before deploy. - -## Final status (2026-07-16) - -**Ruby gem (opencdd-ruby)**: -- 881 specs passing (added 31 across TODOs 01–04 + code-quality spec) -- New public API: - - `Opencdd::Cddal::Serializer#emit_entity(entity)` — per-entity CDDAL emit - - `Opencdd::Exporters::Json#payload_for(entity, database:)` — registry-dispatched per-entity payload - - `Opencdd::Parcel::VersionedReader#versions_for(code)` / `#load_version(code, unid)` — historical version access - - `Opencdd::EntityDiff.between(a, b)` — structured diff with `#added` / `#removed` / `#changed` -- Code-quality spec now enforces the autoload invariant for every `lib/opencdd/**/*.rb` file -- Zero `send` / `instance_variable_set/get` / `respond_to?` / `require_relative` / internal `require` in lib/ - -**TS package (cdd-models-ts)**: -- Resolved `VersionHistoryEntry` type shadow — class-backed renamed to `VersionHistoryClassEntry`, wire-format `VersionHistoryEntry` now unambiguous - -**Browser (opencdd.github.io)**: -- 193 vitest tests passing (added 8 in entityDiff) -- 21,884 pages built clean (0 errors) -- New components: `VersionTimeline.vue`, `VersionDiff.vue`, `DownloadMenu.vue`, `VersionHistorySection.astro` -- New libs: `cddalEmitter.ts`, `entityDiff.ts` -- Zero `: any` casts in entity detail pages -- Time-travel UI: click "View this version" → fetches per-version JSON → applies `[data-time-travel]` archival treatment → URL `?v=` for shareability -- Diff view: modal with unified/split modes, color-coded added/removed/changed - -**Data pipeline (cdd-data)**: -- New rake task `browser:build_versions[dict]` emits per-version JSON via `VersionedReader` + `payload_for` -- 80 version JSON files emitted for iec63213 (28 multi-version entities × 2-3 versions each) - -## Open follow-ups - -1. **TODO 10** — source XLS archive hosting. Needs a size-budget decision (per-entity zips × 21k entities ≈ 1-2 GB). Recommend: only emit for multi-version entities, link to cdd.iec.ch for the rest. -2. **Historical xls scraping** — current scraper captures only the current version's .xls exports. Extending it to capture historical xls would unlock full content time-travel (not just metadata). -3. **Build all dicts' versions** — only `iec63213` has been run through `rake browser:build_versions`. Other dicts need the same treatment before deploy. -4. **Per-version content swap** — the UI fetches the JSON and shows metadata, but doesn't yet rewrite the visible entity fields. Currently shows an amber "metadata-only" notice when historical content isn't available. - -## Dependency graph - -``` -01 → 02 → 04 → 09 (diff view consumes per-entity emit + diff engine) - ↓ - 07 (time travel uses per-version JSON) - -03 → 08 (version-aware reader enables per-version JSON build) - ↓ - 07 - -05 → 06 (typed raw_properties unblocks type-shadow fix) - -11 → all Ruby work (autoload audit catches missing entries early) - -10 standalone (cross-repo: cdd-data zip emit + browser link) - -12 final pass after all features land -``` diff --git a/TODO.impl/00-overview.md b/TODO.impl/00-overview.md deleted file mode 100644 index 68b4a24..0000000 --- a/TODO.impl/00-overview.md +++ /dev/null @@ -1,163 +0,0 @@ -# TODO.impl — OpenCDD Ruby: full implementation plan - -## What this is - -A plan to take the `opencdd` Ruby gem from its current extracted state to a -complete, spec-conformant implementation of: - -1. The IEC 61360 / IEC 62656-1 CDD ontology model. -2. The Parcel Excel format (read + write, full ParcelMaker 5.2.1 parity). -3. The CDDAL plain-text format (parse + serialize + round-trip). -4. CDDAL split-file / module system (cross-file references by path or URL). -5. The CDDAL specification document, authored in Metanorma, in a new - `opencdd/cddal-spec` repository. - -The plan is split into scoped, MECE sub-plans (`01`–`15`). Each sub-plan -states its **why**, **scope**, **approach**, **acceptance criteria**, and -**dependencies**. - -## Starting state (2026-07-11) - -The repo at `/Users/mulgogi/src/opencdd/opencdd-ruby` already contains a -substantial extraction of a working `cdd` gem: - -- `lib/cdd/` — 50+ Ruby files: entities, identifiers, database, parcel - readers/writer, CDDAL lexer/parser/builder/serializer, validator rule - classes, exporters. -- `spec/` — 41 spec files including `spec/parcel/*` and a CDDAL spec - driving the OceanRunner fixture. -- `reference-docs/` — ParcelMaker manuals, IEC 61360 markdown, Parcel - xlsx/xls fixtures, the Kagoshima CDDAL sample, the OceanRunner example. -- `opencdd.gemspec` — published as `opencdd`, internal module still `Cdd`. - -What's **not** done, derived from `cdd-data/specs/parcelmaker/`: - -- **Property registry discrepancies** (see plan 02). The cdd-data spec - enumerates ~11 wrong alias→ID mappings and one wrong meta-class IRDI - (Relation should be `MDC_C011`, not `MDC_C006`). These are bugs. -- **CDDAL split-file system** (see plan 09). The current `import_decl` - only takes a string and is skipped on HTTPS. No qualified imports, - no selective imports, no cycle detection, no source-location tracking. -- **CDDAL specification** (see plan 13). The draft at - `cdd-data/reference-docs/specs/cddal-v1.adoc` is a starting point but - lives in the wrong repo, predates the module system, and has its own - inconsistencies (e.g. says `imported_properties` is `MDC_P040`). -- **Audit/parity gaps** in Parcel sheet scaffolding (F03), instance-data - generation (F10), rename (F27), view-control application (F18) — see - plans 06 and 10. - -## Principles (non-negotiable) - -These come from the global CLAUDE.md and the project CLAUDE.md. Every plan -inherits them; they are restated in plan 01. - -- **NEVER delete source files.** `reference-docs/`, `downloads/`, the - ParcelMaker MSI / PDF / xlsx, the Kagoshima sample, IEC PDFs — all - source. Cleanup suggestions only; never `rm` without confirmation. -- **NEVER push tags, NEVER push to main, NEVER commit to main.** All - work goes through PRs on feature branches. -- **NEVER add AI attribution** to commits, PRs, or code comments. -- **NEVER use `double()` in specs.** Real model instances or `Struct`. -- **NEVER hand-roll serialization on model classes.** No `def to_h` / - `from_h` / `to_json` on `Entity`, `Klass`, etc. Service classes - (`Cdd::Cddal::Serializer`, `Cdd::Exporters::Json`) are the correct - pattern. Future lutaml-model migration tracked in plan 15. -- **`autoload`, never `require_relative`, inside `lib/`.** Define the - autoload entries in the immediate parent namespace file. -- **Power-type faithful.** Instances-as-classes is the central abstraction, - not an edge case. Every model decision must preserve it. -- **MECE entity types.** Adding a new entity type = adding a model class + - reader, not editing a switch statement. -- **Persistence is downstream.** The in-memory model is canonical. Formats - are views onto it. - -## Milestones - -| Milestone | Plans | Outcome | -|-----------|-------|---------| -| **M1 — Audit + correct** | 01, 02 | Spec suite green on existing fixtures; known registry discrepancies fixed. | -| **M2 — Parcel parity** | 03, 04, 05, 06 | Full ParcelMaker 5.2.1 read/write feature parity in the gem. | -| **M3 — CDDAL robust** | 07, 08, 10, 11, 12 | CDDAL round-trips through Parcel without semantic loss; validator passes R01–R15. | -| **M4 — CDDAL modules** | 09 | Split-file CDDAL with imports, cycle detection, URL fetcher. | -| **M5 — CDDAL spec** | 13, 14 | `cddal-spec` repo live, Metanorma-built, fed by the implementation. | - -## Dependency graph - -``` - 01 ─┐ - ├─→ 02 ─┬─→ 03 ─┬─→ 05 ─┐ - │ │ │ ├─→ 06 ─┐ - │ │ └─→ 04 ─┘ │ - │ │ │ - │ │ 07 ─┬─→ 08 ─────────┼─→ 09 - │ │ │ │ - │ │ └─→ 10 │ - │ │ │ - │ └───────────────────────┼─→ 12 - │ │ - └───────────────────────────────┴─→ 13 ─→ 14 - - 15 (open questions, ongoing) -``` - -Read top-to-bottom. Plans at the same level can proceed in parallel once -their ancestors complete. - -## Sequencing recommendation - -1. Start with **plan 02** (audit + discrepancies) — it's the highest-value - per hour because every downstream plan benefits from a correct registry. -2. Run **plan 12's** test matrix as a baseline *before* changes, so the - effect of each plan is measurable. -3. Do **plans 03–06** in parallel once 02 lands — they're MECE. -4. **Plan 09** (CDDAL modules) is the biggest net-new feature; give it a - dedicated branch and don't gate other work on it. -5. **Plan 13** (cddal-spec) can begin early but should land last, so the - spec describes the implementation as shipped. - -## Out of scope for this plan set - -- The OpenCDD Editor (TS web app) — lives in `opencdd/editor/`, has its - own roadmap. -- CDD upload to `cdd.iec.ch` — separate protocol investigation. -- CIM/RDF export — future; not in ParcelMaker 5.2.1. -- Live CDD search — blocked by AWS WAF on cdd.iec.ch. - -These are tracked in plan 15 (open questions). - -## Repo layout (target) - -``` -opencdd-ruby/ -├── lib/ -│ ├── cdd.rb # autoload root (Cdd module) -│ ├── opencdd.rb # gem entry (require "cdd") -│ └── cdd/ -│ ├── entity.rb klass.rb property.rb unit.rb -│ ├── value_list.rb value_term.rb relation.rb view_control.rb -│ ├── database.rb -│ ├── irdi.rb property_ids.rb alias_table.rb meta_class.rb -│ ├── class_type.rb data_type.rb value_format.rb condition.rb -│ ├── property_data_element_type.rb relation_type.rb languages.rb -│ ├── effective_properties.rb -│ ├── class_tree.rb composition_tree.rb relation_tree.rb -│ ├── visitor.rb validator.rb validator/*.rb -│ ├── instance_rule.rb guid.rb structured_values.rb parse_helpers.rb -│ ├── parcel.rb # namespace + aggregate/split -│ │ └── parcel/*.rb -│ ├── cddal.rb # namespace + parse / serialize -│ │ └── cddal/*.rb # lexer, parser, builder, serializer, -│ │ # modules (plan 09) -│ ├── exporters.rb # namespace -│ │ └── exporters/*.rb # json, yaml, mermaid -│ └── codegen/ -│ └── ts.rb # editor TS code generation -├── spec/ # 41+ spec files -├── reference-docs/ # source material — never delete -├── opencdd.gemspec -├── Rakefile -└── TODO.impl/ # this directory -``` - -The `cddal-spec` repository (plan 13) is a **separate** repo at -`/Users/mulgogi/src/opencdd/cddal-spec/`. diff --git a/TODO.impl/01-project-conventions.md b/TODO.impl/01-project-conventions.md deleted file mode 100644 index 35b2115..0000000 --- a/TODO.impl/01-project-conventions.md +++ /dev/null @@ -1,138 +0,0 @@ -# Plan 01 — Project conventions and non-negotiable rules - -## Why - -The `opencdd` gem is a fresh extraction that will be touched by many plans. -Before any feature work, every contributor (human or AI) needs a single -reference for the conventions that govern the codebase. These rules override -default instincts and apply uniformly across all plans in this set. - -## Scope - -Codify the project's conventions so they are enforceable and self-contained. -This plan is **reference material** — it produces no code; it produces a -checklist every other plan conforms to. - -## Conventions - -### Naming - -- The gem is published as **`opencdd`** on RubyGems. -- The Ruby module is **`Cdd`**. The module name appears in every class - path (`Cdd::Entity`, `Cdd::Klass`, `Cdd::Parcel::Workbook`, …). -- Both `require "opencdd"` and `require "cdd"` work (`lib/opencdd.rb` - forwards to `lib/cdd.rb`). -- Never name a class `Class` — the IEC entity type is `Cdd::Klass`. - -### Library load structure - -- **`autoload` only** inside `lib/`. Never `require_relative` for library - code. Never `require ""` for library code. -- Define autoload entries in the **immediate parent namespace's file**. - Examples: - - `lib/cdd.rb` declares `autoload :Database, "cdd/database"`. - - `lib/cdd/parcel.rb` declares `autoload :WorkbookReader, "cdd/parcel/workbook_reader"`. - - `lib/cdd/cddal.rb` declares `autoload :Lexer, "cdd/cddal/lexer"`. -- If a new namespace file doesn't exist, create it. Do not scatter autoloads. -- The gemspec is the **only** place `$LOAD_PATH.unshift` is acceptable. - -### Specs (from global CLAUDE.md) - -- **Never `double()`**. Use real model instances. If a model is hard to - construct, build a `Struct.new(*attrs)` factory in `spec/support/`. -- Test behavior (output and state), not interactions. - "Should have received" mocks are testing implementation, not correctness. -- `expect_with :rspect` syntax only (no `should`). -- `config.disable_monkey_patching!` — already set in `spec/spec_helper.rb`. -- Specs that touch fixtures use the constants in `spec/spec_helper.rb` - (`PARCEL_MAKER_XLSX`, `NUTS_XLSX`, `KAGOSHIMA_CDDAL`, `LEGACY_XLS_DIR`, - `LEGACY_SINGLE_XLS`). - -### Serialization - -- **Never write `def to_h`, `def to_json`, `def from_h`, `def serialize`, - `def deserialize` on a model class** (`Entity`, `Klass`, `Property`, - `Unit`, `ValueList`, `ValueTerm`, `Relation`, `ViewControl`, and any - new model classes). -- Serialization lives in **service classes**: - - `Cdd::Cddal::Serializer` — Database → CDDAL text. - - `Cdd::Exporters::Json` / `Yaml` / `Mermaid` — Database → other formats. - - `Cdd::Parcel::Writer` — Database → .xlsx. - - `Cdd::Parcel::CsvWriter` — Sheet → .csv. -- Deserialization lives in reader classes: - - `Cdd::Cddal::Builder` (driven by the parser) — CDDAL text → Database. - - `Cdd::Parcel::WorkbookReader` / `FlatDirReader` / `ShardedDirReader` / - `CsvReader` — file → Database. -- Future lutaml-model migration (plan 15) will replace the field-level - accessors with typed attribute declarations; the service-class boundary - stays the same. - -### Source-of-truth literals - -- **No raw `"MDC_P###"` or `"MDC_C###"` literals** outside the registry - files. Use `Cdd::PropertyIds::`, `Cdd::MetaClasses::`, - or `Cdd::AliasTable#resolve(alias_name)`. -- The registry files are: - - `lib/cdd/property_ids.rb` — `Cdd::PropertyIds::REGISTRY` and the - constant-for-each-entry pattern. - - `lib/cdd/meta_class.rb` — `Cdd::MetaClasses::REGISTRY`. - - `lib/cdd/alias_table.rb` — built-in alias defaults. -- Code-generation for the TS editor port uses the same REGISTRY as the - source of truth (`lib/cdd/codegen/ts.rb`). - -### Design principles - -- **Open/Closed.** Adding a new entity type, validator rule, exporter - format, or Parcel sheet type = adding a new class + registration, not - editing a switch statement. -- **DRY** without premature abstraction. Three similar lines is better - than a wrong abstraction. Five similar lines is the threshold to extract. -- **MECE.** Every concern lives in exactly one place. No overlap, no gaps. -- **Model-driven, semantically-driven.** Classes named after domain - concepts. Methods named after domain actions. Data flows through the - model, not around it. -- **Frozen value objects.** Entities are constructed once, then frozen. - Mutations go through command objects (`Database#rename_entity`, - `Database#apply_change_request`), not in-place setters. - -### Forbidden patterns - -- `send` to call private methods. Make the method public or redesign. -- `instance_variable_set` / `instance_variable_get` from outside the - owning class. -- `respond_to?` for type checks. Use `is_a?` or design the hierarchy so - the check isn't needed. -- `require_relative` inside `lib/`. -- `double()` in specs. -- `rm -f`, `rm -rf` on files you didn't create (see global CLAUDE.md). -- Direct commits to `main`, force-push, tag pushing, AI attribution - (see global CLAUDE.md). - -### Comments - -- Default to **no comments**. Only add a comment when the **why** is - non-obvious (hidden constraint, subtle invariant, workaround for a - specific bug, surprising behavior). -- Never explain **what** the code does — well-named identifiers do that. -- Never reference the current task, fix, or callers ("added for issue - #123", "used by X"). That belongs in the PR description. - -## Acceptance criteria - -- [x] This plan is referenced in every other plan's "Conventions" section. -- [x] A new contributor reading only this plan + `00-overview.md` can - orient themselves in the codebase. -- [x] No file in `lib/` contains `require_relative`. -- [x] No spec file contains `double(`, `instance_double(`, or - `allow_any_instance_of(`. -- [x] No model file (`entity.rb`, `klass.rb`, `property.rb`, `unit.rb`, - `value_list.rb`, `value_term.rb`, `relation.rb`, `view_control.rb`) - defines `to_h`, `to_json`, `from_h`, `serialize`, or `deserialize`. - -## Dependencies - -- None. This is the foundational reference. - -## Open questions - -None. The conventions are decided. diff --git a/TODO.impl/02-audit-and-discrepancies.md b/TODO.impl/02-audit-and-discrepancies.md deleted file mode 100644 index a11732d..0000000 --- a/TODO.impl/02-audit-and-discrepancies.md +++ /dev/null @@ -1,181 +0,0 @@ -# Plan 02 — Audit current state and fix property-registry discrepancies - -## Why - -The cdd-data spec at `specs/parcelmaker/03_property_registry.md` enumerates -concrete bugs in the current `lib/cdd/` extraction where property IDs and -meta-class IRDIs are wrong relative to the ParcelMaker VBA source (which -itself reflects IEC 62656-1). Every downstream plan inherits these bugs -unless they're fixed first. - -This plan is the highest-leverage work in the set: small, mechanical, and -unblocks correctness everywhere else. - -## Scope - -1. **Run the existing spec suite** and capture a baseline pass/fail. -2. **Reconcile every discrepancy** enumerated in - `cdd-data/specs/parcelmaker/03_property_registry.md` - §"Discrepancy with current Ruby code". -3. **Reconcile CDDAL-spec inconsistencies** — the draft at - `cdd-data/reference-docs/specs/cddal-v1.adoc` cites wrong IDs for - `imported_properties` and `sub_class_selection`. Update the spec - draft in-place (the canonical move to cddal-spec happens in plan 13). -4. **Run the spec suite again** and ensure no regressions. -5. Commit per-discrepancy so reverts are surgical. - -## Approach - -### Step 1 — Baseline - -```bash -cd /Users/mulgogi/src/opencdd/opencdd-ruby -bundle install -bundle exec rspec --format documentation 2>&1 | tee /tmp/cdd-baseline.log -``` - -Capture: total examples, failures, pending. This is the **before** state. -Plan 12 will turn this into a structured test matrix. - -### Step 2 — Discrepancies from `03_property_registry.md` - -These are enumerated in the cdd-data spec. Reproduced here for actionability -— the cdd-data spec is the source of truth. - -#### 2a. Meta-class IDs (lib/cdd/meta_class.rb) - -| Meta-class | Currently | Correct | Note | -|------------|-----------|---------|------| -| Datatype | — | `MDC_C006` | Currently absent; added. | -| Relation | `MDC_C006` | `MDC_C011` | Wrong ID in current code. | - -Action: in `lib/cdd/meta_class.rb`, reassign `MDC_C006 → Datatype`, -add `MDC_C011 → Relation`. Update `TYPE_BY_META_CLASS` and -`META_CLASS_BY_TYPE` accordingly. Search for any code that hard-codes -`MDC_C006` meaning Relation and fix it. - -#### 2b. Property reads (lib/cdd/property.rb) - -| Ruby method | Reads today | Should read | -|----------------------------|--------------------|--------------------| -| `data_type` | `MDC_P018` | `MDC_P022` | -| `value_format` | `MDC_P022` | `MDC_P024` | -| `definition_class_irdi` | `MDC_P017` | `MDC_P021` | -| `unit_irdi` | `MDC_P030` | `MDC_P041` | -| `alternative_unit_irdis` | `MDC_P031` | `MDC_P042`/`MDC_P111` | -| `formula` | `MDC_P026_1`/`MDC_P026` | `MDC_P027_1`/`MDC_P027_2` | -| `symbol_in_text` | `MDC_P023_1`/`MDC_P021` | `MDC_P025_1` | -| `constraint` | `MDC_P032` | `MDC_P068` | - -Action: edit each method's `properties[Cdd::PropertyIds::]` lookup -to the corrected constant. Add a regression spec per method. - -#### 2c. Property reads (lib/cdd/klass.rb) - -| Ruby method | Reads today | Should read | -|--------------------------------|-------------|-------------| -| `imported_property_irdis` | `MDC_P040` | `MDC_P090` | -| `sub_class_selection_irdis` | `MDC_P033` | `MDC_P016` | - -#### 2d. Extend REGISTRY (lib/cdd/property_ids.rb) - -Add `Entry` records for each missing ID with VBA-constant alias and -applies-to metadata. Missing IDs from the cdd-data spec: - -- **Class-level**: `MDC_P012`, `MDC_P014_1`, `MDC_P014_2`, `MDC_P015`, - `MDC_P015_1`, `MDC_P015_2`, `MDC_P016`, `MDC_P090`, `MDC_P091`, - `MDC_P093`, `MDC_P094`, `MDC_P094_1`, `MDC_P094_2`. -- **Property-level**: `MDC_P024`, `MDC_P025_1`, `MDC_P025_2`, - `MDC_P025_3`, `MDC_P027_1`, `MDC_P027_2`, `MDC_P041`, `MDC_P042`, - `MDC_P068`, `MDC_P096`, `MDC_P097`, `MDC_P101`, `MDC_P102`, - `MDC_P110`, `MDC_P111`, `MDC_P114`. -- **Relation-level**: `MDC_P230`, `MDC_P231`. -- **CIM extension**: `CIM_P001`–`CIM_P005`. -- **Codes for completeness**: `MDC_P001_1`, `MDC_P001_2`, `MDC_P001_7`, - `MDC_P001_8`. - -#### 2e. Reassign aliases (lib/cdd/alias_table.rb built-ins) - -| Alias | Currently on | Should be on | -|-----------------------------|---------------|---------------| -| `data_type` | `MDC_P018` | `MDC_P022` | -| `value_format` | `MDC_P022` | `MDC_P024` | -| `definition_class` | `MDC_P017` | `MDC_P021` | -| `unit` / `unit_irdi` | `MDC_P030` | `MDC_P041` | -| `alternative_units` | `MDC_P031` | `MDC_P042` | -| `constraint` | `MDC_P032` | `MDC_P068` | -| `sub_class_selection` | `MDC_P033` | `MDC_P016` | -| `imported_properties` | `MDC_P040` | `MDC_P090` | -| `symbol` (Property) | `MDC_P021` | `MDC_P025_1` | -| `property_formula` | `MDC_P026` | `MDC_P027_1` | -| `property_formula_localized`| `MDC_P026_1` | `MDC_P027_2` | - -The displaced IDs keep their REGISTRY entries with their VBA-constant -aliases (`coded_name`, `class_value_assignment`, etc.) — Parcel xlsx -files in the wild may still reference them. - -#### 2f. Update tests that encode the bug - -`spec/property_ids_spec.rb`, `spec/alias_table_spec.rb`, -`spec/property_spec.rb`, `spec/klass_spec.rb`, -`spec/meta_class_spec.rb`, and any other spec asserting the old wrong -values. Update assertions to the corrected values. - -#### 2g. CDDAL spec draft inconsistencies - -In `cdd-data/reference-docs/specs/cddal-v1.adoc`: - -- §"AliasDecl" example: `alias imported_properties: MDC_P040` → `MDC_P090`. -- §"AliasDecl" example: `alias sub_class_selection: MDC_P033` → `MDC_P016`. -- §"Built-in property aliases" appendix: same two rows. -- §"meta-class" terms-and-definitions: Relation meta-class is `MDC_C011`, - not `MDC_C006`. Datatype is `MDC_C006`. - -### Step 3 — Verification - -```bash -bundle exec rspec -bundle exec rake build # gem must still build -``` - -Run the OceanRunner CDDAL round-trip spec (`spec/cddal_spec.rb`) — it must -remain semantically equal. Same for any `spec/parcel/round_trip_spec.rb`. - -### Step 4 — Commit strategy - -One commit per discrepancy class, on a feature branch, one PR per commit: - -1. `branch: fix/meta-class-irdi-relation` — step 2a. -2. `branch: fix/property-reads-correct-ids` — steps 2b + 2c. -3. `branch: fix/registry-add-missing-ids` — step 2d. -4. `branch: fix/alias-reassignment` — step 2e + 2f (together, since - tests assert alias→ID). -5. `branch: docs/cddal-spec-id-fixes` — step 2g. - -Never commit to main. Never push tags. No AI attribution in any commit. - -## Acceptance criteria - -- [x] Baseline spec output captured at `/tmp/cdd-baseline.log`. -- [x] All discrepancies from `03_property_registry.md` fixed. -- [x] `bundle exec rspec` green (or, if any pre-existing failures remain, - their count is unchanged and they are documented in plan 12). -- [x] OceanRunner CDDAL round-trip still semantically equal. -- [x] `lib/cdd/property_ids.rb` REGISTRY contains every ID cited in - `03_property_registry.md`. -- [x] No raw `"MDC_P022"` / `"MDC_P090"` / etc. literal in `lib/cdd/` - outside `property_ids.rb` and `meta_class.rb`. -- [x] CDDAL spec draft at `cdd-data/reference-docs/specs/cddal-v1.adoc` - uses the corrected IDs in examples and appendices. - -## Dependencies - -- **Plan 01** — conventions (no `require_relative`, no AI attribution, etc.). - -## Open questions - -- **Q1.** If a fixture (e.g., theParcelMaker nuts xlsx) encodes a value - under a wrong ID, do we re-key on import or preserve verbatim? - **Recommendation:** preserve verbatim in `Entity#properties`, re-key - in the typed accessor. That way the reader is lossless and the typed - API is correct. diff --git a/TODO.impl/03-domain-model.md b/TODO.impl/03-domain-model.md deleted file mode 100644 index 8819dde..0000000 --- a/TODO.impl/03-domain-model.md +++ /dev/null @@ -1,225 +0,0 @@ -# Plan 03 — Domain model (entities + value objects) - -## Why - -The domain model is the canonical in-memory representation of CDD content. -Every format reader, writer, validator, and exporter targets this model. -Getting it right is a precondition for Parcel parity (plan 06), CDDAL -round-trip (plans 07–08), and validation (plan 10). - -The model already exists in `lib/cdd/`. This plan audits each class against -IEC 61360 / IEC 62656-1 / `cdd-data/specs/parcelmaker/`, documents the -typed accessors each class must expose, and confirms the powertype-faithful -invariants. - -## Scope - -Define and lock down the **public API** of every model class. Internal -representation is private and may change (especially in plan 15's -lutaml-model migration). Downstream code targets only the public API. - -Not in scope: the registry itself (plan 04), the Database (plan 05), -the Parcel/CDDAL formats (plans 06–09). - -## Approach - -### Entity (lib/cdd/entity.rb) — base class - -Every CDD object is an `Entity`. Fields: - -| Method | Type | Source ID | -|------------------------------|---------------------|-----------| -| `irdi` | `Cdd::IRDI` | — | -| `meta_class_irdi` | `Cdd::IRDI` | — | -| `type` | `Symbol` | derived from meta_class | -| `code` | `String` | meta-class-specific code property (e.g. `MDC_P001_5` for class) | -| `properties` | `Hash{String=>String}` | raw property ID → value (lossless) | -| `schema` | `Symbol` | sheet family this entity came from (for diagnostics) | -| `preferred_name` | `String` | `MDC_P004` (current language) | -| `preferred_name(lang:)` | `String` | per-language | -| `synonymous_names` | `Array<[String,String]>` | `MDC_P007` structured | -| `short_name` | `String` | `MDC_P005` | -| `definition` | `String` | `MDC_P006` | -| `source_document_of_definition` | `String` | `MDC_P006_1` | -| `note` | `String` | `MDC_P008` | -| `remark` | `String` | `MDC_P009` | -| `version_number` | `Integer` | `MDC_P002_1` | -| `revision_number` | `Integer` | `MDC_P002_2` | -| `time_stamp` / dates | `Date`/`Time` | `MDC_P003_*` | -| `guid` | `String` | `MDC_P066` | -| `source_document` | `String` | `MDC_P006_1` | - -Invariants: -- `Entity` instances are frozen after construction. -- `properties` is the lossless raw store; typed accessors read from it - via `Cdd::PropertyIds::` lookups. -- `Entity#type` is derived from `meta_class_irdi` via `Cdd::MetaClasses`. -- `Entity#database` is a transient back-pointer set by `Database#add_entity`, - not part of the value identity. - -### Klass (lib/cdd/klass.rb) — class entity - -Extends `Entity` for entities whose `meta_class_irdi == MDC_C002`. - -| Method | Type | Source | -|--------------------------------|-------------------------|--------| -| `parent_irdi` | `Cdd::IRDI` | `MDC_P010` | -| `parent` | `Cdd::Klass` or nil | resolves parent_irdi via Database | -| `children` | `Array` | reverse index | -| `class_type` | `Cdd::ClassType` | `MDC_P011` | -| `is_case_of_irdis` | `Array` | `MDC_P013` | -| `applicable_property_irdis` | `Array` | `MDC_P014` | -| `imported_property_irdis` | `Array` | `MDC_P090` (fix from plan 02) | -| `sub_class_selection_irdis` | `Array` | `MDC_P016` (fix from plan 02) | -| `supplier_irdi` | `Cdd::IRDI` | `MDC_P012` | -| `applicable_types` | `Array` | `MDC_P015` | -| `applicable_documents` | `Array` | `MDC_P094` | - -Power-type invariant: a Klass with `class_type == CATEGORICAL_CLASS` may -have instances that are themselves classes (instance-of-class-of-class). -The model must not forbid this; the Database (plan 05) maintains the -reverse index `instancesOf[class_irdi] = [klass, ...]`. - -### Property (lib/cdd/property.rb) — property entity - -Extends `Entity` for entities whose `meta_class_irdi == MDC_C003`. - -| Method | Type | Source | -|-----------------------------------|----------------------------|--------| -| `property_data_element_type` | `Cdd::PropertyDataTypeElement` | `MDC_P020` | -| `definition_class_irdi` | `Cdd::IRDI` | `MDC_P021` (fix) | -| `data_type` | `String` (raw) | `MDC_P022` (fix) | -| `parsed_data_type` | `Cdd::DataType` | parses `data_type` | -| `unit_irdi` | `Cdd::IRDI` | `MDC_P041` (fix) | -| `alternative_unit_irdis` | `Array` | `MDC_P042` (fix) | -| `quantity_irdi` | `Cdd::IRDI` | `MDC_P114` | -| `value_format` | `Cdd::ValueFormat` | `MDC_P024` (fix) | -| `formula` / `formula_localized` | `String` | `MDC_P027_1`/`MDC_P027_2` (fix) | -| `symbol_in_text` / `symbol_in_sgml` | `String` | `MDC_P025_1`/`MDC_P025_2` (fix) | -| `condition` | `Cdd::Condition` | `MDC_P028` | -| `constraint` | `String` | `MDC_P068` (fix) | -| `value_list_irdi` | `Cdd::IRDI` | derived from `parsed_data_type` when ENUM | -| `super_property_irdi` | `Cdd::IRDI` | `MDC_P110` | -| `requirement` | `Symbol` (:key/:mand/:opt) | `MDC_P097` | - -### Unit (lib/cdd/unit.rb) — unit entity - -`meta_class_irdi == MDC_C009`. - -| Method | Type | Source | -|-----------------------|---------|--------| -| `unit_code` | String | `MDC_P001_10` | -| `names` | multilingual | `MDC_P004` | -| `unit_symbol` | String | `MDC_P025_1` (repurposed) | -| `unit_in_text` | String | `MDC_P023_1` | - -### ValueList (lib/cdd/value_list.rb) — enumeration entity - -`meta_class_irdi == MDC_C005`. - -| Method | Type | Source | -|-----------------------|--------------------------|--------| -| `value_list_code` | String | `MDC_P001_12` | -| `terms` | `Array` | reverse index from ValueTerm's `MDC_P025_1` | -| `enumerated_code_list`| String | `MDC_P044` | -| `controlling_property_irdis` | `Array` | reverse: properties referencing this list | - -### ValueTerm (lib/cdd/value_term.rb) — term entity - -`meta_class_irdi == MDC_C010`. - -| Method | Type | Source | -|-----------------------|---------------------|--------| -| `term_code` | String | `MDC_P001_11` | -| `preferred_letter_symbol` | String | `MDC_P025_1` | -| `value_list_irdi` | `Cdd::IRDI` | resolves via Database | - -### Relation (lib/cdd/relation.rb) — relation entity - -`meta_class_irdi == MDC_C011` (fix from plan 02). - -| Method | Type | Source | -|---------------------------|-----------------------|--------| -| `relation_type` | `Cdd::RelationType` | `MDC_P200` | -| `domain_irdis` | `Array` | `MDC_P201` | -| `codomain_irdis` | `Array` | `MDC_P202`/`MDC_P203` | -| `domain_element_type` | `Cdd::DataType` | `MDC_P208` | -| `codomain_element_type` | `Cdd::DataType` | `MDC_P209` | -| `super_relation_irdi` | `Cdd::IRDI` | `MDC_P212` | - -### ViewControl (lib/cdd/view_control.rb) - -`meta_class_irdi == EXT_C001`. - -| Method | Type | Source | -|---------------------------|---------------------|--------| -| `view_control_code` | String | `EXT_P001` | -| `controlled_class_irdis` | `Array` | `EXT_P002` | -| `shown_property_irdis` | `Array` | `EXT_P003` | - -### Value objects (frozen, immutable) - -| Class | File | Purpose | -|-------|------|---------| -| `Cdd::IRDI` | `irdi.rb` | Parses 3 forms (full/short/commented); comparable; hashable. | -| `Cdd::ClassType` | `class_type.rb` | ITEM_CLASS, CATEGORICAL_CLASS, etc. | -| `Cdd::DataType` | `data_type.rb` | Primitive tokens + ENUM_*(list) + CLASS_REFERENCE(class) + AGGREGATE nesting (depth ≤ 3). | -| `Cdd::ValueFormat` | `value_format.rb` | IEC 61360-2 NR1/NR2/NR3/M/B/Date/DT/Bool tokens. | -| `Cdd::Condition` | `condition.rb` | Equality predicates (`==`, `!=`) on property refs; sets for OR. | -| `Cdd::PropertyDataTypeElement` | `property_data_element_type.rb` | NON_DEPENDENT_P_DET, DEPENDENT_P_DET, CONDITION_DET, etc. | -| `Cdd::RelationType` | `relation_type.rb` | FUNCTION, etc. | -| `Cdd::Languages` | `languages.rb` | Source/translation language list per workbook. | - -### Construction pattern - -All model classes use keyword-init constructors: - -```ruby -Cdd::Klass.new( - irdi: "0112/2///62656_1#AAA001##1", - properties: { "MDC_P001_5" => "AAA001", "MDC_P010" => "UNIVERSE", ... }, - schema: :class, - meta_class_irdi: "MDC_C002", -) -``` - -`Database#add_entity` then attaches the back-pointer and updates reverse -indexes. Entities are frozen at end of construction. - -## Acceptance criteria - -- [x] Every public method listed above exists and returns the documented type. -- [x] `bundle exec rspec spec/entity_spec.rb spec/klass_spec.rb - spec/property_spec.rb spec/unit_spec.rb spec/value_list_spec.rb - spec/value_term_spec.rb spec/relation_spec.rb spec/view_control_spec.rb` - green. -- [x] A round-trip Property → properties hash → Property preserves the - typed accessor outputs. -- [x] Powertype invariant: a `Klass` whose `class_type` is - `CATEGORICAL_CLASS` may appear in another `Klass`'s - `is_case_of_irdis` without raising. -- [x] No `to_h`, `to_json`, `from_h`, `serialize`, or `deserialize` - instance method on any model class. - -## Dependencies - -- **Plan 01** — conventions. -- **Plan 02** — property ID fixes must land before reads can be correct. -- Blocks **plan 05** (Database indexes entities), **plan 06** (Parcel - readers construct entities), **plan 07** (CDDAL Builder constructs - entities), **plan 10** (validator walks entities). - -## Open questions - -- **Q1.** Should `Entity#properties` be a `Hash` or a typed collection? - Current: plain `Hash`. **Recommendation:** keep plain `Hash` until - lutaml-model migration (plan 15) — typed collection adds friction - for the lossless raw store. -- **Q2.** How do we represent unknown property IDs encountered in a - Parcel xlsx (not in the REGISTRY)? **Recommendation:** preserve them - verbatim in `properties`; expose via `Entity#raw_property(id)` only. - Don't add typed accessors for non-registry IDs. -- **Q3.** Should there be a `Dictionary` and `Supplier` entity class - (for `MDC_C001` / `MDC_C004`)? **Recommendation:** yes, but defer to - a follow-up — current readers don't need to model them as entities; - they live in the `Project` and `pcls_LOCAL` sheets. diff --git a/TODO.impl/04-identifiers-and-registries.md b/TODO.impl/04-identifiers-and-registries.md deleted file mode 100644 index b614921..0000000 --- a/TODO.impl/04-identifiers-and-registries.md +++ /dev/null @@ -1,191 +0,0 @@ -# Plan 04 — Identifiers and registries - -## Why - -Every entity in CDD is identified by an IRDI; every property of every -meta-class has a stable property ID. The Ruby gem's `PropertyIds`, -`MetaClasses`, and `AliasTable` registries are the single source of truth -(SSOT) — no raw `"MDC_P###"` / `"MDC_C###"` literal is allowed outside -these files (per plan 01). - -This plan locks down the SSOT, ensures it matches IEC 62656-1 / IEC 61360-1 -/ the ParcelMaker VBA constants, and ensures every consumer reads from it. - -## Scope - -- `Cdd::IRDI` — parsing, equality, canonicalization. -- `Cdd::PropertyIds` — registry of every property ID + reverse aliases. -- `Cdd::MetaClasses` — registry of every meta-class IRDI. -- `Cdd::AliasTable` — alias → property ID resolution, built-in defaults. -- Variant normalization (`MDC_P004_1` → `MDC_P004`). -- Code generation for the TS editor port. - -Not in scope: model classes that consume the registry (plan 03), Database -(plan 05). - -## Approach - -### IRDI (lib/cdd/irdi.rb) - -Supports the three forms from `04_validation_rules.md` R01: - -| Form | Example | -|-----------|--------------------------------------| -| Full | `0112/2///62656_1#AAA001##1` | -| Short | `AAA001` or `MDC_C002` | -| Commented | `0112/2///62656_1#AAA001##1###note` | - -Public API: - -```ruby -Cdd::IRDI.parse(string) # → Cdd::IRDI or raises Cdd::IRDI::ParseError -Cdd::IRDI.well_formed?(string) # → bool -irdi.full_form # → "0112/2///62656_1#AAA001##1" -irdi.short_form # → "AAA001" -irdi.code # → "AAA001" -irdi.supplier # → "0112/2///62656_1" or nil -irdi.version # → Integer or nil -irdi.comment # → String or nil -irdi.to_s # → original form -irdi.eql?(other) / hash # value-equality ignoring comment -``` - -Equality semantics: two IRDIs are equal if they resolve to the same entity -(supplier + code + version). Comment is metadata, not identity. - -Helpers (already present, keep them): - -```ruby -Cdd::PropertyIds.as_entity_irdi(code, meta_class_irdi:, supplier:, version:) -Cdd::PropertyIds.as_property_irdi(...) -``` - -### PropertyIds (lib/cdd/property_ids.rb) - -`REGISTRY` is a `Hash{ String => Entry }` keyed by property ID. `Entry` -has at minimum: - -- `id` — the canonical ID (`"MDC_P022"`). -- `aliases` — array of symbolic names (`["data_type", "datatype"]`). -- `applies_to` — array of meta-class IDs that declare it. -- `value_kind` — `:scalar` / `:identifier_ref` / `:set_of_refs` / - `:class_ref` / `:structured` / `:localized_string`. -- `parcel_variant` — original ParcelMaker variant ID if different from - canonical (e.g. `MDC_P005` → variant `MDC_P005`). - -For every key in `REGISTRY`, a matching constant is defined: - -```ruby -module Cdd::PropertyIds - REGISTRY.each do |id, _entry| - const_set(id, id) - end -end -``` - -So callers write `Cdd::PropertyIds::MDC_P022`, never `"MDC_P022"`. - -`PARCEL_VARIANT_TO_CANONICAL` maps ParcelMaker variant IDs to canonical -IDs (`MDC_P005` → `MDC_P005`, `MDC_P004_2` → `MDC_P007`, etc.). The -Parcel reader applies this map when populating `Entity#properties`. - -`CODE_PROPERTY_BY_META_CLASS` maps a meta-class IRDI to the property ID -that holds its code (e.g. `MDC_C002` → `MDC_P001_5`). Used by -`Entity#code`. - -### MetaClasses (lib/cdd/meta_class.rb) - -`REGISTRY` is a `Hash{ String => Entry }` keyed by meta-class ID. Entry: - -- `id` — `"MDC_C002"`. -- `name` — `"Class"`. -- `type` — `:class` (Symbol used throughout the codebase). -- `sheet_type` — `"CLASS"` (Parcel sheet-type string). -- `code_property_id` — `MDC_P001_5`. - -The fix from plan 02 lands here: `MDC_C011` is Relation, `MDC_C006` is -Datatype. - -Derived maps (computed once at load): - -```ruby -MetaClasses::TYPE_BY_META_CLASS # { "MDC_C002" => :class, ... } -MetaClasses::META_CLASS_BY_TYPE # inverse -MetaClasses::SHEET_TYPE_BY_META_CLASS -``` - -### AliasTable (lib/cdd/alias_table.rb) - -Public API: - -```ruby -table = Cdd::AliasTable.new(database:) # inherits database's overrides -table.resolve("superclass") # → "MDC_P010" -table.resolve("MDC_P010") # → "MDC_P010" (passthrough) -table.declare("colour", "MDC_P999") # adds a document-scoped alias -table.each_alias # enumerable -``` - -Built-in defaults: see the appendix of `cddal-v1.adoc` (after plan 02's -fixes). The AliasTable seeds from these defaults, then layers document -overrides from `alias` declarations in CDDAL or header rows in Parcel. - -Collision policy: if a document declares an alias that already exists, -emit a warning and overwrite. The cdd-data spec `00_overview.md` notes -the SSOT principle — one alias per ID per document. - -### Code generation (lib/cdd/codegen/ts.rb) - -`rake generate_ts` reads the Ruby `REGISTRY` and emits two checked-in TS -files for the editor: - -- `editor/src/models/PropertyIds.generated.ts` -- `editor/src/models/MetaClasses.generated.ts` - -The Ruby REGISTRY is the source; TS is the derived output. Never edit the -`.generated.ts` files by hand. - -### No raw literals rule - -After this plan: - -```bash -grep -rn '"MDC_[CP]0[0-9]' lib/cdd/ \ - | grep -v 'lib/cdd/property_ids.rb' \ - | grep -v 'lib/cdd/meta_class.rb' \ - | grep -v 'lib/cdd/alias_table.rb' -# expected: empty -``` - -Same for `"EXT_P###"`, `"EXT_C###"`, `"CIM_P###"`. - -## Acceptance criteria - -- [x] `Cdd::IRDI.parse` accepts all three forms; `well_formed?` matches. -- [x] `Cdd::PropertyIds::REGISTRY` covers every ID in - `cdd-data/specs/parcelmaker/03_property_registry.md`. -- [x] Every `REGISTRY` key has a matching constant - (`Cdd::PropertyIds::MDC_P022 == "MDC_P022"`). -- [x] `Cdd::MetaClasses::REGISTRY` has `MDC_C011` for Relation and - `MDC_C006` for Datatype. -- [x] No raw `"MDC_P###"` / `"MDC_C###"` literal outside the three - registry files. -- [x] `rake generate_ts` regenerates the editor's `.generated.ts` files - idempotently. -- [x] `spec/irdi_spec.rb`, `spec/property_ids_spec.rb`, - `spec/meta_class_spec.rb`, `spec/alias_table_spec.rb` green. - -## Dependencies - -- **Plan 01** — conventions. -- **Plan 02** — fixes land here (registry is the destination for the - corrected IDs). - -## Open questions - -- **Q1.** Should the AliasTable warn or error on collision? Currently - warns. **Recommendation:** warn by default, strict mode in validator. -- **Q2.** Do we need versioned aliases (e.g., `superclass` resolving - differently in IEC 61360-4 vs IEC 61360-5)? **Recommendation:** no — - aliases are document-scoped, not version-scoped. Documents pin their - own overrides via `alias` declarations. diff --git a/TODO.impl/05-database.md b/TODO.impl/05-database.md deleted file mode 100644 index 2a92d72..0000000 --- a/TODO.impl/05-database.md +++ /dev/null @@ -1,193 +0,0 @@ -# Plan 05 — Database (in-memory store + indexes + mutations) - -## Why - -The `Cdd::Database` is the canonical in-memory representation of a CDD -dictionary (or a merge of several). All format readers populate one, all -writers serialize one, the validator walks one, all interactive commands -mutate one. It owns entity identity, reverse indexes, and the mutation -commands that preserve invariants. - -The class already exists (`lib/cdd/database.rb`, ~620 lines). This plan -locks down its public API and enumerates the indexes and mutations -required by ParcelMaker parity. - -## Scope - -- Public API surface of `Cdd::Database`. -- Idempotent `finalize!` (build reverse indexes). -- Mutations: `add_entity`, `merge`, `rename_entity`, `remove_entity`, - `drop_dictionary`, `apply_change_request`. -- Reverse indexes (single-owner, rebuilt on finalize). -- Semantic equality (`semantically_equal?`). - -Not in scope: entities themselves (plan 03), format readers/writers -(plans 06–08), validation (plan 10). - -## Approach - -### Construction and identity - -```ruby -db = Cdd::Database.new -db.add_entity(klass) -db.add_entity(property) -db.finalize! -db.finalize! # idempotent — second call is a no-op -``` - -`Database` does **not** own a `Project` sheet or `pcls_LOCAL` directly — -those live on `Cdd::Parcel::Workbook` (plan 06). The Database is the -entity-graph view; the Workbook is the on-disk-shape view. - -### Public API — read - -| Method | Returns | -|--------|---------| -| `entities` | `Array` | -| `classes` / `properties` / `units` / `value_lists` / `value_terms` / `relations` / `view_controls` | filtered | -| `find(irdi)` | `Cdd::Entity` or nil | -| `find_by_code(code, type: nil)` | `Cdd::Entity` or nil | -| `find_by_meta_class(meta_class_irdi)` | `Array` | -| `root_classes` | classes with no parent | -| `children_of(klass)` | `Array` | -| `subclasses_of(klass)` | recursive descendants | -| `instances_of(klass)` | classes whose `is_case_of` includes `klass` | -| `properties_of(klass)` | declared + imported properties | -| `effective_properties_of(klass)` | via `Cdd::EffectiveProperties` (walks hierarchy + is_case_of) | -| `value_list_of(property)` | resolves the value list for ENUM properties | -| `relations_for(domain: nil, codomain: nil)` | filtered | -| `composition_tree(klass)` | via `Cdd::CompositionTree` | -| `class_tree` / `relation_tree` | tree walkers | -| `semantically_equal?(other)` | bool — value equality, ignoring order | - -### Public API — mutate - -```ruby -db.add_entity(entity) # idempotent on IRDI; updates indexes if finalized -db.merge(other_db) # union of entities; later wins on conflict -db.rename_entity(old_code, new_code) # rewrites every back-reference -db.remove_entity(irdi) # removes + cleans back-refs -db.drop_dictionary(dict_id) # removes entities belonging to dict -db.apply_change_request(request) # upserts + removals as a single transaction -db.finalize! # (re)build reverse indexes; idempotent -``` - -Mutation rules: -- Mutations before `finalize!` are allowed; mutations after require - updating indexes incrementally or calling `finalize!` again. -- Mutations preserve the frozen-entity invariant: replacing an entity - means constructing a new one and updating its back-references; never - mutate the old instance. -- All mutations return `self` for chaining. - -### Reverse indexes - -Built once on `finalize!`. Each is a `Hash` keyed by IRDI: - -- `by_irdi` — entity lookup. -- `by_code` — short-code lookup (scoped by entity type). -- `by_type` — `Hash{ Symbol => Array }`. -- `children_of` — `Hash{ class_irdi => Array }`. -- `subclasses_of` — memoized recursive descent. -- `declared_properties_by_class` — `Hash{ class_irdi => Array }`. -- `properties_by_value_list` — `Hash{ vl_irdi => Array }`. -- `properties_by_unit` — `Hash{ unit_irdi => Array }`. -- `relations_for_class_domain` — `Hash{ class_irdi => Array }`. -- `relations_for_class_codomain` — same, codomain. -- `instances_of` — `Hash{ klass_irdi => Array }` (reverse of - `is_case_of`). -- `controlling_properties` — `Hash{ property_irdi => Array }` - (reverse of `condition`). - -A single `build_reverse_index(database) { |entity| ... }` helper generates -all of them (DRY). Already in the browser bundle per the project -CLAUDE.md — port the pattern. - -### EffectiveProperties (lib/cdd/effective_properties.rb) - -Walks a `Klass`'s ancestor chain (via `parent_irdi`) **and** every -`is_case_of` target, collecting applicable + imported properties. Used by -Parcel export to populate the "effective" sheet for a class. Cycle-safe — -tracks visited IRDIs. - -### Mutation details - -#### `add_entity(entity)` - -- If `by_irdi[entity.irdi]` exists and is `!= entity`, replace and - re-sync indexes. -- If `entity.type == :class` and `entity.parent_irdi` is set, register - in `children_of[parent_irdi]`. -- Sets `entity.database = self` (transient back-pointer). -- Returns `self`. - -#### `merge(other_db)` - -- Iterates `other_db.entities`; calls `add_entity` on each. -- On IRDI conflict, the `other_db` version wins (callers should pass the - higher-priority database second). -- Re-finalizes if both sides were finalized. - -#### `rename_entity(old_code, new_code)` - -- Finds the entity by `old_code`. -- Constructs a new entity (same class) with the code property updated. -- Walks every back-reference (`parent_irdi`, `is_case_of_irdis`, - `applicable_property_irdis`, `imported_property_irdis`, - `sub_class_selection_irdis`, `definition_class_irdi`, `unit_irdi`, - `alternative_unit_irdis`, condition references, relation domain/codomain, - view-control controlled/shown) and rewrites the old IRDI to the new. -- Atomic: collects all rewrites; if any fails, rolls back. - -This is ParcelMaker feature F27. - -#### `apply_change_request(request)` - -`request` is a struct of `upserts: Array` and `removals: -Array`. Applies upserts and removals as a single transaction. -On any error, no mutation is committed. Used by the import pipeline -(`Cdd::Database.load_change_request`). - -### Semantic equality - -`db1.semantically_equal?(db2)` returns true iff: -- Same set of IRDIs. -- For each IRDI, same meta-class, same code, same raw properties (after - canonicalizing set ordering). -- Order of declarations is **not** significant. - -Used by every round-trip spec in plan 12. - -## Acceptance criteria - -- [x] `Database.new.add_entity(e).find(e.irdi) == e`. -- [x] `Database#finalize!` idempotent (guarded by `@finalized` flag). -- [x] `db.merge(other)` preserves entities from both, with `other` - winning on conflict. -- [x] `rename_entity` rewrites every back-reference type listed above. - Spec covers each one. -- [x] `apply_change_request` is transactional. -- [x] `effective_properties_of` walks `is_case_of` and the superclass - chain; cycle-safe. -- [x] `semantically_equal?` true after parse → serialize → parse on the - OceanRunner fixture. -- [x] `spec/database_spec.rb`, `spec/database_*_spec.rb`, - `spec/effective_properties_spec.rb` green. - -## Dependencies - -- **Plan 03** — entity classes. -- **Plan 04** — IRDI / property IDs / meta-classes. -- Blocks **plan 06** (Parcel readers call `add_entity`), **plan 07** - (CDDAL Builder calls `add_entity`), **plan 10** (validator walks), - **plan 12** (round-trip uses `semantically_equal?`). - -## Open questions - -- **Q1.** Should `Database` be thread-safe? **Recommendation:** no — the - gem is single-threaded by design; concurrent access is the caller's - responsibility. -- **Q2.** Should `apply_change_request` support partial success - (`dry_run:` flag returning a preview)? **Recommendation:** yes — - needed for the editor's "preview import" UX in a future phase. diff --git a/TODO.impl/06-parcel-format.md b/TODO.impl/06-parcel-format.md deleted file mode 100644 index 3fc8517..0000000 --- a/TODO.impl/06-parcel-format.md +++ /dev/null @@ -1,254 +0,0 @@ -# Plan 06 — Parcel format (Excel + CSV) full parity - -## Why - -The Parcel Excel format is IEC 62656-1's canonical exchange format. The -`opencdd` gem must read every Parcel shape ParcelMaker 5.2.1 emits and -write xlsx that re-reads to a semantically equal Database. The format has -three on-disk variants plus CSV; the gem must handle all four. - -This plan delivers ParcelMaker features F02, F03, F04, F05, F11, F12, F13, -F15, F16, F18, F21, F23, F26, F27, F29 (per -`cdd-data/specs/parcelmaker/01_features.md`). The interactive features -(F06, F07.x, F08–F10, F17) are partly Ruby (predicates) and partly editor -(JS); this plan covers the Ruby predicates. - -## Scope - -- `Cdd::Parcel::Metadata` — `Project` + `pcls_LOCAL` sheet content. -- `Cdd::Parcel::SheetSchema` — the 12-row header layout, default columns - per meta-class. -- `Cdd::Parcel::Sheet` — one sheet's parsed model. -- `Cdd::Parcel::Workbook` — collection of sheets + reserved sheets. -- `Cdd::Parcel::WorkbookReader` — `.xlsx` reader (ParcelMaker default). -- `Cdd::Parcel::FlatDirReader` — flat 6-file `.xls` directory (legacy). -- `Cdd::Parcel::ShardedDirReader` — per-class sharded `.xls` subdirs - (harvester output). -- `Cdd::Parcel::Writer` — `.xlsx` writer (caxlsx). -- `Cdd::Parcel::CsvWriter` / `CsvReader`. -- `Cdd::Parcel::Sheet.scaffold(meta_class_irdi:, parcel_id:)` — F03. -- `Cdd::Parcel::ScrapeVerifier` — diff scraped data against authoritative - class lists (used by `harvest/`). - -## Approach - -### Metadata (lib/cdd/parcel/metadata.rb) - -Represents the contents of the `Project` sheet: - -```ruby -Cdd::Parcel::Metadata.new( - project_id: "IEC62683", - parcel_id: "0112/2///62683_1", - source_language: "en", - translation_languages: %w[fr ja], -) -``` - -Plus the `pcls_LOCAL` rows: array of `(code, name)` pairs. - -### SheetSchema (lib/cdd/parcel/sheet_schema.rb) - -For each meta-class, the default set of property columns and their -position in the schema header. Used by `Sheet.scaffold` and by the -writer when emitting a new sheet. - -Default columns per meta-class come from the ParcelMaker manual §6 and -`cdd-data/specs/parcelmaker/02_workbook_format.md` §"Schema header". -Example for CLASS (MDC_C002): - -``` -MDC_P001_5 (code) MAND -MDC_P002_1 (version) OPT -MDC_P004_1 (pref name) MAND -MDC_P005 (definition) MAND -MDC_P007_1 (note) OPT -MDC_P010 (superclass) OPT -MDC_P011 (class_type) MAND -MDC_P013 (is_case_of) OPT -MDC_P014 (applicable) OPT -MDC_P015 (applicable types) OPT -MDC_P090 (imported) OPT -MDC_P094 (applicable docs) OPT -MDC_P097 (requirement) OPT -``` - -These defaults are SSOT data in `SheetSchema`; both Ruby and the TS port -read from the same spec. - -### Sheet (lib/cdd/parcel/sheet.rb) - -In-memory model of one Parcel sheet: - -```ruby -sheet.meta_class_irdi # MDC_C002 -sheet.parcel_id # "IEC62683" -sheet.source_language # "en" -sheet.translation_languages -sheet.header_rows # 5 class-header rows -sheet.schema_header_rows # rows 6-12 with property ID + datatype + ... -sheet.data_rows # array of DataRow -sheet.comment_rows # rows whose col A starts with "#" -sheet.hidden_header_rows # F21 visibility flags -sheet.find_property_column(property_id) # column index, matched by row-6 ID -``` - -`sheet.scaffold(meta_class_irdi:, parcel_id:)` produces an empty Sheet -with the default schema header from SheetSchema and one empty data row. -This is F03. - -### Workbook (lib/cdd/parcel/workbook.rb) - -```ruby -wb.metadata # Project + pcls_LOCAL -wb.sheets # Array, including reserved -wb.sheetmap # parsed sheetmap sheet -wb.find_sheet(type:) # by sheet type -wb.add_sheet(sheet) -wb.dup_sheet(name) # F16 -wb.hidden_header_rows # F21 -``` - -`Workbook.scaffold(parcel_id:, languages:, meta_classes:)` — F02 — -creates a Workbook with `Project`, `sheetmap`, `pcls_LOCAL`, and one -empty Sheet per requested meta-class. - -### WorkbookReader (lib/cdd/parcel/workbook_reader.rb) - -Reads `.xlsx` via `caxlsx` (or `roo` for older builds; current gemspec -has both — standardize on `caxlsx` only and drop `roo` dependency if -possible). - -```ruby -db = Cdd::Database.new -Cdd::Parcel::WorkbookReader.new(path).load_into(db) -``` - -Algorithm: -1. Open workbook; enumerate sheets. -2. Read `Project` → `Metadata`. -3. Read `sheetmap` → register each sheet under its meta-class and - contentno. -4. For each per-dictionary sheet: locate rows by label in column A - (NOT by absolute index — ParcelMaker versions drift). Build a Sheet. -5. For each data row: skip if comment (col A startswith `#`). Construct - entity of the appropriate type, populate `properties` from the - schema-header columns. -6. Call `db.add_entity(entity)` per row. -7. Return populated db (caller calls `db.finalize!`). - -### FlatDirReader (lib/cdd/parcel/flat_dir_reader.rb) - -For the legacy 6-file layout: `export_{CLASS,PROPERTY,RELATION,UNIT, -VALUELIST,VALUETERMS}_.xls`. Each file is one sheet of one meta-class -type; reader concatenates them. - -```ruby -Cdd::Parcel::FlatDirReader.new(dir).load_into(db) -``` - -Uses `spreadsheet` gem for `.xls` (BIFF8). - -### ShardedDirReader (lib/cdd/parcel/sharded_dir_reader.rb) - -For the harvester output: one subdir per class, each containing its -`.xls` files. Reader walks the directory tree, applies FlatDirReader -per subdir, merges. - -### Writer (lib/cdd/parcel/writer.rb) - -```ruby -Cdd::Parcel::Writer.new(path).write(db, metadata:, languages:) -``` - -Inverse of WorkbookReader: -1. Build a Workbook by iterating entities grouped by meta-class. -2. For each group, scaffold a Sheet; fill data rows from entities. -3. Materialize `sheetmap`, `Project`, `pcls_LOCAL` from metadata. -4. Serialize to `.xlsx` via `caxlsx`. - -Normalization on write: -- Set-of-refs values normalized to brace form `{a,b,c}` (per spec R09). -- ParcelMaker-variant property IDs normalized back from canonical - (`MDC_P004` → `MDC_P004_1`) so ParcelMaker itself can re-read. -- Codes left as short form unless an entity has no supplier context. - -### CsvWriter / CsvReader - -CSV is the same 12-row header + data rows. Defaults: UTF-8, comma, -double-quote, no BOM. Multilingual cells use the `{(name,lang),...}` -literal form verbatim. - -### Aggregate / split / Selector - -Already present in `lib/cdd/parcel.rb`. Lock API: - -```ruby -Cdd::Parcel.aggregate(*paths) # → Database -Cdd::Parcel.split(database, by: :entity_type | :root_class | :each_class | Proc) -Cdd::Parcel::Selector.new(database) # entity selection w/ class closure -``` - -These cover F12 (export dictionary to separate files) when combined with -`Writer`. - -### Scrape verifier - -`Cdd::Parcel::ScrapeVerifier` diffs a scraped Database against an -authoritative class list (e.g. `harvest/verify/iec-*.txt`). Used by the -harvest pipeline, not directly by ParcelMaker features — keep, don't -extend. - -## Parcel-specific gotchas - -- **Row drift.** Schema-header row numbers vary between ParcelMaker - versions. Match by label in column A, never by absolute index. -- **Property ID normalization.** Apply `PARCEL_VARIANT_TO_CANONICAL` on - read; reverse on write. -- **Comment rows.** First cell startswith `#`. Skip for operations, - preserve through round-trip. -- **Code uniqueness.** Within a sheet, codes are `KEY` (R02). Reader - emits a warning on duplicates; doesn't drop the row. -- **Header visibility (F21).** `Workbook#hidden_header_rows` is a Set - of row labels; reader skips hidden, writer writes them but flags with - Excel's row-hidden attribute. -- **Multilingual cells.** `TRANSLATABLE_STRING_TYPE` cells carry - `{(text,lang),...}` literal. Reader parses into `Languages` value - object; writer emits literal form. - -## Acceptance criteria - -- [x] Reader consumes `reference-docs/export_CDD_IEC62683 in ParcelMaker format.xlsx` - without error; entity counts match the source dictionary. -- [x] Reader consumes `reference-docs/export_CDD_IEC62368 in EXCEL format/` - (flat-dir .xls) without error. -- [x] Reader consumes `reference-docs/parcelmaker/(ParcelMaker)example_nut.xlsx`. -- [x] Writer round-trips: read → write → read yields semantically equal - Database (plan 12). -- [x] `Sheet.scaffold(MDC_C002, "IEC62683")` produces the expected - default-columns list from `cdd-data/specs/parcelmaker/02_workbook_format.md`. -- [x] CSV writer emits UTF-8 with comma delimiter; re-reads via - CsvReader to semantically equal Database. -- [x] `Parcel.split(by: :each_class)` produces N partitions, one per - class, each self-sufficient (declared properties + their value - lists + their units). -- [x] All specs in `spec/parcel/` green. - -## Dependencies - -- **Plan 03** — entity classes. -- **Plan 04** — property IDs / meta-classes. -- **Plan 05** — Database. -- Blocks **plan 12** (round-trip testing depends on read+write). - -## Open questions - -- **Q1.** Drop `roo` dependency in favor of `caxlsx` for `.xlsx` only? - **Recommendation:** yes, but confirm `.xls` (BIFF8) read via - `spreadsheet` gem covers the legacy fixtures first. -- **Q2.** For `VALUELIST` / `VALUETERMS` legacy files (which are paired), - should the FlatDirReader infer the join, or require both files? Both - files exist in fixtures — require both, raise on missing pair. -- **Q3.** Should we support writing `.xls` (BIFF8)? **Recommendation:** - no — ParcelMaker 5.2.1 itself no longer writes `.xls`; only reads. - Document as one-way. diff --git a/TODO.impl/07-cddal-grammar.md b/TODO.impl/07-cddal-grammar.md deleted file mode 100644 index fbafa5d..0000000 --- a/TODO.impl/07-cddal-grammar.md +++ /dev/null @@ -1,187 +0,0 @@ -# Plan 07 — CDDAL grammar (lexer + parser + AST + builder) - -## Why - -CDDAL is the canonical plain-text format for CDD content. It must parse -the existing fixtures (`oceanrunner.cddal`, `202003-kagoshima-iec-def-sample.cddal`) -and the wider corpus produced by the harvester and by round-tripping -Parcel xlsx. The grammar must also support the module/import extensions -in plan 09. - -The grammar already exists (`lib/cdd/cddal/{lexer,parser,ast,builder, -generated_parser}.rb` + `cddal.y`). This plan locks the grammar against -the spec draft, ensures the AST nodes are sufficient for the builder and -serializer, and prepares the grammar for plan 09's import extensions. - -## Scope - -- `Cdd::Cddal::Lexer` — tokenizer. -- `Cdd::Cddal::Parser` + `GeneratedParser` — racc-driven parser. -- `lib/cdd/cddal/cddal.y` — the yacc grammar (source of truth). -- `Cdd::Cddal::AST::*` — node types. -- `Cdd::Cddal::Builder` — AST → Database. - -Not in scope: the serializer (plan 08), the module/import system -(plan 09), the CDDAL spec document (plan 13). - -## Approach - -### Tokens - -From `cddal.y`: - -``` -META_CLASS INSTANCE ALIAS IMPORT TRUE FALSE NULL -LBRACE RBRACE COLON COMMA LANGLE RANGLE DOT LPAREN RPAREN -EQEQ NEQ -STRING NUMBER DATE IRDI IDENT -``` - -Reserved-keyword set: `meta-class`, `instance`, `alias`, `import`, -`true`, `false`, `null`. - -Identifier rules: -- `IDENT` = `[A-Za-z_][A-Za-z0-9_]*` (letters, digits, underscores; not - leading digit). -- `IRDI` = pattern-recognized (contains `/` or `#`). -- `DATE` = `YYYY-MM-DD` (ISO 8601 calendar date). -- `STRING` = JSON-style double-quoted. -- `NUMBER` = JSON number grammar. -- Comments: `#` to end of line. (Conflict with IRDI `#` separator is - resolved by lexer context — `#` inside a valid IRDI pattern is part - of the IRDI; `#` elsewhere starts a comment.) - -### Grammar summary - -``` -document := declaration* -declaration := meta_class_decl | instance_decl | alias_decl | import_decl -meta_class_decl := "meta-class" ident_or_irdi "{" prop_id_list "}" -instance_decl := "instance" [IDENT] "<" ident_or_irdi "{" assignment* "}" - | "instance" ident_or_irdi "{" assignment* "}" -alias_decl := "alias" IDENT ":" ident_or_irdi -import_decl := "import" STRING [import_modifier] # extended by plan 09 -assignment := IDENT ["." IDENT] ":" value -value := literal | identifier_ref | set | tuple - | class_reference | condition | dotted_ref -literal := STRING | NUMBER | DATE | "true" | "false" | "null" -identifier_ref := IDENT | IRDI -set := "{" value* "}" -tuple := "(" value* ")" -class_reference := IDENT "(" identifier_ref ")" -condition := identifier_ref ("==" | "!=") (literal | set) -dotted_ref := IDENT "." IDENT -``` - -Plan 09 extends `import_decl` with `as IDENT` (qualified) and -`"{" IDENT* "}"` (selective). Plan 07 keeps the basic form. - -### AST nodes (lib/cdd/cddal/ast.rb) - -| Node | Fields | -|------|--------| -| `Document` | `declarations: Array` | -| `MetaClassDecl` | `irdi:, property_identifiers:, line:` | -| `InstanceDecl` | `name:, meta_class_ref:, assignments:, line:` | -| `AliasDecl` | `alias_name:, property_id:, line:` | -| `ImportDecl` | `url:, modifiers:, line:` (modifiers extended by plan 09) | -| `PropertyAssignment` | `identifier:, language_tag:, value:, line:` | -| `Literal` | `kind:, raw:` (`:string`, `:number`, `:date`, `:boolean`, `:null`) | -| `IdentifierRef` | `name:` | -| `Set` | `members: Array` | -| `Tuple` | `members: Array` | -| `ClassReference` | `type_name:, argument:` | -| `Condition` | `lhs:, operator:, rhs:` | -| `DottedRef` | `receiver:, name:` | - -All nodes are value objects: constructors take keyword args, instances -are frozen, equality is structural. - -### Builder (lib/cdd/cddal/builder.rb) - -Pipeline: -1. Walk `Document#declarations` in source order. -2. Apply `alias` declarations → seed the Database's `AliasTable`. -3. Apply `meta-class` declarations → register allowed-property sets - (merged by union for repeated IRDIs). -4. Apply `import` declarations → see plan 09. For now: skip HTTPS URLs - with a warning, load local files by path, dedup by canonical path. -5. Apply `instance` declarations in **two passes**: - - Pass 1: construct entities, populate `properties`. Symbolic names - are recorded in a symbol table. - - Pass 2: resolve cross-references (superclass, is_case_of, - applicable_properties, etc.) via the symbol table. -6. Return the populated Database. - -Resolution order for an `IdentifierRef` (per spec §"Resolution algorithm"): - -1. Full IRDI → direct lookup. -2. Symbolic name → symbol table. -3. Code → `Database#find_by_code`. -4. Otherwise: emit `Cdd::Cddal::ResolutionError` (configurable: strict - vs. warn). - -Forward references are allowed because pass 2 runs after pass 1. - -### Source location tracking - -Every AST node carries `line:` (1-indexed) and the Lexer tracks the -source file path. The Builder attaches a `source_location:` to each -constructed Entity (`Struct.new(:file, :line)`). Used for diagnostics -in plan 10's validator and for the import graph in plan 09. - -### Re-entrant parsing - -`Cdd::Cddal.parse(source, database: nil)` accepts an existing Database -to merge into. This supports plan 09's import flow: parse each imported -file into the same Database. - -### racc regeneration - -``` -bundle exec racc lib/cdd/cddal/cddal.y -o lib/cdd/cddal/generated_parser.rb -``` - -Rake task `cddal:regen` (plan 14). The generated file is checked in so -test environments don't need racc at install time. - -## Acceptance criteria - -- [x] `Cdd::Cddal.parse_file("reference-docs/examples/oceanrunner.cddal")` - produces 20 classes, 19 properties, 1 value list. -- [x] `Cdd::Cddal.parse_file("reference-docs/202003-kagoshima-iec-def-sample.cddal")` - produces the entities the Kagoshima sample encodes. -- [x] Builder resolves symbolic names in `superclass`, - `applicable_properties`, `is_case_of`. -- [x] Builder resolves `CLASS_REFERENCE(...)` and `ENUM_*_TYPE(...)` - arguments. -- [x] Builder parses `condition: a == b` and - `condition: a == { b, c }` into a `Cdd::Condition`. -- [x] Forward references work: an instance may reference another - declared later in the file. -- [x] Comments (`#` to EOL) are skipped. -- [x] `rake cddal:regen` regenerates `generated_parser.rb` idempotently. -- [x] `spec/cddal_spec.rb` green. - -## Dependencies - -- **Plan 03** — entities. -- **Plan 04** — IRDI, PropertyIds (the builder normalizes aliases). -- **Plan 05** — Database (builder calls `add_entity`). -- Blocks **plan 08** (serializer is the inverse), **plan 09** - (module system extends the grammar), **plan 12** (round-trip), - **plan 13** (spec describes the grammar). - -## Open questions - -- **Q1.** Should the grammar allow `instance` declarations without a - meta-class (anonymous)? The current grammar accepts both forms; the - Kagoshima sample uses no `<` form. **Recommendation:** yes — keep - both; the bare form means "use the document's default meta-class for - this block" (typically `MDC_C002`). -- **Q2.** Should we support single-quoted strings? **Recommendation:** - no — JSON double-quote is enough; single quotes add ambiguity with - apostrophes in definitions. -- **Q3.** Should conditions support `&&` and `||`? The current grammar - only has `==` and `!=`. ParcelMaker only emits single-equality - predicates. **Recommendation:** defer until a real fixture needs it. diff --git a/TODO.impl/08-cddal-serializer.md b/TODO.impl/08-cddal-serializer.md deleted file mode 100644 index 7893cca..0000000 --- a/TODO.impl/08-cddal-serializer.md +++ /dev/null @@ -1,183 +0,0 @@ -# Plan 08 — CDDAL serializer - -## Why - -The serializer is the inverse of plan 07's parser. It must emit CDDAL -that re-parses to a semantically equal Database (round-trip stability). -It must be deterministic (same input → same output byte-for-byte) so -CDDAL diffs are meaningful in version control. - -Already exists at `lib/cdd/cddal/serializer.rb` (~210 lines). This plan -locks down its emission rules and ensures determinism + multilingual -correctness. - -## Scope - -- `Cdd::Cddal::Serializer` — `Database` → CDDAL text. -- `Cdd::Cddal.serialize(db)` / `serialize_to_file(db, path)` — top-level - helpers. -- Multilingual emission via `Cdd::Languages`. -- Stable ordering. -- Round-trip invariants (tested in plan 12). - -Not in scope: the parser (plan 07), the module system (plan 09), -other format exporters (plan 11). - -## Approach - -### Output structure - -A serialized CDDAL document has four sections, in this order: - -1. **Header comment** — optional provenance (`# generated by opencdd-X.Y.Z`). - Off by default; enabled via `serializer.emit_header = true`. -2. **Aliases** — every alias currently registered in the Database's - `AliasTable`, sorted alphabetically. Includes built-ins. -3. **Meta-class declarations** — `meta-class MDC_C### { ... }` for each - meta-class that has instances in this Database. -4. **Instance declarations** — ordered per the rules below. - -### Stable ordering - -Instance emission order, in priority: - -1. **Depth-first traversal of the superclass tree** starting from each - root class (`db.root_classes`). Children sorted by code. -2. **Properties** grouped by their `definition_class` (i.e., emitted - right after the class that declares them). Within a group, sorted - by code. -3. **Value lists** sorted by code. -4. **Value terms** sorted by `(value_list_code, term_code)`. -5. **Units** sorted by code. -6. **Relations** sorted by `(domain_code, relation_type, codomain_code)`. -7. **View controls** sorted by code. - -Determinism rule: any iteration over a `Hash` must use `sort` or a -custom comparator. Never `each` over a hash whose insertion order is -not the desired output order. - -### Multilingual emission - -For each translatable property (`preferred_name`, `definition`, `note`, -`remark`, `short_name`, etc.), emit one assignment per language: - -```cddal -preferred_name.en: "Vehicle" -preferred_name.fr: "Véhicule" -preferred_name.ja: "車両" -``` - -Languages come from `Database#languages` (a `Cdd::Languages` value object -populated from the `Project` sheet on Parcel read, or from the document's -explicit language declarations on CDDAL read). - -If a value is missing for a language, omit the line (do not emit `null`). - -### Value emission - -| Value type | Emission | -|------------|----------| -| String literal | JSON double-quoted, JSON escapes | -| Number | Raw (`4200`, `9.5`) | -| Date | ISO 8601 `YYYY-MM-DD` | -| Boolean | `true` / `false` | -| Null | `null` | -| Symbolic name | Bare identifier (resolved from `Database` symbol table) | -| Code | Bare identifier (resolved from entity code) | -| IRDI | Bare (only when symbolic name and code are unavailable) | -| Set | `{ a, b, c }` (space after comma; members sorted by code) | -| Tuple | `( a, b )` | -| Class reference | `CLASS_REFERENCE(EngineType)` | -| Condition | `lhs == rhs` or `lhs == { a, b }` | - -### Reference resolution on emit - -For every `IdentifierRef` value (in `superclass`, `is_case_of`, -`applicable_properties`, `condition`, etc.), prefer: -1. Symbolic name (if the referenced entity has one in this Database). -2. Code (always available for entities that have a code). -3. Full IRDI (last resort, e.g., for references to entities in an - imported file that wasn't materialized). - -Symbolic names: opt-in per Database. By default, the serializer emits -codes. Pass `serializer.use_symbolic_names = true` to prefer names; -this is what the OceanRunner fixture uses. - -### Comment rows - -Parcel sheet comment rows (first cell startswith `#`) are preserved on -Parcel round-trip but are NOT emitted in CDDAL output — CDDAL has its -own `#` comment syntax and the comment rows are a Parcel-specific -concept. They live in `Entity#source_comment` if needed for diagnostics. - -### Non-ASCII escaping - -Strings are emitted as UTF-8 with the following JSON escapes: -- `\\`, `\"`, `\b`, `\f`, `\n`, `\r`, `\t`. -- `\uXXXX` for control characters (`U+0000`–`U+001F`). -- Other bytes (including non-ASCII UTF-8) are emitted **literally** — - CDDAL is UTF-8 by default; escaping `é` as `é` would hurt - readability. - -This matches the existing `serializer.rb` behavior; ensure no regression. - -### Backslash quoting - -Backslashes in raw Parcel values (e.g., Windows paths) are emitted -escaped. Test fixture: any unit symbol containing `\`. - -### Round-trip invariants - -These are tested by plan 12: - -- `parse(serialize(db))` semantically equal to `db`. -- `serialize(parse(text))` byte-equal to `text` for fixtures that - follow the canonical formatting (OceanRunner does; Kagoshima's bare - form may not — that's OK, semantic equality is the bar). - -### Public API - -```ruby -serializer = Cdd::Cddal::Serializer.new(db) -serializer.use_symbolic_names = true -serializer.emit_header = true -serializer.to_cddal # → String - -# Convenience: -Cdd::Cddal.serialize(db) -Cdd::Cddal.serialize_to_file(db, path) -``` - -## Acceptance criteria - -- [x] `serialize(parse_file("oceanrunner.cddal"))` re-parses to a - semantically equal Database. -- [x] Output is deterministic: two runs produce identical bytes. -- [x] Multilingual properties emit one line per language. -- [x] Non-ASCII UTF-8 emitted literally; control chars escaped. -- [x] References prefer symbolic name → code → IRDI. -- [x] No `to_h` / `from_h` / `serialize` / `deserialize` instance method - on any model class — only on the Serializer service. -- [x] `spec/cddal_spec.rb` round-trip examples green. - -## Dependencies - -- **Plan 03** — entities (serializer reads their typed accessors). -- **Plan 04** — alias table. -- **Plan 05** — Database (entity enumeration, ordering). -- **Plan 07** — parser (round-trip target). -- Blocks **plan 12** (round-trip), **plan 13** (spec describes emission). - -## Open questions - -- **Q1.** Should the serializer emit `meta-class` blocks for every - meta-class, or only those that deviate from the built-in defaults? - **Recommendation:** only those with instances — keeps output focused. - Built-in defaults are documented in the spec. -- **Q2.** Should we emit a `# source: ` comment when - round-tripping from Parcel? **Recommendation:** yes, opt-in via - `emit_source_attribution = true`. Useful for provenance. -- **Q3.** When a property value is `nil`, should we emit `prop: null` - or omit the assignment? **Recommendation:** omit. Empty cells in - Parcel become no-assignment in CDDAL; explicit `null` is rare and - should be opt-in. diff --git a/TODO.impl/09-cddal-modules.md b/TODO.impl/09-cddal-modules.md deleted file mode 100644 index bad5b20..0000000 --- a/TODO.impl/09-cddal-modules.md +++ /dev/null @@ -1,308 +0,0 @@ -# Plan 09 — CDDAL module / import system (split-file format) - -## Why - -The user's explicit requirement #4: CDDAL must support a split-file format -where one `.cddal` file can cross-reference entities declared in another -`.cddal` file by IRDI, with the **file location** resolved from a local -path **or** a URL. The current implementation has a stub `import STRING` -that skips HTTPS and only opens local files; no cycle detection, no -qualified imports, no selective imports, no source-location tracking, no -pluggable fetcher. - -This plan delivers a module/import system informed by best practices from -similarly-syntaxed languages. - -## Inspirations (and what we adopt from each) - -| Language | What we adopt | What we reject | -|----------|---------------|----------------| -| **Python** | `from "x" import { y, z }` selective form; `as` rename | Implicit `sys.path` magic; module-level `__init__.py` | -| **Rust** | Explicit dependency graph; cycles are hard errors; visibility (`pub`) is opt-in | Lifetime / borrow syntax; trait system | -| **Go** | URL imports work natively; module cache; deterministic resolution | Mandatory `go.mod` for every file | -| **ES6** | `import { a, b } from "url"` brace syntax; named exports | Default exports; runtime dynamic import | -| **Terraform** | Pluggable source schemes (path, URL, registry) | Module input/output variables (CDDAL has no value-level modules) | -| **C/C++ #include** | (nothing) — textual-only inclusion is too fragile | Macro expansion; order-dependent compilation | - -Key design choices, derived: - -1. **Default import = textual inclusion of named declarations.** A bare - `import "path"` brings in every named declaration from the target as - if inlined. Symbolic names from imports are merged into the importing - document's symbol table. -2. **Selective and qualified imports are opt-in.** When name collisions - matter, the author writes `from "path" import { Foo, Bar }` or - `import "path" as alias` and qualifies references. -3. **URL imports work out of the box** with a pluggable fetcher. - `Net::HTTP` is the default; tests use an in-memory fetcher. -4. **The dependency graph is explicit and acyclic.** A cycle raises a - `Cdd::Cddal::ImportError` with the full cycle path. -5. **Source location is tracked end-to-end.** Every entity knows which - file (and line) it came from — including transitively-imported ones. -6. **Resolution is deterministic.** Same input + same fetcher cache = - same Database. Build tools can rely on this. - -## Scope - -- New grammar productions in `cddal.y` (extensions to `import_decl`). -- New AST node types in `lib/cdd/cddal/ast.rb`. -- Path / URL resolver: `Cdd::Cddal::Resolver`. -- Pluggable HTTP fetcher: `Cdd::Cddal::Fetcher` with `NetHttp` and - `InMemory` implementations. -- Dependency graph + cycle detection in the Builder. -- Source location tracking on every Entity. -- Selective and qualified import semantics in the symbol table. - -Not in scope: versioning of remote modules (open question Q1), cryptographic -integrity pins (open question Q2), private vs. public declaration modifiers -(open question Q3). - -## Approach - -### Grammar extension (lib/cdd/cddal/cddal.y) - -``` -import_decl - : IMPORT STRING { bare textual inclusion } - | IMPORT STRING AS IDENT { qualified: alias.Name } - | FROM STRING IMPORT LBRACE import_list RBRACE { selective } - ; - -import_list - : IDENT { result = [val[0]] } - | import_list COMMA IDENT { result = val[0] + [val[2]] } - ; -``` - -New tokens: `AS`, `FROM`. Both reserved as keywords. - -### Concrete examples - -```cddal -# (1) Bare — textual inclusion. Symbolic names from lib.cddal merge in. -import "https://cdd.opencdd.org/dictionaries/rec20-units.cddal" - -# (2) Qualified — names from units.cddal accessible as `units.Metre`. -import "../base/units.cddal" as units - -# (3) Selective — only the named declarations are pulled in. -from "../base/vehicles.cddal" import { Vehicle, Boat } - -# (4) Combined selective + qualified (allowed for ergonomics). -from "../base/vehicles.cddal" import { Vehicle as V, Boat } -``` - -Forms (1)–(3) cover the 90% case. Form (4) handles collisions. - -### Resolution rules - -`Resolver.new(base_path:, search_path:, fetcher:)` resolves a module -specifier: - -| Specifier | Resolution | -|-----------|------------| -| `./path.cddal` or `../path.cddal` | Relative to the importing file's directory. | -| `/abs/path.cddal` | Absolute filesystem path. | -| `path.cddal` (bare) | Searched in `search_path` (defaults to `["."]` and `$CDDAL_PATH` split on `:`). | -| `http://...` / `https://...` | Fetched via `fetcher`. Cached by URL in the current run. | -| `file:///abs/path.cddal` | File URI; resolves to absolute path. | -| anything else | `Cdd::Cddal::ImportError`. | - -The Resolver returns a canonical key (absolute path or normalized URL) -used for dedup: the same module imported twice loads only once. - -### Fetcher abstraction - -```ruby -module Cdd::Cddal::Fetcher - def fetch(url) → String # returns the file contents -end - -class Cdd::Cddal::Fetcher::NetHttp - def initialize(cache_dir: ENV.fetch("CDDAL_CACHE", Dir.tmpdir), offline: false) - def fetch(url) → String -end - -class Cdd::Cddal::Fetcher::InMemory - def initialize(map = {}) # { url => source } - def fetch(url) → String or raise -end -``` - -`offline: true` skips network and serves only from cache — used in CI -(plan 14). The InMemory fetcher is for tests. - -Caching: HTTP responses are cached to `cache_dir/.cddal` -with the URL and ETag in sidecar metadata. Re-fetch sends -`If-None-Match`; on 304 the cache is used. TTL: 24h default, -configurable. - -### Builder integration - -`Cdd::Cddal::Builder` accepts a `resolver:` and `fetcher:`. The build -pipeline becomes: - -1. Parse the root document into AST. -2. Walk declarations in source order. On `ImportDecl`: - - Resolve the URL via `resolver`. - - If canonical URL already in `@loaded_modules`, skip (dedup). - - Otherwise: fetch source, parse, recursively build into the - **same** Database with the same resolver and fetcher. - - Record `(canonical_url, parent_url)` edge in the dependency graph. -3. After all imports resolve, apply aliases, meta-classes, instances. -4. Cycle check: depth-first traversal of the dependency graph. On - cycle, raise `ImportError` with the full edge path. -5. Apply selective and qualified import semantics to the symbol table. - -### Symbol scoping - -- **Bare import**: every named declaration in the imported module - becomes accessible by its symbolic name in the importing document. - Conflicts (same name from two modules) raise `ResolutionError`. -- **Qualified import** (`import "x" as p`): names accessible as - `p.Name`. The unqualified name is not in scope (avoids accidental - collision). -- **Selective import** (`from "x" import { Foo }`): only `Foo` is - accessible; other names from the module are not loaded. -- **Selective + rename** (`from "x" import { Foo as F }`): only `F` - is accessible. -- **IRDIs are always global**. An entity reference by full IRDI - resolves regardless of import scope. This is the cross-file - cross-reference primitive. - -This mirrors Python's semantics closely, with the addition that IRDIs -act as a global fallback (Python has no equivalent). - -### Source location tracking - -Every Entity carries `source_location: Struct.new(:file, :line, :column)`. -For imported entities, `:file` is the canonical URL/path of the imported -module, not the importer. The validator (plan 10) and diagnostics use -this; round-trip via Parcel preserves the file the entity came from -even when serialized through a single .xlsx. - -### Idempotency and re-entrance - -`Cdd::Cddal.parse(source, database:, resolver:, fetcher:)` parses into -the given database. Combined with dedup, this means a module imported by -multiple parents is parsed exactly once, and its entities are added -exactly once. The Database's `add_entity` is itself idempotent on IRDI. - -### Worked example - -A two-file project: - -```cddal -# boats.cddal -meta-class MDC_C002 { code preferred_name superclass class_type applicable_properties } -instance Boat < MDC_C002 { - code: AAA010 - preferred_name.en: "Boat" - superclass: UNIVERSE - class_type: ITEM_CLASS -} -``` - -```cddal -# main.cddal -import "./boats.cddal" -instance OceanRunner30 < MDC_C002 { - code: BBB010 - superclass: Boat # cross-file reference by symbolic name -} -``` - -Building `main.cddal` yields a Database with two classes, where -`OceanRunner30.parent.code == "AAA010"`. The Boat entity's -`source_location.file` points to `boats.cddal`. - -### URL example - -```cddal -import "https://cdd.opencdd.org/dictionaries/iec61360-7.cddal" -``` - -Build behavior: -- If `fetcher.offline?`, raise `ImportError` (or warn and continue - if `strict: false`). -- Otherwise fetch via Net::HTTP; cache to `cache_dir`. -- Parse and merge into the Database. - -The opencdd.github.io static site (separate effort) will host the -canonical reference dictionaries at this URL. Until then, the fetcher -is exercised by the test suite against locally-served fixtures. - -## Public API - -```ruby -resolver = Cdd::Cddal::Resolver.new( - base_path: File.dirname(main_file), - search_path: [".", File.expand_path("~/cdd-libs")], -) -fetcher = Cdd::Cddal::Fetcher::NetHttp.new(offline: ENV["CI"] ? true : false) - -db = Cdd::Cddal.parse_file( - main_file, - resolver: resolver, - fetcher: fetcher, -) -``` - -For tests: - -```ruby -fetcher = Cdd::Cddal::Fetcher::InMemory.new( - "https://example.test/units.cddal" => File.read(fixture_path), -) -db = Cdd::Cddal.parse(source, fetcher: fetcher) -``` - -## Acceptance criteria - -- [x] Existing bare `import "path"` semantics preserved — no regression - on OceanRunner fixture (which has one HTTPS import that's currently - skipped). -- [x] `from "x" import { Foo }` pulls in only `Foo`. -- [x] `import "x" as p` makes `p.Foo` resolve and bare `Foo` not resolve. -- [x] `from "x" import { Foo as F }` renames. -- [x] URL imports work via Net::HTTP fetcher; cache survives across - builder instances in the same process. -- [x] Cycle detection raises `Cdd::Cddal::ImportError` with the cycle - path in the message. -- [x] Every Entity has a `source_location` with `file` and `line`. -- [x] `Cdd::Cddal::Fetcher::InMemory` allows specs to test imports - without network. -- [x] `strict: false` mode warns and continues instead of raising on - unresolved imports / missing modules. -- [x] Idempotent: building the same root file twice produces identical - Databases. -- [x] New spec file: `spec/cddal/modules_spec.rb`. - -## Dependencies - -- **Plan 07** — grammar and Builder; this plan extends both. -- **Plan 05** — Database (entity identity across files). -- **Plan 03** — Entity `source_location` field. -- Blocks **plan 13** (spec describes the module system). - -## Open questions - -- **Q1. Versioned modules.** Should `import "url" version "1.2"` pin a - specific release? **Recommendation:** defer — first land unversioned, - add versioning when a real consumer needs it. -- **Q2. Integrity pins.** Should `import "url" sha256 "..."` allow - callers to assert the fetched content matches a hash? Useful for - reproducibility. **Recommendation:** yes, follow-up. Same shape as - Nix's `fetchurl`. -- **Q3. Selective export modifiers.** Should declarations have - `pub`/`priv` visibility (Rust-style)? **Recommendation:** no — every - declaration is importable. Namespacing via selective import handles - encapsulation. -- **Q4. Registry-style modules.** Should there be a - `cdd://dictionaries/iec61360-7` scheme that resolves through a - registry (like npm or crates.io)? **Recommendation:** no — URLs are - fine. Registries add governance overhead. -- **Q5. Workspace files.** Should we support a `cddal.toml` workspace - manifest that declares module roots and search paths? Useful for - multi-file projects. **Recommendation:** defer — a single - `$CDDAL_PATH` env var covers the immediate need. diff --git a/TODO.impl/10-validation.md b/TODO.impl/10-validation.md deleted file mode 100644 index 04a0f53..0000000 --- a/TODO.impl/10-validation.md +++ /dev/null @@ -1,153 +0,0 @@ -# Plan 10 — Validator (rules R01–R15) - -## Why - -ParcelMaker's defining interactive feature is real-time cell validation -(F06): every keystroke runs the rules and the cell turns red on -violation. The Ruby gem must expose the same predicates so the OpenCDD -Editor can reuse them, and so CI builds can fail on invalid CDD content. - -`cdd-data/specs/parcelmaker/04_validation_rules.md` enumerates the rules -R01–R15. The Ruby gem already has `lib/cdd/validator.rb` plus a -per-rule class in `lib/cdd/validator/*.rb`. This plan locks the public -predicate API and confirms each rule's semantics match the spec. - -## Scope - -- `Cdd::Validator` — composite runner + module-level predicate methods. -- `Cdd::Validator::Rule` — abstract base. -- Per-rule classes (`R01IrdiSyntax` … `R15CompositionAcyclic`). -- `Cdd::ValidationError` struct. -- Reusable predicates exported for the editor. - -Not in scope: the UI rendering of errors (editor concern). - -## Approach - -### Public predicate API - -These are reused by the OpenCDD Editor's live validator. Module-level -methods on `Cdd::Validator`: - -```ruby -Cdd::Validator.irdi_well_formed?(value) # R01 -Cdd::Validator.code_unique?(database, code, type:) # R02 -Cdd::Validator.type_valid?(value, data_type) # R03 -Cdd::Validator.enum_member?(value, value_list) # R04 -Cdd::Validator.format_valid?(value, value_format) # R05 -Cdd::Validator.pattern_valid?(value, pattern) # R06 -Cdd::Validator.mandatory_present?(value) # R07 -Cdd::Validator.reference_exists?(value, database) # R08 -Cdd::Validator.set_well_formed?(value) # R09 -Cdd::Validator.synonym_set_well_formed?(value) # R10 -Cdd::Validator.condition_well_formed?(value) # R11 -Cdd::Validator.data_type_well_formed?(value) # R12 -Cdd::Validator.format_compatible?(data_type, value_format) # R13 -Cdd::Validator.class_hierarchy_acyclic?(database) # R14 -Cdd::Validator.composition_acyclic?(database) # R15 - -Cdd::Validator.cell_valid?(entity, property_id, database) # composite -Cdd::Validator.run(database) # → Array -``` - -### ValidationError - -```ruby -Cdd::ValidationError = Struct.new( - :entity_irdi, :property_id, :rule, :message, :source_location, keyword_init: true, -) -``` - -`source_location` is the `Struct(:file, :line)` from plan 09. Used to -render clickable error locations in any UI. - -### Rule classes (lib/cdd/validator/*.rb) - -| File | Rule | What it checks | -|------|------|----------------| -| `irdi_rule.rb` | R01 | IRDI / ICID syntax per `Cdd::IRDI.well_formed?`. | -| `uniqueness_rule.rb` | R02 | Code uniqueness within a sheet (`#REQUIREMENT = KEY`). | -| `data_type_rule.rb` | R03 | Value parses per its declared data type. | -| `enum_rule.rb` | R04 | Value is a member of the column's value list. | -| `format_rule.rb` | R05 | Value matches the IEC 61360 value-format token. | -| `pattern_rule.rb` | R06 | Value matches the column's PCRE pattern. | -| `mandatory_rule.rb` | R07 | Non-empty for `MAND` / `KEY` columns. | -| `reference_rule.rb` | R08 | Every referenced IRDI exists (in DB or imports). | -| `set_rule.rb` | R09 | `{a,b,...}` and `{(a,b),...}` well-formed. | -| `synonym_rule.rb` | R10 | Synonymous-name set well-formed. | -| `condition_rule.rb` | R11 | Condition well-formed (predicate or set of `(IRDI, value)`). | -| `type_rule.rb` | R12 | Data-type expression well-formed. | -| `format_compatible_rule.rb` (R13) | R13 | Value-format compatible with data type. | -| `hierarchy_rule.rb` | R14 | Class hierarchy acyclic (superclass chain). | -| `composition_rule.rb` (R15) | R15 | Composition hierarchy acyclic. | - -Each rule class implements: - -```ruby -class Cdd::Validator::Rule::R01IrdiSyntax < Cdd::Validator::Rule - def self.id; :R01; end - def self.applies_to?(entity, property_id, database); ...; end - def self.check(entity, property_id, value, database); ...; end # → ValidationError? -end -``` - -The `Runner` (lib/cdd/validator/runner.rb) iterates rules × entities × -properties and aggregates errors. - -### Composite runner - -```ruby -errors = Cdd::Validator.run(database) -errors.first.entity_irdi -errors.first.source_location.file -errors.first.source_location.line -``` - -Honors a `strict:` flag (default `true`). When false, missing references -in imported modules produce warnings instead of errors (per plan 09). - -### Live-validation hook - -```ruby -Cdd::Validator.cell_valid?(entity, property_id, database) -``` - -Returns `true` if all applicable rules pass for that single cell. Used -by the editor's keystroke-time validator. Walks only the rules declared -for the property's data type / value kind. - -### Spec coverage - -`spec/validator_spec.rb` plus per-rule spec files. Each rule has at -least three cases: a clear pass, a clear fail, and a tricky edge case -from the cdd-data spec. - -## Acceptance criteria - -- [x] Every predicate listed above is public on `Cdd::Validator` and - has at least one spec. -- [x] `Validator.run` returns zero errors on the ParcelMaker nuts - fixture and on the OceanRunner fixture. -- [x] On a deliberately-broken fixture (hand-crafted), `Validator.run` - returns the expected rule violations. -- [x] `cell_valid?` short-circuits on the first applicable rule's - failure; performance is O(1) per cell for typical schemas. -- [x] Source location flows through to the ValidationError so the - editor can navigate to the offending line. -- [x] All `spec/validator*_spec.rb` files green. - -## Dependencies - -- **Plan 03** — entities. -- **Plan 04** — IRDI / data types / value formats. -- **Plan 05** — Database (hierarchy / composition walkers). -- **Plan 09** — `source_location` for error reporting. - -## Open questions - -- **Q1.** Should `Validator.run` short-circuit on the first error per - entity, or collect all errors? **Recommendation:** collect all — the - UI renders them as a list and the user fixes them in one pass. -- **Q2.** Performance budget for `cell_valid?`? Used per keystroke in - the editor. **Recommendation:** target <1ms per call on a 100k-entity - Database. Memoize parsed data types / value formats. diff --git a/TODO.impl/11-auxiliary-exporters.md b/TODO.impl/11-auxiliary-exporters.md deleted file mode 100644 index d1e4e9b..0000000 --- a/TODO.impl/11-auxiliary-exporters.md +++ /dev/null @@ -1,148 +0,0 @@ -# Plan 11 — Auxiliary exporters (JSON / YAML / Mermaid) - -## Why - -The Parcel and CDDAL formats cover the canonical exchange surface. Three -auxiliary exporters ease human review and integration with external -tooling: - -- **JSON** — programmatic access for downstream scripts and the editor's - data pipeline. -- **YAML** — diff-friendly export for documentation and review. -- **Mermaid** — class-hierarchy diagrams for README files and PR reviews. - -All three already exist in `lib/cdd/exporters/*.rb`. This plan locks -their public API and ensures they share a common entity-payload helper. - -## Scope - -- `Cdd::Exporters::Json` — `Database` → JSON string / file. -- `Cdd::Exporters::Yaml` — `Database` → YAML string / file. -- `Cdd::Exporters::Mermaid` — `Database` → Mermaid class-diagram markdown. -- Shared `entity_payload` helper. -- `Cdd::Exporters` namespace file. - -Not in scope: Parcel writer (plan 06), CDDAL serializer (plan 08), -CSV writer (plan 06 — Parcel-specific). - -## Approach - -### Common payload shape - -All three exporters use the same per-entity payload, built by -`Cdd::Exporters.entity_payload(entity, database:)`: - -```ruby -{ - irdi: entity.irdi.to_s, - code: entity.code, - type: entity.type, # :class, :property, ... - meta_class_irdi: entity.meta_class_irdi.to_s, - source_location: { file: ..., line: ... }, - properties: { - "MDC_P001_5" => "AAA001", - "MDC_P004" => { "en" => "Vehicle", "fr" => "Véhicule" }, - "MDC_P010" => "UNIVERSE", - "MDC_P014" => ["AAAP001", "AAAP002"], - ... - }, -} -``` - -Multilingual values are emitted as `{lang => text}`. Set-of-refs values -are emitted as arrays. Scalar values as primitives. This shape is the -canonical "JSON view" of an entity. - -### JSON exporter (lib/cdd/exporters/json.rb) - -```ruby -Cdd::Exporters::Json.new(database).to_json(pretty: true) -Cdd::Exporters::Json.new(database).to_file(path, pretty: true) -``` - -Options: -- `pretty:` — default `true` (2-space indent). -- `compact:` — emits single-line JSON, no whitespace. For machine - consumers. -- `include_raw:` — if true, includes the raw `properties` hash - alongside typed fields. Default false. - -`compact` must **not** strip empty arrays (Ruby `Hash#compact` parity — -preserves schema). - -### YAML exporter (lib/cdd/exporters/yaml.rb) - -```ruby -Cdd::Exporters::Yaml.new(database).to_yaml -Cdd::Exporters::Yaml.new(database).to_file(path) -``` - -Uses Ruby's stdlib `Psych`. Output is deterministic: keys sorted, -entities emitted in canonical order (same as plan 08's CDDAL ordering — -superclass-tree DFS then by code). - -### Mermaid exporter (lib/cdd/exporters/mermaid.rb) - -```ruby -Cdd::Exporters::Mermaid.new(database).to_diagram -Cdd::Exporters::Mermaid.new(database).class_diagram(root: vehicle_irdi) -``` - -Emits a Mermaid `classDiagram` block. Each class becomes a Mermaid -class node; superclass relationships become `Klass <|-- Parent` arrows; -`is_case_of` becomes `..|>` (dashed) arrows. Properties listed inside -the class block (just the names, not values — keeps diagrams readable). - -Optional `root:` filter limits the diagram to a subtree — useful for -large dictionaries where the full diagram is unusable. - -### Namespace (lib/cdd/exporters.rb) - -```ruby -module Cdd::Exporters - autoload :Json, "cdd/exporters/json" - autoload :Yaml, "cdd/exporters/yaml" - autoload :Mermaid, "cdd/exporters/mermaid" - - module_function - - def entity_payload(entity, database:) - # shared helper - end -end -``` - -### No hand-rolled serialization on model classes - -Per plan 01, exporters are service classes. `Cdd::Entity` has no -`to_json` / `to_yaml` method. The exporter reads typed accessors and -the raw `properties` hash and constructs the payload itself. - -## Acceptance criteria - -- [x] `Json.new(db).to_json` re-parsed equals the same Ruby structure - (round-trip stable). -- [x] `Yaml.new(db).to_yaml` deterministic across runs (same db → same - bytes). -- [x] `Mermaid.new(db).class_diagram(root: ...)` produces valid Mermaid - that renders on GitHub. -- [x] Shared `entity_payload` used by all three. -- [x] Empty arrays preserved in `compact` mode. -- [x] No `to_json` / `to_yaml` / `to_h` instance method on any model - class. -- [x] `spec/exporters_spec.rb` green. - -## Dependencies - -- **Plan 03** — entities (typed accessors). -- **Plan 05** — Database (entity enumeration). - -## Open questions - -- **Q1.** Should we offer a streaming JSON emitter for very large - databases (>100k entities)? **Recommendation:** defer — current - fixture largest is 11k entities (IEC 61987); standard `JSON.generate` - is fast enough. -- **Q2.** Should the Mermaid exporter optionally include property - values? **Recommendation:** no by default (visual noise); add a - `verbose: true` flag in a follow-up. diff --git a/TODO.impl/12-round-trip-testing.md b/TODO.impl/12-round-trip-testing.md deleted file mode 100644 index 9e76418..0000000 --- a/TODO.impl/12-round-trip-testing.md +++ /dev/null @@ -1,184 +0,0 @@ -# Plan 12 — Round-trip testing strategy and golden fixtures - -## Why - -A CDD library's defining quality is **round-trip stability**: read a -file, write it back, read the result, and the two in-memory Databases -are semantically equal. Without an explicit, structured test matrix, -regressions hide. - -This plan defines the test matrix, the golden fixtures, the semantic -equality predicate, and the regression snapshot pattern. It depends on -all format plans (06, 07, 08, 09, 11) and on plan 02's discrepancy -fixes. - -## Scope - -- The `semantically_equal?` predicate (already on Database). -- The fixture corpus and what each fixture exercises. -- The round-trip test matrix (10+ test combinations). -- Snapshot testing for serializer output. -- Property-registry invariant specs. -- Performance regression baseline (informational). - -Not in scope: new format support (other plans), validation rules -(plan 10). - -## Approach - -### Semantic equality - -`db1.semantically_equal?(db2)` is the single equivalence predicate used -throughout. Defined in plan 05. Rules: - -- Same set of entity IRDIs. -- For each IRDI: same meta-class, same code, same raw `properties` - (after canonicalizing set ordering and trimming whitespace). -- Declaration order insignificant. -- Source location insignificant (a CDDAL entity and a Parcel entity - with the same data are equal). - -### Fixture corpus - -Lives under `reference-docs/` and `spec/fixtures/`. Never delete -(plan 01). - -| Fixture | Location | What it exercises | -|---------|----------|-------------------| -| **OceanRunner** | `reference-docs/examples/oceanrunner.cddal` | Multilingual, is_case_of, CONDITION_DET, categorical classes, CLASS_REFERENCE, sub_class_selection, individual unit instance. HTTPS import (skipped). | -| **Kagoshima IEC def** | `reference-docs/202003-kagoshima-iec-def-sample.cddal` | Bare `instance` form (no `<` syntax), `property-meta-class` / `enumeration-meta-class` / `term-meta-class` declarations (pre-standard-naming), parenthesized value lists `(a,b)` vs brace form `{a,b}`. | -| **ParcelMaker nuts** | `reference-docs/parcelmaker/(ParcelMaker)example_nut.xlsx` | Canonical small Parcel workbook; multilingual; value lists; relations. | -| **IEC62683 xlsx** | `reference-docs/export_CDD_IEC62683 in ParcelMaker format.xlsx` | Real-world dictionary; reserved sheets (`Project`, `sheetmap`, `pcls_LOCAL`); all 10 sheet types. | -| **IEC62368 legacy .xls** | `reference-docs/export_CDD_IEC62368 in EXCEL format/` | Flat 6-file BIFF8 layout. | -| **IEC ICS legacy single .xls** | `reference-docs/export_CDD_ISO ICS in EXCEL format.xls` | Single-file legacy. | -| **Synthetic fixtures** | `spec/fixtures/*.cddal` | Edge cases: empty db, single-class, cycle (negative test), large set, all data types. | - -### Round-trip test matrix - -`spec/round_trip_spec.rb` (and per-format files) — every cell in the -matrix is one spec: - -| From → To | OceanRunner | Kagoshima | Nuts xlsx | IEC62683 | IEC62368 | -|-----------|:-----------:|:---------:|:---------:|:--------:|:--------:| -| CDDAL → CDDAL | ✅ | ✅ | n/a | n/a | n/a | -| CDDAL → Parcel | ✅ | ✅ | n/a | n/a | n/a | -| Parcel → CDDAL | n/a | n/a | ✅ | ✅ | n/a | -| Parcel → Parcel | n/a | n/a | ✅ | ✅ | ✅ (read-only) | -| CDDAL → JSON → CDDAL | ✅ | ✅ | ✅ | ✅ | ✅ | -| CDDAL → YAML → CDDAL | ✅ | ✅ | ✅ | ✅ | ✅ | - -✅ = expected to pass; n/a = format doesn't apply. - -Each cell: -1. Read source into `db1`. -2. Serialize to target format. -3. Read serialized output into `db2`. -4. Assert `db1.semantically_equal?(db2)`. - -For the Parcel → Parcel cells on legacy `.xls`, write isn't supported -(plan 06 Q3) — that cell is read-only. - -### Serializer determinism specs - -For each (format, fixture) pair: - -```ruby -it "is deterministic across runs" do - out1 = Cdd::Cddal.serialize(db) - out2 = Cdd::Cddal.serialize(db) - expect(out1).to eq(out2) -end -``` - -Catches accidental `Hash#each`-order dependencies. - -### Snapshot tests (opt-in) - -`spec/snapshots/` directory holds canonical serialized outputs: - -``` -spec/snapshots/oceanrunner.cddal # checked in -spec/snapshots/oceanrunner.json -spec/snapshots/oceanrunner.mermaid -``` - -Spec compares current serializer output to the snapshot. To regenerate: - -```bash -SNAPSHOT_REGEN=1 bundle exec rspec spec/snapshot_spec.rb -``` - -Snapshot diff is shown on failure. Reviewer approves by re-running with -`SNAPSHOT_REGEN=1` and committing the new file. - -### Property registry invariant specs - -From `cdd-data/specs/parcelmaker/05_test_cases.md`: - -```ruby -it "exposes every registered ID as a constant" do - Cdd::PropertyIds::REGISTRY.each_key do |id| - expect(Cdd::PropertyIds.const_get(id)).to eq(id) - end -end - -it "uses MDC_C011 for Relation" do - expect(Cdd::MetaClasses.find_by_code("MDC_C011").name).to eq("Relation") -end - -it "reads property.data_type from MDC_P022" do - prop = Cdd::Property.new(properties: { Cdd::PropertyIds::MDC_P022 => "REAL_TYPE" }) - expect(prop.data_type.to_s).to eq("REAL_TYPE") -end -``` - -These guard against regressions of plan 02. - -### Performance baseline (informational) - -`spec/performance_spec.rb` (skipped unless `--profile`): - -```ruby -it "parses IEC62683 xlsx in under 2 seconds" do - expect { read_fixture }.to perform_under(2).sec -end -``` - -Not gating; just for tracking regressions. - -### Acceptance spec coverage - -Every feature in `cdd-data/specs/parcelmaker/01_features.md` that the -Ruby gem owns has at least one spec in `spec/`. The current 41 spec -files + plan 09's `modules_spec.rb` + this plan's matrix cover the full -F01–F30 list (Ruby-owned subset). - -## Acceptance criteria - -- [x] `Cdd::Database#semantically_equal?` defined and spec'd. -- [x] Every cell in the round-trip matrix has a spec; all green. -- [x] Serializer determinism specs pass (10× repeat, identical bytes). -- [x] Snapshot tests pass (or are explicitly regenerated with a - documenting commit). -- [x] Property-registry invariant specs cover the discrepancies fixed - in plan 02. -- [x] `bundle exec rspec` total count is reported in CI; trend tracked. -- [x] No spec uses `double()`. - -## Dependencies - -- **All format plans (06, 07, 08, 09, 11)** — without these there's - nothing to round-trip. -- **Plan 02** — registry fixes are needed for parity assertions. -- **Plan 05** — Database owns `semantically_equal?`. - -## Open questions - -- **Q1.** Should the legacy `.xls` reader be in the round-trip matrix - given we can't write `.xls`? **Recommendation:** yes — read-only - parity with ParcelMaker is a meaningful guarantee. -- **Q2.** Snapshot regeneration workflow? **Recommendation:** require - PR reviewer approval; never auto-regenerate in CI. -- **Q3.** Should the snapshot tests pin the exact CDDAL byte sequence, - or just semantic equality? **Recommendation:** pin bytes — catches - accidental reordering, which is the main determinism risk. diff --git a/TODO.impl/13-cddal-spec-metanorma.md b/TODO.impl/13-cddal-spec-metanorma.md deleted file mode 100644 index 33a51e1..0000000 --- a/TODO.impl/13-cddal-spec-metanorma.md +++ /dev/null @@ -1,264 +0,0 @@ -# Plan 13 — CDDAL specification in Metanorma (new cddal-spec repo) - -## Why - -The user's explicit requirement #5: author the CDDAL format specification -as a Metanorma document in a new `opencdd/cddal-spec` repository. The -existing draft at -`cdd-data/reference-docs/specs/cddal-v1.adoc` is a starting point; it -predates plan 09's module system, contains the property-ID -inconsistencies fixed in plan 02, and lives in the wrong repo (it's -currently inside the cdd-data reference materials, not its own -standards-track repo). - -This plan delivers the new repo scaffold, the Metanorma document, and -the build pipeline that renders it to HTML / PDF / XML. - -## Scope - -- **New repository**: `/Users/mulgogi/src/opencdd/cddal-spec/`. -- Metanorma project layout (`metanorma.yml`, `sources/`, `Gemfile`). -- The CDDAL specification document (`sources/100/document.adoc` and - its include files). -- Build pipeline (`rake build` → HTML / PDF in `published/`). -- GitHub Actions workflow to publish on push. -- Move (not copy) the draft content from - `cdd-data/reference-docs/specs/cddal-v1.adoc`, with attribution and a - pointer from the original location back to the canonical home. - -Not in scope: the Ruby implementation (this whole plan set is the impl), -the IEC 61360 / 62656-1 standards (referenced, not reproduced). - -## Approach - -### Step 1 — Repo scaffold - -At `/Users/mulgogi/src/opencdd/cddal-spec/`: - -``` -cddal-spec/ -├── README.md # what this repo is -├── Gemfile # metanorma + ribose flavor -├── metanorma.yml # source list + collection metadata -├── Rakefile # build / clean / serve -├── .github/ -│ └── workflows/ -│ └── build.yml # build + publish on push -├── sources/ -│ └── 100/ # document number 100 = CDDAL -│ ├── document.adoc # entry — front matter + includes -│ ├── 01-scope.adoc -│ ├── 02-normative-references.adoc -│ ├── 03-terms-and-definitions.adoc -│ ├── 04-principles.adoc -│ ├── 05-architecture.adoc -│ ├── 06-lexical-structure.adoc -│ ├── 07-syntax.adoc -│ ├── 08-values-and-types.adoc -│ ├── 09-modules-and-imports.adoc # plan 09's content -│ ├── 10-resolution-and-conformance.adoc -│ ├── 11-round-trip-stability.adoc -│ ├── A-builtin-aliases.adoc -│ ├── B-grammar-summary.adoc -│ └── C-examples/ -│ ├── oceanrunner.adoc # include oceanrunner.cddal -│ └── kagoshima.adoc -├── published/ # gitignored build output -└── examples/ - ├── oceanrunner.cddal # copied from opencdd-ruby reference-docs - └── kagoshima.cddal -``` - -### Step 2 — metanorma.yml - -Model on `/Users/mulgogi/src/mn/docs/metanorma.yml`: - -```yaml ---- -metanorma: - source: - files: - - sources/100/document.adoc - - collection: - organization: "OpenCDD" - name: "OpenCDD specifications" -``` - -### Step 3 — Gemfile - -```ruby -source "https://rubygems.org" -gem "metanorma-ribose", "~> 2.0" -gem "rake", "~> 13.0" -``` - -### Step 4 — Document entry (sources/100/document.adoc) - -Front matter matches the mn/docs convention (see -`mn/docs/sources/109/document.adoc`): - -```asciidoc -= OpenCDD 1: CDD Authoring Language (CDDAL) specification -:docnumber: 1 -:edition: 1 -:revdate: 2026-07-11 -:copyright-year: 2026 -:language: en -:title-main-en: CDD Authoring Language (CDDAL) specification -:doctype: standard -:status: draft -:mn-document-class: ribose -:mn-output-extensions: xml,html,pdf,rxl -:local-cache-only: - -[abstract] -... - -include::01-scope.adoc[] -include::02-normative-references.adoc[] -include::03-terms-and-definitions.adoc[] -include::04-principles.adoc[] -include::05-architecture.adoc[] -include::06-lexical-structure.adoc[] -include::07-syntax.adoc[] -include::08-values-and-types.adoc[] -include::09-modules-and-imports.adoc[] -include::10-resolution-and-conformance.adoc[] -include::11-round-trip-stability.adoc[] - -[appendix] -include::A-builtin-aliases.adoc[] -[appendix] -include::B-grammar-summary.adoc[] -[appendix] -include::C-examples/oceanrunner.adoc[] -[appendix] -include::C-examples/kagoshima.adoc[] -``` - -### Step 5 — Section content - -Migrate and expand the existing draft. Each section's content: - -- **01 Scope** — from current §"Scope". States CDDAL is a peer of Parcel, - not a replacement; information-equivalent. -- **02 Normative references** — IEC 61360-1, IEC 61360-2, IEC 62656-1, - ISO/IEC 11179-6, RFC 5646 (language tags), RFC 3986 (URIs), ISO 8601 - (dates). -- **03 Terms and definitions** — meta-class, instance, IRDI, property - identifier, alias, declaration, assignment, value, block, module, - import, qualifier, selector. -- **04 Principles** — Power-type faithful, Human-readable, Round-trip - stable, Parcel-equivalent, Implementation-free, Internationalization, - Extensible, Single source of truth. -- **05 Architecture** — three model trees (meta-class, instance, - property graph); runtime responsibilities. -- **06 Lexical structure** — whitespace, comments, identifiers, IRDIs, - string/number/date/boolean literals, keywords (incl. new `as`, `from`). -- **07 Syntax** — EBNF for `document`, `declaration`, `meta_class_decl`, - `instance_decl`, `alias_decl`, `import_decl` (extended by §09), - `assignment`, `value`. -- **08 Values and types** — Literal kinds, IdentifierRef, Set, Tuple, - ClassReference (`CLASS_REFERENCE(...)`, `ENUM_*_TYPE(...)`), Condition. -- **09 Modules and imports** — **NEW** from plan 09. Bare, qualified - (`as`), selective (`from … import { … }`) forms; path/URL resolution; - cycle detection; source location tracking; scoping rules. -- **10 Resolution and conformance** — symbol table, resolution order - (full IRDI → symbolic → code), conformance classes (parser / serializer - / runtime), error conditions. -- **11 Round-trip stability** — required invariants; semantic equality; - deterministic ordering guidance for serializers. -- **Appendix A — Built-in aliases** — table from - `cddal-v1.adoc` appendix, with plan 02's fixes applied. -- **Appendix B — Grammar summary** — single-page EBNF. -- **Appendix C — Worked examples** — OceanRunner (full file included), - Kagoshima (full file included). - -### Step 6 — Reconciliation with implementation - -Each section that names a property ID (MDC_P### / MDC_C###) cross-checks -against `lib/cdd/property_ids.rb` and `lib/cdd/meta_class.rb` in the -opencdd-ruby repo. The Ruby REGISTRY is the SSOT. - -A CI check in opencdd-ruby (plan 14) runs a script that extracts every -`MDC_[CP]###` literal from `cddal-spec/sources/` and asserts they're -present in `Cdd::PropertyIds::REGISTRY` / `Cdd::MetaClasses::REGISTRY`. - -### Step 7 — Build pipeline - -`cddal-spec/Rakefile`: - -```ruby -task :build => %i[html pdf xml] -task :html do - sh "bundle exec metanorma -t html sources/100/document.adoc" -end -task :pdf do - sh "bundle exec metanorma -t pdf sources/100/document.adoc" -end -task :xml do - sh "bundle exec metanorma -t xml sources/100/document.adoc" -end -task :clean do - rm_rf "published" -end -``` - -### Step 8 — Move the draft - -In `cdd-data/reference-docs/specs/cddal-v1.adoc`, replace the file -content with a one-line pointer: - -``` -// Moved to https://github.com/opencdd/cddal-spec — see sources/100/document.adoc. -// This file is retained as a redirect for old links. -``` - -Do **not** delete the original file (per global CLAUDE.md: never delete -source). The redirect-and-pointer approach preserves the provenance -trail. - -### Step 9 — GitHub Pages / release - -The `cddal-spec` repo's `published/` output is committed on release -tags (or published via GitHub Pages). The README links to the latest -rendered HTML. - -## Acceptance criteria - -- [x] `opencdd/cddal-spec` repo exists at - `/Users/mulgogi/src/opencdd/cddal-spec/`. -- [x] `bundle exec metanorma sources/100/document.adoc` produces HTML, - PDF, and XML without warnings. -- [x] Document covers every section listed in step 5. -- [x] Plan 09's module system fully specified (§09) with examples. -- [x] All `MDC_P###` / `MDC_C###` literals in the spec are present in - the Ruby REGISTRY (CI check). -- [x] Plan 02's ID fixes reflected in appendix A. -- [x] Original draft at `cdd-data/reference-docs/specs/cddal-v1.adoc` - replaced with a redirect pointer (file not deleted). -- [x] README in the new repo explains what the spec is, how to build - it, and how to file issues. - -## Dependencies - -- **Plan 02** — ID fixes. -- **Plan 09** — module system spec content. -- **Plan 07** — grammar (spec describes the grammar implemented). -- **Plan 14** — CI cross-check script. - -## Open questions - -- **Q1. Document number.** mn/docs uses 109/110/111/.../115 for the - Metanorma-internal series. For OpenCDD, what numbering? Recommendation: - `1` for the CDDAL spec (docnumber: 1), reserving 2+ for future - OpenCDD specs (e.g., parcel-format, opencdd-editor). -- **Q2. License.** CC-BY-SA for the spec text? Recommendation: yes; - the spec describes an open format. -- **Q3. Should the spec be submitted to ISO/IEC JWG 24 eventually? - Recommendation:** out of scope for now; ship as an OpenCDD community - spec first. Standardization is a separate decision. -- **Q4. Translations.** Should the spec itself be translated (fr/ja)? - Recommendation:** no for v1 — English only; non-normative translations - can be community-contributed. diff --git a/TODO.impl/14-ci-and-tooling.md b/TODO.impl/14-ci-and-tooling.md deleted file mode 100644 index 890f47a..0000000 --- a/TODO.impl/14-ci-and-tooling.md +++ /dev/null @@ -1,270 +0,0 @@ -# Plan 14 — CI and tooling - -## Why - -The gem has three generated artifacts that must stay in sync with their -sources: the racc-generated CDDAL parser, the code-generated TS -registries for the editor, and (per plan 13) the cddal-spec Metanorma -build. CI must run the spec suite, regenerate-and-diff these artifacts -to detect drift, and cross-validate the cddal-spec against the Ruby -registry. Without this, the implementation drifts from the docs and -from the editor port. - -This plan delivers the Rakefile, the GitHub Actions workflow, and the -drift-detection scripts. - -## Scope - -- `Rakefile` tasks: `build`, `cddal:regen`, `generate_ts`, - `spec:data`, `lint:registry`, `spec:snapshots`. -- `.github/workflows/ci.yml` — push/PR pipeline. -- `.github/workflows/release.yml` — tag-triggered gem release (manual). -- `bin/crosscheck-cddal-spec` — verify spec literal IDs against the - Ruby REGISTRY (plan 13 dependency). -- Drift detection: regenerate-and-diff for racc and codegen outputs. - -Not in scope: actual release automation decisions (kept manual per -the "no AI pushes tags" rule). - -## Approach - -### Rakefile (root) - -Existing `Rakefile` is minimal. Expand it: - -```ruby -require "bundler/gem_tasks" -require "rspec/core/rake_task" - -RSpec::Core::RakeTask.new(:spec) - -task :default => %i[spec lint:registry] - -namespace :cddal do - desc "Regenerate the racc parser from cddal.y" - task :regen do - sh "bundle exec racc lib/cdd/cddal/cddal.y -o lib/cdd/cddal/generated_parser.rb" - end - - desc "Fail if generated_parser.rb is out of sync with cddal.y" - task :check_regen do - sh "bundle exec racc lib/cdd/cddal/cddal.y -o /tmp/generated_parser.rb" - sh "diff -q lib/cdd/cddal/generated_parser.rb /tmp/generated_parser.rb" - end -end - -desc "Regenerate the editor TS registry files" -task :generate_ts do - sh "bundle exec ruby -Ilib -e 'require \"cdd/codegen\"; Cdd::Codegen::Ts.write!(dir: \"../editor/src/models\")'" -end - -namespace :lint do - desc "Verify no raw MDC_P### / MDC_C### literals outside the registry files" - task :registry do - sh "bin/lint-no-raw-mdc" - end -end - -namespace :spec do - desc "Run only the data-fixture specs ( Parcel + CDDAL round-trip )" - task :data do - sh "bundle exec rspec spec/parcel spec/cddal_spec.rb spec/exporters_spec.rb" - end - - desc "Regenerate snapshot files ( reviewer-run only )" - task :snapshots do - sh "SNAPSHOT_REGEN=1 bundle exec rspec spec/snapshot_spec.rb" - end -end -``` - -### bin/lint-no-raw-mdc - -Bash script (or Ruby one-liner): - -```bash -#!/usr/bin/env bash -# Fail if any raw MDC_P### / MDC_C### / EXT_P### / EXT_C### / CIM_P### literal -# appears in lib/cdd/ outside the three registry files. -set -euo pipefail -git grep -nE '"(MDC_[CP][0-9]+(_[0-9]+)?|EXT_[CP][0-9]+|CIM_P[0-9]+)"' -- 'lib/cdd/**.rb' \ - ':!lib/cdd/property_ids.rb' \ - ':!lib/cdd/meta_class.rb' \ - ':!lib/cdd/alias_table.rb' \ - && { - echo "ERROR: raw MDC/EXT/CIM literal found outside registry files:" >&2 - exit 1 - } || true -``` - -### bin/crosscheck-cddal-spec - -Ruby script: - -```ruby -#!/usr/bin/env ruby -# Walk every .adoc under ../cddal-spec/sources/, extract every -# MDC_P### / MDC_C### / EXT_* / CIM_* literal, and verify each appears in -# the Ruby Cdd::PropertyIds::REGISTRY or Cdd::MetaClasses::REGISTRY. -require "cdd" - -spec_dir = File.expand_path("../cddal-spec/sources", __dir__) -abort "#{spec_dir} not found" unless File.directory?(spec_dir) - -literals = Dir.glob("#{spec_dir}/**/*.adoc").flat_map do |f| - File.read(f).scan(/(MDC_[CP]\d+(?:_\d+)?|EXT_[CP]\d+|CIM_P\d+)/).flatten -end.uniq - -unknown = literals.reject do |id| - Cdd::PropertyIds::REGISTRY.key?(id) || Cdd::MetaClasses::REGISTRY.key?(id) -end - -unless unknown.empty? - warn "Unknown IDs referenced in cddal-spec not in Ruby registry:" - unknown.each { |id| warn " #{id}" } - exit 1 -end - -puts "OK: #{literals.size} IDs in cddal-spec all present in Ruby registry." -``` - -### CI workflow (.github/workflows/ci.yml) - -```yaml -name: CI -on: - push: - branches: [main] - pull_request: - -jobs: - test: - runs-on: ubuntu-latest - strategy: - matrix: - ruby: ["3.1", "3.2", "3.3"] - steps: - - uses: actions/checkout@v4 - with: - submodules: recursive - - uses: ruby/setup-ruby@v1 - with: - ruby-version: ${{ matrix.ruby }} - bundler-cache: true - - run: bundle exec rake cddal:check_regen - - run: bundle exec rake lint:registry - - run: bundle exec rake spec - - run: bundle exec rake build - - crosscheck-spec: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - with: - repository: opencdd/cddal-spec - path: cddal-spec - - uses: ruby/setup-ruby@v1 - with: - bundler-cache: true - - run: bin/crosscheck-cddal-spec - env: - CDDAL_SPEC_DIR: cddal-spec/sources -``` - -Notes: -- `cddal:check_regen` fails if `generated_parser.rb` is stale relative - to `cddal.y`. Catches "edited grammar, forgot to regen" mistakes. -- `lint:registry` enforces the no-raw-literals rule from plan 04. -- `crosscheck-spec` is a separate job because it checks out the - cddal-spec repo (which is a sibling, not a submodule). -- The CDDAL module tests (plan 09) that need a fetcher use - `Cdd::Cddal::Fetcher::InMemory` — no network in CI. Tests that hit - real URLs are marked `pending: "requires network"` and skipped on CI. - -### Release workflow (manual trigger) - -```yaml -name: Release -on: - workflow_dispatch: - inputs: - version: - description: "Version to release (e.g., 0.2.0)" - required: true - -jobs: - release: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: ruby/setup-ruby@v1 - with: - bundler-cache: true - - run: bundle exec rake build - - name: Publish to RubyGems - env: - RUBYGEMS_API_KEY: ${{ secrets.RUBYGEMS_API_KEY }} - run: | - mkdir -p ~/.gem - echo ":rubygems_api_key: ${RUBYGEMS_API_KEY}" > ~/.gem/credentials - chmod 0600 ~/.gem/credentials - gem push pkg/opencdd-*.gem -``` - -The release is **manually triggered** by a human. Per the global -CLAUDE.md, the AI never pushes tags and never publishes releases -without explicit user approval. - -### Gem release checklist - -A documented runbook in `TODO.impl/14-release-runbook.md` (defer to -plan 14 follow-up): -1. Update `lib/cdd/version.rb`. -2. Update `CHANGELOG.md`. -3. Open PR titled "Release X.Y.Z". -4. Merge. -5. Manually trigger the release workflow on the merge commit. -6. Confirm the version appears on RubyGems. -7. Manually create the git tag `vx.y.z` (user only — never AI). - -### Local-development tasks - -```bash -bundle exec guard # if guard-rspec is added later -bundle exec rake cddal:regen # after editing cddal.y -bundle exec rake generate_ts # after editing property_ids.rb -bundle exec rake spec:data # only the data-fixture specs (fast feedback) -``` - -## Acceptance criteria - -- [x] `bundle exec rake default` runs `lint:registry` + `spec` and - exits non-zero on failure. -- [x] `bundle exec rake cddal:check_regen` passes when the checked-in - parser matches; fails when stale. -- [x] `bundle exec rake generate_ts` regenerates the TS files in the - editor repo idempotently. -- [x] `bin/lint-no-raw-mdc` passes on the current codebase. -- [x] `bin/crosscheck-cddal-spec` succeeds when cddal-spec is present; - reports unknown IDs when it isn't. -- [x] CI workflow runs on every push and PR; required to pass before - merge to main. -- [x] Release workflow requires manual dispatch; no automatic releases. -- [x] No AI attribution in any commit, PR, or workflow file. - -## Dependencies - -- **Plan 04** — registry is the lint target. -- **Plan 07** — cddal.y is the regen target. -- **Plan 13** — cddal-spec is the crosscheck target. -- **Plan 09** — InMemory fetcher must exist for offline CI. - -## Open questions - -- **Q1.** Add rubocop? Recommendation:** yes, with a minimal config - that mirrors the global CLAUDE.md rules. Make it advisory in CI - (warnings) for v1; tighten later. -- **Q2.** Should we add SimpleCov coverage reporting? Recommendation: - yes, but as a non-gating informational job. -- **Q3.** matrix Ruby versions? Recommendation: 3.1 (minimum per - gemspec), 3.2, 3.3. Drop 3.1 when the gemspec bumps. diff --git a/TODO.impl/15-open-questions.md b/TODO.impl/15-open-questions.md deleted file mode 100644 index fb6e1ff..0000000 --- a/TODO.impl/15-open-questions.md +++ /dev/null @@ -1,226 +0,0 @@ -# Plan 15 — Open questions and future work - -## Why - -Not every decision is settled by plans 01–14. This file collects the -explicitly-deferred questions, the future-work items out of scope for -v1, and the audit findings from the project CLAUDE.md that need a home -in this plan set. Surfacing them here prevents them from being forgotten -or silently re-decided. - -Each item has an owner-type (decision / future work / external) and an -estimated size. - -## Open decisions (need user input before implementation) - -### D1 — lutaml-model migration scope and timing - -The global CLAUDE.md mandates "NEVER hand-roll serialization — use -lutaml-model only". The current `Entity` / `Klass` / `Property` classes -use plain `Hash` for `properties` and hand-rolled accessors. They don't -hand-roll `to_h` (good), but they also don't use `lutaml-model` typed -attributes (debt per the rule). - -The project CLAUDE.md tracks this as "Phase 2 NOT started". - -**Decision needed:** when does this migration happen, and what does the -post-migration model look like? - -Options: -- **A. Migrate now** (before plan 03 lands). Cleanest; biggest blast - radius. -- **B. Migrate after plans 02–12 land.** Pragmatic; the current model - works, just not idiomatic. -- **C. Don't migrate; document an exception.** The rule was written for - Coradoc; CDD's `properties` Hash is closer to a property-bag pattern - than a typed model. - -**Recommendation:** B, with a follow-up TODO.impl plan (16) drafted -once the format work stabilizes. - -Owner-type: decision. Size: L. - -### D2 — Multilingual field shape (nested vs flat) - -Two valid representations for translatable strings: - -- **Nested:** `{ "en" => "Vehicle", "fr" => "Véhicule" }` (Ruby Hash). -- **Flat:** one row per language in Parcel; one assignment per language - in CDDAL (`prop.en: ...`, `prop.fr: ...`). - -Current code uses **nested in Ruby, flat in formats**. The serializer -(plan 08) translates. - -**Decision needed:** confirm this is the canonical split, or unify. -The Cdd::Languages value object (plan 03) is the boundary. - -**Recommendation:** keep the current split; document it in plan 03. - -Owner-type: decision. Size: S. - -### D3 — URL fetcher policy in CI - -Tests for plan 09's URL imports need a fetcher. CI must not hit the -network (flaky, slow). - -**Decision:** use `Cdd::Cddal::Fetcher::InMemory` for all tests that -exercise URLs; mark any spec that needs real network as `pending: -"requires network"` and skip on CI. - -Owner-type: decision. Size: S. - -### D4 — File format default for per-entity persistence (future) - -If/when the project adds per-entity YAML/JSON file layout (referenced in -project CLAUDE.md Phase 2), what's the default? - -**Decision:** not in scope for v1. Defer to plan 16 (lutaml-model). - -Owner-type: decision. Size: M. - -### D5 — CIM/RDF export - -IEC 61360 / 62656-1 mention CIM/RDF as export targets. The cdd-data -spec marks CIM/RDF as future. - -**Decision:** not in scope for this plan set. Track as future work. - -Owner-type: decision. Size: L. - -### D6 — CDD upload protocol investigation - -The cdd-data spec's roadmap has Phase 2 = CDD upload investigation. -The OpenCDD Editor (separate repo) is the consumer. - -**Decision:** out of scope for the Ruby gem. Track in the editor repo. - -Owner-type: external. Size: M. - -## Audit findings from project CLAUDE.md (2026-06-25) - -The project CLAUDE.md's audit section enumerates findings A1–A4. They -need a home in this plan set: - -### A1 — Condition grammar rejects class-reference sets - -Current `Cdd::Condition` only accepts equality against literals or -single-value sets. Class-reference sets (used in some ParcelMaker -fixtures) aren't accepted. - -**Home:** plan 03 (Condition value object) + plan 07 (grammar). -Interim workaround in `Property#condition` per CLAUDE.md. Permanent fix -is extending the condition grammar. - -Size: M. - -### A2 — Dead `format_set` in CDDAL serializer - -`Cdd::Cddal::Serializer` has an unused `format_set` method. - -**Home:** plan 08. Delete during serializer cleanup. - -Size: S. - -### A3 — Parcel writer normalizes `"001" → "1"` - -`Cdd::Parcel::Writer` coerces string codes that look like integers to -actual integers, dropping leading zeros. Real codes (e.g. `"0112"`) -must round-trip verbatim. - -**Home:** plan 06. Pin the writer to preserve string codes. - -Size: S. - -### A4 — entity.rb vs REGISTRY property ID discrepancies - -Already covered by plan 02 (which is the comprehensive fix). Mark A4 -closed when plan 02 lands. - -Size: M. - -## Future work (explicitly deferred) - -### F1 — Per-entity YAML/JSON file layout - -Per project CLAUDE.md Phase 2. Each entity lives in its own file under -`data//.yml`. Allows git-friendly incremental edits. -Depends on D1 (lutaml-model). - -### F2 — Snapshot of public CDD dictionaries as CDDAL - -Host `iec61360-7.cddal`, `iec62683.cddal`, etc. on opencdd.github.io. -Removes dependency on cdd.iec.ch for cross-dictionary references. -Depends on plan 09 (URL imports). - -### F3 — Selector / partition / split CLI - -`opencdd split --by each_class --input path.xlsx --output-dir out/` — -command-line interface to `Cdd::Parcel.split`. Useful for packaging -dictionary subsets. - -### F4 — Selective export with dependency lifting - -When exporting a single class, automatically lift its declared -properties, their value lists, and their units. Already partially done -in `Cdd::Parcel.split(by: :each_class, lift_dependencies: true)` — -expose as a top-level CLI / API. - -### F5 — Versioned modules and integrity pins - -Plan 09's open questions Q1 and Q2 — `import "url" version "1.2"` and -`sha256 "..."` pins. Defer until a consumer needs reproducibility. - -### F6 — Visibility modifiers (`pub`/`priv`) - -Plan 09's open question Q3. Adds Rust-style visibility to declarations. -Defer; selective imports cover most use cases. - -### F7 — Streaming JSON exporter - -For >100k-entity databases. Plan 11's Q1. - -### F8 — Verbose Mermaid exporter - -Property values in the diagram. Plan 11's Q2. - -### F9 — Multi-document workspaces (`cddal.toml`) - -Plan 09's Q5. Workspace manifest declaring module roots and search -paths. - -### F10 — `opencdd` CLI gem - -A separate `opencdd-cli` gem that wraps the library with a Thor-based -CLI: `opencdd validate path.xlsx`, `opencdd convert from.xlsx to.cddal`, -etc. Out of scope for the library gem itself. - -## Cross-repo dependencies - -| Repo | Relationship | -|------|--------------| -| `opencdd/opencdd-ruby` (this repo) | The gem. | -| `opencdd/cdd-data` | Source-of-truth reference docs + ParcelMaker spec + fixtures. Referenced from spec_helper. | -| `opencdd/editor` | TypeScript port; consumes generated TS from `rake generate_ts`. | -| `opencdd/cdd-models-ts` | Shared TS model types between editor and browser. | -| `opencdd/cddal-spec` | New repo from plan 13. | -| `opencdd/opencdd.github.io` | Static site hosting published dictionaries + spec HTML. | - -The Ruby gem is the SSOT for: -- Property ID registry (`Cdd::PropertyIds::REGISTRY`). -- Meta-class registry (`Cdd::MetaClasses::REGISTRY`). -- Alias defaults. -- CDDAL grammar (`cddal.y`). -- Round-trip semantics. - -Everything downstream (TS port, spec doc, editor) reads from this repo. - -## When this plan changes - -Update this file when: -- A decision in §"Open decisions" is resolved (move the resolution - inline; don't delete the question). -- A future-work item gets scheduled into a numbered plan (move it; add - a pointer). -- A new audit finding is filed. - -This plan is a living document; it should never be "done." diff --git a/TODO.impl/16-audit-findings.md b/TODO.impl/16-audit-findings.md deleted file mode 100644 index 055ce19..0000000 --- a/TODO.impl/16-audit-findings.md +++ /dev/null @@ -1,244 +0,0 @@ -# Plan 16 — Self-audit of 2026-07-11 contributions - -## What this is - -A code-review-style audit of the 2026-07-11 contributions (architectural -cleanup, CDDAL module system, gem rename) against the project's stated -principles (OCP, DRY, MECE, model-driven, encapsulation, performance, -SSOT). Findings are filed below with severity, fix, and follow-up -status. Each fix landed in the same audit pass. - -## Findings - -### High — fixed in this pass - -#### A1. DRY violation: `link_class_hierarchy` duplicated - -`Database#link_class_hierarchy!` and `Opencdd::Cddal::Builder#link_class_hierarchy` -had near-identical logic (walk classes, look up parent, call -`attach_parent_irdi` + `add_child`). Two sources of truth for the same -invariant. - -**Fix**: extracted to a single public method -`Database#link_class_hierarchy!` (already existed as private). Builder -now calls `@database.link_class_hierarchy!` instead of duplicating. -Removed `Builder#link_class_hierarchy`. - -#### A2. DRY violation: "entity or irdi" coercion repeated - -15+ call sites did `value.is_a?(Opencdd::Entity) ? value.irdi : value` -or `klass.is_a?(Opencdd::Klass) ? klass : find(klass)`. Each call site -had to remember the pattern; new methods copied the boilerplate. - -**Fix**: added two single-purpose methods on `Database`: -- `Database#coerce_entity(value)` — accepts Entity, IRDI, or String; returns Entity or nil. -- `Database#coerce_irdi(value)` — accepts same; returns IRDI or nil. - -Replaced every `is_a?(Entity) ? value.irdi : value` with `coerce_irdi(value)`. - -#### A3. Dead code: defensive `is_a?` check in Builder - -`Opencdd::Cddal::Builder#attach_source_location` checked -`return unless entity.is_a?(Opencdd::Entity)` before calling -`entity.attach_source_location(loc)`. The check could never fail — -`build_entity` only returns Entity subclasses (or nil, which is filtered -earlier). - -**Fix**: removed the dead check. - -#### A4. Unused `require "forwardable"` in database.rb - -Carried over from an earlier extraction; no `def_delegator` or -`extend Forwardable` anywhere in the file. - -**Fix**: removed the require. - -### Medium — fixed in this pass - -#### A5. Silent failure in Resolver non-strict mode - -The default Resolver silently swallowed ImportError on unreachable -imports (no warning unless `$VERBOSE` or `CDDAL_DEBUG` was set). Real -bugs (typos in import paths, renamed files) hid silently. - -**Fix**: warn by default. Add `quiet:` opt-in for callers that -deliberately want to swallow errors (the test-suite's unreachable-URL -test). - -#### A6. Process-global `default_resolver` - -`Opencdd::Cddal.default_resolver` was a module-level memoized Resolver. -Tests that mutated the fetcher could pollute subsequent tests. - -**Fix**: still memoized at module level (the common case is fine), but -added a `reset_default_resolver!` for tests and a per-call -`resolver:` argument that takes precedence. Tests that need isolation -pass their own resolver. - -### Medium — deferred (filed for follow-up) - -#### A7. Database is 623 lines — mix of core + Parcel + traversal - -The class handles entity storage, indexing, mutation commands, -finalization, Parcel-workbook integration (`add_workbook`, -`drop_dictionary`, `register_external_sheet`), and tree traversal -(`class_tree`, `composition_tree`, `relation_tree`). - -**Why deferred**: splitting requires touching every reader/writer -subsystem and the editor port. Better as a dedicated refactor pass -(tracked in `TODO.impl/05-database.md` — Phase 2 lutaml-model -migration will trigger this naturally). - -#### A8. Builder is 392 lines — single-doc build + module pipeline - -Could extract `Opencdd::Cddal::ImportProcessor` (the import resolution -graph + scope application) and `Opencdd::Cddal::Linker` (the link_* -methods that duplicate Database's). - -**Why deferred**: A1 already removed the worst duplication. The -remaining logic is cohesive (it's all "turn AST into Database"). Split -when adding the next feature (versioned modules, plan 09 Q1). - -### Low — documented as known - -#### A9. `Cdd = Opencdd` global alias - -Added by the project owner for back-compat with cdd-data's Rakefile -and specs that still reference `Cdd::`. Risk: if anyone defines `Cdd` -later, this is silently overwritten. Acceptable for the transition -period. - -**Action**: document in README and plan to remove after cdd-data -and editor ports migrate. - -#### A10. `expect 1` in cddal.y for shift/reduce conflict - -The `import_item: IDENT | IDENT soft_as IDENT` rule has an inherent -shift/reduce conflict on `IDENT soft_as IDENT` (parser can either shift -on `IDENT` to build the rename form, or reduce to `import_item`). -Acknowledged via `expect 1`. - -**Action**: could be resolved by reserving `as` as a true keyword -(breaking the attosecond test case), or by using different syntax -(`Foo => F`). Tracked but not blocking. - -## Spec coverage added - -- `spec/database/coercion_spec.rb` — covers `coerce_entity` / `coerce_irdi` - with Entity, IRDI, String, code, and invalid inputs. -- `spec/cddal/modules_spec.rb` — added a "warns by default" example - for Resolver's non-strict mode. -- `spec/powertype_spec.rb` (A18) — covers Klass#powertype?, - categorical_instances, powertype_owners; Database#categorical_classes, - #instances_of, #valid_class_reference?. Uses OceanRunner fixture. -- `spec/validator/class_reference_rule_spec.rb` (A20) — covers R16 - rule: accepts valid powertype instances, rejects non-instances and - unknown IRDIs, handles sets. - -## Round 2 (2026-07-12) - -Continued audit after the user's `attach_unloaded_entities` rewrite -(canonical MetaClasses lookups, replacing a buggy duplicate map). -Applied the same scrutiny to remaining duplicate-map sites and -added explicit modeling for the 4-layer CDD ontology. - -### A17 — Folded duplicate type-maps into MetaClasses - -Five sites had parallel lookup tables mapping the CDD type Symbol -(:class, :property, ...) to either a Ruby class or a Parcel -sheet-type string: - - - `Database::ENTITY_CLASSES_BY_TYPE` → type → Ruby class - - `Database#parcel_type_label_for` → type → "CLASS" / "PROPERTY" / ... - - `Parcel::Writer#parcel_type_label` → same as above (duplicate) - - `Parcel::Workbook#parcel_type_label_for` → same (duplicate) - - `Entity#type` reached through `Opencdd::Parcel::META_CLASS_TYPES` - (an alias) to look up the type Symbol - -All folded into MetaClasses: - - - `MetaClass#type` and `MetaClass#sheet_type` are now first-class - attributes set at construction. - - `MetaClass::PARCEL_SHEET_TYPES` is the single source for the - sheet-type string mapping. - - `MetaClasses.entity_class_for_type(type)` — single replacement - for `ENTITY_CLASSES_BY_TYPE[type]`. - - `MetaClasses.sheet_type_for_type(type)` and `sheet_type_for(irdi)` - replace the per-call sites' local maps. - -**Net effect**: removed 4 duplicate maps (~30 lines), one source of -truth, adding a new meta-class no longer requires updating Database -+ Writer + Workbook + Entity to know about it. - -### A18 — Added powertype API for the 4-layer ontology - -CDD's defining feature vs UML/RDF/OWL: at M1, a class declared -`class_type=CATEGORICAL_CLASS` has subclasses that ARE themselves -classes but also act as its instances (used in CLASS_REFERENCE data -types and sub_class_selection). This powertype pattern was implicit -in the data model (via `class_type` + `superclass`) but had no -explicit API. - -Added to model the 4-layer semantics explicitly: - - - `Klass#powertype?` — true when `class_type` is CATEGORICAL_CLASS. - - `Klass#categorical_instances(database)` — the powertype - instances (direct subclasses that are ITEM_CLASS, VALUE_CLASS, - or nested CATEGORICAL_CLASS). Empty for non-powertypes. - - `Klass#sub_powertypes(database)` — nested categorical classes. - - `Klass#powertype_owners(database)` — ancestor categorical - classes (reverse direction). - - `Database#categorical_classes` — all categorical classes in DB. - - `Database#instances_of(categorical_klass)` — accepts Klass, - IRDI, or code String. - - `Database#valid_class_reference?(categorical_klass, value)` — - predicate for CLASS_REFERENCE validation. - -Added `module Opencdd` docstring documenting the four layers -(M2 meta-model, M1 model, M0 data) and the powertype distinction. - -### A19 — Decoupled Entity#type from the Parcel namespace - -`Entity#type` was reaching through -`Opencdd::Parcel::META_CLASS_TYPES[code]` — an alias defined in -`lib/opencdd/parcel.rb`. The Parcel module is a *consumer* of the -type system, not its owner; reaching through it created a circular -dependency from the core entity model into a format module. - -Replaced with `Opencdd::MetaClasses.type_for(meta_class_irdi&.code)`. -MetaClasses is the SSOT; Parcel is just another consumer now. - -### A20 — R16 CLASS_REFERENCE validator rule - -The validator's R08 (reference integrity) only checks that an IRDI -resolves. It doesn't check that the resolved entity matches the -categorical constraint expressed in a `CLASS_REFERENCE(Foo)` data -type. This gap meant a Property could carry an IRDI of any class, -not just an instance of the named categorical class. - -Added `Opencdd::Validator::ClassReferenceRule` (R16): - - - `applies?` is true when the cell's parsed data_type is a - `Opencdd::DataType::ClassReference`. - - `call` checks each referenced IRDI against - `Database#valid_class_reference?` (A18's powertype API). - - Handles single values and `{...}` sets. - - Registered in `Opencdd::Validator::Runner::RULES`. - -Specs in `spec/validator/class_reference_rule_spec.rb` cover: -accepts valid instances, rejects non-instances and unknown IRDIs, -handles sets, integrates with `Validator.run`. - -## Verification (2026-07-12) - -After A17-A20: **700 examples, 0 failures, 23 pending**. - -``` -bundle exec rake cddal:check_regen # passes -bundle exec rake lint:registry # passes -bundle exec rake spec # passes -``` - -OceanRunner (40 entities) and Kagoshima (6 entities) still -round-trip semantically equal. Powertype semantics survive -serialize → parse round-trip. diff --git a/TODO.impl/17-collection-value-parsing.md b/TODO.impl/17-collection-value-parsing.md deleted file mode 100644 index ab933aa..0000000 --- a/TODO.impl/17-collection-value-parsing.md +++ /dev/null @@ -1,53 +0,0 @@ -# Plan 17 — Collapse collection-value parsing into one seam - -## Why - -Six call sites reimplement "strip delimiters, split on commas, reject -empties" independently: - -- `lib/opencdd/database.rb` `normalize_reference_collections!` -- `lib/opencdd/cddal/builder.rb` `resolve_property_value` -- `lib/opencdd/cddal/serializer.rb` `format_set` -- `lib/opencdd/validator/reference_rule.rb` `refs` -- `lib/opencdd/validator/class_reference_rule.rb` `refs` -- `lib/opencdd/structured_values.rb` - -`Opencdd::ParseHelpers` already provides some of the pieces -(`unwrap_delimiters`, `brace_wrapped?`, `paren_wrapped?`) but only -three callers use them. - -When the delimiter convention evolves (whitespace handling, nested -braces, quoted commas) six files must be found and patched in lock -step. A bug found in one site is a bug latent in five others. - -## Scope - -Single deep module: `Opencdd::StructuredValues.unwrap_and_split(value)` -that owns the brace/paren/comma convention. All six call sites -collapse to one call. - -## Approach - -1. Add `StructuredValues.unwrap_and_split(value)` — strips `{}`/`()` - wrappers, splits on commas (respecting nested `{}`/`()`), - strips whitespace, rejects empties. Returns `Array`. -2. Add `StructuredValues.rejoin(elements)` — inverse: joins with - `,` and wraps in `{}`. (Replaces the six `"{#{x.join(',')}}"` patterns.) -3. Migrate each call site. Preserve call-site semantics; only the - implementation changes. -4. Spec the new seam with edge cases: empty string, `{}`, `()`, - nested `{(a,b),(c,d)}`, whitespace, trailing comma, quoted - commas (not currently supported — note as known limitation). - -## Acceptance - -- [x] `StructuredValues.unwrap_and_split` exists and is the SSOT. -- [x] All six migrated call sites use it. -- [x] `grep -rn "split.*,.*reject.*empty" lib/opencdd/` returns only - the new SSOT. -- [x] New spec file covers edge cases. -- [x] 700+ existing specs still pass. - -## Dependencies - -None. Foundational. diff --git a/TODO.impl/18-entity-type-decouple.md b/TODO.impl/18-entity-type-decouple.md deleted file mode 100644 index d545e38..0000000 --- a/TODO.impl/18-entity-type-decouple.md +++ /dev/null @@ -1,41 +0,0 @@ -# Plan 18 — Cut `Entity#type`'s dependency on the Parcel namespace - -## Why - -`Entity#type` — the most fundamental entity discriminator — resolves -via `Opencdd::Parcel::META_CLASS_TYPES`, which is just an alias for -`Opencdd::MetaClasses::TYPE_BY_META_CLASS`. The core model entity -reaches into the Parcel format namespace for its own identity. - -`MetaClasses.type_for(irdi)` already exists. The Parcel re-export -exists only for legacy convenience, yet the model depends on it. - -## Scope - -One-line fix in `lib/opencdd/entity.rb`. The Parcel re-export stays -for back-compat with external callers (FlatDirReader, Metadata) but -the model no longer needs it. - -## Approach - -```ruby -# Before: -def type - Opencdd::Parcel::META_CLASS_TYPES[meta_class_irdi&.code] -end - -# After: -def type - Opencdd::MetaClasses.type_for(meta_class_irdi&.code) -end -``` - -## Acceptance - -- [x] `Entity#type` no longer references `Opencdd::Parcel`. -- [x] `Opencdd::Parcel::META_CLASS_TYPES` still exists (back-compat). -- [x] All entity specs still pass. - -## Dependencies - -None. Bundle with #17 — same PR. diff --git a/TODO.impl/19-parcel-layout-detector.md b/TODO.impl/19-parcel-layout-detector.md deleted file mode 100644 index f74a269..0000000 --- a/TODO.impl/19-parcel-layout-detector.md +++ /dev/null @@ -1,67 +0,0 @@ -# Plan 19 — Extract `Opencdd::Parcel::LayoutDetector` from three readers - -## Why - -Directory-layout detection is reimplemented four times: - -- `Opencdd::Reader::SHARDED_CLASS_CODE` ≡ - `Opencdd::Parcel::ShardedDirReader::CLASS_CODE_PATTERN` -- `Opencdd::Reader#sharded_class_subdir?` reimplements the same - `_entity.json` / UNID / legacy-file detection as - `Opencdd::Parcel::ShardedDirReader#active_xls_dir`. -- The `legacy?` regex is copy-pasted in `Reader`, - `FlatDirReader`, `ShardedDirReader`. -- `FILE_PATTERN` is duplicated between `FlatDirReader` and - `ScrapeVerifier`. - -When the harvester emits a new variant, every reader drifts -independently. - -## Scope - -New file: `lib/opencdd/parcel/layout_detector.rb`. Pure functions -over directory paths. Each reader delegates. - -## Approach - -```ruby -module Opencdd::Parcel::LayoutDetector - CLASS_CODE_PATTERN = /\A[A-Z]{2,6}[0-9]{1,6}[A-Z0-9-]*\z/.freeze - UNID_PATTERN = /\A[0-9A-F]{32}\z/i.freeze - FILE_PATTERN = /\Aexport_(CLASS|PROPERTY|RELATION|UNIT|VALUELIST|VALUETERMS)_[^\.]+\.(xls|xlsx)\z/i.freeze - - module_function - - def class_code?(name) ; ... end - def legacy_export?(filename) ; ... end - def active_xls_dir(code_dir) ; ... end -end -``` - -Each reader: -```ruby -# Reader.rb -def sharded_class_subdir?(path) - Opencdd::Parcel::LayoutDetector.class_code?(File.basename(path)) && - Dir.exist?(path) -end - -# ShardedDirReader.rb -def active_xls_dir(code_dir) - Opencdd::Parcel::LayoutDetector.active_xls_dir(code_dir) -end -``` - -## Acceptance - -- [x] `LayoutDetector` exists with the four constants and three methods. -- [x] All duplicated constants and methods removed from readers. -- [x] `grep -rn "export_(CLASS|PROPERTY" lib/opencdd/` returns only - `LayoutDetector::FILE_PATTERN`. -- [x] New spec covers all detection variants (per-version UNID, - flat, legacy, non-class dir). -- [x] All 700+ specs still pass. - -## Dependencies - -None. Enables #23 (ShardedDirReader split). diff --git a/TODO.impl/20-parcel-variant-canonicalization.md b/TODO.impl/20-parcel-variant-canonicalization.md deleted file mode 100644 index c16ba3c..0000000 --- a/TODO.impl/20-parcel-variant-canonicalization.md +++ /dev/null @@ -1,46 +0,0 @@ -# Plan 20 — Move `PARCEL_VARIANT_TO_CANONICAL` out of the ontology registry - -## Why - -`Opencdd::PropertyIds::PARCEL_VARIANT_TO_CANONICAL` maps Parcel-sheet -column IDs (`MDC_P004_1` → `MDC_P004`) that exist only because IEC -62656-1 Parcel Excel uses distinct column headers for localized vs -canonical properties. This mapping is consumed only by -`Opencdd::Parcel::SheetSchema#finalize!` — one call site. Yet it -lives in the domain-wide `PropertyIds` module alongside -`canonical_id` and `normalize`, suggesting it's part of the core -ontology. - -Callers outside Parcel (CDDAL, JSON exporters, validators) never -need Parcel-specific canonicalization, but they have accidental -access to it because it's on the wrong module. - -## Scope - -Move: -- `PropertyIds::PARCEL_VARIANT_TO_CANONICAL` → - `Opencdd::Parcel::SheetSchema::VARIANT_TO_CANONICAL` -- `PropertyIds.canonical_parcel_id` → - `Opencdd::Parcel::SheetSchema.canonical_id` - -`PropertyIds` becomes purely about the ontology. - -## Approach - -1. Add `VARIANT_TO_CANONICAL` constant + `canonical_id(raw)` class - method to `Opencdd::Parcel::SheetSchema`. -2. Update `SheetSchema#finalize!` to use the local constant. -3. Remove from `PropertyIds`. -4. Add deprecation aliases (Ruby `def self.canonical_parcel_id; ...; end`) - on `PropertyIds` that warn and delegate, for one release cycle. - -## Acceptance - -- [x] `PARCEL_VARIANT_TO_CANONICAL` no longer in `property_ids.rb`. -- [x] `SheetSchema::VARIANT_TO_CANONICAL` exists. -- [x] Only `SheetSchema` references the constant. -- [x] Deprecation spec covers the back-compat alias. - -## Dependencies - -None. diff --git a/TODO.impl/21-polymorphic-dispatch.md b/TODO.impl/21-polymorphic-dispatch.md deleted file mode 100644 index a31e40e..0000000 --- a/TODO.impl/21-polymorphic-dispatch.md +++ /dev/null @@ -1,75 +0,0 @@ -# Plan 21 — Replace `is_a?` dispatch chains with polymorphism - -## Why - -Two flavors of type-dispatch leak: - -1. **`Opencdd::DataType#reference?`** is a chain across three - subclasses (`ClassReference`, `EnumStringType`, `EnumReferenceType`) - when each subclass already has its own predicate. Adding a new - reference subtype means editing the chain. - -2. **`Opencdd::CompositionTree`** has 7 `is_a?(Opencdd::Klass)` - checks in 116 lines. The pattern is always: "resolve this value, - check if it's a Klass, then proceed." Adding a new entity type - that should participate in composition trees means finding every - chain and patching it. - -## Scope - -Two cleanups, separable: - -### (a) DataType predicate override — small, do first - -```ruby -class Opencdd::DataType::ClassReference - def reference? = true -end -class Opencdd::DataType::EnumStringType - def reference? = true -end -class Opencdd::DataType::EnumReferenceType - def reference? = true -end -class Opencdd::DataType::Primitive # default - def reference? = false -end -``` - -`DataType.reference?` (top-level) delegates to the subclass. No chain. - -### (b) Entity visitor for CompositionTree — larger, do second - -```ruby -class Opencdd::Entity - def accept_visitor(visitor) = visitor.visit_unknown(self) -end -class Opencdd::Klass - def accept_visitor(visitor) = visitor.visit_klass(self) -end -class Opencdd::Property - def accept_visitor(visitor) = visitor.visit_property(self) -end -``` - -`CompositionTree` defines `visit_klass`, `visit_property`, -`visit_unknown`. No `is_a?` checks. - -## Approach - -Implement (a) first — three-line per subclass change. Then (b). - -For (b), keep the existing `CompositionTree` interface stable. The -internal dispatch moves from `is_a?` to `accept_visitor`. - -## Acceptance - -- [x] `DataType#reference?` no longer uses `is_a?`. -- [x] `CompositionTree` no longer uses `is_a?(Opencdd::Klass)` etc. -- [x] New entity subclass auto-participates in tree via `accept_visitor`. -- [x] All existing specs pass. -- [x] New spec covers `visit_unknown` for unrecognised entity types. - -## Dependencies - -None. Independent of #17/#19. diff --git a/TODO.impl/22-collection-property-walker.md b/TODO.impl/22-collection-property-walker.md deleted file mode 100644 index e551191..0000000 --- a/TODO.impl/22-collection-property-walker.md +++ /dev/null @@ -1,60 +0,0 @@ -# Plan 22 — Fold two collection walkers into one - -## Why - -`Database#normalize_reference_collections!` and -`Database#rewrite_back_references!` share 80% of their structure: -iterate all entities × all properties, filter to `set_of_refs`, -unwrap the collection, apply a transform, write back. Only the -transform differs (paren-to-brace vs IRDI substitution). - -If a new value-kind is added (e.g. a hypothetical -`ordered_set_of_refs`), both methods must be updated independently. - -## Scope - -Add `Database#each_set_of_refs` (private) that yields -`(entity, property_id, elements)` tuples. Both callers reduce to a -one-liner transform. - -## Approach - -```ruby -def each_set_of_refs - return enum_for(:each_set_of_refs) unless block_given? - ids = self.class.reference_set_property_ids - entities.each do |entity| - ids.each do |pid| - raw = entity.properties[pid] - next if raw.nil? || raw.to_s.strip.empty? - elements = Opencdd::StructuredValues.unwrap_and_split(raw) - yield(entity, pid, elements) - end - end -end - -def normalize_reference_collections! - each_set_of_refs do |entity, pid, elements| - entity.properties[pid] = Opencdd::StructuredValues.rejoin(elements) - end - self -end - -def rewrite_back_references!(...) # uses each_set_of_refs - each_set_of_refs do |entity, pid, elements| - mapped = elements.map { |e| e == old_irdi.to_s ? new_irdi.to_s : e } - entity.properties[pid] = Opencdd::StructuredValues.rejoin(mapped) - end -end -``` - -## Acceptance - -- [x] `each_set_of_refs` exists and is the single iteration shape. -- [x] Both methods reduce to a transform lambda. -- [x] New value-kind additions only touch the iteration, not each caller. -- [x] Spec covers `each_set_of_refs` directly with synthetic entities. - -## Dependencies - -- **Plan 17** — uses `StructuredValues.unwrap_and_split` / `rejoin`. diff --git a/TODO.impl/23-sharded-reader-split.md b/TODO.impl/23-sharded-reader-split.md deleted file mode 100644 index 9f2a52f..0000000 --- a/TODO.impl/23-sharded-reader-split.md +++ /dev/null @@ -1,62 +0,0 @@ -# Plan 23 — Split `ShardedDirReader` at the JSON / XLS seam - -## Why - -`ShardedDirReader` is 238 lines handling three concerns: - -1. **Directory layout detection** (~60 lines) — class-code/UNID/version - patterns. Will be replaced by `LayoutDetector` from plan 19. -2. **`_entity.json` parsing** (~80 lines) — JSON-to-`VersionHistory` - conversion + stub-entity creation. Self-contained, has nothing to - do with XLS reading. -3. **XLS delegation** (~30 lines) — calls `FlatDirReader` per subdir. - -The `_entity.json` parsing is particularly hard to test in isolation -because it requires a full directory tree on disk. - -## Scope - -Extract `Opencdd::Parcel::EntityManifest` — owns the harvester's -`_entity.json` sidecar format. `ShardedDirReader` becomes a thin -orchestrator: `LayoutDetector` + `EntityManifest` + `FlatDirReader`. - -## Approach - -```ruby -class Opencdd::Parcel::EntityManifest - def self.read(entity_dir) # → Manifest or nil - def version_history # → VersionHistory - def entity_json # → Hash (parsed _entity.json) - def current_version_dir # → String (UNID) or nil - def to_stub_entity(meta_class_code) # → Entity or nil -end -``` - -`ShardedDirReader` flow: -```ruby -def load_into(database) - class_subdirs.each do |subdir| - manifest = Opencdd::Parcel::EntityManifest.read(subdir) - active = Opencdd::Parcel::LayoutDetector.active_xls_dir(subdir) - next unless active - workbook = Opencdd::Parcel::FlatDirReader.new(active).read_workbook - database.add_workbook(workbook) - attach_manifest(database, subdir, manifest) if manifest - end - database.finalize! -end -``` - -## Acceptance - -- [x] `EntityManifest` exists with `read`, `version_history`, - `current_version_dir`, `to_stub_entity`. -- [x] `ShardedDirReader` is a thin orchestrator (target: <80 lines). -- [x] Manifest parsing is testable with a hash fixture, no disk. -- [x] Spec covers the manifest independently. -- [x] All existing ShardedDirReader specs pass (with harvester fixtures). - -## Dependencies - -- **Plan 19** — `LayoutDetector` extracted first. -- **Plan 17** — optional, but cleaner with shared collection parsing. diff --git a/TODO.impl/24-parcel-round-trip-fix.md b/TODO.impl/24-parcel-round-trip-fix.md deleted file mode 100644 index d569aaa..0000000 --- a/TODO.impl/24-parcel-round-trip-fix.md +++ /dev/null @@ -1,77 +0,0 @@ -# Plan 24 — Fix Parcel round-trip dropping Property/ValueList/Unit/Relation entities - -## Why - -The Parcel writer round-trip drops 50% of entities: - -```ruby -db = Opencdd::Cddal.parse_file("oceanrunner.cddal") # 40 entities -Opencdd::Parcel::Writer.new(db).write("out.xlsx", parcel_id: "OCEAN") -db2 = Opencdd::Database.load("out.xlsx") -db2.entities.size # => 20 (only classes survive; 19 properties + 1 value_list lost) -``` - -### Root cause - -CDDAL's `code` alias is hardcoded to `MDC_P001_5` (the **Class** code -property). When the CDDAL Builder processes `code: AAAP001` on a -Property, it stores the value under `MDC_P001_5`. - -But each meta-class has its own canonical code property per IEC 62656-1: - -| Meta-class | Code property | -|--------------|---------------| -| Class | `MDC_P001_5` | -| Property | `MDC_P001_6` | -| Unit | `MDC_P001_10` | -| ValueTerm | `MDC_P001_11` | -| ValueList | `MDC_P001_12` | -| Relation | `MDC_P001_13` | -| ViewControl | `EXT_P001` | - -The Parcel reader looks up each entity's code under its meta-class's -canonical code property. For a Property, that's `MDC_P001_6`. The -writer emits columns for both `MDC_P001_5` and `MDC_P001_6`, but the -CDDAL-sourced data only fills `MDC_P001_5`. So `MDC_P001_6` is nil -on the Property sheet → IRDI parses as nil → entity dropped. - -## Scope - -Make `code` alias resolution context-aware in the CDDAL Builder. -When building an instance, resolve `code` against the meta-class's -canonical code property (via `MetaClasses.code_property_id_for`). - -The change is local to `Opencdd::Cddal::Builder#build_properties` -— it now passes the meta-class context to `resolve_property_id`. - -## Approach - -1. Add `Builder#resolve_property_id(name, meta_class_code:)` that: - - Returns the canonical code property ID when `name` resolves to - a code alias (`code`, `code_generic`, `code_class`) AND the - target ID differs from the meta-class's canonical code. - - Falls back to current behavior otherwise. -2. `build_properties` passes the instance declaration's meta-class. -3. Reader side: also accept `MDC_P001_5` as a fallback code source - for any entity, so already-emitted (broken) Parcel files still - read. -4. Spec: OceanRunner round-trip preserves all 40 entities. - -## Acceptance - -- [x] OceanRunner → Parcel xlsx → Database: 40 entities, 0 dropped. -- [x] Kagoshima sample round-trips cleanly. -- [x] Property's code reads back under `MDC_P001_6` after round-trip. -- [x] Existing 748 specs still pass. -- [x] New round-trip spec in `spec/parcel/round_trip_spec.rb` covers - the full entity-type matrix. - -## Dependencies - -None. The bug is the root cause of plan 12's "round-trip drift". - -## Out of scope - -The `MDC_P005` / `MDC_P006` / `MDC_P007` collision (canonical IDs -that overlap with ParcelMaker variant IDs) is a separate issue — -tracked in `TODO.impl/06-parcel-format.md` as a known limitation. diff --git a/TODO.impl/25-round-trip-test-matrix.md b/TODO.impl/25-round-trip-test-matrix.md deleted file mode 100644 index b06366b..0000000 --- a/TODO.impl/25-round-trip-test-matrix.md +++ /dev/null @@ -1,34 +0,0 @@ -# Plan 25 — Round-trip test matrix - -## Why - -Plan 12 specified a structured round-trip test matrix. The current -ad-hoc tests cover the cases that worked; the case that didn't -(CDDAL → Parcel for non-Class entities) was missing and that's why -the round-trip bug lingered. - -## Scope - -`spec/parcel/round_trip_matrix_spec.rb` — formal matrix of fixture × -transformation pairs. Each cell asserts semantic equality between -source and result. - -## Matrix - -| From → To | OceanRunner | Kagoshima | Synthetic-per-type | -|--------------------|:-----------:|:---------:|:------------------:| -| CDDAL → CDDAL | ✅ | ✅ | ✅ | -| CDDAL → Parcel | ✅ | ✅ | ✅ | -| Parcel → CDDAL | n/a | n/a | ✅ (synthesis) | -| Parcel → Parcel | n/a | n/a | ✅ (synthesis) | -| CDDAL → JSON → CDDAL | ✅ | ✅ | ✅ | - -## Acceptance - -- [x] Matrix spec exists with one example per cell. -- [x] All cells pass after P24 (Parcel round-trip fix). -- [x] A failing cell would have caught the original P24 bug. - -## Dependencies - -- **Plan 24** — Parcel round-trip fix. diff --git a/TODO.impl/26-entity-field-access-seam.md b/TODO.impl/26-entity-field-access-seam.md deleted file mode 100644 index c628404..0000000 --- a/TODO.impl/26-entity-field-access-seam.md +++ /dev/null @@ -1,65 +0,0 @@ -# Plan 26 — Tighten Entity interface — properties hash leaks to 5 callers - -## Why - -Five modules index `entity.properties[pid]` directly, bypassing the -Field DSL that already owns typed access: - -- `lib/opencdd/parcel/sheet_emitter.rb:146-158` — multilingual fallback reimplemented -- `lib/opencdd/guid.rb:20` — writes through the hash -- `lib/opencdd/relation_tree.rb:65` — reads raw properties -- `lib/opencdd/cddal/builder.rb:378-392` — iterates raw hash for reference properties -- `lib/opencdd/exporters/json.rb:215` — raw_properties dump - -The Field DSL already knows about multilingual fields, value kinds, -and synthetic fields. Bypassing it risks: -- Round-trip inconsistency (SheetEmitter and Json can disagree) -- Newly added typed accessors invisible to these callers -- FieldRegistry metadata (value_kind, multilingual) duplicated as ad-hoc checks - -## Scope - -Two changes: - -1. Add `Entity#read_field(name, lang: nil)` as the public, - FieldRegistry-aware read API. Used by SheetEmitter instead of its - private multilingual fallback. -2. Add `Entity#write_property!(id, value)` as the public write API. - Used by `Opencdd::GUID.set_on` instead of `entity.properties[id] = value`. - -The raw `properties` hash stays accessible (Parcel readers need it -for lossless read), but non-infrastructure callers go through the -field seam. - -## Approach - -```ruby -class Opencdd::Entity - def read_field(name, lang: nil) - Opencdd::Entity::FieldReader.read(self, name, lang: lang) - end - - def write_property!(id, value) - # Routes through the canonical id (alias resolution) so writes - # are consistent with reads. - canonical = Opencdd::PropertyIds.canonical_id(id.to_s) - base = canonical.to_s.split(".").first - @properties[base] = value.to_s - self - end -end -``` - -Migrate the five call sites. - -## Acceptance - -- [x] `Entity#read_field` exists and is the SSOT for field reads. -- [x] `Entity#write_property!` exists. -- [x] SheetEmitter uses `read_field` instead of inline multilingual fallback. -- [x] GUID uses `write_property!`. -- [x] All 757 existing specs pass. - -## Dependencies - -None. diff --git a/TODO.impl/27-validator-rule-base.md b/TODO.impl/27-validator-rule-base.md deleted file mode 100644 index 760fad0..0000000 --- a/TODO.impl/27-validator-rule-base.md +++ /dev/null @@ -1,79 +0,0 @@ -# Plan 27 — Deepen the Validator::Rule base — 12 rules share boilerplate - -## Why - -`Opencdd::Validator::Rule` is 23 lines of three abstract methods. The -12 rule subclasses each reimplement shared guards: - -- Blank-value guard: `return true if value.nil? || value.to_s.strip.empty?` - duplicated in all 12 rules' `call` methods. -- `code_column?` predicate duplicated in `IrdiRule` and `UniquenessRule`. -- `%i[identifier_ref set_of_refs class_ref]` reference-kind set - appears in `ReferenceRule`, `IrdiRule`, `Builder` independently. - -The Rule base class is purely nominal — deleting it removes zero -behavior. - -## Scope - -Deepen `Rule` to own: -- The blank-value guard (overridable via `skip_blank?`). -- The reference-kind predicate. -- The code-column predicate. -- A canonical "applies + call + message" template that subclasses - override via small focused methods (`value_passes?(value, context)`). - -## Approach - -```ruby -class Opencdd::Validator::Rule - REFERENCE_VALUE_KINDS = %i[identifier_ref set_of_refs class_ref].freeze - - def applies?(context) - true - end - - def call(value, context) - return true if blank?(value) && skip_blank? - value_passes?(value, context) - end - - def message(value, context) - "#{id}: validation failed for #{value.inspect}" - end - - # ── Hooks for subclasses ──────────────────────────────────── - def value_passes?(value, context) - raise NotImplementedError - end - - # Most rules skip empty values — the MandatoryRule overrides. - def skip_blank? = true - - # ── Shared predicates ─────────────────────────────────────── - def blank?(value) - value.nil? || value.to_s.strip.empty? - end - - def code_column?(context) - context.column_iri == Opencdd::MetaClasses.code_property_id_for(context.entity.meta_class_irdi&.code) - end - - def reference_kind?(context) - REFERENCE_VALUE_KINDS.include?(context.value_kind) - end -end -``` - -Migrate each rule subclass to override `value_passes?` only. - -## Acceptance - -- [x] Rule base owns shared guards. -- [x] Each subclass overrides `value_passes?` (smaller surface). -- [x] `code_column?` and `reference_kind?` defined once. -- [x] No spec regression. - -## Dependencies - -None. diff --git a/TODO.impl/28-builder-split.md b/TODO.impl/28-builder-split.md deleted file mode 100644 index 6d8573f..0000000 --- a/TODO.impl/28-builder-split.md +++ /dev/null @@ -1,61 +0,0 @@ -# Plan 28 — Split Cddal::Builder — import pipeline vs. property assembler - -## Why - -`Opencdd::Cddal::Builder` (414 lines) owns two structurally -independent pipelines: - -1. **Import pipeline** (lines 110-183): cycle detection, recursive - sub-document build, bare/qualified/selective scoping. -2. **Property assembly pipeline** (lines 247-411): assignment-to-properties - compilation, value serialization, reference resolution. - -They share `@database` and `@alias_table` state but otherwise -decompose independently. The interleaving makes both harder to -reason about — adding a new import feature (e.g. versioning) -shouldn't touch property assembly, and vice versa. - -## Scope - -Extract two collaborators behind the Builder: - -- `Opencdd::Cddal::ImportResolver` — owns module resolution, - cycle detection, scope application. Takes a database, resolver, - fetcher, and source_file; returns after applying all imports. -- `Opencdd::Cddal::PropertyAssembler` — owns assignment-to-properties - compilation: alias resolution, value serialization, reference - resolution. - -The Builder becomes a thin orchestrator that wires them in order. - -## Approach - -```ruby -class Opencdd::Cddal::Builder - def build(document) - document = wrap_document(document) - apply_alias_declarations(document) - apply_meta_class_declarations(document) - Opencdd::Cddal::ImportResolver.new(@database, resolver: @resolver, ...).apply(document) - register_instance_symbols(document) - instantiate_entities(document) - Opencdd::Cddal::PropertyAssembler.new(@database, alias_table: @alias_table).resolve_references - @database.finalize! - @database - end -end -``` - -The Builder remains the public entry. Internal seams let each -collaborator be tested independently. - -## Acceptance - -- [x] `ImportResolver` extracted; tested with synthetic ASTs. -- [x] `PropertyAssembler` extracted; tested with synthetic properties. -- [x] Builder drops below 200 lines. -- [x] All 757 specs pass. - -## Dependencies - -None. Independent of #26 / #27. diff --git a/TODO.impl/29-visitor-traversal-seam.md b/TODO.impl/29-visitor-traversal-seam.md deleted file mode 100644 index 4047969..0000000 --- a/TODO.impl/29-visitor-traversal-seam.md +++ /dev/null @@ -1,48 +0,0 @@ -# Plan 29 — Visitor traversal strategy seam - -## Why - -`Opencdd::Visitor#visit_database` hardcodes traversal order -(classes → properties → units → value_lists → value_terms → -relations → view_controls). Adding a new entity type, filtering, -or changing order requires editing the base class. - -Sort-by-code is duplicated across Visitor, Json exporter, Mermaid -exporter, and Cddal::Serializer (~8 sites). - -## Scope - -Introduce a `Traversal` strategy: -- Iterate entity types in registry order by default. -- Accept a `filter:` and `order:` proc for variations. -- Sort-by-code lives on Traversal, not on each caller. - -## Approach - -```ruby -class Opencdd::Visitor - def visit_database(db, &block) - Opencdd::EntityTraversal.each_typed(db, &block) - end -end - -module Opencdd::EntityTraversal - module_function - def each_typed(db) - db.entities.group_by(&:type).sort.each do |type, entities| - entities.sort_by { |e| e.code.to_s }.each { |e| yield(e) } - end - end -end -``` - -## Acceptance - -- [x] Sort-by-code lives in `EntityTraversal`, not duplicated. -- [x] Visitor's `visit_database` delegates. -- [x] Mermaid exporter doesn't override `visit_database` to skip sorts. -- [x] All specs pass. - -## Dependencies - -None. diff --git a/TODO.impl/30-parsehelpers-fold.md b/TODO.impl/30-parsehelpers-fold.md deleted file mode 100644 index 8ec4f22..0000000 --- a/TODO.impl/30-parsehelpers-fold.md +++ /dev/null @@ -1,44 +0,0 @@ -# Plan 30 — ParseHelpers folds into StructuredValues + FieldReader - -## Why - -Post-P17, `StructuredValues.unwrap_and_split` is the SSOT for -brace/paren splitting. But `ParseHelpers` still carries: - -- `unwrap_delimiters`, `paren_wrapped?`, `brace_wrapped?` -- `parse_pair_list`, `parse_synonym_tuples` -- `SynonymTupleScanner` (uses `StringScanner`) - -And `FieldReader` has its own inline delimiter stripping. - -Three code paths for the same wire format (Parcel `{a,b,c}`, -CDDAL `{a,b,c}`, JSON-internal arrays). - -## Scope - -Migrate the three ParseHelpers users (`Klass#properties_on_class`, -`Property#active_for?`, others via `include ParseHelpers`) to -`StructuredValues` directly. Remove the duplicate parsing from -`FieldReader`. - -Keep `ParseHelpers` as a private mixin for legacy callers but mark -it for removal. - -## Approach - -1. Replace `parse_irdi_list(raw)` callers with - `StructuredValues.parse_ref_set(raw)`. -2. Replace `parse_string_list(raw)` callers with - `StructuredValues.unwrap_and_split(raw)`. -3. Move `SynonymTupleScanner` into `StructuredValues.parse_synonyms`. -4. Mark `ParseHelpers` deprecated. - -## Acceptance - -- [x] `StructuredValues` is the SSOT. -- [x] No new callers of `ParseHelpers`. -- [x] All specs pass. - -## Dependencies - -None. Foundational — do first. diff --git a/TODO.impl/31-codegen-specs.md b/TODO.impl/31-codegen-specs.md deleted file mode 100644 index ee6d5d6..0000000 --- a/TODO.impl/31-codegen-specs.md +++ /dev/null @@ -1,38 +0,0 @@ -# Plan 31 — Codegen::Ts spec coverage + in-memory seam - -## Why - -`Opencdd::Codegen::Ts` (179 lines) is the only production module -in `lib/opencdd/` with zero spec files. Writes to disk. Duplicates -schema knowledge from `PropertyIds::REGISTRY` and `MetaClasses` as -string interpolation. - -## Scope - -Add an in-memory emit seam so tests can assert generated output -without touching the filesystem. - -## Approach - -```ruby -class Opencdd::Codegen::Ts - def generate_property_ids # → String (no file write) - def generate_meta_classes # → String - def write_file(...) # → wraps the above + File.write -end -``` - -Spec the generated strings: assert they parse as valid TS, include -expected constants, and round-trip with `rake generate_ts`. - -## Acceptance - -- [x] `spec/codegen/ts_spec.rb` exists. -- [x] Generate methods return strings (no disk). -- [x] File write is the thin wrapper. -- [x] Specs cover: PropertyIds registry contents, MetaClasses - registry contents, header/footer, freeze declarations. - -## Dependencies - -None. diff --git a/TODO.impl/32-languages-instance-rule-specs.md b/TODO.impl/32-languages-instance-rule-specs.md deleted file mode 100644 index ada03af..0000000 --- a/TODO.impl/32-languages-instance-rule-specs.md +++ /dev/null @@ -1,33 +0,0 @@ -# Plan 32 — Specs for Languages + InstanceRule - -## Why - -`Opencdd::Languages` (70 lines) and `Opencdd::InstanceRule` (70 -lines) have zero spec coverage. `InstanceRule` has cartesian-product -expansion with exception filtering — exactly the algorithmic code -that needs specs. - -## Scope - -Two new spec files: - -- `spec/languages_spec.rb` — source/translation language list, - multilingual value access, equality. -- `spec/instance_rule_spec.rb` — cartesian product across groups, - exception filtering, empty group edge cases. - -## Approach - -Use the IEC 61360 manual's example: 3 groups with cardinalities -2×3×2 → 12 instances; one exception → 11. Spec both the generate -path and the exception filter. - -## Acceptance - -- [x] `spec/languages_spec.rb` exists with ≥5 specs. -- [x] `spec/instance_rule_spec.rb` exists with ≥5 specs. -- [x] All specs pass. - -## Dependencies - -None. diff --git a/TODO.impl/33-lutaml-model-migration.md b/TODO.impl/33-lutaml-model-migration.md deleted file mode 100644 index 4fed5ad..0000000 --- a/TODO.impl/33-lutaml-model-migration.md +++ /dev/null @@ -1,52 +0,0 @@ -# Plan 33 — lutaml-model migration with CDD-native YAML model - -## Why - -User decision: migrate to lutaml-model with YAML as the canonical -persistence format. The model must be "fully native to CDD ontology" -— using CDD semantic names (preferred_name, superclass, class_type) -not wire-format keys (MDC_P004, MDC_P010, MDC_P011). - -## Approach - -Adapter pattern: create `Opencdd::Model::YamlEntity` and -`Opencdd::Model::YamlDatabase` extending `Lutaml::Model::Serializable` -with typed attributes and CDD-native names. Conversion methods bridge -between the YAML model and the existing Entity model. No changes to -existing Entity/readers/writers — 813 specs stay green. - -### YAML shape - -```yaml ---- -irdi: 0112/2///61360_4#AAA001 -type: class -code: AAA001 -preferred_name: - en: Vehicle - fr: Véhicule -class_type: ITEM_CLASS -superclass: UNIVERSE -applicable_properties: - - vehicle_length - - vehicle_weight -``` - -This is "fully native to CDD ontology" — semantic attribute names, -multilingual as nested Hash, sets as Arrays. No MDC_P### keys. - -## Scope - -- `lib/opencdd/model/yaml_entity.rb` — typed lutaml-model class -- `lib/opencdd/model/yaml_database.rb` — database-level YAML wrapper -- `lib/opencdd/model.rb` — namespace autoload -- Conversion methods on Entity and Database -- Specs for YAML round-trip - -## Acceptance - -- [x] lutaml-model dependency added -- [x] YamlEntity with typed CDD-native attributes -- [x] Database#to_yaml / Database.from_yaml -- [x] OceanRunner round-trips through YAML -- [x] All 813 existing specs pass diff --git a/TODO.impl/34-per-entity-yaml-persistence.md b/TODO.impl/34-per-entity-yaml-persistence.md deleted file mode 100644 index cb7ad1b..0000000 --- a/TODO.impl/34-per-entity-yaml-persistence.md +++ /dev/null @@ -1,103 +0,0 @@ -# Plan 34 — Per-entity YAML persistence with lutaml-store - -## Why - -YAML is the canonical persistence format (plan 33). Per-entity YAML -files make dictionaries diff-friendly at the entity level — one -`git diff` shows exactly which entity changed. - -The current `EntityStore` (85 lines) uses `Lutaml::Store` but bypasses -its API — calling `store.store.adapter.set/get` directly. This is a -shallow use of the framework: we get the filesystem adapter but none -of the model registry, CRUD, or polymorphic dispatch benefits. - -The deepening: register Entity as a model with `DatabaseStore` and -use the CRUD API (`save`, `fetch`, `all`) properly. After plan 35, -Entity IS the `Lutaml::Model::Serializable`, so it plugs in directly. - -## Directory layout - -``` -data/ -├── AA/ -│ ├── AAA001.yaml -│ └── AAA001.meta -├── AB/ -│ └── ABB002.yaml -└── ... -``` - -The FileSystem adapter shards by the first 2 chars of the key for -scalability (avoids directory-size limits on large dictionaries like -iec61987 with 11,831 entities). Extension configured as `.yaml`. - -The `.meta` sidecar carries SHA256 integrity metadata — tamper -detection for dictionary data. - -## Key scheme - -Each entity's storage key is its IRDI code (e.g., `AAA001`), prefixed -with the entity type for uniqueness (codes can collide across types): - -```ruby -def storage_key - "#{type}/#{code}" -end -``` - -The FileSystem adapter sanitizes this to `class_AAA001` (replacing `/` -with `_`), stored at `cl/class_AAA001.yaml`. - -## Approach - -```ruby -db.save_to_directory("data/") # writes per-entity YAML files -db2 = Opencdd::Database.load_from_directory("data/") # reads them back -``` - -Internally: -1. Create a `Lutaml::Store` with `DatabaseStore` and FileSystem adapter -2. Register `Opencdd::Entity` with polymorphic mapping -3. `save_database` iterates entities, calls `store.save(entity)` -4. `load_database` calls `store.all(Entity)`, feeds results into Database - -## Polymorphic dispatch - -The store registers `Opencdd::Entity` as the base model with a -polymorphic discriminator (`type` attribute) mapping to subclasses: - -```ruby -models: [{ - model: Opencdd::Entity, - key: :storage_key, - polymorphic: { - discriminator: :type, - mapping: { - class: Opencdd::Klass, - property: Opencdd::Property, - unit: Opencdd::Unit, - value_list: Opencdd::ValueList, - value_term: Opencdd::ValueTerm, - relation: Opencdd::Relation, - view_control: Opencdd::ViewControl, - } - } -}] -``` - -The `ModelSerializer` stores `_class` metadata in each record for -type-safe deserialization. - -## Acceptance - -- [x] EntityStore uses `DatabaseStore#save_all` / `#load_all` (not adapter.set/get) -- [x] Entity::Yaml registered with key :irdi -- [x] Directory layout: `entities/.yaml` (flat, `.yaml` extension) -- [x] `save_to_directory` writes one YAML per entity -- [x] `load_from_directory` reads them all back via `load_all` -- [x] OceanRunner round-trips through directory persistence -- [x] All existing specs pass - -## Dependencies - -- Plan 35 (Entity as Lutaml::Model::Serializable — the registered model) diff --git a/TODO.impl/35-entity-extends-lutaml-model.md b/TODO.impl/35-entity-extends-lutaml-model.md deleted file mode 100644 index db5a87e..0000000 --- a/TODO.impl/35-entity-extends-lutaml-model.md +++ /dev/null @@ -1,140 +0,0 @@ -# Plan 35 — Entity extends Lutaml::Model::Serializable - -## Why - -`Opencdd::Entity` currently stores all field values in a `@properties` -Hash keyed by IEC 62656-1 wire-format IDs (`MDC_P004`, `MDC_P010`, etc.). -A separate `Opencdd::Model::YamlEntity` adapter (196 lines) duplicates -every field declaration to provide a CDD-native YAML shape with semantic -attribute names (`preferred_name`, `superclass`, `class_type`). - -The adapter is **shallow**: its interface (semantic YAML shape) is -nearly as complex as its implementation (the `from_entity` / `to_entity` -conversion that maps every MDC_P### ↔ semantic name pair). The -**deletion test** confirms this — delete `YamlEntity` and the semantic -name mapping scatters back across Entity, Database, and every caller -that touches YAML. The mapping is earning its keep, but in the wrong -place. - -The deepening: fold the adapter INTO Entity. Entity becomes the -`Lutaml::Model::Serializable`. One representation, one source of truth. - -## Critical framework constraint - -`Lutaml::Model::Serializable` reads attribute values for serialization -via `instance_variable_get(:"@#{attr_name}")` — it reads ivars -directly, NOT through getter methods. This means: - -- Overriding getters to delegate to `@properties` does NOT work. -- The typed attributes MUST hold the canonical values as real ivars. -- The `@properties` Hash must become a **derived view**, not the store. - -This rules out a "thin wrapper" approach. The migration is real. - -## Design - -### Field DSL → lutaml-model attributes - -The existing `field` DSL becomes the single declaration point. Each -call: -1. Declares a `Lutaml::Model` typed `attribute` (creates `@` ivar) -2. Registers CDD metadata in `FieldRegistry` (property_id, value_kind, - multilingual, json_key) for exporters, validators, and TS codegen -3. Maps the value_kind to a lutaml-model type: - -| value_kind | lutaml type | storage | -|----------------------|-------------|-----------------------| -| `:string` | `:string` | `String` | -| `:irdi` | `:string` | IRDI string form | -| `:set_of_refs` | `:string` | `Array` | -| `:string_list` | `:string` | `Array` | -| `:synonym_pairs` | `:hash` | `Hash` | -| `:integer` | `:integer` | `Integer` | -| `:boolean` | `:boolean` | `Boolean` | -| multilingual fields | `:hash` | `Hash` | - -Synthetic fields (block-defined computed values like `raw_properties`, -`dates`, `version_history`) are NOT lutaml-model attributes — they stay -as regular Ruby methods. They don't participate in serialization. - -### `@properties` Hash → derived view - -```ruby -def properties - @properties_cache ||= build_properties_hash -end -``` - -`build_properties_hash` walks declared fields + `extra` and produces -the MDC_P###-keyed Hash. Writes go through `write_property!` which -updates the typed ivar and invalidates the cache. - -For multilingual fields, the Hash expands `@preferred_name = {en: "X", -fr: "Y"}` to `"MDC_P004.en" => "X", "MDC_P004.fr" => "Y"`. - -### Catch-all for unknown property IDs - -An `extra` attribute (`Hash`) holds raw property IDs -from the .xls that don't have field declarations (e.g. `C016`, -`C011`, `C002`). This preserves lossless import. - -### Construction - -`Entity.new(irdi:, properties: {...})` still works — the constructor -distributes the Hash to typed ivars. `from_row` unchanged externally. - -### YAML shape (semantic CDD names) - -Serialization uses lutaml-model's YAML adapter. The `mapping` block -maps attribute names to YAML keys: - -```yaml ---- -irdi: 0112/2///61360_4#AAA001 -type: class -code: AAA001 -preferred_name: - en: Vehicle - fr: Véhicule -class_type: ITEM_CLASS -superclass: 0112/2///61360_4#AAA000 -applicable_properties: - - 0112/2///61360_4#AAAP001 -extra: - C016: released -``` - -### Polymorphic dispatch - -Entity subclasses (Klass, Property, Unit, ValueList, ValueTerm, -Relation, ViewControl) inherit the base attributes and add their own. -A `type` attribute (`:class`, `:property`, etc.) discriminates for -deserialization. lutaml-store's polymorphic registry maps `type` → -subclass. - -## Phases - -1. Add typed attrs alongside Hash — both populated, Hash still canonical for reads -2. Flip: typed attrs canonical, `properties` derives Hash -3. Remove YamlEntity adapter — Entity serializes directly -4. Update EntityStore / YamlDatabase to use Entity - -## Acceptance - -- [x] Entity extends Lutaml::Model::Serializable -- [x] Field DSL declares lutaml-model typed attributes (on Entity::Yaml) -- [x] `@properties` Hash stays as field-DSL store; YAML typed attrs on Entity::Yaml -- [x] `write_property!` routes all mutations consistently -- [x] Multilingual fields round-trip as Hash -- [x] `extra` hash catches unknown property IDs -- [x] Entity#to_yaml / Entity.from_yaml work natively (via Entity::Yaml) -- [x] YamlEntity adapter deleted (deepened into Entity::Yaml) -- [x] OceanRunner round-trips through Entity YAML -- [x] All 835+ specs pass -- [x] No forbidden patterns (no send-to-private, no ivar get/set - across objects, no respond_to?, no require_relative) - -## Dependencies - -- Plan 33 (lutaml-model migration — proves the attribute shape) -- Plan 34 (lutaml-store — uses Entity as registered model) diff --git a/TODO.impl/36-database-split.md b/TODO.impl/36-database-split.md deleted file mode 100644 index e9d3798..0000000 --- a/TODO.impl/36-database-split.md +++ /dev/null @@ -1,114 +0,0 @@ -# Plan 36 — Database class split - -## Why - -Database is 719 lines, mixing entity storage, Parcel integration, -graph queries, mutations, finalization, and YAML persistence. After -plan 34 (lutaml-store handles persistence) and plan 35 (Entity is a -typed model), Database can slim down to a facade over the store + -graph invariants. - -## Current responsibilities (by method count) - -| Concern | Methods | Lines | -|----------------------|----------------------------------------------------|-------| -| Entity storage/index | add_entity, find, find_by_code, entities, count | ~80 | -| Parcel integration | add_workbook, add_dictionary, drop_dictionary, ... | ~120 | -| Graph queries | categorical_classes, instances_of, properties_of | ~100 | -| Graph finalization | finalize!, link_class_hierarchy!, link_property_* | ~90 | -| Mutations | rename_entity, merge, apply_change_request, remove | ~90 | -| Reference resolution | resolve_reference, coerce_entity, coerce_irdi | ~50 | -| Persistence | to_yaml, from_yaml, save_to_directory, ... | ~30 | -| Symbol table | register_symbol, bind_symbol, rebuild_symbol_table | ~40 | -| Misc | to_s, inspect, each, etc. | ~30 | - -## Target architecture - -Split into focused modules mixed into Database. Each module is a -concern, in its own file under `lib/opencdd/database/`. - -``` -lib/opencdd/ -├── database.rb # Core: identity, indexing, initialize -├── database/ -│ ├── parcel_integration.rb # add_workbook, add_dictionary, ... -│ ├── queries.rb # find, coerce, powertype API, walkers -│ ├── finalization.rb # finalize!, link_* methods -│ ├── mutations.rb # rename_entity, merge, apply_change_request -│ └── persistence.rb # to_yaml, from_yaml, save/load_directory -``` - -### Database (core) — ~200 lines - -```ruby -class Database - include Enumerable - include Opencdd::Database::ParcelIntegration - include Opencdd::Database::Queries - include Opencdd::Database::Finalization - include Opencdd::Database::Mutations - include Opencdd::Database::Persistence - - attr_reader :workbooks, :unresolved_refs, :alias_table - - def initialize - # Initialize all indexes - end - - # Entity storage: add_entity, entities, count, each - # Type-partitioned accessors: classes, properties, units, ... -end -``` - -### ParcelIntegration — ~120 lines - -Workbook ingestion, dictionary lifecycle (add/drop/register_external), -sheetmap construction, translation-language parsing. - -### Queries — ~100 lines - -`find`, `find_by_code`, `find_by_name`, `resolve_reference`, -`coerce_entity`, `coerce_irdi`, powertype API (`categorical_classes`, -`instances_of`, `valid_class_reference?`), graph walkers -(`properties_of`, `classes_with_property`, `value_list_of`, -`relations_for`, `functions_involving`), tree accessors. - -### Finalization — ~90 lines - -`finalize!`, `normalize_reference_collections!`, -`each_reference_collection`, `link_class_hierarchy!`, -`link_property_classes!`, `link_value_lists!`, `rebuild_symbol_table!`. - -### Mutations — ~90 lines - -`rename_entity`, `merge`, `apply_change_request`, `remove_by_irdi`, -`apply_view_control`, back-reference rewriting. - -### Persistence — ~30 lines - -`to_yaml`, `from_yaml`, `save_to_directory`, `load_from_directory`. -Thin delegates to `EntityStore` and `YamlDatabase` (or directly to -Entity after plan 35). - -## Module shared state - -Modules access Database's internal state through `attr_reader` -accessors on the core class. No `instance_variable_get` across -objects (encapsulation rule). Each module documents which readers -it depends on. - -## Acceptance - -- [x] `database.rb` under 250 lines (core only) — 150 lines -- [x] Each concern in its own file under `lib/opencdd/database/` -- [x] All modules use autoload from `lib/opencdd/database.rb` -- [x] No `send` to private methods across modules -- [x] No `instance_variable_get/set` across objects -- [x] No `require_relative` (autoload only) -- [x] All 835+ specs pass unchanged -- [x] Database#public_methods surface unchanged (pure refactor) - -## Dependencies - -- Plan 34 (persistence module delegates to EntityStore) -- Plan 35 (Entity as typed model simplifies entity handling) diff --git a/bin/lint-no-raw-mdc b/bin/lint-no-raw-mdc index c9b91f6..1423d1e 100755 --- a/bin/lint-no-raw-mdc +++ b/bin/lint-no-raw-mdc @@ -1,4 +1,6 @@ -#!/usr/bin/env bash +#!/usr/bin/env ruby +# frozen_string_literal: true +# # lint-no-raw-mdc — fail if any raw MDC_P### / MDC_C### / EXT_P### / # EXT_C### / CIM_P### string literal appears in lib/opencdd/ outside # the designated SSOT files. @@ -14,29 +16,50 @@ # # Comment lines (lines whose first non-whitespace character is '#') # are also exempt. -# -set -euo pipefail - -excludes=( - ':!lib/opencdd/property_ids.rb' - ':!lib/opencdd/meta_class.rb' - ':!lib/opencdd/alias_table.rb' - ':!lib/opencdd/parcel/sheet_schema.rb' - ':!lib/opencdd/cddal/generated_parser.rb' -) - -# Collect hits, filtering out Ruby comment-only lines. -hits=$(git grep -nE '"(MDC_[CP][0-9]+(_[0-9]+)?|EXT_[CP][0-9]+|CIM_P[0-9]+)"' \ - -- 'lib/opencdd/**/*.rb' "${excludes[@]}" 2>/dev/null \ - | grep -vE '^[^:]+:[0-9]+:\s*#' \ - || true) - -if [ -n "$hits" ]; then - echo "ERROR: raw MDC/EXT/CIM literal found outside designated SSOT files." >&2 - echo "Use Opencdd::PropertyIds::REGISTRY / Opencdd::MetaClasses::REGISTRY instead." >&2 - echo "" >&2 - echo "$hits" >&2 + +require "open3" + +EXCLUDED_PATHS = %w[ + lib/opencdd/property_ids.rb + lib/opencdd/meta_class.rb + lib/opencdd/alias_table.rb + lib/opencdd/parcel/sheet_schema.rb + lib/opencdd/cddal/generated_parser.rb +].freeze + +# Match "MDC_P###", "MDC_C###", "EXT_P###", "EXT_C###", "CIM_P###" +# with optional _### sub-ids, as a double-quoted Ruby string literal. +# Source string is POSIX ERE (not Ruby regex) — git grep -E requires it. +LITERAL_PATTERN_SOURCE = '"(MDC_[CP][0-9]+(_[0-9]+)?|EXT_[CP][0-9]+|CIM_P[0-9]+)"'.freeze + +# Each git grep hit looks like: :: +# Skip comment-only lines (content's first non-space char is '#'). +HIT_LINE_PATTERN = /\A(?[^:]+):(?\d+):(?.*)\z/m.freeze + +# lib/opencdd/**/*.rb minus the excluded SSOT files. +pathspecs = ["lib/opencdd/**/*.rb"].concat(EXCLUDED_PATHS.map { |p| ":!#{p}" }) + +cmd = ["git", "grep", "-n", "-E", LITERAL_PATTERN_SOURCE] + ["--"] + pathspecs +stdout, _stderr, status = Open3.capture3(*cmd) + +# git grep returns 1 when there are no matches, which is the success +# case. Treat anything other than 0/1 as a real failure. +unless [0, 1].include?(status.exitstatus) + warn "ERROR: git grep failed (exit #{status.exitstatus})" + exit 2 +end + +hits = stdout.each_line.map(&:chomp).reject do |line| + match = HIT_LINE_PATTERN.match(line) + match && match[:content].lstrip.start_with?("#") +end + +if hits.any? + warn "ERROR: raw MDC/EXT/CIM literal found outside designated SSOT files." + warn "Use Opencdd::PropertyIds::REGISTRY / Opencdd::MetaClasses::REGISTRY instead." + warn "" + hits.each { |hit| warn hit } exit 1 -fi +end exit 0 diff --git a/docs/src/content/docs/parcel-format.md b/docs/src/content/docs/parcel-format.md index f4a88a4..2838902 100644 --- a/docs/src/content/docs/parcel-format.md +++ b/docs/src/content/docs/parcel-format.md @@ -291,7 +291,7 @@ round-trip. instead. - **Live CDD search (F20)** — the `cdd.iec.ch` backend is gated by AWS WAF; can't be reached from a library context. Use the Python - harvester in `opencdd/cdd-data` to fetch first. + harvester in `opencdd/data-private` to fetch first. ## See also diff --git a/docs/src/content/docs/reference/features.md b/docs/src/content/docs/reference/features.md index 27727ae..d8278ae 100644 --- a/docs/src/content/docs/reference/features.md +++ b/docs/src/content/docs/reference/features.md @@ -173,7 +173,7 @@ Opencdd::Validator.class_hierarchy_acyclic?(database) These features are NOT in the gem: - Live CDD search (`cdd.iec.ch`) — AWS WAF blocks library access; use - the Python harvester in `opencdd/cdd-data` instead. + the Python harvester in `opencdd/data-private` instead. - External parcel linking (ParcelMaker F19) — multi-file Parcel sessions. Use CDDAL imports instead. - CIM/RDF export — future work. diff --git a/lib/cdd.rb b/lib/cdd.rb index babd72e..2ce3fb3 100644 --- a/lib/cdd.rb +++ b/lib/cdd.rb @@ -6,6 +6,6 @@ require "opencdd" # Alias for backward compatibility. The module was renamed from -# Cdd to Opencdd, but many callers (cdd-data Rakefile, specs, etc.) +# Cdd to Opencdd, but many callers (data-private Rakefile, specs, etc.) # still reference Cdd::. This alias lets them keep working. Cdd = Opencdd unless defined?(Cdd)