CONCAT(K) reintroduced as the folding operation; JOIN(K) redefined as the record product

CHANGELOG.md

@@ -7,6 +7,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed (breaking)
+- **`JOIN(K)` is now the record product; the folding operation is `CONCAT(K)`.** Up to 0.5.x
+  `JOIN(K)` *folded* records: the records of the provided anchors were concatenated to the anchor's
+  record (one wide record, the number of records strictly decreased) — what `pandas.concat(axis=1)`
+  does, not what a join does. The operation keeps that semantics under its original name
+  `CONCAT(K)` (RTL `CONCAT`, `CONCAT(k1, k2, …)`; `ActionSpec.concat(…)`, `Rtl.concat(…)`,
+  `ConcatOperation`, `OperationType.CONCAT`, `WorkingState.applyConcat`). `JOIN(K)` is redefined
+  as the **record product**: every record of the anchor is combined with every record of the
+  provided anchors — a cross product for `K = ∅`, an equi-join on the key positions `K` otherwise;
+  a named attribute shared by the two records is a natural-join condition (the pair is kept only
+  if the values agree, the attribute occurs once). The provided anchors become *joined-away*
+  (`J`): they are excluded from the recordset but their records stay available, so several
+  anchors may join the same records irrespective of action order. Migration: replace `JOIN(K)`
+  by `CONCAT(K)` in every existing pattern (`JOIN` → `CONCAT`, `JOIN(0)` → `CONCAT(0)`, …); the
+  corpus tasks 016, 023, 025, 033, 046, 047, 050, 053, 069, 094, 097, 098 were migrated this way
+  and produce byte-identical recordsets
+- **`CONCAT(K)` no longer deduplicates named attributes.** The former `JOIN(K)` silently kept the
+  first occurrence of a named attribute shared by the concatenated records, which hid
+  specification errors (a pattern that joined several stacked cells to one token lost all but
+  the first `value`). Now a shared named attribute (apart from the key positions `K`) is a
+  precondition violation: the action has **no effect** — both records survive — and a
+  `Diagnostic` is recorded (`TableInterpreter.diagnostics()`); `withStrictPreconditions(true)`
+  raises an `IllegalStateException` instead. The key positions are likewise checked (`compat_K`).
+  In the corpus this changed one pattern: task 098 lists its full group key,
+  `(BW&STR)*->CONCAT(0,1,2,3)` instead of `JOIN(0,1)` — the named attributes `A`/`B` repeated on
+  every row of a group are part of the key, not duplicates to drop; the expected recordsets are unchanged
+- **Working state: `rec` is multi-valued, recordsets are multisets.** `rec(ι)` is a non-empty
+  sequence of item-based records (`WorkingState.rec(item)` → `List<List<Item>>`, a single record
+  until a join multiplies it); the working state gains the component `J` (`WorkingState.allJoined()`,
+  `isJoined(item)`), and `WorkingState.allRec()` returns the **live** anchors `dom(rec) \ J` only —
+  exactly what recordset extraction sees. `SchemaConstructionStrategy.buildVisitOrder` visits
+  `(anchor, record, position)` triples. The order of records in a `Recordset` is a documented
+  default (anchor visit order, then nested-loop order of the join), not part of the semantics
+- Grammar: `concatOp : CONCAT (LPAREN INT (COMMA INT)* RPAREN)?`, keyword `CONCAT`; the ATP→RTL
+  serializer emits `CONCAT(k1, k2)`; the VS Code grammar highlights `CONCAT`
+
+### Added
+- `Diagnostic` (`ru.icc.regtab.itm.semantics`), `WorkingState.diagnostics()`,
+  `TableInterpreter.diagnostics()`, `TableInterpreter.withStrictPreconditions(boolean)`,
+  `WorkingState(boolean strictPreconditions)`
+- Conformance corpus, semantic section: `concat_by_key` (task 016 shape), `join_product`
+  (explode × stack — six records from two rows), `join_equi_key` (`JOIN(0)` on a positional key)
+- Docs: Example 6 (record join) and a `CONCAT` vs `JOIN` comparison in `examples.md`;
+  `rtl-reference.md`, `model/itm.md`, `model/atp.md`, `api.md`, `architecture.md`,
+  `embedded-rtl.md` updated
+
 ## [0.5.3] - 2026-08-27
 
 ### Changed

conformance/VERSION

@@ -1,2 +1,5 @@
-generated: 2026-08-26
+generated: 2026-08-28
 sources: RtlTask001..150 + curated extras
+note: CONCAT(K) reintroduced (the former JOIN(K)); JOIN(K) redefined as the record product;
+      semantic cases concat_by_key, join_product, join_equi_key added; task_098 lists its full
+      group key CONCAT(0,1,2,3)

conformance/positive/task_016.expected.rtl

@@ -1 +1 @@
-[ [ VAL : RT->REC, (BW & STR)*->JOIN(0) ] [ VAL ] ]+
+[ [ VAL : RT->REC, (BW & STR)*->CONCAT(0) ] [ VAL ] ]+

conformance/positive/task_016.rtl

@@ -1 +1 @@
-[ [VAL : RT->REC, BW&STR*->JOIN(0)] [VAL] ]+
+[ [VAL : RT->REC, BW&STR*->CONCAT(0)] [VAL] ]+

conformance/positive/task_023.expected.rtl

@@ -1 +1 @@
-{ [ [ VAL : ''->AVP, SR*->REC, (BW & STR)*->JOIN(0) ] [ ATTR : RT->SUFFIX ] [ AUX ] [ VAL : SR->AVP ] ]{3} }+
+{ [ [ VAL : ''->AVP, SR*->REC, (BW & STR)*->CONCAT(0) ] [ ATTR : RT->SUFFIX ] [ AUX ] [ VAL : SR->AVP ] ]{3} }+

conformance/positive/task_023.rtl

@@ -1 +1 @@
-{ [ [VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+
+{ [ [VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+

conformance/positive/task_025.expected.rtl

@@ -1 +1 @@
-<SPLIT("/")> [ [ VAL : RT->SUFFIX("/"), (RT & C+2..)*->REC('/'), (BW & STR)*->JOIN(0) ] [ VAL ]+ ]+
+<SPLIT("/")> [ [ VAL : RT->SUFFIX("/"), (RT & C+2..)*->REC('/'), (BW & STR)*->CONCAT(0) ] [ VAL ]+ ]+

conformance/positive/task_025.rtl

@@ -1 +1 @@
-[ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->JOIN(0)] [VAL]+ ]+
+[ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->CONCAT(0)] [VAL]+ ]+

conformance/positive/task_033.expected.rtl

@@ -1 +1 @@
-[ [ VAL : SR*->REC, (BW & STR)*->JOIN(0) ] [ VAL ]+ ]+
+[ [ VAL : SR*->REC, (BW & STR)*->CONCAT(0) ] [ VAL ]+ ]+

conformance/positive/task_033.rtl

@@ -1 +1 @@
-[ [VAL : SR*->REC, BW&STR*->JOIN(0)] [VAL]+ ]+
+[ [VAL : SR*->REC, BW&STR*->CONCAT(0)] [VAL]+ ]+

conformance/positive/task_046.expected.rtl

@@ -1 +1 @@
-{ [ [ !BLANK? VAL : ''->AVP, SR*->REC, (BW & STR)*->JOIN(0) ] [ !BLANK? ATTR ] [ !BLANK? VAL : SR->AVP ] ]+ }+
+{ [ [ !BLANK? VAL : ''->AVP, SR*->REC, (BW & STR)*->CONCAT(0) ] [ !BLANK? ATTR ] [ !BLANK? VAL : SR->AVP ] ]+ }+

conformance/positive/task_046.rtl

@@ -1 +1 @@
-{ [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+
+{ [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+

conformance/positive/task_047.expected.rtl

@@ -1 +1 @@
-{ [ [ !BLANK? VAL : SR*->REC, (BW & STR)*->JOIN(0) ] [ !BLANK? VAL ] ]+ }+
+{ [ [ !BLANK? VAL : SR*->REC, (BW & STR)*->CONCAT(0) ] [ !BLANK? VAL ] ]+ }+

conformance/positive/task_047.rtl

@@ -1 +1 @@
-{ [ [!BLANK? VAL : SR*->REC, BW&STR*->JOIN(0)] [!BLANK? VAL] ]+ }+
+{ [ [!BLANK? VAL : SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? VAL] ]+ }+

conformance/positive/task_050.expected.rtl

@@ -1 +1 @@
-[ [ !BLANK? VAL : ''->AVP, SR*->REC, (BW & STR)*->JOIN(0) ] [ !BLANK? ATTR ] [ !BLANK? VAL : SR->AVP ] ]+
+[ [ !BLANK? VAL : ''->AVP, SR*->REC, (BW & STR)*->CONCAT(0) ] [ !BLANK? ATTR ] [ !BLANK? VAL : SR->AVP ] ]+

conformance/positive/task_050.rtl

@@ -1 +1 @@
-[ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+
+[ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+

conformance/positive/task_053.expected.rtl

@@ -1 +1 @@
-[ [] [ AUX ]+ ] [ [ VAL : ROW*->REC, (BW & STR)->JOIN(0), 'ID'->AVP ] { [ ATTR : AV->PREFIX("_") ] [ VAL : SR->AVP ] }+ ]+
+[ [] [ AUX ]+ ] [ [ VAL : ROW*->REC, (BW & STR)->CONCAT(0), 'ID'->AVP ] { [ ATTR : AV->PREFIX("_") ] [ VAL : SR->AVP ] }+ ]+

conformance/positive/task_053.rtl

@@ -1,3 +1,3 @@
 [ [] [AUX]+ ]
-[ [VAL : ROW*->REC, BW&STR->JOIN(0), 'ID'->AVP]
+[ [VAL : ROW*->REC, BW&STR->CONCAT(0), 'ID'->AVP]
   {[ATTR : AV->PREFIX('_')] [VAL : SR->AVP]}+]+

conformance/positive/task_069.expected.rtl

@@ -1 +1 @@
-[ { [ ATTR : SR->AVP, BW*->REC ] [ VAL #'1' : SR->AVP, BW*->REC, (ROW & #'1')*->JOIN ] [ VAL #'2' : SR->AVP, BW*->REC, (ROW & #'2')*->JOIN ] }* ] [ { [ ATTR : SR->AVP ] [ VAL : SR->AVP ]{2} }* ]*
+[ { [ ATTR : SR->AVP, BW*->REC ] [ VAL #'1' : SR->AVP, BW*->REC, (ROW & #'1')*->CONCAT ] [ VAL #'2' : SR->AVP, BW*->REC, (ROW & #'2')*->CONCAT ] }* ] [ { [ ATTR : SR->AVP ] [ VAL : SR->AVP ]{2} }* ]*

conformance/positive/task_069.rtl

@@ -1,3 +1,3 @@
 SR->AVP
-[ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->JOIN][VAL#'2': ROW&#'2'*->JOIN] }* ]
+[ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->CONCAT][VAL#'2': ROW&#'2'*->CONCAT] }* ]
 [          { [ATTR] [VAL]{2} }* ]*

conformance/positive/task_094.expected.rtl

@@ -1 +1 @@
-{ [ { [ !BLANK? VAL : COL*->REC, (ROW & C+1.. & STR)*->JOIN(0) ]+ [ BLANK ]? }+ ] } { [ { [ !BLANK? VAL ]+ [ BLANK ]? }+ ]+ [ [ BLANK ]+ ]? }+
+{ [ { [ !BLANK? VAL : COL*->REC, (ROW & C+1.. & STR)*->CONCAT(0) ]+ [ BLANK ]? }+ ] } { [ { [ !BLANK? VAL ]+ [ BLANK ]? }+ ]+ [ [ BLANK ]+ ]? }+

conformance/positive/task_094.rtl

@@ -1,3 +1,3 @@
-  [ { [!BLANK? VAL: COL*->REC, ROW&C+1..&STR*->JOIN(0)]+ [BLANK]? }+ ]
+  [ { [!BLANK? VAL: COL*->REC, ROW&C+1..&STR*->CONCAT(0)]+ [BLANK]? }+ ]
 { [ { [!BLANK? VAL]+ [BLANK]? }+ ]+
   [ [BLANK]+ ]? }+

conformance/positive/task_097.expected.rtl

@@ -1 +1 @@
-[ [ VAL : RT*->REC, (BW & STR)*->JOIN(0, 1) ] [ VAL ]+ ]+
+[ [ VAL : RT*->REC, (BW & STR)*->CONCAT(0, 1) ] [ VAL ]+ ]+

conformance/positive/task_097.rtl

@@ -1 +1 @@
-[ [VAL: RT*->REC, (BW&STR)*->JOIN(0,1)] [VAL]+ ]+
+[ [VAL: RT*->REC, (BW&STR)*->CONCAT(0,1)] [VAL]+ ]+

conformance/positive/task_098.expected.rtl

@@ -1 +1 @@
-[ [] [] [ ATTR ]+ ] [ [ VAL : RT*->REC, (BW & STR)*->JOIN(0, 1) ] [ VAL ] [ VAL : COL->AVP ]{2} [ VAL ]+ ]+
+[ [] [] [ ATTR ]+ ] [ [ VAL : RT*->REC, (BW & STR)*->CONCAT(0, 1, 2, 3) ] [ VAL ] [ VAL : COL->AVP ]{2} [ VAL ]+ ]+

conformance/positive/task_098.rtl

@@ -1,2 +1,2 @@
 [ []                                    []    [ATTR]+ ]
-[ [VAL: RT*->REC, (BW&STR)*->JOIN(0,1)] [VAL] [VAL: COL->AVP]{2} [VAL]+ ]+
+[ [VAL: RT*->REC, (BW&STR)*->CONCAT(0,1,2,3)] [VAL] [VAL: COL->AVP]{2} [VAL]+ ]+

conformance/semantic/concat_by_key/expected.csv

@@ -0,0 +1,3 @@
+"book","5","6","7"
+"cat","2","3","8"
+"dog","2","3","4"

conformance/semantic/concat_by_key/input.csv

@@ -0,0 +1,9 @@
+"book","5"
+"book","6"
+"book","7"
+"cat","2"
+"cat","3"
+"cat","8"
+"dog","2"
+"dog","3"
+"dog","4"

conformance/semantic/concat_by_key/options.json

@@ -0,0 +1 @@
+{ "expectedHasHeader": false }

conformance/semantic/concat_by_key/pattern.rtl

@@ -0,0 +1 @@
+[ [VAL : RT->REC, BW&STR*->CONCAT(0)] [VAL] ]+

conformance/semantic/join_equi_key/expected.csv

@@ -0,0 +1,3 @@
+"k","v","u"
+"X","5","kg"
+"Y","8","pc"

conformance/semantic/join_equi_key/input.csv

@@ -0,0 +1,3 @@
+"k","v","k","u"
+"X","5","X","kg"
+"Y","8","Y","pc"

conformance/semantic/join_equi_key/options.json

@@ -0,0 +1 @@
+{ "expectedHasHeader": true }

conformance/semantic/join_equi_key/pattern.rtl

@@ -0,0 +1,2 @@
+[ [ATTR]+ ]
+[ [VAL: COL->AVP, RT->REC, C2*->JOIN(0)] [VAL: COL->AVP] [VAL: COL->AVP, RT->REC] [VAL: COL->AVP] ]+

conformance/semantic/join_product/expected.csv

@@ -0,0 +1,7 @@
+"id","value","var"
+"a","1","x"
+"a","2","y"
+"b","1","x"
+"b","2","y"
+"c","3","x"
+"c","4","y"

conformance/semantic/join_product/input.csv

@@ -0,0 +1,3 @@
+"id","x","y"
+"a;b","1","2"
+"c","3","4"

conformance/semantic/join_product/options.json

@@ -0,0 +1 @@
+{ "expectedHasHeader": true }

conformance/semantic/join_product/pattern.rtl

@@ -0,0 +1,2 @@
+[ [ATTR] [VAL: 'var'->AVP]+ ]
+[ [(VAL: COL->AVP, ()->REC, RT*->JOIN){';'}] [VAL: 'value'->AVP, COL->REC]+ ]+

docs/api.md

@@ -235,6 +235,7 @@ Specifies how an item participates in the semantic layer. Actions are attached t
 ```java
 ActionSpec.rec(ProviderSpec.val(ItemFilterConditionSpec.sameRow()))   // REC
 ActionSpec.avp("AIRLINE")                                             // AVP with literal attribute
+ActionSpec.concat(0, ProviderSpec.val(...))                           // CONCAT(0)
 ActionSpec.join(ProviderSpec.val(...))                                // JOIN
 ActionSpec.fill("/", ProviderSpec.val(...))                           // FILL
 ActionSpec.prefix(" ", ProviderSpec.val(...))                         // PREFIX
@@ -247,8 +248,10 @@ ActionSpec.suffix(" ", ProviderSpec.val(...))                         // SUFFIX
 | `rec(int anchorPos, ProviderSpec... providers)` | REC with schema anchor at position N. |
 | `avp(ProviderSpec provider)` | Attribute-value pair via provider. |
 | `avp(String literal)` | AVP with constant attribute name. |
-| `join(ProviderSpec... providers)` | JOIN: merge co-anchored items. |
-| `join(Set<Integer> keyPositions, ProviderSpec... providers)` | JOIN with key positions. |
+| `concat(ProviderSpec... providers)` | CONCAT: fold the provided records into the anchor's record (one wider record). |
+| `concat(int keyPosition, ProviderSpec... providers)`, `concat(Set<Integer> keyPositions, ProviderSpec... providers)` | CONCAT with key positions K (must agree, not repeated). Was `join(…)` up to 0.5.x. |
+| `join(ProviderSpec... providers)` | JOIN: the record product — one record per (anchor record × provided record). |
+| `join(int keyPosition, ProviderSpec... providers)`, `join(Set<Integer> keyPositions, ProviderSpec... providers)` | Equi-join on the key positions K. |
 | `fill(String delimiter, ProviderSpec... providers)` | Fill gap in REC sequence. |
 | `prefix(String delimiter, ProviderSpec... providers)` | Prepend to anchor value. |
 | `suffix(String delimiter, ProviderSpec... providers)` | Append to anchor value. |
@@ -361,6 +364,8 @@ Recordset rs = new TableInterpreter()
 | `withMissingValueHandler(MissingValueHandler h)` | Handling of missing attribute values (default: `NULL_HANDLER`). |
 | `withTransformations(List<RecordsetTransformation> t)` | Post-processing transformations. |
 | `withAnonymousAttributeTemplate(String template)` | Name template for unnamed attributes; `%i` → index. Default: `"$a_%i"`. |
+| `withStrictPreconditions(boolean strict)` | A violated `CONCAT`/`JOIN` precondition (e.g. a named attribute shared by two concatenated records) raises an `IllegalStateException` instead of having no effect. Default: `false`. |
+| `List<Diagnostic> diagnostics()` | The `CONCAT`/`JOIN` actions skipped during the most recent `interpret(...)` because a precondition was violated — each with the anchor, the operation and the reason. Empty when nothing was skipped. |
 
 ---
 

docs/architecture.md

@@ -87,8 +87,8 @@ If the pattern does not match, `AtpMatcher.match` returns `Optional.empty()`.
 | Phase | What happens |
 |---|---|
 | **1. Initialisation** | Each cell-derived and context-derived item of type VAL/ATTR is entered into the working state with its string value |
-| **2. Completion** | Interpretation actions are applied in operation-type order: FILL/PREFIX/SUFFIX → AVP → REC → JOIN; each action uses its providers to retrieve items relative to the anchor and updates the working state |
-| **3. Extraction** | The working state is traversed to build the schema (attribute list) and generate records |
+| **2. Completion** | Interpretation actions are applied in operation-type order: FILL/PREFIX/SUFFIX → AVP → REC → CONCAT → JOIN (records are folded before they are multiplied); each action uses its providers to retrieve items relative to the anchor and updates the working state. A violated CONCAT/JOIN precondition has no effect and is reported through `TableInterpreter.diagnostics()` |
+| **3. Extraction** | The live anchors of the working state (joined-away anchors excluded) are traversed to build the schema (attribute list) and generate records — one per item-based record, several per anchor after a JOIN |
 | **4. Transformation** | Optional post-processing steps are applied: `WhitespaceNormalization`, `FieldSplitting`, `SchemaReordering` |
 
 ---
@@ -139,7 +139,7 @@ The round-trip property — serialize then compile gives back the original patte
 | `RowPattern`, `SubrowPattern`, `CellPattern` | `[ ... ]q`, `{ ... }q`, `[ ... ]q` |
 | `AtomicContentSpec` with tags | `VAL #'tag'` |
 | `AtomicContentSpec` with extractor | `VAL = TRIM` |
-| `ActionSpec` (avp, rec, join, fill, prefix, suffix) | `'NAME'->AVP`, `(prov…)->REC`, etc. |
+| `ActionSpec` (avp, rec, concat, join, fill, prefix, suffix) | `'NAME'->AVP`, `(prov…)->REC`, etc. |
 | `ProviderSpec` with traversal order | leading `-` / `^` / `-^` |
 | `ProviderSpec` with cardinality | `{n}` / `*` |
 | `RecordsetTransformation` settings | `<NORM>`, `<ANCH(n)>`, `<SPLIT("d")>` |

docs/embedded-rtl.md

@@ -98,9 +98,10 @@ compiler.
 |---|---|
 | `(…)->REC` / `REC(n)` / `REC('s')` | `rec(…)` / `rec(n, …)` / `recSplit("s", …)` |
 | `prov->AVP` / `'NAME'->AVP` | `avp(prov)` / `avp("NAME")` |
+| `(…)->CONCAT` / `CONCAT(k)` | `concat(…)` / `concat(k, …)` |
 | `(…)->JOIN` / `JOIN(k)` | `join(…)` / `join(k, …)` |
 | `(…)->FILL('d')`, `PREFIX`, `SUFFIX` | `fill("d", …)`, `prefix(…)`, `suffix(…)` (delimiter optional) |
-| `'EUR'` context literal | `lit("EUR")` (VALUE under REC/JOIN, ATTRIBUTE otherwise — as in the compiler) |
+| `'EUR'` context literal | `lit("EUR")` (VALUE under REC/CONCAT/JOIN, ATTRIBUTE otherwise — as in the compiler) |
 | `@'K'='V'` | `ctxAvp("K", "V")` |
 
 Provider kinds (VAL/ATTR/UNRESTRICTED) are inferred from the action, exactly as in the