From aeb25f589e0bf1e670c150fcaa03256659734f4d Mon Sep 17 00:00:00 2001 From: "Alexey O. Shigarov" Date: Fri, 28 Aug 2026 17:41:45 +0800 Subject: [PATCH 1/3] CONCAT(K) reintroduced as the folding operation; JOIN(K) redefined as the record product MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The operation spelled JOIN(K) up to 0.5.x concatenated the records of the provided anchors to the anchor's record (one wide record, the number of records strictly decreasing) — a fold, what pandas.concat(axis=1) does, not a join. It keeps that semantics under its original name CONCAT(K); JOIN(K) now multiplies: every record of the anchor is combined with every provided record (cross product for K = ∅, equi-join on the key positions otherwise, a shared named attribute is a natural-join condition). The provided anchors become joined-away (J): excluded from the recordset, but their records stay available so that several anchors may join the same records irrespective of action order. Working state: rec is multi-valued (a non-empty sequence of records per anchor), the component J is added, allRec() returns the live anchors only; SchemaConstructionStrategy visits (anchor, record, position) triples; record generation emits one record per item-based record. CONCAT(K) no longer deduplicates named attributes: a named attribute shared by the concatenated records (apart from the key) violates a precondition — the action has no effect and a Diagnostic is recorded (TableInterpreter.diagnostics(); withStrictPreconditions(true) raises). Task 098 lists its full group key, CONCAT(0,1,2,3), where the former JOIN(0,1) silently dropped the repeated A/B attributes; expected recordsets of all migrated tasks are unchanged. Grammar: concatOp / CONCAT keyword; serializer emits CONCAT(k1, k2); VS Code grammar highlights CONCAT. Conformance: 12 positive tasks migrated JOIN(K) -> CONCAT(K); semantic cases concat_by_key, join_product, join_equi_key added. Tests: WorkingStateConcatTest, WorkingStateJoinTest, TableInterpreterMultiRecordTest. --- conformance/VERSION | 5 +- conformance/positive/task_016.expected.rtl | 2 +- conformance/positive/task_016.rtl | 2 +- conformance/positive/task_023.expected.rtl | 2 +- conformance/positive/task_023.rtl | 2 +- conformance/positive/task_025.expected.rtl | 2 +- conformance/positive/task_025.rtl | 2 +- conformance/positive/task_033.expected.rtl | 2 +- conformance/positive/task_033.rtl | 2 +- conformance/positive/task_046.expected.rtl | 2 +- conformance/positive/task_046.rtl | 2 +- conformance/positive/task_047.expected.rtl | 2 +- conformance/positive/task_047.rtl | 2 +- conformance/positive/task_050.expected.rtl | 2 +- conformance/positive/task_050.rtl | 2 +- conformance/positive/task_053.expected.rtl | 2 +- conformance/positive/task_053.rtl | 2 +- conformance/positive/task_069.expected.rtl | 2 +- conformance/positive/task_069.rtl | 2 +- conformance/positive/task_094.expected.rtl | 2 +- conformance/positive/task_094.rtl | 2 +- conformance/positive/task_097.expected.rtl | 2 +- conformance/positive/task_097.rtl | 2 +- conformance/positive/task_098.expected.rtl | 2 +- conformance/positive/task_098.rtl | 2 +- .../semantic/concat_by_key/expected.csv | 3 + conformance/semantic/concat_by_key/input.csv | 9 + .../semantic/concat_by_key/options.json | 1 + .../semantic/concat_by_key/pattern.rtl | 1 + .../semantic/join_equi_key/expected.csv | 3 + conformance/semantic/join_equi_key/input.csv | 3 + .../semantic/join_equi_key/options.json | 1 + .../semantic/join_equi_key/pattern.rtl | 2 + .../semantic/join_product/expected.csv | 7 + conformance/semantic/join_product/input.csv | 3 + .../semantic/join_product/options.json | 1 + conformance/semantic/join_product/pattern.rtl | 2 + ide/vscode/syntaxes/rtl.tmLanguage.json | 2 +- src/main/antlr4/ru/icc/regtab/rtl/RTL.g4 | 4 +- .../regtab/atp/match/SemanticConstructor.java | 1 + .../ru/icc/regtab/atp/spec/ActionSpec.java | 30 ++- .../ru/icc/regtab/atp/spec/OperationType.java | 4 +- src/main/java/ru/icc/regtab/dsl/Rtl.java | 34 ++- .../interpret/SchemaConstructionStrategy.java | 46 ++-- .../regtab/interpret/TableInterpreter.java | 78 ++++-- .../icc/regtab/itm/semantics/Diagnostic.java | 28 ++ .../regtab/itm/semantics/WorkingState.java | 240 ++++++++++++++++-- .../semantics/operation/ConcatOperation.java | 25 ++ .../semantics/operation/JoinOperation.java | 19 +- .../operation/WorkingStateOperation.java | 4 +- .../ru/icc/regtab/rtl/AtpToRtlSerializer.java | 6 + .../icc/regtab/rtl/internal/ATPBuilder.java | 7 +- .../internal/ProviderTemplateResolver.java | 2 +- .../ru/icc/regtab/atp/AtpTask016Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask023Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask025Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask033Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask046Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask047Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask050Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask053Test.java | 2 +- .../ru/icc/regtab/atp/AtpTask069Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask094Test.java | 6 +- .../ru/icc/regtab/atp/AtpTask097Test.java | 4 +- .../ru/icc/regtab/atp/AtpTask098Test.java | 6 +- .../java/ru/icc/regtab/dsl/DslSpikeTest.java | 20 +- .../icc/regtab/interpret/EquipmentTest.java | 4 +- .../regtab/interpret/SchemaFlexibleTest.java | 4 +- .../TableInterpreterMultiRecordTest.java | 119 +++++++++ .../itm/semantics/WorkingStateConcatTest.java | 150 +++++++++++ .../itm/semantics/WorkingStateJoinTest.java | 176 +++++++++++++ .../ru/icc/regtab/rtl/RtlCompilerTest.java | 2 +- .../ru/icc/regtab/rtl/RtlTask016Test.java | 8 +- .../ru/icc/regtab/rtl/RtlTask023Test.java | 8 +- .../ru/icc/regtab/rtl/RtlTask025Test.java | 8 +- .../ru/icc/regtab/rtl/RtlTask033Test.java | 8 +- .../ru/icc/regtab/rtl/RtlTask046Test.java | 8 +- .../ru/icc/regtab/rtl/RtlTask047Test.java | 8 +- .../ru/icc/regtab/rtl/RtlTask050Test.java | 8 +- .../ru/icc/regtab/rtl/RtlTask053Test.java | 6 +- .../ru/icc/regtab/rtl/RtlTask069Test.java | 8 +- .../ru/icc/regtab/rtl/RtlTask094Test.java | 6 +- .../ru/icc/regtab/rtl/RtlTask097Test.java | 6 +- .../ru/icc/regtab/rtl/RtlTask098Test.java | 2 +- 84 files changed, 1030 insertions(+), 198 deletions(-) create mode 100644 conformance/semantic/concat_by_key/expected.csv create mode 100644 conformance/semantic/concat_by_key/input.csv create mode 100644 conformance/semantic/concat_by_key/options.json create mode 100644 conformance/semantic/concat_by_key/pattern.rtl create mode 100644 conformance/semantic/join_equi_key/expected.csv create mode 100644 conformance/semantic/join_equi_key/input.csv create mode 100644 conformance/semantic/join_equi_key/options.json create mode 100644 conformance/semantic/join_equi_key/pattern.rtl create mode 100644 conformance/semantic/join_product/expected.csv create mode 100644 conformance/semantic/join_product/input.csv create mode 100644 conformance/semantic/join_product/options.json create mode 100644 conformance/semantic/join_product/pattern.rtl create mode 100644 src/main/java/ru/icc/regtab/itm/semantics/Diagnostic.java create mode 100644 src/main/java/ru/icc/regtab/itm/semantics/operation/ConcatOperation.java create mode 100644 src/test/java/ru/icc/regtab/interpret/TableInterpreterMultiRecordTest.java create mode 100644 src/test/java/ru/icc/regtab/itm/semantics/WorkingStateConcatTest.java create mode 100644 src/test/java/ru/icc/regtab/itm/semantics/WorkingStateJoinTest.java diff --git a/conformance/VERSION b/conformance/VERSION index 1820d49b..363016c2 100644 --- a/conformance/VERSION +++ b/conformance/VERSION @@ -1,2 +1,5 @@ -generated: 2026-08-26 +generated: 2026-08-28 sources: RtlTask001..150 + curated extras +note: CONCAT(K) reintroduced (the former JOIN(K)); JOIN(K) redefined as the record product; + semantic cases concat_by_key, join_product, join_equi_key added; task_098 lists its full + group key CONCAT(0,1,2,3) diff --git a/conformance/positive/task_016.expected.rtl b/conformance/positive/task_016.expected.rtl index 40267fa4..175e5779 100644 --- a/conformance/positive/task_016.expected.rtl +++ b/conformance/positive/task_016.expected.rtl @@ -1 +1 @@ -[ [ VAL : RT->REC, (BW & STR)*->JOIN(0) ] [ VAL ] ]+ +[ [ VAL : RT->REC, (BW & STR)*->CONCAT(0) ] [ VAL ] ]+ diff --git a/conformance/positive/task_016.rtl b/conformance/positive/task_016.rtl index 42add13f..86bd6e6d 100644 --- a/conformance/positive/task_016.rtl +++ b/conformance/positive/task_016.rtl @@ -1 +1 @@ -[ [VAL : RT->REC, BW&STR*->JOIN(0)] [VAL] ]+ +[ [VAL : RT->REC, BW&STR*->CONCAT(0)] [VAL] ]+ diff --git a/conformance/positive/task_023.expected.rtl b/conformance/positive/task_023.expected.rtl index 7aa71664..24865b55 100644 --- a/conformance/positive/task_023.expected.rtl +++ b/conformance/positive/task_023.expected.rtl @@ -1 +1 @@ -{ [ [ VAL : ''->AVP, SR*->REC, (BW & STR)*->JOIN(0) ] [ ATTR : RT->SUFFIX ] [ AUX ] [ VAL : SR->AVP ] ]{3} }+ +{ [ [ VAL : ''->AVP, SR*->REC, (BW & STR)*->CONCAT(0) ] [ ATTR : RT->SUFFIX ] [ AUX ] [ VAL : SR->AVP ] ]{3} }+ diff --git a/conformance/positive/task_023.rtl b/conformance/positive/task_023.rtl index e4337024..ac2a9464 100644 --- a/conformance/positive/task_023.rtl +++ b/conformance/positive/task_023.rtl @@ -1 +1 @@ -{ [ [VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+ +{ [ [VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+ diff --git a/conformance/positive/task_025.expected.rtl b/conformance/positive/task_025.expected.rtl index f4789e30..b587f190 100644 --- a/conformance/positive/task_025.expected.rtl +++ b/conformance/positive/task_025.expected.rtl @@ -1 +1 @@ - [ [ VAL : RT->SUFFIX("/"), (RT & C+2..)*->REC('/'), (BW & STR)*->JOIN(0) ] [ VAL ]+ ]+ + [ [ VAL : RT->SUFFIX("/"), (RT & C+2..)*->REC('/'), (BW & STR)*->CONCAT(0) ] [ VAL ]+ ]+ diff --git a/conformance/positive/task_025.rtl b/conformance/positive/task_025.rtl index 27dd7e8d..c69aaae5 100644 --- a/conformance/positive/task_025.rtl +++ b/conformance/positive/task_025.rtl @@ -1 +1 @@ -[ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->JOIN(0)] [VAL]+ ]+ +[ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->CONCAT(0)] [VAL]+ ]+ diff --git a/conformance/positive/task_033.expected.rtl b/conformance/positive/task_033.expected.rtl index 2a91346b..bd6c2d3b 100644 --- a/conformance/positive/task_033.expected.rtl +++ b/conformance/positive/task_033.expected.rtl @@ -1 +1 @@ -[ [ VAL : SR*->REC, (BW & STR)*->JOIN(0) ] [ VAL ]+ ]+ +[ [ VAL : SR*->REC, (BW & STR)*->CONCAT(0) ] [ VAL ]+ ]+ diff --git a/conformance/positive/task_033.rtl b/conformance/positive/task_033.rtl index 40670b73..ce800079 100644 --- a/conformance/positive/task_033.rtl +++ b/conformance/positive/task_033.rtl @@ -1 +1 @@ -[ [VAL : SR*->REC, BW&STR*->JOIN(0)] [VAL]+ ]+ +[ [VAL : SR*->REC, BW&STR*->CONCAT(0)] [VAL]+ ]+ diff --git a/conformance/positive/task_046.expected.rtl b/conformance/positive/task_046.expected.rtl index df99b505..ec0c856c 100644 --- a/conformance/positive/task_046.expected.rtl +++ b/conformance/positive/task_046.expected.rtl @@ -1 +1 @@ -{ [ [ !BLANK? VAL : ''->AVP, SR*->REC, (BW & STR)*->JOIN(0) ] [ !BLANK? ATTR ] [ !BLANK? VAL : SR->AVP ] ]+ }+ +{ [ [ !BLANK? VAL : ''->AVP, SR*->REC, (BW & STR)*->CONCAT(0) ] [ !BLANK? ATTR ] [ !BLANK? VAL : SR->AVP ] ]+ }+ diff --git a/conformance/positive/task_046.rtl b/conformance/positive/task_046.rtl index 606b0c20..546cb6b6 100644 --- a/conformance/positive/task_046.rtl +++ b/conformance/positive/task_046.rtl @@ -1 +1 @@ -{ [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+ +{ [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+ diff --git a/conformance/positive/task_047.expected.rtl b/conformance/positive/task_047.expected.rtl index f43e7972..feaf3f09 100644 --- a/conformance/positive/task_047.expected.rtl +++ b/conformance/positive/task_047.expected.rtl @@ -1 +1 @@ -{ [ [ !BLANK? VAL : SR*->REC, (BW & STR)*->JOIN(0) ] [ !BLANK? VAL ] ]+ }+ +{ [ [ !BLANK? VAL : SR*->REC, (BW & STR)*->CONCAT(0) ] [ !BLANK? VAL ] ]+ }+ diff --git a/conformance/positive/task_047.rtl b/conformance/positive/task_047.rtl index beb3d232..bea0aa8c 100644 --- a/conformance/positive/task_047.rtl +++ b/conformance/positive/task_047.rtl @@ -1 +1 @@ -{ [ [!BLANK? VAL : SR*->REC, BW&STR*->JOIN(0)] [!BLANK? VAL] ]+ }+ +{ [ [!BLANK? VAL : SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? VAL] ]+ }+ diff --git a/conformance/positive/task_050.expected.rtl b/conformance/positive/task_050.expected.rtl index 3381e199..f74bb47f 100644 --- a/conformance/positive/task_050.expected.rtl +++ b/conformance/positive/task_050.expected.rtl @@ -1 +1 @@ -[ [ !BLANK? VAL : ''->AVP, SR*->REC, (BW & STR)*->JOIN(0) ] [ !BLANK? ATTR ] [ !BLANK? VAL : SR->AVP ] ]+ +[ [ !BLANK? VAL : ''->AVP, SR*->REC, (BW & STR)*->CONCAT(0) ] [ !BLANK? ATTR ] [ !BLANK? VAL : SR->AVP ] ]+ diff --git a/conformance/positive/task_050.rtl b/conformance/positive/task_050.rtl index c9ffe93b..593f633d 100644 --- a/conformance/positive/task_050.rtl +++ b/conformance/positive/task_050.rtl @@ -1 +1 @@ -[ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ +[ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ diff --git a/conformance/positive/task_053.expected.rtl b/conformance/positive/task_053.expected.rtl index e1a8e872..37924410 100644 --- a/conformance/positive/task_053.expected.rtl +++ b/conformance/positive/task_053.expected.rtl @@ -1 +1 @@ -[ [] [ AUX ]+ ] [ [ VAL : ROW*->REC, (BW & STR)->JOIN(0), 'ID'->AVP ] { [ ATTR : AV->PREFIX("_") ] [ VAL : SR->AVP ] }+ ]+ +[ [] [ AUX ]+ ] [ [ VAL : ROW*->REC, (BW & STR)->CONCAT(0), 'ID'->AVP ] { [ ATTR : AV->PREFIX("_") ] [ VAL : SR->AVP ] }+ ]+ diff --git a/conformance/positive/task_053.rtl b/conformance/positive/task_053.rtl index cab6584e..0724d429 100644 --- a/conformance/positive/task_053.rtl +++ b/conformance/positive/task_053.rtl @@ -1,3 +1,3 @@ [ [] [AUX]+ ] -[ [VAL : ROW*->REC, BW&STR->JOIN(0), 'ID'->AVP] +[ [VAL : ROW*->REC, BW&STR->CONCAT(0), 'ID'->AVP] {[ATTR : AV->PREFIX('_')] [VAL : SR->AVP]}+]+ diff --git a/conformance/positive/task_069.expected.rtl b/conformance/positive/task_069.expected.rtl index 1eae93e8..f0fc9598 100644 --- a/conformance/positive/task_069.expected.rtl +++ b/conformance/positive/task_069.expected.rtl @@ -1 +1 @@ -[ { [ ATTR : SR->AVP, BW*->REC ] [ VAL #'1' : SR->AVP, BW*->REC, (ROW & #'1')*->JOIN ] [ VAL #'2' : SR->AVP, BW*->REC, (ROW & #'2')*->JOIN ] }* ] [ { [ ATTR : SR->AVP ] [ VAL : SR->AVP ]{2} }* ]* +[ { [ ATTR : SR->AVP, BW*->REC ] [ VAL #'1' : SR->AVP, BW*->REC, (ROW & #'1')*->CONCAT ] [ VAL #'2' : SR->AVP, BW*->REC, (ROW & #'2')*->CONCAT ] }* ] [ { [ ATTR : SR->AVP ] [ VAL : SR->AVP ]{2} }* ]* diff --git a/conformance/positive/task_069.rtl b/conformance/positive/task_069.rtl index d055ce99..d9ccba56 100644 --- a/conformance/positive/task_069.rtl +++ b/conformance/positive/task_069.rtl @@ -1,3 +1,3 @@ SR->AVP -[ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->JOIN][VAL#'2': ROW&#'2'*->JOIN] }* ] +[ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->CONCAT][VAL#'2': ROW&#'2'*->CONCAT] }* ] [ { [ATTR] [VAL]{2} }* ]* diff --git a/conformance/positive/task_094.expected.rtl b/conformance/positive/task_094.expected.rtl index d2d1e733..d63701c8 100644 --- a/conformance/positive/task_094.expected.rtl +++ b/conformance/positive/task_094.expected.rtl @@ -1 +1 @@ -{ [ { [ !BLANK? VAL : COL*->REC, (ROW & C+1.. & STR)*->JOIN(0) ]+ [ BLANK ]? }+ ] } { [ { [ !BLANK? VAL ]+ [ BLANK ]? }+ ]+ [ [ BLANK ]+ ]? }+ +{ [ { [ !BLANK? VAL : COL*->REC, (ROW & C+1.. & STR)*->CONCAT(0) ]+ [ BLANK ]? }+ ] } { [ { [ !BLANK? VAL ]+ [ BLANK ]? }+ ]+ [ [ BLANK ]+ ]? }+ diff --git a/conformance/positive/task_094.rtl b/conformance/positive/task_094.rtl index 1786fde8..cba7a15f 100644 --- a/conformance/positive/task_094.rtl +++ b/conformance/positive/task_094.rtl @@ -1,3 +1,3 @@ - [ { [!BLANK? VAL: COL*->REC, ROW&C+1..&STR*->JOIN(0)]+ [BLANK]? }+ ] + [ { [!BLANK? VAL: COL*->REC, ROW&C+1..&STR*->CONCAT(0)]+ [BLANK]? }+ ] { [ { [!BLANK? VAL]+ [BLANK]? }+ ]+ [ [BLANK]+ ]? }+ diff --git a/conformance/positive/task_097.expected.rtl b/conformance/positive/task_097.expected.rtl index 494b9eac..4c3a4928 100644 --- a/conformance/positive/task_097.expected.rtl +++ b/conformance/positive/task_097.expected.rtl @@ -1 +1 @@ -[ [ VAL : RT*->REC, (BW & STR)*->JOIN(0, 1) ] [ VAL ]+ ]+ +[ [ VAL : RT*->REC, (BW & STR)*->CONCAT(0, 1) ] [ VAL ]+ ]+ diff --git a/conformance/positive/task_097.rtl b/conformance/positive/task_097.rtl index 73151032..cf59eda5 100644 --- a/conformance/positive/task_097.rtl +++ b/conformance/positive/task_097.rtl @@ -1 +1 @@ -[ [VAL: RT*->REC, (BW&STR)*->JOIN(0,1)] [VAL]+ ]+ +[ [VAL: RT*->REC, (BW&STR)*->CONCAT(0,1)] [VAL]+ ]+ diff --git a/conformance/positive/task_098.expected.rtl b/conformance/positive/task_098.expected.rtl index 606004f8..7f3bfd2b 100644 --- a/conformance/positive/task_098.expected.rtl +++ b/conformance/positive/task_098.expected.rtl @@ -1 +1 @@ -[ [] [] [ ATTR ]+ ] [ [ VAL : RT*->REC, (BW & STR)*->JOIN(0, 1) ] [ VAL ] [ VAL : COL->AVP ]{2} [ VAL ]+ ]+ +[ [] [] [ ATTR ]+ ] [ [ VAL : RT*->REC, (BW & STR)*->CONCAT(0, 1, 2, 3) ] [ VAL ] [ VAL : COL->AVP ]{2} [ VAL ]+ ]+ diff --git a/conformance/positive/task_098.rtl b/conformance/positive/task_098.rtl index 9cfbd4a6..6a4252b0 100644 --- a/conformance/positive/task_098.rtl +++ b/conformance/positive/task_098.rtl @@ -1,2 +1,2 @@ [ [] [] [ATTR]+ ] -[ [VAL: RT*->REC, (BW&STR)*->JOIN(0,1)] [VAL] [VAL: COL->AVP]{2} [VAL]+ ]+ +[ [VAL: RT*->REC, (BW&STR)*->CONCAT(0,1,2,3)] [VAL] [VAL: COL->AVP]{2} [VAL]+ ]+ diff --git a/conformance/semantic/concat_by_key/expected.csv b/conformance/semantic/concat_by_key/expected.csv new file mode 100644 index 00000000..efc5836e --- /dev/null +++ b/conformance/semantic/concat_by_key/expected.csv @@ -0,0 +1,3 @@ +"book","5","6","7" +"cat","2","3","8" +"dog","2","3","4" diff --git a/conformance/semantic/concat_by_key/input.csv b/conformance/semantic/concat_by_key/input.csv new file mode 100644 index 00000000..848a3afa --- /dev/null +++ b/conformance/semantic/concat_by_key/input.csv @@ -0,0 +1,9 @@ +"book","5" +"book","6" +"book","7" +"cat","2" +"cat","3" +"cat","8" +"dog","2" +"dog","3" +"dog","4" diff --git a/conformance/semantic/concat_by_key/options.json b/conformance/semantic/concat_by_key/options.json new file mode 100644 index 00000000..0a01d42d --- /dev/null +++ b/conformance/semantic/concat_by_key/options.json @@ -0,0 +1 @@ +{ "expectedHasHeader": false } diff --git a/conformance/semantic/concat_by_key/pattern.rtl b/conformance/semantic/concat_by_key/pattern.rtl new file mode 100644 index 00000000..86bd6e6d --- /dev/null +++ b/conformance/semantic/concat_by_key/pattern.rtl @@ -0,0 +1 @@ +[ [VAL : RT->REC, BW&STR*->CONCAT(0)] [VAL] ]+ diff --git a/conformance/semantic/join_equi_key/expected.csv b/conformance/semantic/join_equi_key/expected.csv new file mode 100644 index 00000000..d813e435 --- /dev/null +++ b/conformance/semantic/join_equi_key/expected.csv @@ -0,0 +1,3 @@ +"k","v","u" +"X","5","kg" +"Y","8","pc" diff --git a/conformance/semantic/join_equi_key/input.csv b/conformance/semantic/join_equi_key/input.csv new file mode 100644 index 00000000..c2c6a4fd --- /dev/null +++ b/conformance/semantic/join_equi_key/input.csv @@ -0,0 +1,3 @@ +"k","v","k","u" +"X","5","X","kg" +"Y","8","Y","pc" diff --git a/conformance/semantic/join_equi_key/options.json b/conformance/semantic/join_equi_key/options.json new file mode 100644 index 00000000..41659efd --- /dev/null +++ b/conformance/semantic/join_equi_key/options.json @@ -0,0 +1 @@ +{ "expectedHasHeader": true } diff --git a/conformance/semantic/join_equi_key/pattern.rtl b/conformance/semantic/join_equi_key/pattern.rtl new file mode 100644 index 00000000..c8c1fa95 --- /dev/null +++ b/conformance/semantic/join_equi_key/pattern.rtl @@ -0,0 +1,2 @@ +[ [ATTR]+ ] +[ [VAL: COL->AVP, RT->REC, C2*->JOIN(0)] [VAL: COL->AVP] [VAL: COL->AVP, RT->REC] [VAL: COL->AVP] ]+ diff --git a/conformance/semantic/join_product/expected.csv b/conformance/semantic/join_product/expected.csv new file mode 100644 index 00000000..e18920b1 --- /dev/null +++ b/conformance/semantic/join_product/expected.csv @@ -0,0 +1,7 @@ +"id","value","var" +"a","1","x" +"a","2","y" +"b","1","x" +"b","2","y" +"c","3","x" +"c","4","y" diff --git a/conformance/semantic/join_product/input.csv b/conformance/semantic/join_product/input.csv new file mode 100644 index 00000000..20d7ad12 --- /dev/null +++ b/conformance/semantic/join_product/input.csv @@ -0,0 +1,3 @@ +"id","x","y" +"a;b","1","2" +"c","3","4" diff --git a/conformance/semantic/join_product/options.json b/conformance/semantic/join_product/options.json new file mode 100644 index 00000000..41659efd --- /dev/null +++ b/conformance/semantic/join_product/options.json @@ -0,0 +1 @@ +{ "expectedHasHeader": true } diff --git a/conformance/semantic/join_product/pattern.rtl b/conformance/semantic/join_product/pattern.rtl new file mode 100644 index 00000000..35732b3a --- /dev/null +++ b/conformance/semantic/join_product/pattern.rtl @@ -0,0 +1,2 @@ +[ [ATTR] [VAL: 'var'->AVP]+ ] +[ [(VAL: COL->AVP, ()->REC, RT*->JOIN){';'}] [VAL: 'value'->AVP, COL->REC]+ ]+ diff --git a/ide/vscode/syntaxes/rtl.tmLanguage.json b/ide/vscode/syntaxes/rtl.tmLanguage.json index e7ffeeac..b1833a7e 100644 --- a/ide/vscode/syntaxes/rtl.tmLanguage.json +++ b/ide/vscode/syntaxes/rtl.tmLanguage.json @@ -67,7 +67,7 @@ ] }, "action": { - "match": "(?i)\\b(FILL|PREFIX|SUFFIX|AVP|REC|JOIN)\\b", + "match": "(?i)\\b(FILL|PREFIX|SUFFIX|AVP|REC|CONCAT|JOIN)\\b", "name": "keyword.control.action.rtl" }, "provider": { diff --git a/src/main/antlr4/ru/icc/regtab/rtl/RTL.g4 b/src/main/antlr4/ru/icc/regtab/rtl/RTL.g4 index d9d17c1a..3725c63e 100644 --- a/src/main/antlr4/ru/icc/regtab/rtl/RTL.g4 +++ b/src/main/antlr4/ru/icc/regtab/rtl/RTL.g4 @@ -103,17 +103,19 @@ actSpecs : actSpec (COMMA actSpec)* ; // Interpretation action specification actSpec : provSpecs RIGHT_ARROW op ; -op : fillOp | prefixOp | suffixOp | AVP | recOp | joinOp ; +op : fillOp | prefixOp | suffixOp | AVP | recOp | concatOp | joinOp ; fillOp : FILL (LPAREN STRING RPAREN)? ; prefixOp : PREFIX (LPAREN STRING RPAREN)? ; suffixOp : SUFFIX (LPAREN STRING RPAREN)? ; recOp : REC (LPAREN (INT | STRING) RPAREN)? ; +concatOp : CONCAT (LPAREN INT (COMMA INT)* RPAREN)? ; joinOp : JOIN (LPAREN INT (COMMA INT)* RPAREN)? ; FILL : 'FILL' ; PREFIX : 'PREFIX' ; SUFFIX : 'SUFFIX' ; AVP : 'AVP' ; REC : 'REC' ; +CONCAT : 'CONCAT' ; JOIN : 'JOIN' ; provSpecs : provSpec | (LPAREN provSpec (COMMA provSpec)* RPAREN) | LPAREN RPAREN ; diff --git a/src/main/java/ru/icc/regtab/atp/match/SemanticConstructor.java b/src/main/java/ru/icc/regtab/atp/match/SemanticConstructor.java index 7c1256a5..cdae155d 100644 --- a/src/main/java/ru/icc/regtab/atp/match/SemanticConstructor.java +++ b/src/main/java/ru/icc/regtab/atp/match/SemanticConstructor.java @@ -266,6 +266,7 @@ private static WorkingStateOperation createOperation(ActionSpec as) { case SUFFIX -> new SuffixOperation(delim); case AVP -> new AvpOperation(); case REC -> new RecOperation(); + case CONCAT -> new ConcatOperation(as.keyPositions()); case JOIN -> new JoinOperation(as.keyPositions()); }; } diff --git a/src/main/java/ru/icc/regtab/atp/spec/ActionSpec.java b/src/main/java/ru/icc/regtab/atp/spec/ActionSpec.java index b97ea3e4..ada3c1e6 100644 --- a/src/main/java/ru/icc/regtab/atp/spec/ActionSpec.java +++ b/src/main/java/ru/icc/regtab/atp/spec/ActionSpec.java @@ -21,11 +21,11 @@ * {@link TablePattern#of(SubtablePattern...)} collects these automatically. * * @param operationType working-state update operation op - * @param delimiter delimiter for FILL/PREFIX/SUFFIX (empty string if none); null for AVP/REC/JOIN + * @param delimiter delimiter for FILL/PREFIX/SUFFIX (empty string if none); null for AVP/REC/CONCAT/JOIN * @param providers sequence of item provider specifications ⟨S_prov¹, …, S_provⁿ⟩ * @param anchorPos inline anchor position for REC (null = none) * @param splitDelimiter inline split delimiter for REC (null = none) - * @param keyPositions key positions K for JOIN (empty = K=∅); null treated as empty + * @param keyPositions key positions K for CONCAT/JOIN (empty = K=∅); null treated as empty * @param inherited true if this action was inherited from a parent scope (row/subrow/subtable level), * false if explicitly specified on the cell's own contSpec */ @@ -52,9 +52,9 @@ public ActionSpec(OperationType operationType, String delimiter, if (p.isContextLiteral()) { var ctxType = p.contextLiteral().type(); boolean isConstAvp = p.contextLiteral().constValue() != null; - if (operationType == OperationType.JOIN) + if (operationType == OperationType.JOIN || operationType == OperationType.CONCAT) throw new IllegalArgumentException( - "JOIN action does not allow context literals"); + operationType + " action does not allow context literals"); if (operationType == OperationType.REC && !isConstAvp && ctxType != ItemType.VALUE) throw new IllegalArgumentException( "REC action requires a VALUE context literal, got " + ctxType); @@ -63,7 +63,8 @@ public ActionSpec(OperationType operationType, String delimiter, "AVP action requires an ATTRIBUTE context literal, got " + ctxType); } else { var kind = p.targetItemKind(); - if ((operationType == OperationType.REC || operationType == OperationType.JOIN) + if ((operationType == OperationType.REC || operationType == OperationType.CONCAT + || operationType == OperationType.JOIN) && kind != CellDerivedProviderKind.VAL) throw new IllegalArgumentException( operationType + " action requires a VAL provider, got " + kind); @@ -115,12 +116,27 @@ public static ActionSpec avp(String literal) { return new ActionSpec(OperationType.AVP, null, List.of(ProviderSpec.ctxAttr(literal)), null, null); } - /** Convenience: JOIN action with K=∅ (include all positions, then dedup). */ + /** Convenience: CONCAT action with K=∅ — fold the provided records into the anchor's record. */ + public static ActionSpec concat(ProviderSpec... providers) { + return new ActionSpec(OperationType.CONCAT, null, List.of(providers), null, null, Set.of(), false); + } + + /** Convenience: CONCAT^K action with explicit key positions as a Set (mirrors RTL {@code CONCAT(k1,k2,...)}). */ + public static ActionSpec concat(Set keyPositions, ProviderSpec... providers) { + return new ActionSpec(OperationType.CONCAT, null, List.of(providers), null, null, keyPositions, false); + } + + /** Convenience: CONCAT^K action with a single key position (mirrors RTL {@code CONCAT(k)}). */ + public static ActionSpec concat(int keyPosition, ProviderSpec... providers) { + return new ActionSpec(OperationType.CONCAT, null, List.of(providers), null, null, Set.of(keyPosition), false); + } + + /** Convenience: JOIN action with K=∅ — the record product (cross product with the provided records). */ public static ActionSpec join(ProviderSpec... providers) { return new ActionSpec(OperationType.JOIN, null, List.of(providers), null, null, Set.of(), false); } - /** Convenience: JOIN^K action with explicit key positions as a Set (mirrors RTL {@code JOIN(k1,k2,...)}). */ + /** Convenience: JOIN^K action — equi-join on the key positions K (mirrors RTL {@code JOIN(k1,k2,...)}). */ public static ActionSpec join(Set keyPositions, ProviderSpec... providers) { return new ActionSpec(OperationType.JOIN, null, List.of(providers), null, null, keyPositions, false); } diff --git a/src/main/java/ru/icc/regtab/atp/spec/OperationType.java b/src/main/java/ru/icc/regtab/atp/spec/OperationType.java index 3e5c2556..00b47151 100644 --- a/src/main/java/ru/icc/regtab/atp/spec/OperationType.java +++ b/src/main/java/ru/icc/regtab/atp/spec/OperationType.java @@ -15,6 +15,8 @@ public enum OperationType { AVP, /** O_rec: construct an item-based record. */ REC, - /** O_join^K: join item-based records with key-position dropping and deduplication. */ + /** O_concat^K: concatenate item-based records into one wide record (key positions K not repeated). */ + CONCAT, + /** O_join^K: record product — every record of the anchor by every joined record (equi-join on K). */ JOIN } diff --git a/src/main/java/ru/icc/regtab/dsl/Rtl.java b/src/main/java/ru/icc/regtab/dsl/Rtl.java index 0c114587..d7a92509 100644 --- a/src/main/java/ru/icc/regtab/dsl/Rtl.java +++ b/src/main/java/ru/icc/regtab/dsl/Rtl.java @@ -463,7 +463,7 @@ public static CellPredicate where(String description, Predicate predicate) // ==== context providers (RTL: 'text', @'ATTR'='VALUE') ==== - /** Context literal (RTL {@code 'EUR'}): VALUE under REC/JOIN, ATTRIBUTE otherwise. */ + /** Context literal (RTL {@code 'EUR'}): VALUE under REC/CONCAT/JOIN, ATTRIBUTE otherwise. */ public static Ctx lit(String text) { return new Ctx(text); } /** Constant attribute-value pair (RTL {@code @'ATTR'='VALUE'}). */ @@ -503,19 +503,37 @@ public static ActionSpec avp(String literal) { return ActionSpec.avp(literal); } - /** RTL {@code (…)->JOIN}. */ + /** RTL {@code (…)->CONCAT} — fold the provided records into the anchor's record. */ + public static ActionSpec concat(ProvArg... providers) { + return new ActionSpec(OperationType.CONCAT, null, resolve(providers, OperationType.CONCAT), + null, null, Set.of(), false); + } + + /** RTL {@code (…)->CONCAT(k)} — with a key position. */ + public static ActionSpec concat(int keyPosition, ProvArg... providers) { + return new ActionSpec(OperationType.CONCAT, null, resolve(providers, OperationType.CONCAT), + null, null, Set.of(keyPosition), false); + } + + /** RTL {@code (…)->CONCAT(k1,k2,…)} — with key positions. */ + public static ActionSpec concat(Set keyPositions, ProvArg... providers) { + return new ActionSpec(OperationType.CONCAT, null, resolve(providers, OperationType.CONCAT), + null, null, keyPositions, false); + } + + /** RTL {@code (…)->JOIN} — the record product (cross product with the provided records). */ public static ActionSpec join(ProvArg... providers) { return new ActionSpec(OperationType.JOIN, null, resolve(providers, OperationType.JOIN), null, null, Set.of(), false); } - /** RTL {@code (…)->JOIN(k)} — with a key position. */ + /** RTL {@code (…)->JOIN(k)} — equi-join on a key position. */ public static ActionSpec join(int keyPosition, ProvArg... providers) { return new ActionSpec(OperationType.JOIN, null, resolve(providers, OperationType.JOIN), null, null, Set.of(keyPosition), false); } - /** RTL {@code (…)->JOIN(k1,k2,…)} — with key positions. */ + /** RTL {@code (…)->JOIN(k1,k2,…)} — equi-join on key positions. */ public static ActionSpec join(Set keyPositions, ProvArg... providers) { return new ActionSpec(OperationType.JOIN, null, resolve(providers, OperationType.JOIN), null, null, keyPositions, false); @@ -552,7 +570,7 @@ private static List resolve(ProvArg[] providers, OperationType op) for (ProvArg arg : providers) { result.add(switch (arg) { case Prov p -> p.spec(kindFor(op)); - case Ctx c -> (op == OperationType.REC || op == OperationType.JOIN) + case Ctx c -> (op == OperationType.REC || op == OperationType.CONCAT || op == OperationType.JOIN) ? ProviderSpec.ctxVal(c.text()) : ProviderSpec.ctxAttr(c.text()); case CtxAvp x -> ProviderSpec.ctxAvp(x.attribute(), x.value()); @@ -563,9 +581,9 @@ private static List resolve(ProvArg[] providers, OperationType op) private static CellDerivedProviderKind kindFor(OperationType op) { return switch (op) { - case REC, JOIN -> CellDerivedProviderKind.VAL; - case AVP -> CellDerivedProviderKind.ATTR; - default -> CellDerivedProviderKind.UNRESTRICTED; + case REC, CONCAT, JOIN -> CellDerivedProviderKind.VAL; + case AVP -> CellDerivedProviderKind.ATTR; + default -> CellDerivedProviderKind.UNRESTRICTED; }; } } diff --git a/src/main/java/ru/icc/regtab/interpret/SchemaConstructionStrategy.java b/src/main/java/ru/icc/regtab/interpret/SchemaConstructionStrategy.java index 3a22c71c..ee6ccff1 100644 --- a/src/main/java/ru/icc/regtab/interpret/SchemaConstructionStrategy.java +++ b/src/main/java/ru/icc/regtab/interpret/SchemaConstructionStrategy.java @@ -9,55 +9,67 @@ /** * Schema construction strategy Γ (sec:itm:table-interpretation): defines the - * order in which (anchor, position) pairs are visited when constructing the schema. + * order in which (anchor, record, position) triples are visited when constructing the schema. + * An anchor may carry several records after a join (the record product); the records of + * an anchor are always visited in their sequence order. */ public enum SchemaConstructionStrategy { /** - * Record-first (Γ_rec): iterates over anchors, for each anchor iterates over positions. + * Record-first (Γ_rec): iterates over anchors, for each anchor over its records, + * for each record over positions. */ RECORD_FIRST { @Override public List buildVisitOrder( List anchors, - Map> allRec) { - List pairs = new ArrayList<>(); + Map>> allRec) { + List triples = new ArrayList<>(); for (int a = 0; a < anchors.size(); a++) { - List seq = allRec.get(anchors.get(a)); - for (int i = 1; i < seq.size(); i++) { - pairs.add(new int[]{a, i}); + List> records = allRec.get(anchors.get(a)); + for (int r = 0; r < records.size(); r++) { + List seq = records.get(r); + for (int i = 1; i < seq.size(); i++) { + triples.add(new int[]{a, r, i}); + } } } - return pairs; + return triples; } }, /** - * Position-first (Γ_pos): iterates over positions, for each position iterates over anchors. + * Position-first (Γ_pos): iterates over positions, for each position over anchors + * and their records. */ POSITION_FIRST { @Override public List buildVisitOrder( List anchors, - Map> allRec) { - List pairs = new ArrayList<>(); + Map>> allRec) { + List triples = new ArrayList<>(); int maxLen = 0; for (CellDerivedItem anchor : anchors) { - maxLen = Math.max(maxLen, allRec.get(anchor).size()); + for (List seq : allRec.get(anchor)) { + maxLen = Math.max(maxLen, seq.size()); + } } for (int i = 1; i < maxLen; i++) { for (int a = 0; a < anchors.size(); a++) { - pairs.add(new int[]{a, i}); + List> records = allRec.get(anchors.get(a)); + for (int r = 0; r < records.size(); r++) { + triples.add(new int[]{a, r, i}); + } } } - return pairs; + return triples; } }; /** - * Builds the visit order of (anchorIndex, positionIndex) pairs - * for schema construction. + * Builds the visit order of (anchorIndex, recordIndex, positionIndex) triples + * for schema construction. Positions beyond the end of a record are skipped by the caller. */ public abstract List buildVisitOrder( List anchors, - Map> allRec); + Map>> allRec); } diff --git a/src/main/java/ru/icc/regtab/interpret/TableInterpreter.java b/src/main/java/ru/icc/regtab/interpret/TableInterpreter.java index 3fc741a6..d144ad2a 100644 --- a/src/main/java/ru/icc/regtab/interpret/TableInterpreter.java +++ b/src/main/java/ru/icc/regtab/interpret/TableInterpreter.java @@ -1,6 +1,7 @@ package ru.icc.regtab.interpret; import ru.icc.regtab.itm.InterpretableTable; +import ru.icc.regtab.itm.semantics.Diagnostic; import ru.icc.regtab.itm.semantics.TableSemantics; import ru.icc.regtab.itm.semantics.WorkingState; import ru.icc.regtab.itm.semantics.action.InterpretationAction; @@ -18,6 +19,12 @@ /** * Table interpreter: derives a recordset from an InterpretableTable * by executing 4 phases (Sec. 3.3). + *

+ * Working state completion applies the actions in operation-type order + * {@code FILL/PREFIX/SUFFIX → AVP → REC → CONCAT → JOIN}: records are folded by concatenation + * before they are multiplied by joins. Preconditions violated by {@code CONCAT}/{@code JOIN} + * leave the working state unchanged and are reported through {@link #diagnostics()} + * (or raise an exception under {@link #withStrictPreconditions(boolean) strict preconditions}). */ public final class TableInterpreter { @@ -28,6 +35,8 @@ public final class TableInterpreter { private MissingValueHandler missingValueHandler = MissingValueHandler.NULL_HANDLER; private List transformations = List.of(); private String anonymousAttributeTemplate = DEFAULT_ANONYMOUS_ATTRIBUTE_TEMPLATE; + private boolean strictPreconditions = false; + private List diagnostics = List.of(); public TableInterpreter withStrategy(SchemaConstructionStrategy strategy) { this.strategy = Objects.requireNonNull(strategy); @@ -64,6 +73,27 @@ public TableInterpreter withAnonymousAttributeTemplate(String template) { return this; } + /** + * Strict preconditions: a {@code CONCAT} / {@code JOIN} action whose precondition is violated + * (e.g. a named attribute shared by two records being concatenated) raises an + * {@link IllegalStateException} instead of having no effect. Default: {@code false} — + * by the formal model the operation has no effect, and the violation is reported through + * {@link #diagnostics()}. + */ + public TableInterpreter withStrictPreconditions(boolean strict) { + this.strictPreconditions = strict; + return this; + } + + /** + * Diagnostics of the most recent {@link #interpret(InterpretableTable)} call: every + * {@code CONCAT} / {@code JOIN} action that was skipped because its precondition was violated. + * Empty if nothing was skipped (or before the first call). + */ + public List diagnostics() { + return diagnostics; + } + /** * Interprets the given table and returns the resulting recordset. */ @@ -75,6 +105,7 @@ public Recordset interpret(InterpretableTable table) { // Phase 2: Working state completion completeWorkingState(ws, sem.actions()); + diagnostics = List.copyOf(ws.diagnostics()); // Phase 3: Recordset extraction Recordset recordset = extractRecordset(ws); @@ -88,7 +119,7 @@ public Recordset interpret(InterpretableTable table) { // --- Phase 1: Working state initialization --- private WorkingState initWorkingState(TableSemantics sem) { - WorkingState ws = new WorkingState(); + WorkingState ws = new WorkingState(strictPreconditions); for (CellDerivedItem item : sem.cellDerivedItems()) { switch (item.type()) { @@ -113,6 +144,7 @@ private void completeWorkingState(WorkingState ws, List ac List strActions = new ArrayList<>(); List avpActions = new ArrayList<>(); List recActions = new ArrayList<>(); + List concatActions = new ArrayList<>(); List joinActions = new ArrayList<>(); for (InterpretationAction action : actions) { @@ -122,6 +154,7 @@ private void completeWorkingState(WorkingState ws, List ac case SuffixOperation ignored -> strActions.add(action); case AvpOperation ignored -> avpActions.add(action); case RecOperation ignored -> recActions.add(action); + case ConcatOperation ignored -> concatActions.add(action); case JoinOperation ignored -> joinActions.add(action); } } @@ -130,11 +163,13 @@ private void completeWorkingState(WorkingState ws, List ac strActions.sort(cmp); avpActions.sort(cmp); recActions.sort(cmp); + concatActions.sort(cmp); joinActions.sort(cmp); for (InterpretationAction action : strActions) applyAction(ws, action); for (InterpretationAction action : avpActions) applyAction(ws, action); for (InterpretationAction action : recActions) applyAction(ws, action); + for (InterpretationAction action : concatActions) applyAction(ws, action); for (InterpretationAction action : joinActions) applyAction(ws, action); } @@ -158,7 +193,8 @@ private void applyAction(WorkingState ws, InterpretationAction action) { // Empty items (e.g. lenient inherited provider on incompatible anchor) → skip case AvpOperation ignored -> { if (!items.isEmpty()) ws.applyAvp(anchor, items); } case RecOperation ignored -> ws.applyRec((CellDerivedItem) anchor, items); - case JoinOperation op -> { if (!items.isEmpty()) ws.applyJoin((CellDerivedItem) anchor, items, op.keyPositions()); } + case ConcatOperation op -> { if (!items.isEmpty()) ws.applyConcat((CellDerivedItem) anchor, items, op.keyPositions()); } + case JoinOperation op -> { if (!items.isEmpty()) ws.applyJoin((CellDerivedItem) anchor, items, op.keyPositions()); } } } @@ -176,7 +212,7 @@ private Recordset extractRecordset(WorkingState ws) { } private Schema constructSchema(WorkingState ws) { - Map> allRec = ws.allRec(); + Map>> allRec = ws.allRec(); // live anchors only List anchors = new ArrayList<>(allRec.keySet()); List schemaAttrs = new ArrayList<>(); @@ -198,13 +234,15 @@ private Schema constructSchema(WorkingState ws) { } schemaAttrs.add(a1); - List pairs = strategy.buildVisitOrder(anchors, allRec); + List triples = strategy.buildVisitOrder(anchors, allRec); Set inSchema = new LinkedHashSet<>(schemaAttrs); - for (int[] pair : pairs) { - CellDerivedItem anchor = anchors.get(pair[0]); - int posIdx = pair[1]; - List sequence = allRec.get(anchor); + for (int[] triple : triples) { + CellDerivedItem anchor = anchors.get(triple[0]); + List> records = allRec.get(anchor); + if (triple[1] >= records.size()) continue; + List sequence = records.get(triple[1]); + int posIdx = triple[2]; if (posIdx >= sequence.size()) continue; Item item = sequence.get(posIdx); @@ -238,18 +276,20 @@ private List generateRecords(WorkingState ws, Schema schema) { List records = new ArrayList<>(); List attrs = schema.attributes(); - for (var entry : ws.allRec().entrySet()) { - String[] values = new String[attrs.size()]; - for (int i = 0; i < values.length; i++) { - values[i] = missingValueHandler.handle(attrs.get(i)); - } - for (Item item : entry.getValue()) { - String a = ws.assoc(item); - if (a == null) continue; - int i = schema.indexOf(a); - if (i >= 0) values[i] = ws.val(item); + for (var entry : ws.allRec().entrySet()) { // live anchors only + for (List sequence : entry.getValue()) { + String[] values = new String[attrs.size()]; + for (int i = 0; i < values.length; i++) { + values[i] = missingValueHandler.handle(attrs.get(i)); + } + for (Item item : sequence) { + String a = ws.assoc(item); + if (a == null) continue; + int i = schema.indexOf(a); + if (i >= 0) values[i] = ws.val(item); + } + records.add(new Record(schema, values)); } - records.add(new Record(schema, values)); } return records; } diff --git a/src/main/java/ru/icc/regtab/itm/semantics/Diagnostic.java b/src/main/java/ru/icc/regtab/itm/semantics/Diagnostic.java new file mode 100644 index 00000000..9e14fde7 --- /dev/null +++ b/src/main/java/ru/icc/regtab/itm/semantics/Diagnostic.java @@ -0,0 +1,28 @@ +package ru.icc.regtab.itm.semantics; + +import ru.icc.regtab.itm.semantics.item.CellDerivedItem; + +import java.util.Objects; + +/** + * A diagnostic recorded during working state completion when an operation is not applicable + * to its anchor (a violated precondition, e.g. a named attribute shared by two records that + * are being concatenated). By the formal model the operation then has no effect; the + * diagnostic makes the silent no-op visible to the pattern author. + * + * @param anchor the anchor item of the action that was skipped + * @param operation the operation name ({@code CONCAT}, {@code JOIN}, …) + * @param message what precondition was violated and why + */ +public record Diagnostic(CellDerivedItem anchor, String operation, String message) { + public Diagnostic { + Objects.requireNonNull(anchor, "anchor"); + Objects.requireNonNull(operation, "operation"); + Objects.requireNonNull(message, "message"); + } + + @Override + public String toString() { + return operation + " skipped at " + anchor + ": " + message; + } +} diff --git a/src/main/java/ru/icc/regtab/itm/semantics/WorkingState.java b/src/main/java/ru/icc/regtab/itm/semantics/WorkingState.java index eaadc1b1..30070c15 100644 --- a/src/main/java/ru/icc/regtab/itm/semantics/WorkingState.java +++ b/src/main/java/ru/icc/regtab/itm/semantics/WorkingState.java @@ -8,9 +8,17 @@ import java.util.*; /** - * Working state of an ITM instance (def:working-state). + * Working state of an ITM instance (def:working-state): + * {@code ws = (V, A, val, attr, avp, rec, J)}. * Tracks values, attributes, attribute-value pairs, and item-based records * as they are built up during table interpretation. + *

+ * {@code rec} maps an anchor item to a non-empty sequence of item-based records: + * a single record after {@code O_rec}/{@code O_concat}, several after {@code O_join} + * (the record product). {@code J} — the joined-away anchors — are items whose records + * have been consumed by a join; they stay in {@code rec} (a later join may consume the same + * records again, irrespective of action order) but are excluded from recordset extraction. + * {@link #allRec()} therefore returns the live anchors {@code dom(rec) \ J} only. */ public final class WorkingState { @@ -20,24 +28,68 @@ public final class WorkingState { private final Map attr = new IdentityHashMap<>(); private final Map avp = new IdentityHashMap<>(); /** Insertion order of rec defines the order of records — keep it. */ - private final Map> rec = new LinkedHashMap<>(); + private final Map>> rec = new LinkedHashMap<>(); + /** J: joined-away anchors (identity semantics, like the items themselves). */ + private final Set joined = Collections.newSetFromMap(new IdentityHashMap<>()); + /** Preconditions violated during completion; the operations had no effect. */ + private final List diagnostics = new ArrayList<>(); + private final boolean strictPreconditions; + + public WorkingState() { + this(false); + } + + /** + * @param strictPreconditions if {@code true}, a violated precondition of {@code O_concat} / + * {@code O_join} raises an {@link IllegalStateException} instead of + * being recorded as a {@link Diagnostic} with no effect + */ + public WorkingState(boolean strictPreconditions) { + this.strictPreconditions = strictPreconditions; + } // --- Accessors --- public String val(Item item) { return val.get(item); } public String attr(Item item) { return attr.get(item); } public AttributeValuePair avp(Item item) { return avp.get(item); } - public List rec(CellDerivedItem item) { return rec.get(item); } + + /** rec(ι): the records of the anchor, also for joined-away anchors; {@code null} if ι ∉ dom(rec). */ + public List> rec(CellDerivedItem item) { + List> records = rec.get(item); + return records == null ? null : Collections.unmodifiableList(records); + } public boolean hasVal(Item item) { return val.containsKey(item); } public boolean hasAttr(Item item) { return attr.containsKey(item); } public boolean hasAvp(Item item) { return avp.containsKey(item); } public boolean hasRec(CellDerivedItem item) { return rec.containsKey(item); } + /** ι ∈ J. */ + public boolean isJoined(CellDerivedItem item) { return joined.contains(item); } public Map allVal() { return Collections.unmodifiableMap(val); } public Map allAttr() { return Collections.unmodifiableMap(attr); } public Map allAvp() { return Collections.unmodifiableMap(avp); } - public Map> allRec() { return Collections.unmodifiableMap(rec); } + + /** + * The records of the live anchors, {@code dom(rec) \ J}, in insertion order — + * exactly what recordset extraction sees. Joined-away anchors are reachable through + * {@link #rec(CellDerivedItem)} and {@link #allJoined()}. + */ + public Map>> allRec() { + if (joined.isEmpty()) return Collections.unmodifiableMap(rec); + Map>> live = new LinkedHashMap<>(); + for (var entry : rec.entrySet()) { + if (!joined.contains(entry.getKey())) live.put(entry.getKey(), entry.getValue()); + } + return Collections.unmodifiableMap(live); + } + + /** J: the joined-away anchors. */ + public Set allJoined() { return Collections.unmodifiableSet(joined); } + + /** Preconditions violated so far (the corresponding operations had no effect). */ + public List diagnostics() { return Collections.unmodifiableList(diagnostics); } /** * Derived function: assoc(iota) = a iff avp(iota) = (a, v). @@ -105,7 +157,7 @@ public void applyAvp(Item anchor, List items) { avp.put(anchor, new AttributeValuePair(a, v)); } - // --- O_rec: rec(anchor) := --- + // --- O_rec: rec(anchor) := <> --- public void applyRec(CellDerivedItem anchor, List items) { if (!val.containsKey(anchor)) return; @@ -119,23 +171,99 @@ public void applyRec(CellDerivedItem anchor, List items) { } sequence.add(item); } - rec.put(anchor, sequence); + List> records = new ArrayList<>(1); + records.add(sequence); + rec.put(anchor, records); } - // --- O_join^K: rec(anchor) := dedup(rec(anchor) · drop_K(rec(i1)) · ... · drop_K(rec(in))) --- + // --- O_concat^K: rec(anchor) := ; rec.remove(i_k) --- - public void applyJoin(CellDerivedItem anchor, List items, Set keyPositions) { - List anchorRec = rec.get(anchor); - if (anchorRec == null || items.isEmpty()) return; - List result = new ArrayList<>(anchorRec); + /** + * Concatenates the records of the provided anchors to the anchor's record (one wide record) + * and removes them from dom(rec). Applicable iff (i) the anchor and at least one provided + * item have records, (ii) all records agree at the key positions K, and (iii) apart from the + * key no named attribute occurs in more than one of the concatenated records; otherwise + * the operation has no effect and a {@link Diagnostic} is recorded. + */ + public void applyConcat(CellDerivedItem anchor, List items, Set keyPositions) { + List> anchorRecs = rec.get(anchor); + if (anchorRecs == null || items.isEmpty()) return; + List others = new ArrayList<>(); for (Item item : items) { - if (!(item instanceof CellDerivedItem cdi)) continue; - List otherRec = rec.get(cdi); - if (otherRec == null) continue; + if (item instanceof CellDerivedItem cdi && cdi != anchor && rec.containsKey(cdi) && !others.contains(cdi)) { + others.add(cdi); + } + } + if (others.isEmpty()) return; // (i) + + List anchorRec = anchorRecs.getFirst(); + List result = new ArrayList<>(anchorRec); + for (CellDerivedItem other : others) { + List otherRec = rec.get(other).getFirst(); + String problem = keyMismatch(anchorRec, otherRec, keyPositions); // (ii) + if (problem != null) { + skip(anchor, "CONCAT", problem); + return; + } result.addAll(dropK(otherRec, keyPositions)); - rec.remove(cdi); } - rec.put(anchor, dedup(result)); + String duplicate = duplicateAttribute(result); // (iii) + if (duplicate != null) { + skip(anchor, "CONCAT", "named attribute '" + duplicate + + "' occurs in more than one of the concatenated records"); + return; + } + List> records = new ArrayList<>(1); + records.add(result); + rec.put(anchor, records); + for (CellDerivedItem other : others) { + rec.remove(other); + joined.remove(other); + } + } + + // --- O_join^K: rec(anchor) := ; J := J ∪ {i_k} --- + + /** + * Multiplies every record of the anchor by every record of the provided anchors + * (a cross product for K = ∅, an equi-join on the key positions K otherwise; a named + * attribute shared by two records acts as a natural-join condition) and marks the provided + * anchors as joined-away. Pairs whose key positions differ or whose shared attributes + * disagree are dropped; if no pair survives, the anchor keeps its records (left outer join). + */ + public void applyJoin(CellDerivedItem anchor, List items, Set keyPositions) { + List> anchorRecs = rec.get(anchor); + if (anchorRecs == null || items.isEmpty()) return; + List others = new ArrayList<>(); + List> joinedRecs = new ArrayList<>(); + for (Item item : items) { + if (item instanceof CellDerivedItem cdi && cdi != anchor && rec.containsKey(cdi) && !others.contains(cdi)) { + others.add(cdi); + joinedRecs.addAll(rec.get(cdi)); + } + } + if (others.isEmpty()) return; + + List> result = new ArrayList<>(); + int dropped = 0; + for (List rho : anchorRecs) { + for (List rho2 : joinedRecs) { + if (keyMismatch(rho, rho2, keyPositions) != null || !agree(rho, rho2)) { + dropped++; + continue; + } + List combined = new ArrayList<>(rho); + combined.addAll(dropK(rho2, keyPositions)); + result.add(dedup(combined)); + } + } + if (result.isEmpty()) { + skip(anchor, "JOIN", "none of the " + dropped + " record pairs satisfies the key/attribute conditions; " + + "the anchor keeps its records"); + } else { + rec.put(anchor, result); + } + joined.addAll(others); } /** drop_K(ρ̄): returns sequence with items at positions k ∈ K removed (0-based). */ @@ -159,15 +287,73 @@ private List dedup(List sequence) { return result; } + /** + * compat_K(ρ, ρ'): {@code null} if for every k ∈ K both records are long enough and the items + * at k are compatible (both with the same attribute-value pair, or both unnamed with the same + * value); otherwise a description of the first mismatch. + */ + private String keyMismatch(List rho, List rho2, Set keyPositions) { + for (int k : keyPositions) { + if (k >= rho.size() || k >= rho2.size()) { + return "key position " + k + " is beyond the end of a record"; + } + Item a = rho.get(k), b = rho2.get(k); + AttributeValuePair pa = avp.get(a), pb = avp.get(b); + if (pa != null && pb != null) { + if (!pa.equals(pb)) return "key position " + k + " differs: " + pa + " vs " + pb; + } else if (pa == null && pb == null) { + if (!Objects.equals(val.get(a), val.get(b))) + return "key position " + k + " differs: '" + val.get(a) + "' vs '" + val.get(b) + "'"; + } else { + return "key position " + k + " mixes a named and an unnamed item"; + } + } + return null; + } + + /** agree(ρ, ρ'): every named attribute appearing in both records carries the same value. */ + private boolean agree(List rho, List rho2) { + Map named = new HashMap<>(); + for (Item item : rho) { + AttributeValuePair p = avp.get(item); + if (p != null) named.putIfAbsent(p.attribute(), p.value()); + } + for (Item item : rho2) { + AttributeValuePair p = avp.get(item); + if (p == null) continue; + String v = named.get(p.attribute()); + if (v != null && !v.equals(p.value())) return false; + } + return true; + } + + /** The first named attribute occurring twice in the sequence, or {@code null}. */ + private String duplicateAttribute(List sequence) { + Set seen = new HashSet<>(); + for (Item item : sequence) { + String a = assoc(item); + if (a != null && !seen.add(a)) return a; + } + return null; + } + + private void skip(CellDerivedItem anchor, String operation, String message) { + Diagnostic d = new Diagnostic(anchor, operation, message); + diagnostics.add(d); + if (strictPreconditions) throw new IllegalStateException(d.toString()); + } + // --- Consistency checks --- /** - * For every iota in dom(rec): rec(iota)[0] == iota. + * For every iota in dom(rec) and every record of it: record[0] == iota. */ public boolean isRecAnchored() { for (var entry : rec.entrySet()) { - if (entry.getValue().isEmpty() || entry.getValue().getFirst() != entry.getKey()) { - return false; + for (List record : entry.getValue()) { + if (record.isEmpty() || record.getFirst() != entry.getKey()) { + return false; + } } } return true; @@ -194,13 +380,14 @@ public boolean isConsistent() { } /** - * All anchors in dom(rec) either have no associated attribute, + * All live anchors in dom(rec) \ J either have no associated attribute, * or share the same attribute. */ public boolean isAnchorAttributeUniform() { String commonAttr = null; boolean found = false; for (CellDerivedItem anchor : rec.keySet()) { + if (joined.contains(anchor)) continue; String a = assoc(anchor); if (a != null) { if (!found) { @@ -215,17 +402,14 @@ public boolean isAnchorAttributeUniform() { } /** - * For every item-based record, the attributes of items + * For every item-based record of every live anchor, the attributes of items * that have associated attributes are pairwise distinct. */ public boolean isRecordAttributesDistinct() { - for (List sequence : rec.values()) { - Set seen = new HashSet<>(); - for (Item item : sequence) { - String a = assoc(item); - if (a != null && !seen.add(a)) { - return false; - } + for (var entry : rec.entrySet()) { + if (joined.contains(entry.getKey())) continue; + for (List record : entry.getValue()) { + if (duplicateAttribute(record) != null) return false; } } return true; diff --git a/src/main/java/ru/icc/regtab/itm/semantics/operation/ConcatOperation.java b/src/main/java/ru/icc/regtab/itm/semantics/operation/ConcatOperation.java new file mode 100644 index 00000000..0400e2c3 --- /dev/null +++ b/src/main/java/ru/icc/regtab/itm/semantics/operation/ConcatOperation.java @@ -0,0 +1,25 @@ +package ru.icc.regtab.itm.semantics.operation; + +import java.util.Objects; +import java.util.Set; + +/** + * O_concat^K: concatenates the item-based records of the provided items to the anchor's record + * (one wide record), dropping the items at the key positions K from each concatenated record, + * and removes the concatenated anchors from dom(rec). The number of records strictly decreases. + *

+ * Applicable only if the key positions agree across all records and no named attribute + * (apart from the key) occurs in more than one of the concatenated records; otherwise the + * operation has no effect and a diagnostic is recorded. + *

+ * RTL: {@code CONCAT}, {@code CONCAT(k1, k2, …)}. Up to jRegTab 0.5.x this operation was + * called {@code JOIN(K)}; {@code JOIN} now denotes the record product ({@link JoinOperation}). + * + * @param keyPositions K ⊆ ℕ₀; items at these positions are dropped from the concatenated records. + * Empty set means no positions are dropped (all items included). + */ +public record ConcatOperation(Set keyPositions) implements WorkingStateOperation { + public ConcatOperation { + keyPositions = Set.copyOf(Objects.requireNonNull(keyPositions, "keyPositions")); + } +} diff --git a/src/main/java/ru/icc/regtab/itm/semantics/operation/JoinOperation.java b/src/main/java/ru/icc/regtab/itm/semantics/operation/JoinOperation.java index 27d3c67c..a2ba27c5 100644 --- a/src/main/java/ru/icc/regtab/itm/semantics/operation/JoinOperation.java +++ b/src/main/java/ru/icc/regtab/itm/semantics/operation/JoinOperation.java @@ -4,12 +4,21 @@ import java.util.Set; /** - * O_join^K: joins item-based records into the anchor's record, - * dropping items at key positions K from each joined record, then deduplicating - * by named attribute, and removing the joined records from dom(rec). + * O_join^K: the record product. Every record of the anchor is combined with every record of the + * provided anchors — a cross product for K = ∅, an equi-join on the key positions K otherwise; + * the key items of the joined record are dropped, and a named attribute shared by the two + * records acts as a natural-join condition (the pair is kept only if the values agree, and the + * attribute occurs once in the result). The provided anchors are marked as joined-away and are + * excluded from recordset extraction; their records stay available, so that several anchors may + * join the same records irrespective of the order in which the actions are applied. + * If no record pair satisfies the conditions, the anchor keeps its records (left outer join). + *

+ * The width of the records stays fixed while their number grows — the counterpart of + * {@code CROSS JOIN} / {@code LATERAL}. The folding operation that was called {@code JOIN(K)} + * up to jRegTab 0.5.x is now {@link ConcatOperation} ({@code CONCAT(K)}). * - * @param keyPositions K ⊆ ℕ₀; items at these positions are dropped from joined records before merge. - * Empty set means no positions are dropped (all items included). + * @param keyPositions K ⊆ ℕ₀; positions at which a record pair must agree, dropped from the + * joined record. Empty set means a cross product. */ public record JoinOperation(Set keyPositions) implements WorkingStateOperation { public JoinOperation { diff --git a/src/main/java/ru/icc/regtab/itm/semantics/operation/WorkingStateOperation.java b/src/main/java/ru/icc/regtab/itm/semantics/operation/WorkingStateOperation.java index 1b41b70a..e01f9737 100644 --- a/src/main/java/ru/icc/regtab/itm/semantics/operation/WorkingStateOperation.java +++ b/src/main/java/ru/icc/regtab/itm/semantics/operation/WorkingStateOperation.java @@ -2,9 +2,9 @@ /** * Working-state update operation (def:ws-update-operation). - * Sealed interface with six permitted implementations. + * Sealed interface with seven permitted implementations. */ public sealed interface WorkingStateOperation permits FillOperation, PrefixOperation, SuffixOperation, - AvpOperation, RecOperation, JoinOperation { + AvpOperation, RecOperation, ConcatOperation, JoinOperation { } diff --git a/src/main/java/ru/icc/regtab/rtl/AtpToRtlSerializer.java b/src/main/java/ru/icc/regtab/rtl/AtpToRtlSerializer.java index d486a777..bbd39186 100644 --- a/src/main/java/ru/icc/regtab/rtl/AtpToRtlSerializer.java +++ b/src/main/java/ru/icc/regtab/rtl/AtpToRtlSerializer.java @@ -245,6 +245,12 @@ private static String serializeOp(ActionSpec as) { if (as.splitDelimiter() != null) yield "REC('" + escapeString(as.splitDelimiter()) + "')"; yield "REC"; } + case CONCAT -> { + Set kp = as.keyPositions(); + if (kp.isEmpty()) yield "CONCAT"; + String args = kp.stream().sorted().map(Object::toString).collect(Collectors.joining(", ")); + yield "CONCAT(" + args + ")"; + } case JOIN -> { Set kp = as.keyPositions(); if (kp.isEmpty()) yield "JOIN"; diff --git a/src/main/java/ru/icc/regtab/rtl/internal/ATPBuilder.java b/src/main/java/ru/icc/regtab/rtl/internal/ATPBuilder.java index c96a070c..8c98532d 100644 --- a/src/main/java/ru/icc/regtab/rtl/internal/ATPBuilder.java +++ b/src/main/java/ru/icc/regtab/rtl/internal/ATPBuilder.java @@ -338,7 +338,7 @@ private ProviderSpec buildProvSpec(RTLParser.ProvSpecContext ctx, return ProviderSpec.ctxAvp(name, value); } String literal = StringExtractorFactory.parseStringLiteral(ctx.ctxProvSpec().STRING().getText()); - if (op != null && (op.recOp() != null || op.joinOp() != null)) + if (op != null && (op.recOp() != null || op.concatOp() != null || op.joinOp() != null)) return ProviderSpec.ctxVal(literal); return ProviderSpec.ctxAttr(literal); } @@ -351,6 +351,11 @@ private static ActionSpec buildOp(RTLParser.OpContext ctx, List pr String splitDelimiter = rec.STRING() != null ? StringExtractorFactory.parseStringLiteral(rec.STRING().getText()) : null; return new ActionSpec(OperationType.REC, null, providers, anchorPos, splitDelimiter); } + if (ctx.concatOp() != null) { + Set kp = new LinkedHashSet<>(); + for (var t : ctx.concatOp().INT()) kp.add(Integer.parseInt(t.getText())); + return new ActionSpec(OperationType.CONCAT, null, providers, null, null, Set.copyOf(kp), false); + } if (ctx.joinOp() != null) { Set kp = new LinkedHashSet<>(); for (var t : ctx.joinOp().INT()) kp.add(Integer.parseInt(t.getText())); diff --git a/src/main/java/ru/icc/regtab/rtl/internal/ProviderTemplateResolver.java b/src/main/java/ru/icc/regtab/rtl/internal/ProviderTemplateResolver.java index b40f7846..6aa0b497 100644 --- a/src/main/java/ru/icc/regtab/rtl/internal/ProviderTemplateResolver.java +++ b/src/main/java/ru/icc/regtab/rtl/internal/ProviderTemplateResolver.java @@ -83,7 +83,7 @@ private static TraversalOrder parseTraversalOrder(RTLParser.TraversalOrderMarkCo private static CellDerivedProviderKind inferKind(RTLParser.OpContext op, ItemDerivationDirective anchorType) { if (op == null) return CellDerivedProviderKind.UNRESTRICTED; - if (op.recOp() != null || op.joinOp() != null) return CellDerivedProviderKind.VAL; + if (op.recOp() != null || op.concatOp() != null || op.joinOp() != null) return CellDerivedProviderKind.VAL; if (op.AVP() != null) return CellDerivedProviderKind.ATTR; return CellDerivedProviderKind.UNRESTRICTED; } diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask016Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask016Test.java index 6e9ad03c..97e30390 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask016Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask016Test.java @@ -14,7 +14,7 @@ /** * Task 16: flat table where each anchor cell collects one value to the right - * via REC and joins same-string cells below via JOIN(0). + * via REC and joins same-string cells below via CONCAT(0). *

* Fixtures: {@code src/test/resources/tasks/task_016/} * RTL: {@link ru.icc.regtab.rtl.RtlTask016Test} @@ -36,7 +36,7 @@ protected TablePattern buildPattern() { RowPattern.of(Quantifier.oneOrMore(), CellPattern.of(AtomicContentSpec.val( ActionSpec.rec(ProviderSpec.val(1, RIGHT_OF)), - ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(AtomicContentSpec.val()) ) diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask023Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask023Test.java index 0f298e2d..d2aa0ceb 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask023Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask023Test.java @@ -14,7 +14,7 @@ /** - * Task 23: repeated subtables of exactly 3 rows, each combining AVP, REC, JOIN(0), + * Task 23: repeated subtables of exactly 3 rows, each combining AVP, REC, CONCAT(0), * and SUFFIX actions across same-row and below-same-string providers. *

* Fixtures: {@code src/test/resources/tasks/task_023/} @@ -39,7 +39,7 @@ protected TablePattern buildPattern() { CellPattern.of(AtomicContentSpec.val( ActionSpec.avp(""), ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, SAME_SUBROW)), - ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(AtomicContentSpec.attr( ActionSpec.suffix("", ProviderSpec.any(1, TraversalOrder.ROW_MAJOR, RIGHT_OF)) diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask025Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask025Test.java index 9342369f..920a23a2 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask025Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask025Test.java @@ -14,7 +14,7 @@ /** * Task 25: flat table where each row's first cell uses SUFFIX, slash-delimited REC - * over values to the right, and JOIN(0) to group rows with the same ID string. + * over values to the right, and CONCAT(0) to group rows with the same ID string. *

* Fixtures: {@code src/test/resources/tasks/task_025/} * RTL: {@link ru.icc.regtab.rtl.RtlTask025Test} @@ -40,7 +40,7 @@ protected TablePattern buildPattern() { CellPattern.of(AtomicContentSpec.val( ActionSpec.suffix(SEP, ProviderSpec.any(1, RIGHT_OF)), ActionSpec.rec(SEP, ProviderSpec.val(ProviderSpec.UNBOUNDED, SUBROW_AFTER_ANCHOR)), - ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(Quantifier.oneOrMore(), AtomicContentSpec.val()) ) diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask033Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask033Test.java index 25d2aac5..9140883e 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask033Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask033Test.java @@ -14,7 +14,7 @@ /** * Task 33: flat table where each row's anchor collects same-row values via REC - * and groups rows with the same ID string via JOIN(0). + * and groups rows with the same ID string via CONCAT(0). *

* Fixtures: {@code src/test/resources/tasks/task_033/} * RTL: {@link ru.icc.regtab.rtl.RtlTask033Test} @@ -36,7 +36,7 @@ protected TablePattern buildPattern() { RowPattern.of(Quantifier.oneOrMore(), CellPattern.of(AtomicContentSpec.val( ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, SAME_SUBROW)), - ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(Quantifier.oneOrMore(), AtomicContentSpec.val()) ) diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask046Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask046Test.java index 3261ddb6..25ed16aa 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask046Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask046Test.java @@ -16,7 +16,7 @@ /** * Task 46: repeated subtables with one-or-more non-blank three-cell rows — - * anchor VAL (AVP + same-row REC + below-same-string JOIN(0)), ATTR, and AVP VAL. + * anchor VAL (AVP + same-row REC + below-same-string CONCAT(0)), ATTR, and AVP VAL. *

* Fixtures: {@code src/test/resources/tasks/task_046/} * RTL: {@link ru.icc.regtab.rtl.RtlTask046Test} @@ -41,7 +41,7 @@ protected TablePattern buildPattern() { CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.val( ActionSpec.avp(""), ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, SAME_SUBROW)), - ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.attr()), CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.val( diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask047Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask047Test.java index adc7a350..98afbc25 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask047Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask047Test.java @@ -16,7 +16,7 @@ /** * Task 47: repeated subtables with one-or-more non-blank two-cell rows — - * anchor VAL with same-row REC and below-same-string JOIN(0), plus a plain VAL. + * anchor VAL with same-row REC and below-same-string CONCAT(0), plus a plain VAL. *

* Fixtures: {@code src/test/resources/tasks/task_047/} * RTL: {@link ru.icc.regtab.rtl.RtlTask047Test} @@ -40,7 +40,7 @@ protected TablePattern buildPattern() { RowPattern.of(Quantifier.oneOrMore(), CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.val( ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, SAME_SUBROW)), - ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.val()) ) diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask050Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask050Test.java index aac406b5..c33d4ec5 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask050Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask050Test.java @@ -16,7 +16,7 @@ /** * Task 50: single (non-repeating) flat table with non-blank three-cell rows — - * anchor VAL (AVP + same-row REC + below-same-string JOIN(0)), ATTR, and AVP VAL. + * anchor VAL (AVP + same-row REC + below-same-string CONCAT(0)), ATTR, and AVP VAL. *

* Fixtures: {@code src/test/resources/tasks/task_050/} * RTL: {@link ru.icc.regtab.rtl.RtlTask050Test} @@ -41,7 +41,7 @@ protected TablePattern buildPattern() { CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.val( ActionSpec.avp(""), ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, SAME_SUBROW)), - ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.attr()), CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.val( diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask053Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask053Test.java index d98ed36c..4134fbdb 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask053Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask053Test.java @@ -41,7 +41,7 @@ protected TablePattern buildPattern() { SubrowPattern.of( CellPattern.of(AtomicContentSpec.val( ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, SAME_ROW)), - ActionSpec.join(0, ProviderSpec.val(1, BELOW_STR)), + ActionSpec.concat(0, ProviderSpec.val(1, BELOW_STR)), ActionSpec.avp("ID") )) ), diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask069Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask069Test.java index a5845278..735a2ade 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask069Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask069Test.java @@ -45,11 +45,11 @@ protected TablePattern buildPattern() { CellPattern.of(AtomicContentSpec.attr()), CellPattern.of(AtomicContentSpec.valTagged("#1", avpSR, recBW, - ActionSpec.join(ProviderSpec.val(ProviderSpec.UNBOUNDED, ROW_TAG1)) + ActionSpec.concat(ProviderSpec.val(ProviderSpec.UNBOUNDED, ROW_TAG1)) )), CellPattern.of(AtomicContentSpec.valTagged("#2", avpSR, recBW, - ActionSpec.join(ProviderSpec.val(ProviderSpec.UNBOUNDED, ROW_TAG2)) + ActionSpec.concat(ProviderSpec.val(ProviderSpec.UNBOUNDED, ROW_TAG2)) )) ) ), diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask094Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask094Test.java index 06e8a57a..67bc1ee9 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask094Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask094Test.java @@ -17,13 +17,13 @@ /** * Task 94: single header row (groups separated by optional blank) above one-or-more * blank-separated data blocks; COL*->REC collects all same-column VALs regardless of - * subtable boundaries, (ROW & C+1.. & STR)*->JOIN(0) merges sibling header columns + * subtable boundaries, (ROW & C+1.. & STR)*->CONCAT(0) merges sibling header columns * into one record. *

* Fixtures: {@code src/test/resources/tasks/task_094/} * RTL: {@link ru.icc.regtab.rtl.RtlTask094Test} * 

- * [ { [!BLANK? VAL: COL*->REC, (ROW & C+1.. & STR)*->JOIN(0)]+ [BLANK?]? }+ ]
+ * [ { [!BLANK? VAL: COL*->REC, (ROW & C+1.. & STR)*->CONCAT(0)]+ [BLANK?]? }+ ]
  * { [ { [!BLANK? VAL]+ [BLANK?]? }+ ]+
  *   [ [BLANK?]+ ]? }+
  *
@@ -46,7 +46,7 @@ protected TablePattern buildPattern() { ); ActionSpec colRec = ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, sameCol)); - ActionSpec rowJoin = ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, rowColRightStr)); + ActionSpec rowJoin = ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, rowColRightStr)); CellPattern headerCell = CellPattern.of(NOT_BLANK, Quantifier.oneOrMore(), AtomicContentSpec.val(colRec, rowJoin)); diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask097Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask097Test.java index be49b094..51689674 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask097Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask097Test.java @@ -16,7 +16,7 @@ /** * Task 97: flat table where each row's anchor VAL collects same-subrow values to the right * via RT*->REC, and joins all records of same-string VALs below in the same subcol - * via (BW&STR)*->JOIN(0,1), dropping key positions {0,1} from each joined record. + * via (BW&STR)*->CONCAT(0,1), dropping key positions {0,1} from each joined record. *

* Fixtures: {@code src/test/resources/tasks/task_097/} * RTL: {@link ru.icc.regtab.rtl.RtlTask097Test} @@ -37,7 +37,7 @@ protected TablePattern buildPattern() { RowPattern.of(Quantifier.oneOrMore(), CellPattern.of(AtomicContentSpec.val( ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, RIGHT_OF)), - ActionSpec.join(Set.of(0, 1), ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(Set.of(0, 1), ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(Quantifier.oneOrMore(), AtomicContentSpec.val()) ) diff --git a/src/test/java/ru/icc/regtab/atp/AtpTask098Test.java b/src/test/java/ru/icc/regtab/atp/AtpTask098Test.java index 0a93debc..5810e3d2 100644 --- a/src/test/java/ru/icc/regtab/atp/AtpTask098Test.java +++ b/src/test/java/ru/icc/regtab/atp/AtpTask098Test.java @@ -16,7 +16,7 @@ /** * Task 98: headed flat table with one header row (two blank cells + ATTR+ header cells), * then data rows where anchor VAL at col 0 collects all right-of cells via RT*->REC, - * joins same-string rows below via (BW&STR)*->JOIN(0,1), and each VAL at cols 2+ + * joins same-string rows below via (BW&STR)*->CONCAT(0,1,2,3), and each VAL at cols 2+ * carries a COL->AVP action that maps it to its column header attribute. *

* Fixtures: {@code src/test/resources/tasks/task_098/} @@ -42,11 +42,11 @@ protected TablePattern buildPattern() { CellPattern.skip(), CellPattern.of(Quantifier.oneOrMore(), AtomicContentSpec.attr()) ), - // Data rows: [VAL: RT*->REC, (BW&STR)*->JOIN(0,1)] [VAL] [VAL: COL->AVP]{2} [VAL]+ + // Data rows: [VAL: RT*->REC, (BW&STR)*->CONCAT(0,1,2,3)] [VAL] [VAL: COL->AVP]{2} [VAL]+ RowPattern.of(Quantifier.oneOrMore(), CellPattern.of(AtomicContentSpec.val( ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, RIGHT_OF)), - ActionSpec.join(Set.of(0, 1), ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(Set.of(0, 1, 2, 3), ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), CellPattern.of(AtomicContentSpec.val()), CellPattern.of(Quantifier.exactly(2), AtomicContentSpec.val( diff --git a/src/test/java/ru/icc/regtab/dsl/DslSpikeTest.java b/src/test/java/ru/icc/regtab/dsl/DslSpikeTest.java index 283da10b..4526041a 100644 --- a/src/test/java/ru/icc/regtab/dsl/DslSpikeTest.java +++ b/src/test/java/ru/icc/regtab/dsl/DslSpikeTest.java @@ -81,13 +81,13 @@ void task015() { } @Test - @DisplayName("016: REC + JOIN(0) with bare conjunction BW&STR*") + @DisplayName("016: REC + CONCAT(0) with bare conjunction BW&STR*") void task016() { assertMirrors(/* language=RTL */ """ - [ [VAL : RT->REC, BW&STR*->JOIN(0)] [VAL] ]+ + [ [VAL : RT->REC, BW&STR*->CONCAT(0)] [VAL] ]+ """, table(subtable( - row(cell(VAL, rec(RT), join(0, BW.and(STR).unbounded())), cell(VAL)) + row(cell(VAL, rec(RT), concat(0, BW.and(STR).unbounded())), cell(VAL)) .oneOrMore()))); } @@ -109,11 +109,11 @@ void task022() { @DisplayName("023: empty context AVP, SUFFIX, AUX, provider-based AVP") void task023() { assertMirrors(/* language=RTL */ """ - { [ [VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+ + { [ [VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+ """, table( subtable( - row(cell(VAL, avp(""), rec(SR.unbounded()), join(0, BW.and(STR).unbounded())), + row(cell(VAL, avp(""), rec(SR.unbounded()), concat(0, BW.and(STR).unbounded())), cell(ATTR, suffix(RT)), cell(AUX), cell(VAL, avp(SR)) @@ -196,12 +196,12 @@ void task013() { @DisplayName("025: SUFFIX('/'), REC('/') split, relative open column range C+2..*") void task025() { assertMirrors(/* language=RTL */ """ - [ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->JOIN(0)] [VAL]+ ]+ + [ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->CONCAT(0)] [VAL]+ ]+ """, table(subtable(row( cell(VAL, suffix("/", RT), recSplit("/", RT.and(CrelFrom(2)).unbounded()), - join(0, BW.and(STR).unbounded())), + concat(0, BW.and(STR).unbounded())), cell(VAL).oneOrMore()).oneOrMore()))); } @@ -222,12 +222,12 @@ void task029() { @DisplayName("069: row-level inherited REC merged down into subrow atoms") void task069() { assertMirrors(/* language=RTL */ """ - [ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->JOIN][VAL#'2': ROW&#'2'*->JOIN] }* ] + [ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->CONCAT][VAL#'2': ROW&#'2'*->CONCAT] }* ] """, table(subtable(row(acts(rec(BW.unbounded())), subrow(cell(ATTR), - cell(val(join(ROW.and(tag("1")).unbounded())).tagged("1")), - cell(val(join(ROW.and(tag("2")).unbounded())).tagged("2"))) + cell(val(concat(ROW.and(tag("1")).unbounded())).tagged("1")), + cell(val(concat(ROW.and(tag("2")).unbounded())).tagged("2"))) .zeroOrMore())))); } diff --git a/src/test/java/ru/icc/regtab/interpret/EquipmentTest.java b/src/test/java/ru/icc/regtab/interpret/EquipmentTest.java index 3bead3d1..974401e3 100644 --- a/src/test/java/ru/icc/regtab/interpret/EquipmentTest.java +++ b/src/test/java/ru/icc/regtab/interpret/EquipmentTest.java @@ -125,14 +125,14 @@ void testEquipment() { List.of(new CellDerivedItemProvider( (a, cand) -> cand == target1, TraversalOrder.ROW_MAJOR, allCdi, 1)), - new JoinOperation(Set.of(0)))); + new ConcatOperation(Set.of(0)))); CellDerivedItem target3 = identity[3]; actions.add(new InterpretationAction(identity[2], List.of(new CellDerivedItemProvider( (a, cand) -> cand == target3, TraversalOrder.ROW_MAJOR, allCdi, 1)), - new JoinOperation(Set.of(0)))); + new ConcatOperation(Set.of(0)))); TableSemantics semantics = new TableSemantics(allCdi, allCtx, actions); InterpretableTable itm = new InterpretableTable(syntax, semantics); diff --git a/src/test/java/ru/icc/regtab/interpret/SchemaFlexibleTest.java b/src/test/java/ru/icc/regtab/interpret/SchemaFlexibleTest.java index b3c1325e..cf66f09d 100644 --- a/src/test/java/ru/icc/regtab/interpret/SchemaFlexibleTest.java +++ b/src/test/java/ru/icc/regtab/interpret/SchemaFlexibleTest.java @@ -88,7 +88,7 @@ void testSchemaFlexible() { List.of(new CellDerivedItemProvider( sameNameBelow, TraversalOrder.ROW_MAJOR, allCdi)), - new JoinOperation(Set.of(0)))); + new ConcatOperation(Set.of(0)))); } TableSemantics semantics = new TableSemantics(allCdi, Set.of(), actions); @@ -176,7 +176,7 @@ void testCustomAnonymousAttributeTemplate() { List.of(new CellDerivedItemProvider( sameNameBelow, TraversalOrder.ROW_MAJOR, allCdi)), - new JoinOperation(Set.of(0)))); + new ConcatOperation(Set.of(0)))); } TableSemantics semantics = new TableSemantics(allCdi, Set.of(), actions); diff --git a/src/test/java/ru/icc/regtab/interpret/TableInterpreterMultiRecordTest.java b/src/test/java/ru/icc/regtab/interpret/TableInterpreterMultiRecordTest.java new file mode 100644 index 00000000..6a562365 --- /dev/null +++ b/src/test/java/ru/icc/regtab/interpret/TableInterpreterMultiRecordTest.java @@ -0,0 +1,119 @@ +package ru.icc.regtab.interpret; + +import org.junit.jupiter.api.Test; +import ru.icc.regtab.atp.AtpMatcher; +import ru.icc.regtab.atp.spec.TablePattern; +import ru.icc.regtab.itm.InterpretableTable; +import ru.icc.regtab.itm.semantics.Diagnostic; +import ru.icc.regtab.itm.syntax.TableSyntax; +import ru.icc.regtab.recordset.Record; +import ru.icc.regtab.recordset.Recordset; +import ru.icc.regtab.rtl.RtlCompiler; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * End-to-end: anchors carrying several records after {@code JOIN} (the record product) flow + * through schema construction (both strategies) and record generation; joined-away anchors + * are excluded from the recordset and from recordset-consistency; {@code CONCAT} conflicts + * surface through {@link TableInterpreter#diagnostics()} or, under strict preconditions, + * as an exception. + */ +class TableInterpreterMultiRecordTest { + + /** id | x | y / a;b | 1 | 2 / c | 3 | 4 — explode × stack, six records. */ + private static TableSyntax explodeStack() { + TableSyntax s = new TableSyntax(3, 3); + String[][] rows = {{"id", "x", "y"}, {"a;b", "1", "2"}, {"c", "3", "4"}}; + for (int r = 0; r < 3; r++) + for (int c = 0; c < 3; c++) s.getCell(r, c).setText(rows[r][c]); + return s; + } + + private static final String JOIN_PRODUCT = """ + [ [ATTR] [VAL: 'var'->AVP]+ ] + [ [(VAL: COL->AVP, ()->REC, RT*->JOIN){';'}] [VAL: 'value'->AVP, COL->REC]+ ]+ + """; + + private static Recordset run(TableInterpreter interpreter, String rtl, TableSyntax syntax) { + TablePattern pattern = RtlCompiler.compile(rtl); + InterpretableTable itm = AtpMatcher.match(pattern, syntax).orElseThrow(); + return pattern.transform(interpreter.interpret(itm)); + } + + private static List row(Record r, String... attrs) { + return java.util.Arrays.stream(attrs).map(r::get).toList(); + } + + @Test + void joinProduct_recordFirst() { + TableInterpreter interpreter = new TableInterpreter(); + Recordset rs = run(interpreter, JOIN_PRODUCT, explodeStack()); + + assertEquals(List.of("id", "value", "var"), rs.schema().attributes()); + assertEquals(6, rs.size()); + List> rows = rs.records().stream().map(r -> row(r, "id", "var", "value")).toList(); + assertEquals(List.of( + List.of("a", "x", "1"), List.of("a", "y", "2"), + List.of("b", "x", "1"), List.of("b", "y", "2"), + List.of("c", "x", "3"), List.of("c", "y", "4")), rows); + assertTrue(interpreter.diagnostics().isEmpty()); + } + + @Test + void joinProduct_positionFirst_sameRecords() { + TableInterpreter interpreter = new TableInterpreter().withStrategy(SchemaConstructionStrategy.POSITION_FIRST); + Recordset rs = run(interpreter, JOIN_PRODUCT, explodeStack()); + + assertEquals(List.of("id", "value", "var"), rs.schema().attributes()); + assertEquals(6, rs.size()); + assertEquals(List.of("b", "y", "2"), row(rs.get(3), "id", "var", "value")); + } + + /** k | v / A | 5 / A | 7 — the two rows share the named attribute v: a CONCAT conflict. */ + private static TableSyntax conflict() { + TableSyntax s = new TableSyntax(3, 2); + String[][] rows = {{"k", "v"}, {"A", "5"}, {"A", "7"}}; + for (int r = 0; r < 3; r++) + for (int c = 0; c < 2; c++) s.getCell(r, c).setText(rows[r][c]); + return s; + } + + private static final String CONCAT_CONFLICT = """ + [ [ATTR]+ ] + [ [VAL: COL->AVP, RT->REC, BW&STR*->CONCAT(0)] [VAL: COL->AVP] ]+ + """; + + @Test + void concatConflict_noEffect_bothRecordsSurvive_diagnosticReported() { + TableInterpreter interpreter = new TableInterpreter(); + Recordset rs = run(interpreter, CONCAT_CONFLICT, conflict()); + + assertEquals(2, rs.size(), "neither row is folded, nothing is lost silently"); + assertEquals(List.of("A", "5"), row(rs.get(0), "k", "v")); + assertEquals(List.of("A", "7"), row(rs.get(1), "k", "v")); + assertEquals(1, interpreter.diagnostics().size()); + Diagnostic d = interpreter.diagnostics().getFirst(); + assertEquals("CONCAT", d.operation()); + assertTrue(d.message().contains("'v'"), d.message()); + } + + @Test + void concatConflict_strictPreconditions_throws() { + TableInterpreter interpreter = new TableInterpreter().withStrictPreconditions(true); + IllegalStateException e = assertThrows(IllegalStateException.class, + () -> run(interpreter, CONCAT_CONFLICT, conflict())); + assertTrue(e.getMessage().contains("CONCAT"), e.getMessage()); + } + + @Test + void diagnosticsAreResetPerInterpretation() { + TableInterpreter interpreter = new TableInterpreter(); + run(interpreter, CONCAT_CONFLICT, conflict()); + assertEquals(1, interpreter.diagnostics().size()); + run(interpreter, JOIN_PRODUCT, explodeStack()); + assertTrue(interpreter.diagnostics().isEmpty()); + } +} diff --git a/src/test/java/ru/icc/regtab/itm/semantics/WorkingStateConcatTest.java b/src/test/java/ru/icc/regtab/itm/semantics/WorkingStateConcatTest.java new file mode 100644 index 00000000..e0bf8253 --- /dev/null +++ b/src/test/java/ru/icc/regtab/itm/semantics/WorkingStateConcatTest.java @@ -0,0 +1,150 @@ +package ru.icc.regtab.itm.semantics; + +import org.junit.jupiter.api.Test; +import ru.icc.regtab.itm.semantics.item.CellDerivedItem; +import ru.icc.regtab.itm.semantics.item.Item; +import ru.icc.regtab.itm.semantics.item.ItemType; +import ru.icc.regtab.itm.syntax.TableSyntax; + +import java.util.List; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * O_concat^K (RTL {@code CONCAT(K)}): folds the records of items sharing a key into one wide + * record. Pins the preconditions: the key positions must agree, and — apart from the key — + * no named attribute may occur in more than one of the concatenated records; a violation + * leaves the working state unchanged and is reported as a {@link Diagnostic} + * (or raises under strict preconditions). Up to 0.5.x a shared attribute was silently + * deduplicated, which hid the conflict. + */ +class WorkingStateConcatTest { + + private final TableSyntax syntax = new TableSyntax(4, 4); + + private CellDerivedItem val(int r, int c, String text) { + syntax.getCell(r, c).setText(text); + return new CellDerivedItem(text, 0, syntax.getCell(r, c), ItemType.VALUE); + } + + private static void name(WorkingState ws, CellDerivedItem item, String attribute) { + ws.setAvp(item, attribute, item.str()); + } + + private static WorkingState init(CellDerivedItem... items) { + WorkingState ws = new WorkingState(); + for (CellDerivedItem i : items) ws.initVal(i, i.str()); + return ws; + } + + @Test + void foldsRecordsWithDistinctAttributes_keyNotRepeated() { + // T-1 | D16 rec(t1a) = + // T-1 | 001 rec(t1b) = + CellDerivedItem t1a = val(1, 0, "T-1"), d16 = val(1, 1, "D16"); + CellDerivedItem t1b = val(2, 0, "T-1"), s001 = val(2, 1, "001"); + WorkingState ws = init(t1a, d16, t1b, s001); + name(ws, t1a, "ID"); name(ws, d16, "REF_TP"); + name(ws, t1b, "ID"); name(ws, s001, "REF_SN"); + ws.applyRec(t1a, List.of(d16)); + ws.applyRec(t1b, List.of(s001)); + + ws.applyConcat(t1a, List.of(t1b), Set.of(0)); + + assertEquals(List.of(List.of(t1a, d16, s001)), ws.rec(t1a)); + assertFalse(ws.hasRec(t1b), "the concatenated anchor is removed from dom(rec)"); + assertEquals(Set.of(t1a), ws.allRec().keySet()); + assertTrue(ws.diagnostics().isEmpty()); + } + + @Test + void foldsUnnamedRecords_task016Shape() { + // book | 5 ; book | 6 ; book | 7 -> book,5,6,7 + CellDerivedItem b1 = val(1, 0, "book"), five = val(1, 1, "5"); + CellDerivedItem b2 = val(2, 0, "book"), six = val(2, 1, "6"); + CellDerivedItem b3 = val(3, 0, "book"), seven = val(3, 1, "7"); + WorkingState ws = init(b1, five, b2, six, b3, seven); + ws.applyRec(b1, List.of(five)); + ws.applyRec(b2, List.of(six)); + ws.applyRec(b3, List.of(seven)); + + ws.applyConcat(b1, List.of(b2, b3), Set.of(0)); + + assertEquals(List.of(List.of(b1, five, six, seven)), ws.rec(b1)); + assertFalse(ws.hasRec(b2)); + assertFalse(ws.hasRec(b3)); + assertTrue(ws.diagnostics().isEmpty()); + } + + @Test + void sharedNamedAttribute_noEffectAndDiagnostic() { + // A | 5 rec = + // A | 7 rec = -- Qty occurs in both: not a key, a conflict + CellDerivedItem a1 = val(1, 0, "A"), five = val(1, 1, "5"); + CellDerivedItem a2 = val(2, 0, "A"), seven = val(2, 1, "7"); + WorkingState ws = init(a1, five, a2, seven); + name(ws, a1, "ID"); name(ws, five, "Qty"); + name(ws, a2, "ID"); name(ws, seven, "Qty"); + ws.applyRec(a1, List.of(five)); + ws.applyRec(a2, List.of(seven)); + + ws.applyConcat(a1, List.of(a2), Set.of(0)); + + assertEquals(List.of(List.of(a1, five)), ws.rec(a1), "anchor record unchanged"); + assertEquals(List.of(List.of(a2, seven)), ws.rec(a2), "the other record is kept: both survive"); + assertEquals(2, ws.allRec().size()); + assertEquals(1, ws.diagnostics().size()); + Diagnostic d = ws.diagnostics().getFirst(); + assertSame(a1, d.anchor()); + assertEquals("CONCAT", d.operation()); + assertTrue(d.message().contains("Qty"), d.message()); + } + + @Test + void keyPositionMismatch_noEffectAndDiagnostic() { + CellDerivedItem x = val(1, 0, "X"), five = val(1, 1, "5"); + CellDerivedItem y = val(2, 0, "Y"), seven = val(2, 1, "7"); + WorkingState ws = init(x, five, y, seven); + ws.applyRec(x, List.of(five)); + ws.applyRec(y, List.of(seven)); + + ws.applyConcat(x, List.of(y), Set.of(0)); + + assertEquals(List.of(List.of(x, five)), ws.rec(x)); + assertTrue(ws.hasRec(y)); + assertEquals(1, ws.diagnostics().size()); + assertTrue(ws.diagnostics().getFirst().message().contains("key position 0")); + } + + @Test + void strictPreconditions_throwWithTheSameMessage() { + CellDerivedItem a1 = val(1, 0, "A"), five = val(1, 1, "5"); + CellDerivedItem a2 = val(2, 0, "A"), seven = val(2, 1, "7"); + WorkingState ws = new WorkingState(true); + for (CellDerivedItem i : List.of(a1, five, a2, seven)) ws.initVal(i, i.str()); + name(ws, a1, "ID"); name(ws, five, "Qty"); + name(ws, a2, "ID"); name(ws, seven, "Qty"); + ws.applyRec(a1, List.of(five)); + ws.applyRec(a2, List.of(seven)); + + IllegalStateException e = assertThrows(IllegalStateException.class, + () -> ws.applyConcat(a1, List.of(a2), Set.of(0))); + assertTrue(e.getMessage().contains("Qty"), e.getMessage()); + assertEquals(1, ws.diagnostics().size(), "the diagnostic is recorded before throwing"); + } + + @Test + void itemsWithoutRecordsAreIgnored() { + CellDerivedItem b1 = val(1, 0, "book"), five = val(1, 1, "5"); + CellDerivedItem stray = val(2, 0, "book"); + WorkingState ws = init(b1, five, stray); + ws.applyRec(b1, List.of(five)); + List> before = ws.rec(b1); + + ws.applyConcat(b1, List.of(stray), Set.of(0)); + + assertEquals(before, ws.rec(b1)); + assertTrue(ws.diagnostics().isEmpty(), "no record to concatenate is not a violation"); + } +} diff --git a/src/test/java/ru/icc/regtab/itm/semantics/WorkingStateJoinTest.java b/src/test/java/ru/icc/regtab/itm/semantics/WorkingStateJoinTest.java new file mode 100644 index 00000000..d4c62ab1 --- /dev/null +++ b/src/test/java/ru/icc/regtab/itm/semantics/WorkingStateJoinTest.java @@ -0,0 +1,176 @@ +package ru.icc.regtab.itm.semantics; + +import org.junit.jupiter.api.Test; +import ru.icc.regtab.itm.semantics.item.CellDerivedItem; +import ru.icc.regtab.itm.semantics.item.Item; +import ru.icc.regtab.itm.semantics.item.ItemType; +import ru.icc.regtab.itm.syntax.TableSyntax; + +import java.util.List; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * O_join^K (RTL {@code JOIN(K)}): the record product. Pins the cross product (K = ∅), the + * nested-loop order of the resulting records, the equi-join on key positions, the natural-join + * treatment of a shared named attribute (agree → kept once, disagree → pair dropped), the + * left-outer behaviour when no pair survives, lazy consumption through J (several anchors may + * join the same records), and composition (a second join multiplies further). + */ +class WorkingStateJoinTest { + + private final TableSyntax syntax = new TableSyntax(5, 5); + + private CellDerivedItem val(int r, int c, String text) { + return val(r, c, text, 0); + } + + private CellDerivedItem val(int r, int c, String text, int index) { + syntax.getCell(r, c).setText(syntax.getCell(r, c).text() == null ? text : syntax.getCell(r, c).text()); + return new CellDerivedItem(text, index, syntax.getCell(r, c), ItemType.VALUE); + } + + private static void name(WorkingState ws, CellDerivedItem item, String attribute) { + ws.setAvp(item, attribute, item.str()); + } + + private static WorkingState init(CellDerivedItem... items) { + WorkingState ws = new WorkingState(); + for (CellDerivedItem i : items) ws.initVal(i, i.str()); + return ws; + } + + // id | x | y tokens a, b of one cell; rec(1) = , rec(2) = + // a;b | 1 | 2 + private CellDerivedItem a, b, c1, c2, hx, hy; + + private WorkingState explodeStack() { + a = val(1, 0, "a", 0); b = val(1, 0, "b", 1); + c1 = val(1, 1, "1"); c2 = val(1, 2, "2"); + hx = val(0, 1, "x"); hy = val(0, 2, "y"); + WorkingState ws = init(a, b, c1, c2, hx, hy); + name(ws, a, "id"); name(ws, b, "id"); + name(ws, c1, "value"); name(ws, c2, "value"); + name(ws, hx, "var"); name(ws, hy, "var"); + ws.applyRec(a, List.of()); + ws.applyRec(b, List.of()); + ws.applyRec(c1, List.of(hx)); + ws.applyRec(c2, List.of(hy)); + return ws; + } + + @Test + void crossProduct_oneRecordPerJoinedRecord_inNestedLoopOrder() { + WorkingState ws = explodeStack(); + + ws.applyJoin(a, List.of(c1, c2), Set.of()); + + assertEquals(List.of(List.of(a, c1, hx), List.of(a, c2, hy)), ws.rec(a)); + assertTrue(ws.isJoined(c1)); + assertTrue(ws.isJoined(c2)); + assertEquals(Set.of(c1, c2), ws.allJoined()); + assertNotNull(ws.rec(c1), "a joined-away anchor keeps its records"); + assertEquals(List.of(a, b), List.copyOf(ws.allRec().keySet()), "allRec() = live anchors only"); + assertTrue(ws.diagnostics().isEmpty()); + } + + @Test + void lazyConsumption_secondAnchorJoinsTheSameRecords() { + WorkingState ws = explodeStack(); + + ws.applyJoin(a, List.of(c1, c2), Set.of()); + ws.applyJoin(b, List.of(c1, c2), Set.of()); + + assertEquals(2, ws.rec(a).size()); + assertEquals(List.of(List.of(b, c1, hx), List.of(b, c2, hy)), ws.rec(b)); + assertTrue(ws.isRecordsetConsistent(), "the 'value' attribute of the joined-away anchors does not break uniformity"); + } + + @Test + void composition_secondJoinMultipliesFurther() { + WorkingState ws = explodeStack(); + CellDerivedItem d1 = val(2, 3, "p"), d2 = val(2, 4, "q"); + ws.initVal(d1, "p"); ws.initVal(d2, "q"); + name(ws, d1, "w"); name(ws, d2, "w"); + ws.applyRec(d1, List.of()); + ws.applyRec(d2, List.of()); + + ws.applyJoin(a, List.of(c1, c2), Set.of()); + ws.applyJoin(a, List.of(d1, d2), Set.of()); + + assertEquals(List.of( + List.of(a, c1, hx, d1), List.of(a, c1, hx, d2), + List.of(a, c2, hy, d1), List.of(a, c2, hy, d2)), ws.rec(a)); + } + + @Test + void equiJoinOnKeyPosition_dropsMismatchedPairsAndTheJoinedKey() { + // rec(p1) = rec(q1) = rec(q2) = + CellDerivedItem p1 = val(1, 0, "X"), qty = val(1, 1, "5"); + CellDerivedItem q1 = val(1, 2, "X"), kg = val(1, 3, "kg"); + CellDerivedItem q2 = val(2, 2, "Z"), pc = val(2, 3, "pc"); + WorkingState ws = init(p1, qty, q1, kg, q2, pc); + name(ws, qty, "Qty"); name(ws, kg, "Unit"); name(ws, pc, "Unit"); + ws.applyRec(p1, List.of(qty)); + ws.applyRec(q1, List.of(kg)); + ws.applyRec(q2, List.of(pc)); + + ws.applyJoin(p1, List.of(q1, q2), Set.of(0)); + + assertEquals(List.of(List.of(p1, qty, kg)), ws.rec(p1)); + assertTrue(ws.diagnostics().isEmpty()); + } + + @Test + void noSurvivingPair_anchorKeepsItsRecords_leftOuter() { + CellDerivedItem p2 = val(1, 0, "Y"), qty = val(1, 1, "8"); + CellDerivedItem q1 = val(1, 2, "X"), kg = val(1, 3, "kg"); + WorkingState ws = init(p2, qty, q1, kg); + name(ws, qty, "Qty"); name(ws, kg, "Unit"); + ws.applyRec(p2, List.of(qty)); + ws.applyRec(q1, List.of(kg)); + + ws.applyJoin(p2, List.of(q1), Set.of(0)); + + assertEquals(List.of(List.of(p2, qty)), ws.rec(p2)); + assertTrue(ws.isJoined(q1), "J is still extended"); + assertEquals(1, ws.diagnostics().size()); + assertEquals("JOIN", ws.diagnostics().getFirst().operation()); + } + + @Test + void sharedNamedAttribute_isANaturalJoinCondition() { + // rec(r1) = ; rec(s1) = ; rec(s2) = + CellDerivedItem r1 = val(1, 0, "S1"), y1 = val(1, 1, "2024"); + CellDerivedItem s1 = val(2, 0, "2024"), sales10 = val(2, 1, "10"); + CellDerivedItem s2 = val(3, 0, "2025"), sales12 = val(3, 1, "12"); + WorkingState ws = init(r1, y1, s1, sales10, s2, sales12); + name(ws, r1, "Store"); name(ws, y1, "Year"); + name(ws, s1, "Year"); name(ws, sales10, "Sales"); + name(ws, s2, "Year"); name(ws, sales12, "Sales"); + ws.applyRec(r1, List.of(y1)); + ws.applyRec(s1, List.of(sales10)); + ws.applyRec(s2, List.of(sales12)); + + ws.applyJoin(r1, List.of(s1, s2), Set.of()); + + List> records = ws.rec(r1); + assertEquals(1, records.size(), "the 2025 pair disagrees on Year and is dropped"); + assertEquals(List.of(r1, y1, sales10), records.getFirst(), "Year occurs once (dedup)"); + assertTrue(ws.diagnostics().isEmpty()); + } + + @Test + void itemsWithoutRecordsAreIgnored() { + WorkingState ws = explodeStack(); + CellDerivedItem stray = val(3, 0, "z"); + ws.initVal(stray, "z"); + + ws.applyJoin(a, List.of(stray), Set.of()); + + assertEquals(List.of(List.of(a)), ws.rec(a)); + assertTrue(ws.allJoined().isEmpty()); + assertTrue(ws.diagnostics().isEmpty()); + } +} diff --git a/src/test/java/ru/icc/regtab/rtl/RtlCompilerTest.java b/src/test/java/ru/icc/regtab/rtl/RtlCompilerTest.java index c5d030f4..9c63276b 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlCompilerTest.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlCompilerTest.java @@ -154,7 +154,7 @@ void parse_fillPrefixSuffixOps() { compile("[ [VAL : (CL)->FILL('/')] ]"); compile("[ [VAL : (CL)->PREFIX(' ')] ]"); compile("[ [VAL : (CL)->SUFFIX(',')] ]"); - compile("[ [VAL : (CL)->JOIN(0)] ]"); + compile("[ [VAL : (CL)->CONCAT(0)] ]"); } @Test diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask016Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask016Test.java index 00799392..08d1976e 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask016Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask016Test.java @@ -2,15 +2,15 @@ /** * Task 16: flat table where each anchor cell in column 0 collects one value - * to the right via REC and concatenates same-string cells below via JOIN(0). + * to the right via REC and concatenates same-string cells below via CONCAT(0). *

* Fixtures: {@code src/test/resources/tasks/task_016/} * ATP: {@link ru.icc.regtab.atp.AtpTask016Test} * 

- * [ [VAL : RT->REC, BW&STR*->JOIN(0)] [VAL] ]+
+ * [ [VAL : RT->REC, BW&STR*->CONCAT(0)] [VAL] ]+
  *
* Data rows: anchor VAL uses RT->REC (1 value immediately to the right) and - * BW&STR*->JOIN(0) (unbounded concatenation of cells that are both below + * BW&STR*->CONCAT(0) (unbounded concatenation of cells that are both below * and have the same string as the anchor), followed by a plain VAL cell. */ public class RtlTask016Test extends RtlTaskBase { @@ -21,7 +21,7 @@ public class RtlTask016Test extends RtlTaskBase { @Override protected String buildRtl() { return /* language=RTL */ """ - [ [VAL : RT->REC, BW&STR*->JOIN(0)] [VAL] ]+ + [ [VAL : RT->REC, BW&STR*->CONCAT(0)] [VAL] ]+ """; } } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask023Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask023Test.java index 222667d3..da67e8c6 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask023Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask023Test.java @@ -2,15 +2,15 @@ /** * Task 23: repeated subtables of exactly 3 rows, each with four cells combining - * AVP, REC, JOIN(0), and SUFFIX actions across same-row and below-same-string providers. + * AVP, REC, CONCAT(0), and SUFFIX actions across same-row and below-same-string providers. *

* Fixtures: {@code src/test/resources/tasks/task_023/} * ATP: {@link ru.icc.regtab.atp.AtpTask023Test} * 

- * { [ [VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+
+ * { [ [VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+
  *
* Each of the 3 rows has: (1) VAL anchor with empty-literal AVP, unbounded same-subrow - * REC, and unbounded below-same-string JOIN(0); (2) ATTR cell that appends the adjacent + * REC, and unbounded below-same-string CONCAT(0); (2) ATTR cell that appends the adjacent * AUX to its value via RT->SUFFIX; (3) an AUX cell; (4) a VAL cell whose attribute is * looked up from the same-subrow ATTR (SR->AVP). */ @@ -22,7 +22,7 @@ public class RtlTask023Test extends RtlTaskBase { @Override protected String buildRtl() { return /* language=RTL */ """ - { [ [VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+ + { [ [VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [ATTR : RT->SUFFIX] [AUX] [VAL : SR->AVP] ]{3} }+ """; } } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask025Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask025Test.java index cee337b0..f45b6b82 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask025Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask025Test.java @@ -3,16 +3,16 @@ /** * Task 25: flat table where each row's first cell is an ID anchor with a * slash-delimited SUFFIX, REC over values two-or-more positions right, and - * JOIN(0) grouping rows with the same ID string. + * CONCAT(0) grouping rows with the same ID string. *

* Fixtures: {@code src/test/resources/tasks/task_025/} * ATP: {@link ru.icc.regtab.atp.AtpTask025Test} * 

- * [ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->JOIN(0)] [VAL]+ ]+
+ * [ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->CONCAT(0)] [VAL]+ ]+
  *
* Each data row: anchor VAL uses RT->SUFFIX('/') (appends the immediately * right cell with '/'), RT&C+2..*->REC('/') (slash-delimited REC of all - * same-row cells from relative column +2 onward), and BW&STR*->JOIN(0) + * same-row cells from relative column +2 onward), and BW&STR*->CONCAT(0) * (concatenates anchors in rows below that share the same ID string). One-or-more * plain VAL cells follow. */ @@ -24,7 +24,7 @@ public class RtlTask025Test extends RtlTaskBase { @Override protected String buildRtl() { return /* language=RTL */ """ - [ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->JOIN(0)] [VAL]+ ]+ + [ [VAL : RT->SUFFIX('/'), RT&C+2..*->REC('/'), BW&STR*->CONCAT(0)] [VAL]+ ]+ """; } } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask033Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask033Test.java index f81ec88c..572da6fb 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask033Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask033Test.java @@ -2,15 +2,15 @@ /** * Task 33: flat table where each row's anchor cell collects same-row values via - * REC and groups rows with the same ID string via JOIN(0). + * REC and groups rows with the same ID string via CONCAT(0). *

* Fixtures: {@code src/test/resources/tasks/task_033/} * ATP: {@link ru.icc.regtab.atp.AtpTask033Test} * 

- * [ [VAL : SR*->REC, BW&STR*->JOIN(0)] [VAL]+ ]+
+ * [ [VAL : SR*->REC, BW&STR*->CONCAT(0)] [VAL]+ ]+
  *
* Each data row: anchor VAL with SR*->REC (unbounded same-subrow collection) - * and BW&STR*->JOIN(0) (unbounded concatenation of cells that are both below + * and BW&STR*->CONCAT(0) (unbounded concatenation of cells that are both below * and share the same string as the anchor). One-or-more plain VAL cells follow. */ public class RtlTask033Test extends RtlTaskBase { @@ -21,7 +21,7 @@ public class RtlTask033Test extends RtlTaskBase { @Override protected String buildRtl() { return /* language=RTL */ """ - [ [VAL : SR*->REC, BW&STR*->JOIN(0)] [VAL]+ ]+ + [ [VAL : SR*->REC, BW&STR*->CONCAT(0)] [VAL]+ ]+ """; } } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask046Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask046Test.java index 4f61b346..79b8f27d 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask046Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask046Test.java @@ -2,15 +2,15 @@ /** * Task 46: repeated subtables with one-or-more non-blank three-cell rows — - * anchor VAL (AVP + same-row REC + below-same-string JOIN(0)), ATTR, and AVP VAL. + * anchor VAL (AVP + same-row REC + below-same-string CONCAT(0)), ATTR, and AVP VAL. *

* Fixtures: {@code src/test/resources/tasks/task_046/} * ATP: {@link ru.icc.regtab.atp.AtpTask046Test} * 

- * { [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+
+ * { [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+
  *
* Each row of each subtable: non-blank anchor VAL with empty-literal AVP, - * SR*->REC (unbounded same-subrow collection), and BW&STR*->JOIN(0) + * SR*->REC (unbounded same-subrow collection), and BW&STR*->CONCAT(0) * (grouping rows with the same string below); non-blank ATTR cell; non-blank * VAL with SR->AVP (same-subrow attribute lookup). */ @@ -22,7 +22,7 @@ public class RtlTask046Test extends RtlTaskBase { @Override protected String buildRtl() { return /* language=RTL */ """ - { [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+ + { [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+ """; } } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask047Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask047Test.java index c2c7c514..b8982f8c 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask047Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask047Test.java @@ -2,15 +2,15 @@ /** * Task 47: repeated subtables with one-or-more non-blank two-cell rows — - * anchor VAL with same-row REC and below-same-string JOIN(0), plus a plain VAL. + * anchor VAL with same-row REC and below-same-string CONCAT(0), plus a plain VAL. *

* Fixtures: {@code src/test/resources/tasks/task_047/} * ATP: {@link ru.icc.regtab.atp.AtpTask047Test} * 

- * { [ [!BLANK? VAL : SR*->REC, BW&STR*->JOIN(0)] [!BLANK? VAL] ]+ }+
+ * { [ [!BLANK? VAL : SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? VAL] ]+ }+
  *
* Each row of each subtable: non-blank anchor VAL with SR*->REC (unbounded - * same-subrow collection) and BW&STR*->JOIN(0) (grouping rows with the + * same-subrow collection) and BW&STR*->CONCAT(0) (grouping rows with the * same string below); followed by a non-blank plain VAL cell. */ public class RtlTask047Test extends RtlTaskBase { @@ -21,7 +21,7 @@ public class RtlTask047Test extends RtlTaskBase { @Override protected String buildRtl() { return /* language=RTL */ """ - { [ [!BLANK? VAL : SR*->REC, BW&STR*->JOIN(0)] [!BLANK? VAL] ]+ }+ + { [ [!BLANK? VAL : SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? VAL] ]+ }+ """; } } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask050Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask050Test.java index 011ad7ba..76d80ea5 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask050Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask050Test.java @@ -2,16 +2,16 @@ /** * Task 50: single (non-repeating) flat table with non-blank three-cell rows — - * anchor VAL (AVP + same-row REC + below-same-string JOIN(0)), ATTR, and AVP VAL. + * anchor VAL (AVP + same-row REC + below-same-string CONCAT(0)), ATTR, and AVP VAL. *

* Fixtures: {@code src/test/resources/tasks/task_050/} * ATP: {@link ru.icc.regtab.atp.AtpTask050Test} * 

- * [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+
+ * [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+
  *
* Same pattern as task 46 but without the outer subtable repetition ({...}+). * Each non-blank row: anchor VAL with empty-literal AVP, SR*->REC (unbounded - * same-subrow collection), and BW&STR*->JOIN(0) (grouping rows with the + * same-subrow collection), and BW&STR*->CONCAT(0) (grouping rows with the * same string below); non-blank ATTR cell; non-blank VAL with SR->AVP (same-subrow * attribute lookup). */ @@ -23,7 +23,7 @@ public class RtlTask050Test extends RtlTaskBase { @Override protected String buildRtl() { return /* language=RTL */ """ - [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ + [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ """; } } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask053Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask053Test.java index 2fe5b42b..5d968570 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask053Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask053Test.java @@ -6,12 +6,12 @@ * Fixtures: {@code src/test/resources/tasks/task_053/} * 
  * [ [] [AUX]+ ]
- * [ [VAL : ROW*->REC, BW&STR->JOIN(0), 'ID'->AVP]
+ * [ [VAL : ROW*->REC, BW&STR->CONCAT(0), 'ID'->AVP]
  *   {[ATTR : AV->PREFIX('_')] [VAL : SR->AVP]}+ ]+
  *
* Header row: group-name cells (REF, SPECS) are AUX; ID column is a bare skip cell. * Data rows come in pairs sharing the same ID. The ID cell is the REC anchor; ROW* collects - * all VAL cells from the same row, while JOIN(0) BW&STR merges the paired row below. + * all VAL cells from the same row, while CONCAT(0) BW&STR merges the paired row below. * Each ATTR qualifier cell gets PREFIX'd with the header cell above it (AV→PREFIX('_')), * forming compound names (REF_TP, SPECS_HV, …). Each VAL cell gets AVP from its ATTR sibling * in the same explicit subrow (SR→AVP). @@ -25,7 +25,7 @@ public class RtlTask053Test extends RtlTaskBase { protected String buildRtl() { return /* language=RTL */ """ [ [] [AUX]+ ] - [ [VAL : ROW*->REC, BW&STR->JOIN(0), 'ID'->AVP] + [ [VAL : ROW*->REC, BW&STR->CONCAT(0), 'ID'->AVP] {[ATTR : AV->PREFIX('_')] [VAL : SR->AVP]}+]+ """; } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask069Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask069Test.java index 6b3efb29..f71e8ada 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask069Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask069Test.java @@ -2,14 +2,14 @@ /** * Task 69: single non-repeating subtable with SR->AVP; first row anchors BW*->REC with - * explicit subrows of ATTR + tagged VAL#1 (JOIN with same-row #1 items) + tagged VAL#2 - * (JOIN with same-row #2 items); subsequent rows have ATTR + two plain VAL cells. + * explicit subrows of ATTR + tagged VAL#1 (CONCAT with same-row #1 items) + tagged VAL#2 + * (CONCAT with same-row #2 items); subsequent rows have ATTR + two plain VAL cells. *

* Fixtures: {@code src/test/resources/tasks/task_069/} * ATP: {@link ru.icc.regtab.atp.AtpTask069Test} * 

  * { SR->AVP
- * [ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->JOIN][VAL#'2': ROW&#'2'*->JOIN] }* ]
+ * [ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->CONCAT][VAL#'2': ROW&#'2'*->CONCAT] }* ]
  * [          { [ATTR] [VAL]{2} }* ]* }
  *
* The SR->AVP subtable-level action propagates attribute lookup from the same subrow. @@ -25,7 +25,7 @@ public class RtlTask069Test extends RtlTaskBase { protected String buildRtl() { return /* language=RTL */ """ SR->AVP - [ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->JOIN][VAL#'2': ROW&#'2'*->JOIN] }* ] + [ BW*->REC { [ATTR] [VAL#'1': ROW&#'1'*->CONCAT][VAL#'2': ROW&#'2'*->CONCAT] }* ] [ { [ATTR] [VAL]{2} }* ]* """; } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask094Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask094Test.java index 4ed8830a..caf2cf03 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask094Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask094Test.java @@ -3,12 +3,12 @@ /** * Task 94: single header row (groups separated by optional blank) above one-or-more * blank-separated data blocks; COL*->REC collects all same-column VALs regardless of subtable - * boundaries, ROW&C+1..&STR*->JOIN(0) merges sibling header columns into one record. + * boundaries, ROW&C+1..&STR*->CONCAT(0) merges sibling header columns into one record. *

* Fixtures: {@code src/test/resources/tasks/task_094/} * ATP: {@link ru.icc.regtab.atp.AtpTask094Test} * 

- * [ { [!BLANK? VAL: COL*->REC, ROW&C+1..&STR*->JOIN(0)]+ [BLANK]? }+ ]
+ * [ { [!BLANK? VAL: COL*->REC, ROW&C+1..&STR*->CONCAT(0)]+ [BLANK]? }+ ]
  * { [ { [!BLANK? VAL]+ [BLANK]? }+ ]+
  *   [ [BLANK]+ ]? }+
  *
@@ -21,7 +21,7 @@ public class RtlTask094Test extends RtlTaskBase { @Override protected String buildRtl() { return /* language=RTL */ """ - [ { [!BLANK? VAL: COL*->REC, ROW&C+1..&STR*->JOIN(0)]+ [BLANK]? }+ ] + [ { [!BLANK? VAL: COL*->REC, ROW&C+1..&STR*->CONCAT(0)]+ [BLANK]? }+ ] { [ { [!BLANK? VAL]+ [BLANK]? }+ ]+ [ [BLANK]+ ]? }+ """; diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask097Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask097Test.java index a492d23f..e07ae7ae 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask097Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask097Test.java @@ -3,12 +3,12 @@ /** * Task 97: flat table where each row's anchor VAL collects same-subrow values to the right * via RT*->REC, and joins all records of same-string VALs below in the same subcol - * via (BW&STR)*->JOIN(0,1), dropping key positions {0,1} from each joined record. + * via (BW&STR)*->CONCAT(0,1), dropping key positions {0,1} from each joined record. *

* Fixtures: {@code src/test/resources/tasks/task_097/} * ATP: {@link ru.icc.regtab.atp.AtpTask097Test} * 

- * [ [VAL: RT*->REC, (BW&STR)*->JOIN(0,1)] [VAL]+ ]+
+ * [ [VAL: RT*->REC, (BW&STR)*->CONCAT(0,1)] [VAL]+ ]+
  *
*/ public class RtlTask097Test extends RtlTaskBase { @@ -18,6 +18,6 @@ public class RtlTask097Test extends RtlTaskBase { @Override protected String buildRtl() { - return /* language=RTL */ "[ [VAL: RT*->REC, (BW&STR)*->JOIN(0,1)] [VAL]+ ]+"; + return /* language=RTL */ "[ [VAL: RT*->REC, (BW&STR)*->CONCAT(0,1)] [VAL]+ ]+"; } } diff --git a/src/test/java/ru/icc/regtab/rtl/RtlTask098Test.java b/src/test/java/ru/icc/regtab/rtl/RtlTask098Test.java index f24aae7e..02429632 100644 --- a/src/test/java/ru/icc/regtab/rtl/RtlTask098Test.java +++ b/src/test/java/ru/icc/regtab/rtl/RtlTask098Test.java @@ -15,7 +15,7 @@ public class RtlTask098Test extends RtlTaskBase { protected String buildRtl() { return /* language=RTL */ """ [ [] [] [ATTR]+ ] - [ [VAL: RT*->REC, (BW&STR)*->JOIN(0,1)] [VAL] [VAL: COL->AVP]{2} [VAL]+ ]+ + [ [VAL: RT*->REC, (BW&STR)*->CONCAT(0,1,2,3)] [VAL] [VAL: COL->AVP]{2} [VAL]+ ]+ """; } } From 269914b4d2e627f1d8f13460d705deb1c6f6b9eb Mon Sep 17 00:00:00 2001 From: "Alexey O. Shigarov" Date: Fri, 28 Aug 2026 17:41:46 +0800 Subject: [PATCH 2/3] Docs and CHANGELOG for CONCAT(K) / JOIN(K); version 0.6.0-SNAPSHOT examples.md: Examples 2 and 3 on CONCAT(0), CONCAT vs JOIN note, new Example 6 (record join: exploding a delimited key against stacked columns); rtl-reference, model/itm, model/atp, api, architecture, embedded-rtl, index updated; CHANGELOG Unreleased: Changed (breaking) with the migration table; pom 0.6.0-SNAPSHOT. --- CHANGELOG.md | 46 +++++++++++++ docs/api.md | 9 ++- docs/architecture.md | 6 +- docs/embedded-rtl.md | 3 +- docs/examples.md | 148 ++++++++++++++++++++++++++++++++++++------ docs/index.md | 2 +- docs/model/atp.md | 10 +-- docs/model/itm.md | 59 +++++++++++------ docs/rtl-reference.md | 19 ++++-- pom.xml | 2 +- 10 files changed, 250 insertions(+), 54 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 65eb4846..a9b2f2e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed (breaking) +- **`JOIN(K)` is now the record product; the folding operation is `CONCAT(K)`.** Up to 0.5.x + `JOIN(K)` *folded* records: the records of the provided anchors were concatenated to the anchor's + record (one wide record, the number of records strictly decreased) — what `pandas.concat(axis=1)` + does, not what a join does. The operation keeps that semantics under its original name + `CONCAT(K)` (RTL `CONCAT`, `CONCAT(k1, k2, …)`; `ActionSpec.concat(…)`, `Rtl.concat(…)`, + `ConcatOperation`, `OperationType.CONCAT`, `WorkingState.applyConcat`). `JOIN(K)` is redefined + as the **record product**: every record of the anchor is combined with every record of the + provided anchors — a cross product for `K = ∅`, an equi-join on the key positions `K` otherwise; + a named attribute shared by the two records is a natural-join condition (the pair is kept only + if the values agree, the attribute occurs once). The provided anchors become *joined-away* + (`J`): they are excluded from the recordset but their records stay available, so several + anchors may join the same records irrespective of action order. Migration: replace `JOIN(K)` + by `CONCAT(K)` in every existing pattern (`JOIN` → `CONCAT`, `JOIN(0)` → `CONCAT(0)`, …); the + corpus tasks 016, 023, 025, 033, 046, 047, 050, 053, 069, 094, 097, 098 were migrated this way + and produce byte-identical recordsets +- **`CONCAT(K)` no longer deduplicates named attributes.** The former `JOIN(K)` silently kept the + first occurrence of a named attribute shared by the concatenated records, which hid + specification errors (a pattern that joined several stacked cells to one token lost all but + the first `value`). Now a shared named attribute (apart from the key positions `K`) is a + precondition violation: the action has **no effect** — both records survive — and a + `Diagnostic` is recorded (`TableInterpreter.diagnostics()`); `withStrictPreconditions(true)` + raises an `IllegalStateException` instead. The key positions are likewise checked (`compat_K`). + In the corpus this changed one pattern: task 098 lists its full group key, + `(BW&STR)*->CONCAT(0,1,2,3)` instead of `JOIN(0,1)` — the named attributes `A`/`B` repeated on + every row of a group are part of the key, not duplicates to drop; the expected recordsets are unchanged +- **Working state: `rec` is multi-valued, recordsets are multisets.** `rec(ι)` is a non-empty + sequence of item-based records (`WorkingState.rec(item)` → `List>`, a single record + until a join multiplies it); the working state gains the component `J` (`WorkingState.allJoined()`, + `isJoined(item)`), and `WorkingState.allRec()` returns the **live** anchors `dom(rec) \ J` only — + exactly what recordset extraction sees. `SchemaConstructionStrategy.buildVisitOrder` visits + `(anchor, record, position)` triples. The order of records in a `Recordset` is a documented + default (anchor visit order, then nested-loop order of the join), not part of the semantics +- Grammar: `concatOp : CONCAT (LPAREN INT (COMMA INT)* RPAREN)?`, keyword `CONCAT`; the ATP→RTL + serializer emits `CONCAT(k1, k2)`; the VS Code grammar highlights `CONCAT` + +### Added +- `Diagnostic` (`ru.icc.regtab.itm.semantics`), `WorkingState.diagnostics()`, + `TableInterpreter.diagnostics()`, `TableInterpreter.withStrictPreconditions(boolean)`, + `WorkingState(boolean strictPreconditions)` +- Conformance corpus, semantic section: `concat_by_key` (task 016 shape), `join_product` + (explode × stack — six records from two rows), `join_equi_key` (`JOIN(0)` on a positional key) +- Docs: Example 6 (record join) and a `CONCAT` vs `JOIN` comparison in `examples.md`; + `rtl-reference.md`, `model/itm.md`, `model/atp.md`, `api.md`, `architecture.md`, + `embedded-rtl.md` updated + ## [0.5.3] - 2026-08-27 ### Changed diff --git a/docs/api.md b/docs/api.md index d152c722..ba716e79 100644 --- a/docs/api.md +++ b/docs/api.md @@ -235,6 +235,7 @@ Specifies how an item participates in the semantic layer. Actions are attached t ```java ActionSpec.rec(ProviderSpec.val(ItemFilterConditionSpec.sameRow())) // REC ActionSpec.avp("AIRLINE") // AVP with literal attribute +ActionSpec.concat(0, ProviderSpec.val(...)) // CONCAT(0) ActionSpec.join(ProviderSpec.val(...)) // JOIN ActionSpec.fill("/", ProviderSpec.val(...)) // FILL ActionSpec.prefix(" ", ProviderSpec.val(...)) // PREFIX @@ -247,8 +248,10 @@ ActionSpec.suffix(" ", ProviderSpec.val(...)) // SUFFIX | `rec(int anchorPos, ProviderSpec... providers)` | REC with schema anchor at position N. | | `avp(ProviderSpec provider)` | Attribute-value pair via provider. | | `avp(String literal)` | AVP with constant attribute name. | -| `join(ProviderSpec... providers)` | JOIN: merge co-anchored items. | -| `join(Set keyPositions, ProviderSpec... providers)` | JOIN with key positions. | +| `concat(ProviderSpec... providers)` | CONCAT: fold the provided records into the anchor's record (one wider record). | +| `concat(int keyPosition, ProviderSpec... providers)`, `concat(Set keyPositions, ProviderSpec... providers)` | CONCAT with key positions K (must agree, not repeated). Was `join(…)` up to 0.5.x. | +| `join(ProviderSpec... providers)` | JOIN: the record product — one record per (anchor record × provided record). | +| `join(int keyPosition, ProviderSpec... providers)`, `join(Set keyPositions, ProviderSpec... providers)` | Equi-join on the key positions K. | | `fill(String delimiter, ProviderSpec... providers)` | Fill gap in REC sequence. | | `prefix(String delimiter, ProviderSpec... providers)` | Prepend to anchor value. | | `suffix(String delimiter, ProviderSpec... providers)` | Append to anchor value. | @@ -361,6 +364,8 @@ Recordset rs = new TableInterpreter() | `withMissingValueHandler(MissingValueHandler h)` | Handling of missing attribute values (default: `NULL_HANDLER`). | | `withTransformations(List t)` | Post-processing transformations. | | `withAnonymousAttributeTemplate(String template)` | Name template for unnamed attributes; `%i` → index. Default: `"$a_%i"`. | +| `withStrictPreconditions(boolean strict)` | A violated `CONCAT`/`JOIN` precondition (e.g. a named attribute shared by two concatenated records) raises an `IllegalStateException` instead of having no effect. Default: `false`. | +| `List diagnostics()` | The `CONCAT`/`JOIN` actions skipped during the most recent `interpret(...)` because a precondition was violated — each with the anchor, the operation and the reason. Empty when nothing was skipped. | --- diff --git a/docs/architecture.md b/docs/architecture.md index 12b3c2de..5fe3cb5b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -87,8 +87,8 @@ If the pattern does not match, `AtpMatcher.match` returns `Optional.empty()`. | Phase | What happens | |---|---| | **1. Initialisation** | Each cell-derived and context-derived item of type VAL/ATTR is entered into the working state with its string value | -| **2. Completion** | Interpretation actions are applied in operation-type order: FILL/PREFIX/SUFFIX → AVP → REC → JOIN; each action uses its providers to retrieve items relative to the anchor and updates the working state | -| **3. Extraction** | The working state is traversed to build the schema (attribute list) and generate records | +| **2. Completion** | Interpretation actions are applied in operation-type order: FILL/PREFIX/SUFFIX → AVP → REC → CONCAT → JOIN (records are folded before they are multiplied); each action uses its providers to retrieve items relative to the anchor and updates the working state. A violated CONCAT/JOIN precondition has no effect and is reported through `TableInterpreter.diagnostics()` | +| **3. Extraction** | The live anchors of the working state (joined-away anchors excluded) are traversed to build the schema (attribute list) and generate records — one per item-based record, several per anchor after a JOIN | | **4. Transformation** | Optional post-processing steps are applied: `WhitespaceNormalization`, `FieldSplitting`, `SchemaReordering` | --- @@ -139,7 +139,7 @@ The round-trip property — serialize then compile gives back the original patte | `RowPattern`, `SubrowPattern`, `CellPattern` | `[ ... ]q`, `{ ... }q`, `[ ... ]q` | | `AtomicContentSpec` with tags | `VAL #'tag'` | | `AtomicContentSpec` with extractor | `VAL = TRIM` | -| `ActionSpec` (avp, rec, join, fill, prefix, suffix) | `'NAME'->AVP`, `(prov…)->REC`, etc. | +| `ActionSpec` (avp, rec, concat, join, fill, prefix, suffix) | `'NAME'->AVP`, `(prov…)->REC`, etc. | | `ProviderSpec` with traversal order | leading `-` / `^` / `-^` | | `ProviderSpec` with cardinality | `{n}` / `*` | | `RecordsetTransformation` settings | ``, ``, `` | diff --git a/docs/embedded-rtl.md b/docs/embedded-rtl.md index f9648ca2..09889046 100644 --- a/docs/embedded-rtl.md +++ b/docs/embedded-rtl.md @@ -98,9 +98,10 @@ compiler. |---|---| | `(…)->REC` / `REC(n)` / `REC('s')` | `rec(…)` / `rec(n, …)` / `recSplit("s", …)` | | `prov->AVP` / `'NAME'->AVP` | `avp(prov)` / `avp("NAME")` | +| `(…)->CONCAT` / `CONCAT(k)` | `concat(…)` / `concat(k, …)` | | `(…)->JOIN` / `JOIN(k)` | `join(…)` / `join(k, …)` | | `(…)->FILL('d')`, `PREFIX`, `SUFFIX` | `fill("d", …)`, `prefix(…)`, `suffix(…)` (delimiter optional) | -| `'EUR'` context literal | `lit("EUR")` (VALUE under REC/JOIN, ATTRIBUTE otherwise — as in the compiler) | +| `'EUR'` context literal | `lit("EUR")` (VALUE under REC/CONCAT/JOIN, ATTRIBUTE otherwise — as in the compiler) | | `@'K'='V'` | `ctxAvp("K", "V")` | Provider kinds (VAL/ATTR/UNRESTRICTED) are inferred from the action, exactly as in the diff --git a/docs/examples.md b/docs/examples.md index bb8dc06e..ba766c6a 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -1,6 +1,7 @@ # Examples -Three worked examples drawn from the benchmark test suite — tasks **052**, **053**, and **046**. +Six worked examples: five drawn from the benchmark test suite — tasks **052**, **053**, **046**, +**116**, and **051** — and one from the conformance corpus (`join_product`, the record product). For each task the ATP pattern and its RTL equivalent are shown side by side. --- @@ -127,10 +128,11 @@ mvn test -Dtest="RtlTask052Test" --- -## Example 2 — Task 053: compound attribute names and paired-row JOIN +## Example 2 — Task 053: compound attribute names and paired-row CONCAT Two physical rows describe one logical record. Attribute names are *composed* from a group header -(`REF`, `SPECS`) and a per-row qualifier (`TP`, `HV`, …), and the paired rows are merged by JOIN. +(`REF`, `SPECS`) and a per-row qualifier (`TP`, `HV`, …), and the paired rows are concatenated +into one record by `CONCAT` — the counterpart of `pandas.concat(axis=1)` aligned on the `ID` key. **Input table** (task 053, variant 1): @@ -190,7 +192,7 @@ TablePattern pattern = TablePattern.of( SubrowPattern.of( CellPattern.of(AtomicContentSpec.val( ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, SAME_ROW)), - ActionSpec.join(0, ProviderSpec.val(1, BELOW_STR)), + ActionSpec.concat(0, ProviderSpec.val(1, BELOW_STR)), ActionSpec.avp("ID") )) ), @@ -212,7 +214,7 @@ TablePattern pattern = TablePattern.of( ```rtl [ [] [AUX]+ ] -[ [VAL : ROW*->REC, BW&STR->JOIN(0), 'ID'->AVP] +[ [VAL : ROW*->REC, BW&STR->CONCAT(0), 'ID'->AVP] {[ATTR : AV->PREFIX('_')] [VAL : SR->AVP]}+ ]+ ``` @@ -221,11 +223,12 @@ TablePattern pattern = TablePattern.of( - **Header subtable** `[ [] [AUX]+ ]`: skip the corner `[]`, then mark the group-name cells (`REF`, `REF`, `SPECS`, `SPECS`) as `AUX` — they are not values, they only supply name prefixes. - **Data subtable** `[ … ]+`: one-or-more rows. The anchor cell - `[VAL : ROW*->REC, BW&STR->JOIN(0), 'ID'->AVP]` is the `ID` value (`T-1`): + `[VAL : ROW*->REC, BW&STR->CONCAT(0), 'ID'->AVP]` is the `ID` value (`T-1`): - `ROW*->REC` collects all `VAL` items in the same row into one record. - `'ID'->AVP` names the anchor's attribute `ID`. - - `BW&STR->JOIN(0)` (`Below` & `SameStr`) merges the next row whose `ID` string is identical below, - so `T-1`'s two physical rows fold into a single record. + - `BW&STR->CONCAT(0)` (`Below` & `SameStr`) concatenates the record of the next row whose `ID` + string is identical below to the anchor's record, so `T-1`'s two physical rows fold into a + single wider record; key position `0` (the `ID`) is not repeated. - The rest of each row is an **explicit subrow** `{[ATTR] [VAL]}+` repeated per qualifier/value pair: - `[ATTR : AV->PREFIX('_')]` — the qualifier cell (`TP`, `HV`, …) becomes an `ATTR`; `AV` (`Above`) prepends the group header from the cell above with `'_'`, forming `REF_TP`, @@ -241,10 +244,22 @@ TablePattern pattern = TablePattern.of( `REF_SN=001`, `SPECS_LV=110` (row 2); `REF_TP=D24`, `SPECS_HV=110` (row 3); `REF_SN=002`, `SPECS_LV=10` (row 4). 3. **REC** (one per row): `⟨T-1, D16, 750⟩`, `⟨T-1, 001, 110⟩`, `⟨T-2, D24, 110⟩`, `⟨T-2, 002, 10⟩`. -4. **JOIN(0)** merges the row below sharing the same `ID`: +4. **CONCAT(0)** concatenates the record of the row below sharing the same `ID` (its position 0, + the second `ID`, is dropped): - `⟨T-1, D16, 750, 001, 110⟩` - `⟨T-2, D24, 110, 002, 10⟩` +!!! note "CONCAT vs JOIN" + `CONCAT` **folds** records: two records become one wider record. `JOIN` **multiplies** them: + every record of the anchor is combined with every provided record, and the width stays fixed. + Here each `ID` has exactly one row below, so `JOIN(0)` would give the same two records — the + two operations coincide whenever a single record is provided. They diverge from two records + on: with three rows per `ID`, `CONCAT(0)` yields one record of width 7, `JOIN(0)` yields two + records of width 5 (see Example 6). A named attribute shared by the concatenated records + (apart from the key) is a conflict: `CONCAT` leaves both records as they are and reports a + diagnostic (`TableInterpreter.diagnostics()`); for `JOIN` the same situation is a natural-join + condition. Up to jRegTab 0.5.x the folding operation was spelled `JOIN(K)`. + ### Running the test ```bash @@ -319,7 +334,7 @@ TablePattern pattern = TablePattern.of( CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.val( ActionSpec.avp(""), // blank-named attribute ActionSpec.rec(ProviderSpec.val(ProviderSpec.UNBOUNDED, SAME_SUBROW)), - ActionSpec.join(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) + ActionSpec.concat(0, ProviderSpec.val(ProviderSpec.UNBOUNDED, BELOW_STR)) )), // Subject cell → ATTR CellPattern.of(NOT_BLANK, Quantifier.one(), AtomicContentSpec.attr()), @@ -335,18 +350,20 @@ TablePattern pattern = TablePattern.of( ### RTL equivalent ```rtl -{ [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+ +{ [ [!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)] [!BLANK? ATTR] [!BLANK? VAL : SR->AVP] ]+ }+ ``` ### How it works - The whole list is matched by one-or-more subtables `{ … }+` of one-or-more three-cell rows `[ … ]+`; every cell is **guarded** `!BLANK?` (must be non-blank). -- Anchor cell `[!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->JOIN(0)]` — the name (`Anna`): +- Anchor cell `[!BLANK? VAL : ''->AVP, SR*->REC, BW&STR*->CONCAT(0)]` — the name (`Anna`): - `''->AVP` binds the name to the **empty-named attribute** (the blank-header name column). - `SR*->REC` collects all same-subrow `VAL` items into the record. - - `BW&STR*->JOIN(0)` merges every following row whose name string is identical below — `Anna`'s two - rows collapse into one record, `Bob`'s two, and so on. + - `BW&STR*->CONCAT(0)` concatenates the records of every following row whose name string is + identical below — `Anna`'s two rows fold into one record, `Bob`'s two, and so on. This is a + fold, not a join: three rows per student would still give one record (see Example 6 for the + record product). - `[!BLANK? ATTR]` — the subject cell (`Math`) becomes an `ATTR` (a schema attribute name). - `[!BLANK? VAL : SR->AVP]` — the score cell (`43`) takes its attribute name from the `ATTR` in the same subrow → `Math=43`. @@ -359,8 +376,9 @@ TablePattern pattern = TablePattern.of( (Bob); `English=79` (Joan); `Math=90`, `French=85` (Tom); `English=87`, `French=92` (Rob). 2. **REC** (one per row, anchored on the name): `⟨Anna,43⟩`, `⟨Anna,78⟩`, `⟨Bob,96⟩`, `⟨Bob,54⟩`, `⟨Joan,79⟩`, `⟨Tom,90⟩`, `⟨Tom,85⟩`, `⟨Rob,87⟩`, `⟨Rob,92⟩`. -3. **JOIN(0)** merges every row whose name repeats directly below, collapsing each student to one - record: `⟨Anna, 43, 78⟩`, `⟨Bob, 96, 54⟩`, `⟨Joan, 79⟩`, `⟨Tom, 90, 85⟩`, `⟨Rob, 87, 92⟩`. +3. **CONCAT(0)** concatenates the records of every row whose name repeats directly below, folding + each student into one record: `⟨Anna, 43, 78⟩`, `⟨Bob, 96, 54⟩`, `⟨Joan, 79⟩`, `⟨Tom, 90, 85⟩`, + `⟨Rob, 87, 92⟩`. **Schema-flexible result.** Different students list different subjects, so the records are *ragged*: the schema is the union `⟨"", Math, French, English⟩`, but a subject/student combination that never @@ -576,14 +594,106 @@ mvn test -Dtest="RtlTask051Test" --- +## Example 6 — Record join (product): exploding a delimited key against stacked columns + +Examples 2 and 3 *fold* records with `CONCAT`. This example *multiplies* them with `JOIN` — the +record product. A key cell lists several identifiers separated by `;`, and the numbers in the row +apply to each of them; every (identifier, column) combination must become a record of its own. + +**Input table** (conformance case `join_product`): + +``` +id | x | y +a;b | 1 | 2 +c | 3 | 4 +``` + +**Schema:** `⟨id, value, var⟩` — six records, |tokens| × |columns| per row: + +``` +id | var | value +a | x | 1 +a | y | 2 +b | x | 1 +b | y | 2 +c | x | 3 +c | y | 4 +``` + +### Item roles + +| | col 0 | col 1 | col 2 | +|-----------|-----------------------------------------|----------------------|----------------------| +| **row 0** | ATTR `id` | VAL `x` → var | VAL `y` → var | +| **row 1** | VAL `a` → id, VAL `b` → id *(one cell)* | VAL `1` → value | VAL `2` → value | +| **row 2** | VAL `c` → id | VAL `3` → value | VAL `4` → value | + +The delimited key cell `a;b` yields **two** cell-derived items (`a` at index 0, `b` at index 1); +each is an anchor of its own. + +### RTL pattern + +```rtl +[ [ATTR] [VAL: 'var'->AVP]+ ] +[ [(VAL: COL->AVP, ()->REC, RT*->JOIN){';'}] [VAL: 'value'->AVP, COL->REC]+ ]+ +``` + +### How it works + +- **Header row** `[ [ATTR] [VAL: 'var'->AVP]+ ]`: `id` is the attribute of the key column; the + column names `x`, `y` are *values* named `var` — they will travel into the records. +- **Data rows** `[ … ]+`: the key cell is **delimited** `(VAL: …){';'}` — one item per token: + - `COL->AVP` names each token `id` (the `ATTR` in the same column); + - `()->REC` gives each token a record of its own, `⟨id:a⟩`, `⟨id:b⟩`, `⟨id:c⟩`; + - `RT*->JOIN` multiplies the token's record by the records of all cells to its right. +- Each number cell `[VAL: 'value'->AVP, COL->REC]` is named `value` and builds the record + `⟨value:1, var:x⟩` with the column name above it (`COL`, cardinality 1, row-major → the header). + +### Derivation + +1. **AVP**: `var=x`, `var=y`; `id=a`, `id=b`, `id=c`; `value=1`, `value=2`, `value=3`, `value=4`. +2. **REC**: `rec(a) = ⟨a⟩`, `rec(b) = ⟨b⟩`, `rec(c) = ⟨c⟩`; `rec(1) = ⟨1, x⟩`, `rec(2) = ⟨2, y⟩`, + `rec(3) = ⟨3, x⟩`, `rec(4) = ⟨4, y⟩`. +3. **JOIN** at `a` (providers: cells `1`, `2`): `rec(a) = ⟨⟨a, 1, x⟩, ⟨a, 2, y⟩⟩` — one record per + provided record, in nested-loop order; the cells `1` and `2` become *joined-away* anchors. +4. **JOIN** at `b`: the records of `1` and `2` are still available (joined-away anchors are + retired lazily, not removed), so `rec(b) = ⟨⟨b, 1, x⟩, ⟨b, 2, y⟩⟩`. With immediate removal, + `b` would have found nothing to join. +5. **JOIN** at `c`: `rec(c) = ⟨⟨c, 3, x⟩, ⟨c, 4, y⟩⟩`. +6. Recordset extraction visits the live anchors `a`, `b`, `c` only — six records. + +### CONCAT vs JOIN + +| | `CONCAT(K)` | `JOIN(K)` | +|---|---|---| +| direction | n records → 1 | 1 record → n | +| what grows | the width of the record | the number of records | +| SQL counterpart | `GROUP BY` + collect into columns, `pandas.concat(axis=1)` | `CROSS JOIN`, `LATERAL`, `pandas.merge` | +| key positions `K` | must agree in all records; not repeated | a record pair is combined only if it agrees there; not repeated | +| shared named attribute | a conflict: no effect + diagnostic | a natural-join condition: kept once if the values agree, pair dropped otherwise | +| one provided record | one wider record | the same record — the two coincide | +| two or more provided records | still one record | one record each — the two diverge | + +### Running the test + +```bash +mvn test -Dtest="RtlSemanticConformanceTest" # conformance/semantic/join_product +mvn test -Dtest="TableInterpreterMultiRecordTest" # the same table, both schema strategies +mvn test -Dtest="WorkingStateJoinTest" # the operation on the working state +``` + +--- + ## Running all examples -All five examples above are benchmark tasks (052, 053, 046, 116, 051). The `*TaskTest` -wildcard runs both the ATP and the RTL test for each: +Examples 1–5 above are benchmark tasks (052, 053, 046, 116, 051). The `*TaskTest` +wildcard runs both the ATP and the RTL test for each; Example 6 is a conformance case: ```bash -# ATP + RTL tests for the five examples on this page +# ATP + RTL tests for the five benchmark examples on this page mvn test -Dtest="*Task052Test,*Task053Test,*Task046Test,*Task116Test,*Task051Test" +# Example 6 +mvn test -Dtest="RtlSemanticConformanceTest,TableInterpreterMultiRecordTest" ``` To run the whole benchmark suite instead, use the `AtpTask*Test` / `RtlTask*Test` globs. diff --git a/docs/index.md b/docs/index.md index 34ec82dc..3ba69684 100644 --- a/docs/index.md +++ b/docs/index.md @@ -99,7 +99,7 @@ Requires **Java 21+**. - **RTL** (Regular Table Language) — compact DSL that compiles to ATP; dramatically reduces pattern verbosity. - **ATP → RTL serializer** — round-trip: serialize any `TablePattern` back to an RTL string. - **Content specs** — atomic, delimited, compound, and conditional cell content. -- **Action specs** — `REC`, `AVP`, `JOIN`, `FILL`, `PREFIX`, `SUFFIX` for rich schema construction. +- **Action specs** — `REC`, `AVP`, `CONCAT`, `JOIN`, `FILL`, `PREFIX`, `SUFFIX` for rich schema construction (fold records with `CONCAT`, multiply them with `JOIN`). - **Named fragments** — reuse recurring sub-patterns in RTL with `$name` definitions. - **Post-processing** — whitespace normalization, field splitting, schema reordering. - **150-task benchmark** — Foofah (50), RegTab (60), and Baikal (40) tasks, 1 500 test variants, 100 % pass rate. diff --git a/docs/model/atp.md b/docs/model/atp.md index 47629697..3a9fd780 100644 --- a/docs/model/atp.md +++ b/docs/model/atp.md @@ -322,8 +322,8 @@ S_act = (op, ⟨S_prov¹, …, S_provⁿ⟩) S_act = (op, s, ⟨S_prov¹, …, S_provⁿ⟩) ``` -In both forms, `op` is one of the six working-state update operations (`FILL`, -`PREFIX`, `SUFFIX`, `AVP`, `REC`, `JOIN`) and `S_prov¹ … S_provⁿ` are item +In both forms, `op` is one of the seven working-state update operations (`FILL`, +`PREFIX`, `SUFFIX`, `AVP`, `REC`, `CONCAT`, `JOIN`) and `S_prov¹ … S_provⁿ` are item provider specifications whose types must satisfy the consistency constraints for the chosen operation (see [ITM — Interpretation actions](itm.md#interpretation-actions)). @@ -335,8 +335,10 @@ chosen operation (see [ITM — Interpretation actions](itm.md#interpretation-act | `REC('s')` | `ActionSpec.rec(String delim, providers…)` | adds `DelimitedFieldSplit` post-step | | `AVP` | `ActionSpec.avp(provider)` | associates VAL anchor with ATTR item | | `AVP "name"` | `ActionSpec.avp("ATTR_NAME")` | context-derived ATTR constant | - | `JOIN` | `ActionSpec.join(providers…)` | joins records, dedup by named attribute | - | `JOIN(K)` | `ActionSpec.join(Set.of(k…), providers…)` | joins with key positions K dropped | + | `CONCAT` | `ActionSpec.concat(providers…)` | folds the provided records into the anchor's record | + | `CONCAT(K)` | `ActionSpec.concat(Set.of(k…), providers…)` | same, key positions K not repeated (was `JOIN(K)` up to 0.5.x) | + | `JOIN` | `ActionSpec.join(providers…)` | record product: one record per (anchor record × provided record) | + | `JOIN(K)` | `ActionSpec.join(Set.of(k…), providers…)` | equi-join on the key positions K | | `FILL` | `ActionSpec.fill(delimiter, providers…)` | fills anchor value from providers | | `PREFIX` | `ActionSpec.prefix(delimiter, providers…)` | prepends provider values | | `SUFFIX` | `ActionSpec.suffix(delimiter, providers…)` | appends provider values | diff --git a/docs/model/itm.md b/docs/model/itm.md index 13f875c4..73ad41a7 100644 --- a/docs/model/itm.md +++ b/docs/model/itm.md @@ -234,10 +234,13 @@ During interpretation, semantic information is accumulated in a **working state* - `val(ι)` — maps each VAL item to a value in `V`; - `attr(ι)` — maps each ATTR item to an attribute in `A`; - `avp(ι)` — partial map from VAL items to `(attribute, value)` pairs; -- `rec(ι)` — partial map from cell-derived VAL items to sequences of VAL items - (the *item-based records*). +- `rec(ι)` — partial map from cell-derived VAL items to a *non-empty sequence* of item-based + records (each a sequence of VAL items); a single record until a join multiplies it; +- `J ⊆ dom(rec)` — the *joined-away anchors*: items whose records were consumed by a join. They + stay in `rec` (a later join may consume the same records again) but are excluded from recordset + extraction; `dom*(rec) = dom(rec) \ J` are the *live* anchors. -Six **working-state update operations** populate or modify the working state: +Seven **working-state update operations** populate or modify the working state: | Operation | Symbol | Effect | |---|---|---| @@ -246,10 +249,11 @@ Six **working-state update operations** populate or modify the working state: | Suffix | `O_suffix^δ` | Appends provider strings (joined by `δ`) to the anchor's value/attribute | | AVP construction | `O_avp` | Creates an attribute-value pair `(attr(ι₁), val(ι_anch))` for the anchor VAL item using the single ATTR item `ι₁` returned by the provider | | Record construction | `O_rec` | Creates an item-based record with the anchor VAL item as its first element and the provided VAL items as the remaining elements | -| Record join | `O_join^K` | Merges previously created records; key positions `K` are dropped from joined records, duplicate named attributes are removed, and the merged result is stored under the anchor | +| Record concatenation | `O_concat^K` | Folds previously created records into one wide record under the anchor: the key positions `K` (at which all records must agree) are not repeated; the concatenated anchors are removed from `dom(rec)`. A named attribute shared by two records (apart from the key) is a precondition violation: no effect, a diagnostic is recorded | +| Record join | `O_join^K` | The record product: every record of the anchor is combined with every record of the provided anchors — a cross product for `K = ∅`, an equi-join on the key positions `K` otherwise; a shared named attribute is a natural-join condition (kept once if the values agree, the pair is dropped otherwise). The provided anchors are added to `J` | ??? note "Java mapping — WorkingState" - **Definition (Working state):** `ws = (V, A, val, attr, avp, rec)`. + **Definition (Working state):** `ws = (V, A, val, attr, avp, rec, J)`. | Formal component | Java | |---|---| @@ -258,10 +262,12 @@ Six **working-state update operations** populate or modify the working state: | `val(ι)` | `WorkingState.val(item)` | | `attr(ι)` | `WorkingState.attr(item)` | | `avp(ι)` | `WorkingState.avp(item)` → `AttributeValuePair(attribute, value)` | - | `rec(ι)` | `WorkingState.rec(item)` → `List` | + | `rec(ι)` | `WorkingState.rec(item)` → `List>` (the records of the anchor, also for joined-away anchors) | + | `J` | `WorkingState.allJoined()`, `WorkingState.isJoined(item)` | + | `dom*(rec) = dom(rec) \ J` | `WorkingState.allRec()` — the **live** anchors only, in insertion order; this is what recordset extraction sees | | Derived `assoc(ι)` | `WorkingState.assoc(item)` — attribute of `avp(ι)`, or `null` | - Six working-state update operations: + Seven working-state update operations: | Formal operation | Java method | Anchor type | |---|---|---| @@ -270,14 +276,19 @@ Six **working-state update operations** populate or modify the working state: | `O_suffix^δ` | `WorkingState.applySuffix(anchor, items, delimiter)` | VAL or ATTR | | `O_avp` | `WorkingState.applyAvp(anchor, items)` | VAL | | `O_rec` | `WorkingState.applyRec(anchor, items)` | cell-derived VAL | + | `O_concat^K` | `WorkingState.applyConcat(anchor, items, keyPositions)` | cell-derived VAL | | `O_join^K` | `WorkingState.applyJoin(anchor, items, keyPositions)` | cell-derived VAL | + A violated precondition of `O_concat^K` / `O_join^K` has no effect and is recorded as a + `Diagnostic` (`WorkingState.diagnostics()`, surfaced as `TableInterpreter.diagnostics()`); + `new WorkingState(true)` / `TableInterpreter.withStrictPreconditions(true)` raise instead. + Consistency predicates: | Predicate | Java method | |---|---| - | Basic consistency: `rec(ι)[0] = ι` and `avp(ι) = (a,v) ⟹ val(ι) = v` | `WorkingState.isConsistent()` | - | Recordset-consistency: uniform anchor attribute + distinct per-record attributes | `WorkingState.isRecordsetConsistent()` | + | Basic consistency: `ρ[0] = ι` for every `ρ ∈ rec(ι)`, and `avp(ι) = (a,v) ⟹ val(ι) = v` | `WorkingState.isConsistent()` | + | Recordset-consistency over the live anchors: uniform anchor attribute + distinct attributes in every record | `WorkingState.isRecordsetConsistent()` | ### Interpretation actions @@ -299,7 +310,8 @@ satisfy the constraints of the chosen operation (Tab. I in the paper): | String modification | VAL or ATTR (cell-derived) | any | ≥ 1 | | AVP construction | VAL (cell- or context-derived) | ATTR provider | = 1 | | Record construction | VAL (cell-derived) | VAL providers | ≥ 0 | -| Record join | VAL (cell-derived) | VAL providers (cell-derived) | ≥ 0 | +| Record concatenation | VAL (cell-derived) | VAL providers (cell-derived) | ≥ 1 | +| Record join | VAL (cell-derived) | VAL providers (cell-derived) | ≥ 1 | ??? note "Java API — ActionSpec" **Action spec:** `S_act = (op, ⟨S_prov¹, …, S_provⁿ⟩)` where `op` is a @@ -309,8 +321,10 @@ satisfy the constraints of the chosen operation (Tab. I in the paper): |---|---|---| | `REC` | `ActionSpec.rec(providers…)` | Anchor item → record; providers supply the remaining fields | | `AVP` | `ActionSpec.avp(provider)` | Associates a VAL item (anchor) with an ATTR item from the provider | - | `JOIN` | `ActionSpec.join(providers…)` | Joins item-based records; dedup by named attribute (K=∅) | - | `JOIN(K)` | `ActionSpec.join(Set.of(0), providers…)` | Joins with key positions K dropped (e.g. `JOIN(0)` drops the anchor position) | + | `CONCAT` | `ActionSpec.concat(providers…)` | Folds the provided records into the anchor's record (K=∅) | + | `CONCAT(K)` | `ActionSpec.concat(Set.of(0), providers…)` | Same, key positions K not repeated (e.g. `CONCAT(0)` drops the anchor position of each provided record) | + | `JOIN` | `ActionSpec.join(providers…)` | Record product: one record per (anchor record × provided record) | + | `JOIN(K)` | `ActionSpec.join(Set.of(0), providers…)` | Equi-join on the key positions K | | `FILL` | `ActionSpec.fill(delimiter, providers…)` | Fills anchor value using provider values | | `PREFIX` | `ActionSpec.prefix(delimiter, providers…)` | Prepends provider values to the anchor | | `SUFFIX` | `ActionSpec.suffix(delimiter, providers…)` | Appends provider values to the anchor | @@ -327,7 +341,12 @@ satisfy the constraints of the chosen operation (Tab. I in the paper): ## Recordset and schema **Definition (Recordset):** given a schema `S = ⟨a₁, …, aₙ⟩`, a *record* is an -n-tuple `⟨(a₁,v₁), …, (aₙ,vₙ)⟩`; a *recordset* is a finite sequence of records. +n-tuple `⟨(a₁,v₁), …, (aₙ,vₙ)⟩`; a *recordset* is a finite **multiset** of records — as in the +relational model, the order of records is not part of the result, whereas duplicates are. +Implementations materialize a recordset as a sequence (e.g. for CSV export); the reference +implementation emits records in the order in which their anchors were visited during working +state completion and, for an anchor carrying several records after a join, in the nested-loop +order of the join. This order is a documented default, not a guarantee of the model. ??? note "Java mapping — Schema, Record, Recordset" | Formal concept | Java class / method | @@ -349,7 +368,7 @@ An initial working state `ws₀` is constructed directly from the semantic layer - each VAL item is assigned its value `val(ι) ∈ V`; - each ATTR item is assigned its attribute `attr(ι) ∈ A`; -- `dom(avp)` and `dom(rec)` are initialised to empty. +- `dom(avp)`, `dom(rec)` and `J` are initialised to empty. ### Phase 2 — Working state completion @@ -358,7 +377,8 @@ The interpretation actions `A` are applied to `ws₀` in a fixed order: 1. String-modification actions (`O_fill`, `O_prefix`, `O_suffix`); 2. AVP-construction actions (`O_avp`); 3. Record-construction actions (`O_rec`); -4. Record-join actions (`O_join`). +4. Record-concatenation actions (`O_concat`) — records are folded … +5. Record-join actions (`O_join`) — … before they are multiplied. Within each phase, actions are applied in *traversal order* over their anchor items. The default strategy visits anchors in row-major order @@ -384,12 +404,13 @@ item-based records: `S = ⟨a₁, a₂, …, aₙ⟩`; unnamed items receive a fresh anonymous attribute for their position. -**Record generation** iterates over `dom(rec)` in the order in which record-construction -actions were applied: +**Record generation** iterates over the live anchors `dom*(rec) = dom(rec) \ J` in the order in +which record-construction actions were applied, and over the records `ρ ∈ rec(ι)` of each anchor +(several after a join): -- For each anchor `ι`, initialise all `n` field values to a *missing value* (via +- For each record `ρ`, initialise all `n` field values to a *missing value* (via an optional user-defined handler `μ`; default: `⊥`). -- For each item `ι'` in `rec(ι)` that has an associated attribute in `S`, fill in +- For each item `ι'` in `ρ` that has an associated attribute in `S`, fill in `val(ι')` at the corresponding position. - Emit the resulting record `⟨(a₁, v₁), …, (aₙ, vₙ)⟩`. diff --git a/docs/rtl-reference.md b/docs/rtl-reference.md index 389876fc..7a8e8264 100644 --- a/docs/rtl-reference.md +++ b/docs/rtl-reference.md @@ -409,18 +409,27 @@ provSpecs -> op | `REC(n)` | `prov->REC(n)` | Same + use attribute at position *n* as the record's attribute name | | `REC('s')` | `prov->REC('s')` | Same + split field values by delimiter *s* | | `AVP` | `prov->AVP` | Associate anchor (VAL) with an attribute from the provider (ATTR) | -| `JOIN` | `prov->JOIN` | Join item-based records: all items included, then dedup by named attribute (K=∅) | -| `JOIN(K)` | `prov->JOIN(0)` | Join with key positions K dropped from each joined record before dedup (e.g. `JOIN(0)` drops the anchor position) | +| `CONCAT` | `prov->CONCAT` | Concatenate the provided records to the anchor's record — one wide record, the provided anchors are removed (K=∅: all items included) | +| `CONCAT(K)` | `prov->CONCAT(0)` | Same, with the key positions K not repeated: dropped from each concatenated record, and all records must agree there (e.g. `CONCAT(0)` drops the anchor of each concatenated record). A named attribute shared by two records (apart from the key) is an error: the action has no effect and a diagnostic is reported | +| `JOIN` | `prov->JOIN` | Record product: every record of the anchor is combined with every provided record (cross product); the provided anchors are joined-away. A named attribute shared by two records acts as a natural-join condition | +| `JOIN(K)` | `prov->JOIN(0)` | Equi-join on the key positions K: a record pair is combined only if it agrees at K, the key of the joined record is not repeated | | `FILL('s')` | `prov->FILL('/')` | Fill anchor value forward from provider, separated by *s* | | `PREFIX('s')` | `prov->PREFIX(' ')` | Prepend provider value to anchor, separated by *s* | | `SUFFIX('s')` | `prov->SUFFIX(' ')` | Append provider value to anchor, separated by *s* | +`CONCAT` folds, `JOIN` multiplies: with a single provided record the two coincide, with two or more +they diverge — `CONCAT` yields one wider record, `JOIN` yields one record per provided record +(see Examples 2 and 6). Up to jRegTab 0.5.x the folding operation was spelled `JOIN(K)`; a pattern +written for 0.5.x must replace `JOIN(K)` by `CONCAT(K)`. + Examples by operation: ```rtl [VAL : ST*->REC] // REC, collect whole subtable (Task 01) [VAL : SR->REC(1)]{2} // REC(1), name the record by attribute at position 1 (Task 03) [VAL: 'AIRLINE'->AVP] // AVP with a literal attribute (Illustrative example) +[VAL : RT->REC, BW&STR*->CONCAT(0)] // CONCAT(0): fold the rows below with the same key into one record (Task 16) +[(VAL: COL->AVP, RT*->JOIN){';'}] // JOIN: one record per token × per cell to the right (Example 6) [VAL: -AV->PREFIX(', ')] // PREFIX: prepend the value above, separator ", " (Task 116) [BLANK ? VAL#'H': -LT&!BLANK->FILL | …] // FILL: copy the nearest non-blank cell to the left (Task 107) ``` @@ -536,7 +545,7 @@ A quoted string literal supplies a fixed string as an attribute or value: ('AIRLINE')->AVP ``` -The item type is inferred from the action: `->AVP` → ATTR, `->REC` → VAL. +The item type is inferred from the action: `->AVP` → ATTR, `->REC` / `->CONCAT` / `->JOIN` → VAL. --- @@ -549,7 +558,9 @@ The item type is inferred from the action: `->AVP` → ATTR, `->REC` → VAL. | `^COL->AVP` | Associate with an attribute from the same column (column-major) | | `('LABEL')->AVP` | Associate with a fixed string attribute | | `(ST*)->REC` (in parentheses) | Same as `ST*->REC` but explicit grouping | -| `CL->JOIN(0)` | Join (drop anchor) another item from the same cell | +| `CL->CONCAT(0)` | Concatenate (drop anchor) the record of another item from the same cell | +| `BW&STR*->CONCAT(0)` | Fold the rows below with the same key into the anchor's record (group by key) | +| `RT*->JOIN` | One record per cell to the right (record product, e.g. explode × stack) | | `(COL)->FILL('/')` | Fill forward from same-column values, delimiter `/` | | `-AV->PREFIX(', ')` | Prepend the nearest value above, separator ", " | diff --git a/pom.xml b/pom.xml index 266ba546..14a86bd9 100644 --- a/pom.xml +++ b/pom.xml @@ -4,7 +4,7 @@ ru.icc.regtab regtab - 0.5.4-SNAPSHOT + 0.6.0-SNAPSHOT jar RegTab From 2e8f032a405c1df184f99e61db9c4bf29392294a Mon Sep 17 00:00:00 2001 From: "Alexey O. Shigarov" Date: Fri, 28 Aug 2026 17:51:25 +0800 Subject: [PATCH 3/3] rtl-reference: how to choose the key positions K of CONCAT(K) K is unbounded and positional; it must list every position that repeats across the rows of a group (task 098: CONCAT(0,1,2,3), not (0,1)); a row that differs at a key position is rejected with a diagnostic instead of being folded silently. --- docs/rtl-reference.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/rtl-reference.md b/docs/rtl-reference.md index 7a8e8264..5262392a 100644 --- a/docs/rtl-reference.md +++ b/docs/rtl-reference.md @@ -422,6 +422,20 @@ they diverge — `CONCAT` yields one wider record, `JOIN` yields one record per (see Examples 2 and 6). Up to jRegTab 0.5.x the folding operation was spelled `JOIN(K)`; a pattern written for 0.5.x must replace `JOIN(K)` by `CONCAT(K)`. +**Choosing the key positions `K`.** `K` is any number of 0-based positions in the item-based +record — `0` is the anchor, the following positions are the items in the order the `REC` +providers supplied them. At every position in `K` all records being concatenated must agree, and +the item is not repeated in the result; every other position is carried over as is, so a named +attribute that occurs at such a position in two records is a conflict. Put into `K` **every** +position that repeats in each row of a group — the anchor plus all fields that are identical +across the group. With rows `k1 | k11 | a1:A | b1:B | c1` and `k1 | k11 | a1:A | b1:B | c2`, the key +is four positions, `CONCAT(0,1,2,3)`, not two: `A` and `B` repeat exactly like `k1` and `k11` +(`CONCAT(0,1)` would report the shared attribute `A` and leave both rows unfolded). A row whose +`A` differs within the group is then rejected with a diagnostic instead of being folded silently. +`K = ∅` is right only when the records share nothing, not even the anchor (task 069). `K` names +positions, not attributes — if the repeated fields sit to the right of the varying ones, the +positions shift accordingly (`CONCAT(0,1,4,5)`). + Examples by operation: ```rtl