[FLINK-39409][connect/postgres] Support tables without primary key for incremental snapshot#4367
[FLINK-39409][connect/postgres] Support tables without primary key for incremental snapshot#4367JNSimba wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds support for PostgreSQL tables without primary keys when running in incremental snapshot mode, aligning behavior with the existing MySQL approach.
Changes:
- Extend incremental snapshot split-key extraction and output buffering to handle Debezium records with
nullkeys (no-PK tables). - Add PostgreSQL-side validation to require
REPLICA IDENTITY FULLfor no-PK tables. - Add end-to-end Postgres IT coverage and new DDL for a no-PK test table.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| flink-cdc-connect/flink-cdc-source-connectors/flink-connector-postgres-cdc/src/test/resources/ddl/inventory_no_pk.sql | Adds a no-PK test table and seed data (with REPLICA IDENTITY FULL). |
| flink-cdc-connect/flink-cdc-source-connectors/flink-connector-postgres-cdc/src/test/java/org/apache/flink/cdc/connectors/postgres/table/PostgreSQLConnectorITCase.java | Adds IT cases for no-PK tables with/without chunk key. |
| flink-cdc-connect/flink-cdc-source-connectors/flink-connector-postgres-cdc/src/main/java/org/apache/flink/cdc/connectors/postgres/source/PostgresDialect.java | Validates REPLICA IDENTITY FULL for no-PK tables during schema discovery. |
| flink-cdc-connect/flink-cdc-source-connectors/flink-cdc-base/src/main/java/org/apache/flink/cdc/connectors/base/utils/SplitKeyUtils.java | Extracts chunk key from value (before/after) when record key is null. |
| flink-cdc-connect/flink-cdc-source-connectors/flink-cdc-base/src/main/java/org/apache/flink/cdc/connectors/base/source/reader/external/JdbcSourceFetchTaskContext.java | Supports buffer rewrite/merge logic for no-PK by using before/after structs as buffer keys. |
| flink-cdc-connect/flink-cdc-source-connectors/flink-cdc-base/src/main/java/org/apache/flink/cdc/connectors/base/source/reader/external/IncrementalSourceScanFetcher.java | Uses value after struct as initial snapshot buffer key when record key is null. |
Comments suppressed due to low confidence (1)
flink-cdc-connect/flink-cdc-source-connectors/flink-cdc-base/src/main/java/org/apache/flink/cdc/connectors/base/source/reader/external/JdbcSourceFetchTaskContext.java:142
- Typo/grammar in exception message: "the the record" should be corrected to "the record" (and/or rephrase for clarity).
throw new IllegalStateException(
String.format(
"Data change record shouldn't use READ operation, the the record is %s.",
changeRecord));
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@lvyanquan Could you help me trigger the ci? This MySqlToIcebergE2eITCase and UdfE2eITCase.testFlinkCompatibleScalarFunctions failure seems unrelated to the changes. Thanks |
Mrart
left a comment
There was a problem hiding this comment.
Can we add e2e test case to cover this change?
Summary
This PR adds support for PostgreSQL tables without primary keys in incremental snapshot mode,
following the same approach as MySQL (PR #2150).
Changes
JDBC Base Layer (flink-cdc-base):
SplitKeyUtils.getSplitKey(): Handle null record key by extracting chunk key from value'safter/before struct instead of record key
IncrementalSourceScanFetcher.pollWithBuffer(): Use after struct as buffer key whenrecord key is null (no-PK table)
JdbcSourceFetchTaskContext.rewriteOutputBuffer(): Support no-PK merge logic — for tableswithout primary key, use before/after struct as buffer key for CREATE/UPDATE/DELETE operations
PostgreSQL Connector (flink-connector-postgres-cdc):
PostgresDialect: Validate that tables without primary key must haveREPLICA IDENTITY FULLset, querying
pg_class.relreplidentand failing fast with a clear error message if notTests:
products_no_pktable DDL (without PK, withREPLICA IDENTITY FULL)testNoPKTableWithChunkKey: End-to-end test for no-PK table withscan.incremental.snapshot.chunk.key-columnconfiguredtestNoPKTableWithoutChunkKey: VerifyValidationExceptionis thrown when chunk key column is not specifiedDesign Decisions
Map<Struct, SourceRecord>inFetchTask.Context.rewriteOutputBuffer()unchanged to avoid breaking MongoDB and other connectors. For no-PK tables, the full row struct
(all columns via REPLICA IDENTITY FULL) serves as the buffer key.
crosses split boundaries, only at-least-once semantics can be guaranteed (consistent with MySQL behavior).
full before image in WAL for UPDATE/DELETE events. The connector validates this at startup.