Package org.apache.flink.cdc.connectors.postgres.source

Class PostgresChunkSplitter

java.lang.Object

org.apache.flink.cdc.connectors.base.source.assigner.splitter.JdbcSourceChunkSplitter

org.apache.flink.cdc.connectors.postgres.source.PostgresChunkSplitter

All Implemented Interfaces:

ChunkSplitter

@Internal
public class PostgresChunkSplitter
extends JdbcSourceChunkSplitter

The splitter to split the table into chunks using primary-key (by default) or a given split key.

Field Summary

Fields inherited from class org.apache.flink.cdc.connectors.base.source.assigner.splitter.JdbcSourceChunkSplitter

dialect, sourceConfig

Constructor Summary

Constructors

Constructor	Description
PostgresChunkSplitter(JdbcSourceConfig sourceConfig, JdbcDataSourceDialect dialect, ChunkSplitterState chunkSplitterState)

Method Summary

Modifier and Type	Method	Description
protected org.apache.flink.table.types.DataType	fromDbzColumn(io.debezium.relational.Column splitColumn)	Get a corresponding Flink data type from a debezium Column.
protected Long	queryApproximateRowCnt(io.debezium.jdbc.JdbcConnection jdbc, io.debezium.relational.TableId tableId)	Approximate total number of entries in the lookup table.
Object	queryMin(io.debezium.jdbc.JdbcConnection jdbc, io.debezium.relational.TableId tableId, io.debezium.relational.Column splitColumn, Object excludedLowerBound)	Postgres chunk split overrides queryMin method to query based on uuid.
Object[]	queryMinMax(io.debezium.jdbc.JdbcConnection jdbc, io.debezium.relational.TableId tableId, io.debezium.relational.Column splitColumn)	Postgres chunk split overrides queryMin method to query based on uuid.
Object	queryNextChunkMax(io.debezium.jdbc.JdbcConnection jdbc, io.debezium.relational.TableId tableId, io.debezium.relational.Column splitColumn, int chunkSize, Object includedLowerBound)	Query the maximum value of the next chunk, and the next chunk must be greater than or equal to includedLowerBound value [min_1, max_1), [min_2, max_2),...

Methods inherited from class org.apache.flink.cdc.connectors.base.source.assigner.splitter.JdbcSourceChunkSplitter

calculateDistributionFactor, close, generateSplits, getCurrentSplittingTableId, getSplitColumn, hasNextChunk, isChunkEndGeMax, isChunkEndLeMax, isEvenlySplitColumn, open, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PostgresChunkSplitter

public PostgresChunkSplitter(JdbcSourceConfig sourceConfig,
                             JdbcDataSourceDialect dialect,
                             ChunkSplitterState chunkSplitterState)

Method Detail

queryNextChunkMax

public Object queryNextChunkMax(io.debezium.jdbc.JdbcConnection jdbc,
                                io.debezium.relational.TableId tableId,
                                io.debezium.relational.Column splitColumn,
                                int chunkSize,
                                Object includedLowerBound)
                         throws SQLException

Description copied from class: JdbcSourceChunkSplitter

Query the maximum value of the next chunk, and the next chunk must be greater than or equal to includedLowerBound value [min_1, max_1), [min_2, max_2),... [min_n, null). Each time this method is called it will return max1, max2...

Each database has different grammar to get limit number of data, for example, `limit N` in mysql or postgres, `top(N)` in sqlserver , `FETCH FIRST %S ROWS ONLY` in DB2.

Specified by:

queryNextChunkMax in class JdbcSourceChunkSplitter

Parameters:

jdbc - JDBC connection.

tableId - table identity.

splitColumn - column.

chunkSize - chunk size.

includedLowerBound - the previous chunk end value.

Returns:

next chunk end value.

Throws:

SQLException

queryMinMax

public Object[] queryMinMax(io.debezium.jdbc.JdbcConnection jdbc,
                            io.debezium.relational.TableId tableId,
                            io.debezium.relational.Column splitColumn)
                     throws SQLException

Postgres chunk split overrides queryMin method to query based on uuid.

Overrides:

queryMinMax in class JdbcSourceChunkSplitter

Parameters:

jdbc - JDBC connection.

tableId - table identity.

splitColumn - column.

Returns:

maximum and minimum value.

Throws:

SQLException

queryMin

public Object queryMin(io.debezium.jdbc.JdbcConnection jdbc,
                       io.debezium.relational.TableId tableId,
                       io.debezium.relational.Column splitColumn,
                       Object excludedLowerBound)
                throws SQLException

Postgres chunk split overrides queryMin method to query based on uuid.

Overrides:

queryMin in class JdbcSourceChunkSplitter

Parameters:

jdbc - JDBC connection.

tableId - table identity.

splitColumn - column.

excludedLowerBound - the minimum value should be greater than this value.

Returns:

minimum value.

Throws:

SQLException

queryApproximateRowCnt

protected Long queryApproximateRowCnt(io.debezium.jdbc.JdbcConnection jdbc,
                                      io.debezium.relational.TableId tableId)
                               throws SQLException

Description copied from class: JdbcSourceChunkSplitter

Approximate total number of entries in the lookup table.

Each database has different system table to lookup up approximate total number. For example, `pg_class` in postgres, `sys.dm_db_partition_stats` in sqlserver, `SYSCAT.TABLE` in db2.

Specified by:

queryApproximateRowCnt in class JdbcSourceChunkSplitter

Parameters:

jdbc - JDBC connection.

tableId - table identity.

Returns:

approximate row count.

Throws:

SQLException

fromDbzColumn

protected org.apache.flink.table.types.DataType fromDbzColumn(io.debezium.relational.Column splitColumn)

Description copied from class: JdbcSourceChunkSplitter

Get a corresponding Flink data type from a debezium Column.

Specified by:

fromDbzColumn in class JdbcSourceChunkSplitter

Parameters:

splitColumn - dbz split column.

Returns:

flink data type