Package org.apache.flink.cdc.connectors.base.source.assigner

Class SnapshotSplitAssigner<C extends SourceConfig>

java.lang.Object

org.apache.flink.cdc.connectors.base.source.assigner.SnapshotSplitAssigner<C>

All Implemented Interfaces:

Closeable, AutoCloseable, SplitAssigner

public class SnapshotSplitAssigner<C extends SourceConfig>
extends Object
implements SplitAssigner

Assigner for snapshot split.

Constructor Summary

Constructors

Constructor	Description
SnapshotSplitAssigner(C sourceConfig, int currentParallelism, List<io.debezium.relational.TableId> remainingTables, boolean isTableIdCaseSensitive, DataSourceDialect<C> dialect, OffsetFactory offsetFactory)
SnapshotSplitAssigner(C sourceConfig, int currentParallelism, SnapshotPendingSplitsState checkpoint, DataSourceDialect<C> dialect, OffsetFactory offsetFactory)

Method Summary

Modifier and Type	Method	Description
void	addSplits(Collection<SourceSplitBase> splits)	Adds a set of splits to this assigner.
void	close()	Called to close the assigner, in case it holds on to any resources, like threads or network connections.
Map<String,SchemalessSnapshotSplit>	getAssignedSplits()
AssignerStatus	getAssignerStatus()	Gets the split assigner status, see AssignerStatus.
List<FinishedSnapshotSplitInfo>	getFinishedSplitInfos()	Gets the finished splits' information.
Optional<SourceSplitBase>	getNext()	Gets the next split.
Map<String,Offset>	getSplitFinishedOffsets()
Map<io.debezium.relational.TableId,io.debezium.relational.history.TableChanges.TableChange>	getTableSchemas()
void	initEnumeratorMetrics(SourceEnumeratorMetrics enumeratorMetrics)	This must be invoked before this class's open() method, because open() starts an asynchronous splitting thread that concurrently mutates remainingSplits.
boolean	isSnapshotMetaReleased()	Returns whether SplitAssigner.releaseSnapshotMetadata() has dropped the snapshot metadata.
boolean	needToDiscoveryTables()	Indicates current assigner need to discovery tables or not.
boolean	noMoreSplits()	Indicates there is no more splits available in this assigner.
void	notifyCheckpointComplete(long checkpointId)	Notifies the listener that the checkpoint with the given checkpointId completed and was committed.
void	onFinishedSplits(Map<String,Offset> splitFinishedOffsets)	Callback to handle the finished splits with finished change log offset.
void	onStreamSplitUpdated()	Callback to handle the stream split has been updated in the newly added tables process.
void	open()	Called to open the assigner to acquire any resources, like threads or network connections.
void	releaseSnapshotMetadata()	Releases the bulk snapshot-split metadata once the reader has assembled it and a checkpoint has covered that assignment.
SnapshotPendingSplitsState	snapshotState(long checkpointId)	Creates a snapshot of the state of this split assigner, to be stored in a checkpoint.
void	startAssignNewlyAddedTables()	Starts assign newly added tables.
boolean	waitingForFinishedSplits()	Whether the split assigner is still waiting for callback of finished splits, i.e.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SnapshotSplitAssigner

public SnapshotSplitAssigner(C sourceConfig,
                             int currentParallelism,
                             List<io.debezium.relational.TableId> remainingTables,
                             boolean isTableIdCaseSensitive,
                             DataSourceDialect<C> dialect,
                             OffsetFactory offsetFactory)

SnapshotSplitAssigner

public SnapshotSplitAssigner(C sourceConfig,
                             int currentParallelism,
                             SnapshotPendingSplitsState checkpoint,
                             DataSourceDialect<C> dialect,
                             OffsetFactory offsetFactory)

Method Detail

open

public void open()

Description copied from interface: SplitAssigner

Called to open the assigner to acquire any resources, like threads or network connections.

Specified by:

open in interface SplitAssigner

initEnumeratorMetrics

public void initEnumeratorMetrics(SourceEnumeratorMetrics enumeratorMetrics)

This must be invoked before this class's open() method, because open() starts an asynchronous splitting thread that concurrently mutates remainingSplits.

getNext

public Optional<SourceSplitBase> getNext()

Description copied from interface: SplitAssigner

Gets the next split.

When this method returns an empty Optional, then the set of splits is assumed to be done and the source will finish once the readers finished their current splits.

Specified by:

getNext in interface SplitAssigner

waitingForFinishedSplits

public boolean waitingForFinishedSplits()

Description copied from interface: SplitAssigner

Whether the split assigner is still waiting for callback of finished splits, i.e. SplitAssigner.onFinishedSplits(Map).

Specified by:

waitingForFinishedSplits in interface SplitAssigner

getFinishedSplitInfos

public List<FinishedSnapshotSplitInfo> getFinishedSplitInfos()

Description copied from interface: SplitAssigner

Gets the finished splits' information. This is useful metadata to generate a stream split that considering finished snapshot splits.

Specified by:

getFinishedSplitInfos in interface SplitAssigner

onFinishedSplits

public void onFinishedSplits(Map<String,Offset> splitFinishedOffsets)

Description copied from interface: SplitAssigner

Callback to handle the finished splits with finished change log offset. This is useful for determine when to generate stream split and what stream split to generate.

Specified by:

onFinishedSplits in interface SplitAssigner

addSplits

public void addSplits(Collection<SourceSplitBase> splits)

Description copied from interface: SplitAssigner

Adds a set of splits to this assigner. This happens for example when some split processing failed and the splits need to be re-added.

Specified by:

addSplits in interface SplitAssigner

snapshotState

public SnapshotPendingSplitsState snapshotState(long checkpointId)

Description copied from interface: SplitAssigner

Creates a snapshot of the state of this split assigner, to be stored in a checkpoint.

The snapshot should contain the latest state of the assigner: It should assume that all operations that happened before the snapshot have successfully completed. For example all splits assigned to readers via SplitAssigner.getNext() don't need to be included in the snapshot anymore.

This method takes the ID of the checkpoint for which the state is snapshotted. Most implementations should be able to ignore this parameter, because for the contents of the snapshot, it doesn't matter for which checkpoint it gets created. This parameter can be interesting for source connectors with external systems where those systems are themselves aware of checkpoints; for example in cases where the enumerator notifies that system about a specific checkpoint being triggered.

Specified by:

snapshotState in interface SplitAssigner

Parameters:

checkpointId - The ID of the checkpoint for which the snapshot is created.

Returns:

an object containing the state of the split enumerator.

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)

Description copied from interface: SplitAssigner

Notifies the listener that the checkpoint with the given checkpointId completed and was committed.

Specified by:

notifyCheckpointComplete in interface SplitAssigner

See Also:

CheckpointListener.notifyCheckpointComplete(long)

releaseSnapshotMetadata

public void releaseSnapshotMetadata()

Releases the bulk snapshot-split metadata once the reader has assembled it and a checkpoint has covered that assignment. Clears the metadata maps but keeps alreadyProcessedTables, the assigner status and the chunk-splitter state so a restore does not re-discover tables or rebuild the stream split. Generalizes the MySQL mechanism (FLINK-39775).

Specified by:

releaseSnapshotMetadata in interface SplitAssigner

isSnapshotMetaReleased

public boolean isSnapshotMetaReleased()

Description copied from interface: SplitAssigner

Returns whether SplitAssigner.releaseSnapshotMetadata() has dropped the snapshot metadata.

Specified by:

isSnapshotMetaReleased in interface SplitAssigner

close

public void close()
           throws IOException

Description copied from interface: SplitAssigner

Called to close the assigner, in case it holds on to any resources, like threads or network connections.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface SplitAssigner

Throws:

IOException

noMoreSplits

public boolean noMoreSplits()

Description copied from interface: SplitAssigner

Indicates there is no more splits available in this assigner.

Specified by:

noMoreSplits in interface SplitAssigner

needToDiscoveryTables

public boolean needToDiscoveryTables()

Indicates current assigner need to discovery tables or not.

getAssignedSplits

public Map<String,SchemalessSnapshotSplit> getAssignedSplits()

getAssignerStatus

public AssignerStatus getAssignerStatus()

Description copied from interface: SplitAssigner

Gets the split assigner status, see AssignerStatus.

Specified by:

getAssignerStatus in interface SplitAssigner

startAssignNewlyAddedTables

public void startAssignNewlyAddedTables()

Description copied from interface: SplitAssigner

Starts assign newly added tables.

Specified by:

startAssignNewlyAddedTables in interface SplitAssigner

onStreamSplitUpdated

public void onStreamSplitUpdated()

Description copied from interface: SplitAssigner

Callback to handle the stream split has been updated in the newly added tables process. This is useful to check the newly added tables has been finished or not.

Specified by:

onStreamSplitUpdated in interface SplitAssigner

getTableSchemas

public Map<io.debezium.relational.TableId,io.debezium.relational.history.TableChanges.TableChange> getTableSchemas()

getSplitFinishedOffsets

public Map<String,Offset> getSplitFinishedOffsets()