Package org.apache.flink.cdc.connectors.base.source.reader

Class IncrementalSourceReader<T,C extends SourceConfig>

java.lang.Object

org.apache.flink.connector.base.source.reader.SourceReaderBase<E,T,SplitT,SplitStateT>

org.apache.flink.connector.base.source.reader.SingleThreadMultiplexSourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader<T,C>

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SourceReader<T,SourceSplitBase>

Direct Known Subclasses:

IncrementalSourceReaderWithCommit

@Experimental
public class IncrementalSourceReader<T,C extends SourceConfig>
extends org.apache.flink.connector.base.source.reader.SingleThreadMultiplexSourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

The multi-parallel source reader for table snapshot phase from SnapshotSplit and then single-parallel source reader for table stream phase from StreamSplit.

Field Summary

Fields

Modifier and Type	Field	Description
protected DataSourceDialect<C>	dialect
protected IncrementalSourceReaderContext	incrementalSourceReaderContext
protected C	sourceConfig
protected StreamSplit	suspendedStreamSplit	Steam split which is suspended reading in split reader and wait for LatestFinishedSplitsNumberEvent from enumerator, will addSpilt and become uncompletedStreamSplits later.
protected Map<String,StreamSplit>	uncompletedStreamSplits	Stream split which lack and wait for snapshot splits information from enumerator, will addSplit and add back to split reader if snapshot split information is enough.

Fields inherited from class org.apache.flink.connector.base.source.reader.SourceReaderBase

config, context, eofRecordEvaluator, options, recordEmitter, splitFetcherManager

Constructor Summary

Constructors

Constructor

Description

IncrementalSourceReader(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<SourceRecords>> elementQueue, java.util.function.Supplier<IncrementalSourceSplitReader<C>> splitReaderSupplier, org.apache.flink.connector.base.source.reader.RecordEmitter<SourceRecords,T,SourceSplitState> recordEmitter, org.apache.flink.configuration.Configuration config, IncrementalSourceReaderContext incrementalSourceReaderContext, C sourceConfig, SourceSplitSerializer sourceSplitSerializer, DataSourceDialect<C> dialect)

Method Summary

Modifier and Type	Method	Description
void	addSplits(List<SourceSplitBase> splits)
Map<String,SnapshotSplit>	getFinishedUnackedSplits()
static int	getNextMetaGroupId(int receivedMetaNum, int metaGroupSize)	Returns next meta group id according to received meta number and meta group size.
void	handleSourceEvents(org.apache.flink.api.connector.source.SourceEvent sourceEvent)
protected SourceSplitState	initializedState(SourceSplitBase split)
protected void	onSplitFinished(Map<String,SourceSplitState> finishedSplitIds)
List<SourceSplitBase>	snapshotState(long checkpointId)
void	start()
protected SourceSplitBase	toSplitType(String splitId, SourceSplitState splitState)
protected void	updateStreamSplitFinishedSplitsSize(LatestFinishedSplitsNumberEvent sourceEvent)

Methods inherited from class org.apache.flink.connector.base.source.reader.SourceReaderBase

close, getNumberOfCurrentlyAssignedSplits, isAvailable, notifyNoMoreSplits, pauseOrResumeSplits, pollNext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.api.connector.source.SourceReader

notifyCheckpointComplete

Field Detail

uncompletedStreamSplits

protected final Map<String,StreamSplit> uncompletedStreamSplits

Stream split which lack and wait for snapshot splits information from enumerator, will addSplit and add back to split reader if snapshot split information is enough.

suspendedStreamSplit

protected volatile StreamSplit suspendedStreamSplit

Steam split which is suspended reading in split reader and wait for LatestFinishedSplitsNumberEvent from enumerator, will addSpilt and become uncompletedStreamSplits later.

sourceConfig

protected final C extends SourceConfig sourceConfig

dialect

protected final DataSourceDialect<C extends SourceConfig> dialect

incrementalSourceReaderContext

protected final IncrementalSourceReaderContext incrementalSourceReaderContext

Constructor Detail

IncrementalSourceReader

public IncrementalSourceReader(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<SourceRecords>> elementQueue,
                               java.util.function.Supplier<IncrementalSourceSplitReader<C>> splitReaderSupplier,
                               org.apache.flink.connector.base.source.reader.RecordEmitter<SourceRecords,T,SourceSplitState> recordEmitter,
                               org.apache.flink.configuration.Configuration config,
                               IncrementalSourceReaderContext incrementalSourceReaderContext,
                               C sourceConfig,
                               SourceSplitSerializer sourceSplitSerializer,
                               DataSourceDialect<C> dialect)

Method Detail

start

public void start()

Specified by:

start in interface org.apache.flink.api.connector.source.SourceReader<T,C extends SourceConfig>

Overrides:

start in class org.apache.flink.connector.base.source.reader.SourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

initializedState

protected SourceSplitState initializedState(SourceSplitBase split)

Specified by:

initializedState in class org.apache.flink.connector.base.source.reader.SourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

snapshotState

public List<SourceSplitBase> snapshotState(long checkpointId)

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SourceReader<T,C extends SourceConfig>

Overrides:

snapshotState in class org.apache.flink.connector.base.source.reader.SourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

onSplitFinished

protected void onSplitFinished(Map<String,SourceSplitState> finishedSplitIds)

Specified by:

onSplitFinished in class org.apache.flink.connector.base.source.reader.SourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

addSplits

public void addSplits(List<SourceSplitBase> splits)

Specified by:

addSplits in interface org.apache.flink.api.connector.source.SourceReader<T,C extends SourceConfig>

Overrides:

addSplits in class org.apache.flink.connector.base.source.reader.SourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

handleSourceEvents

public void handleSourceEvents(org.apache.flink.api.connector.source.SourceEvent sourceEvent)

Specified by:

handleSourceEvents in interface org.apache.flink.api.connector.source.SourceReader<T,C extends SourceConfig>

Overrides:

handleSourceEvents in class org.apache.flink.connector.base.source.reader.SourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

updateStreamSplitFinishedSplitsSize

protected void updateStreamSplitFinishedSplitsSize(LatestFinishedSplitsNumberEvent sourceEvent)

getNextMetaGroupId

public static int getNextMetaGroupId(int receivedMetaNum,
                                     int metaGroupSize)

Returns next meta group id according to received meta number and meta group size.

toSplitType

protected SourceSplitBase toSplitType(String splitId,
                                      SourceSplitState splitState)

Specified by:

toSplitType in class org.apache.flink.connector.base.source.reader.SourceReaderBase<SourceRecords,T,SourceSplitBase,SourceSplitState>

getFinishedUnackedSplits

@VisibleForTesting
public Map<String,SnapshotSplit> getFinishedUnackedSplits()