Package org.apache.flink.cdc.connectors.base.source

Class IncrementalSource<T,C extends SourceConfig>

java.lang.Object

org.apache.flink.cdc.connectors.base.source.IncrementalSource<T,C>

All Implemented Interfaces:

Serializable, org.apache.flink.api.connector.source.Source<T,SourceSplitBase,PendingSplitsState>, org.apache.flink.api.connector.source.SourceReaderFactory<T,SourceSplitBase>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

Direct Known Subclasses:

JdbcIncrementalSource, MongoDBSource

@Experimental
public class IncrementalSource<T,C extends SourceConfig>
extends Object
implements org.apache.flink.api.connector.source.Source<T,SourceSplitBase,PendingSplitsState>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

The basic source of Incremental Snapshot framework for datasource, it is based on FLIP-27 and Watermark Signal Algorithm which supports parallel reading snapshot of table and then continue to capture data change by streaming reading.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected SourceConfig.Factory<C>	configFactory
protected DataSourceDialect<C>	dataSourceDialect
protected DebeziumDeserializationSchema<T>	deserializationSchema
protected OffsetFactory	offsetFactory
protected SnapshotPhaseHooks	snapshotHooks
protected SourceSplitSerializer	sourceSplitSerializer

Constructor Summary

Constructors

Constructor	Description
IncrementalSource(SourceConfig.Factory<C> configFactory, DebeziumDeserializationSchema<T> deserializationSchema, OffsetFactory offsetFactory, DataSourceDialect<C> dataSourceDialect)

Method Summary

Modifier and Type	Method	Description
org.apache.flink.api.connector.source.SplitEnumerator<SourceSplitBase,PendingSplitsState>	createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SourceSplitBase> enumContext)
IncrementalSourceReader<T,C>	createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext)
protected org.apache.flink.connector.base.source.reader.RecordEmitter<SourceRecords,T,SourceSplitState>	createRecordEmitter(SourceConfig sourceConfig, SourceReaderMetrics sourceReaderMetrics)
org.apache.flink.api.connector.source.Boundedness	getBoundedness()
org.apache.flink.core.io.SimpleVersionedSerializer<PendingSplitsState>	getEnumeratorCheckpointSerializer()
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getProducedType()
org.apache.flink.core.io.SimpleVersionedSerializer<SourceSplitBase>	getSplitSerializer()
org.apache.flink.api.connector.source.SplitEnumerator<SourceSplitBase,PendingSplitsState>	restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SourceSplitBase> enumContext, PendingSplitsState checkpoint)
void	setSnapshotHooks(SnapshotPhaseHooks snapshotHooks)	Set snapshot hooks only for test.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

configFactory

protected final SourceConfig.Factory<C extends SourceConfig> configFactory

dataSourceDialect

protected final DataSourceDialect<C extends SourceConfig> dataSourceDialect

offsetFactory

protected final OffsetFactory offsetFactory

deserializationSchema

protected final DebeziumDeserializationSchema<T> deserializationSchema

sourceSplitSerializer

protected final SourceSplitSerializer sourceSplitSerializer

snapshotHooks

protected SnapshotPhaseHooks snapshotHooks

Constructor Detail

IncrementalSource

public IncrementalSource(SourceConfig.Factory<C> configFactory,
                         DebeziumDeserializationSchema<T> deserializationSchema,
                         OffsetFactory offsetFactory,
                         DataSourceDialect<C> dataSourceDialect)

Method Detail

getBoundedness

public org.apache.flink.api.connector.source.Boundedness getBoundedness()

Specified by:

getBoundedness in interface org.apache.flink.api.connector.source.Source<T,SourceSplitBase,PendingSplitsState>

createReader

public IncrementalSourceReader<T,C> createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext)
                                                throws Exception

Specified by:

createReader in interface org.apache.flink.api.connector.source.SourceReaderFactory<T,C extends SourceConfig>

Throws:

Exception

createEnumerator

public org.apache.flink.api.connector.source.SplitEnumerator<SourceSplitBase,PendingSplitsState> createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SourceSplitBase> enumContext)

Specified by:

createEnumerator in interface org.apache.flink.api.connector.source.Source<T,SourceSplitBase,PendingSplitsState>

restoreEnumerator

public org.apache.flink.api.connector.source.SplitEnumerator<SourceSplitBase,PendingSplitsState> restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SourceSplitBase> enumContext,
                                                                                                                         PendingSplitsState checkpoint)

Specified by:

restoreEnumerator in interface org.apache.flink.api.connector.source.Source<T,SourceSplitBase,PendingSplitsState>

getSplitSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<SourceSplitBase> getSplitSerializer()

Specified by:

getSplitSerializer in interface org.apache.flink.api.connector.source.Source<T,SourceSplitBase,PendingSplitsState>

getEnumeratorCheckpointSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<PendingSplitsState> getEnumeratorCheckpointSerializer()

Specified by:

getEnumeratorCheckpointSerializer in interface org.apache.flink.api.connector.source.Source<T,SourceSplitBase,PendingSplitsState>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

createRecordEmitter

protected org.apache.flink.connector.base.source.reader.RecordEmitter<SourceRecords,T,SourceSplitState> createRecordEmitter(SourceConfig sourceConfig,
                                                                                                                                        SourceReaderMetrics sourceReaderMetrics)

setSnapshotHooks

@VisibleForTesting
public void setSnapshotHooks(SnapshotPhaseHooks snapshotHooks)

Set snapshot hooks only for test. The SnapshotPhaseHook should be serializable。

Parameters:

snapshotHooks -