Package org.apache.flink.cdc.connectors.mongodb.source

Class MongoDBSourceBuilder<T>

  • @Experimental
@PublicEvolving
public class MongoDBSourceBuilder<T>
extends Object
    The builder class for MongoDBSource to make it easier for the users to construct a MongoDBSource. 
    
 MongoDBSource
     .<String>builder()
     .hosts("localhost:27017")
     .databaseList("mydb")
     .collectionList("mydb.users")
     .username(username)
     .password(password)
     .deserializer(new JsonDebeziumDeserializationSchema())
     .build();

    Check the Java docs of each individual method to learn more about the settings to build a MongoDBSource.

    • Constructor Detail

      • MongoDBSourceBuilder

        public MongoDBSourceBuilder()

    • Method Detail

      • scheme

        public MongoDBSourceBuilder<T> scheme​(String scheme)
        The protocol connected to MongoDB. For example mongodb or mongodb+srv.

      • hosts

        public MongoDBSourceBuilder<T> hosts​(String hosts)
        The comma-separated list of hostname and port pairs of mongodb servers.

      • connectionOptions

        public MongoDBSourceBuilder<T> connectionOptions​(String connectionOptions)
        Ampersand (i.e. &) separated MongoDB connection options eg replicaSet=test&connectTimeoutMS=300000 https://docs.mongodb.com/manual/reference/connection-string/#std-label-connections-connection-options

      • username

        public MongoDBSourceBuilder<T> username​(String username)
        Name of the database user to be used when connecting to MongoDB.

      • databaseList

        public MongoDBSourceBuilder<T> databaseList​(String... databases)
        Regular expressions list that match database names to be monitored.

      • collectionList

        public MongoDBSourceBuilder<T> collectionList​(String... collections)
        Regular expressions that match fully-qualified collection identifiers for collections to be monitored. Each identifier is of the form <databaseName>.<collectionName>.

      • batchSize

        public MongoDBSourceBuilder<T> batchSize​(int batchSize)
        batch.size

        The cursor batch size. Default: 1024

      • pollAwaitTimeMillis

        public MongoDBSourceBuilder<T> pollAwaitTimeMillis​(int pollAwaitTimeMillis)
        poll.await.time.ms

        The amount of time to wait before checking for new results on the change stream. Default: 1000

      • pollMaxBatchSize

        public MongoDBSourceBuilder<T> pollMaxBatchSize​(int pollMaxBatchSize)
        poll.max.batch.size

        Maximum number of change stream documents to include in a single batch when polling for new data. This setting can be used to limit the amount of data buffered internally in the connector. Default: 1024

      • startupOptions

        public MongoDBSourceBuilder<T> startupOptions​(StartupOptions startupOptions)
        scan.startup.mode

        Optional startup mode for MongoDB CDC consumer, valid enumerations are initial, latest-offset, timestamp. Default: initial

      • heartbeatIntervalMillis

        public MongoDBSourceBuilder<T> heartbeatIntervalMillis​(int heartbeatIntervalMillis)
        heartbeat.interval.ms

        The length of time in milliseconds between sending heartbeat messages. Heartbeat messages contain the post batch resume token and are sent when no source records have been published in the specified interval. This improves the resumability of the connector for low volume namespaces. Use 0 to disable.

      • splitSizeMB

        public MongoDBSourceBuilder<T> splitSizeMB​(int splitSizeMB)
        scan.incremental.snapshot.chunk.size.mb

        The chunk size mb of incremental snapshot. Default: 64mb.

      • splitMetaGroupSize

        public MongoDBSourceBuilder<T> splitMetaGroupSize​(int splitMetaGroupSize)
        The group size of split meta, if the meta size exceeds the group size, the meta will be divided into multiple groups.

      • samplesPerChunk

        public MongoDBSourceBuilder<T> samplesPerChunk​(int samplesPerChunk)
        scan.incremental.snapshot.chunk.samples

        The number of samples to take per chunk. Defaults to 20.

      • closeIdleReaders

        public MongoDBSourceBuilder<T> closeIdleReaders​(boolean closeIdleReaders)
        scan.incremental.close-idle-reader.enabled

        Whether to close idle readers at the end of the snapshot phase. This feature depends on FLIP-147: Support Checkpoints After Tasks Finished. The flink version is required to be greater than or equal to 1.14, and the configuration 'execution.checkpointing.checkpoints-after-tasks-finish.enabled' needs to be set to true.

        See more https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished.

      • scanFullChangelog

        public MongoDBSourceBuilder<T> scanFullChangelog​(boolean enableFullDocPrePostImage)
        scan.full-changelog

        Whether to generate full mode row data by looking up full document pre- and post-image collections. Requires MongoDB >= 6.0.

      • disableCursorTimeout

        public MongoDBSourceBuilder<T> disableCursorTimeout​(boolean disableCursorTimeout)
        Whether disable cursor timeout during snapshot phase. Defaults to true. Only enable this when MongoDB server doesn't support noCursorTimeout option.

      • skipSnapshotBackfill

        public MongoDBSourceBuilder<T> skipSnapshotBackfill​(boolean skipSnapshotBackfill)
        Whether to skip backfill in snapshot reading phase.

        If backfill is skipped, changes on captured tables during snapshot phase will be consumed later in binlog reading phase instead of being merged into the snapshot.

        WARNING: Skipping backfill might lead to data inconsistency because some binlog events happened within the snapshot phase might be replayed (only at-least-once semantic is promised). For example updating an already updated value in snapshot, or deleting an already deleted entry in snapshot. These replayed binlog events should be handled specially.

      • assignUnboundedChunkFirst

        public MongoDBSourceBuilder<T> assignUnboundedChunkFirst​(boolean assignUnboundedChunkFirst)
        Whether the MongoDBSource should assign the unbounded chunks first or not during snapshot reading phase.

      • releaseSnapshotMetadataEnabled

        public MongoDBSourceBuilder<T> releaseSnapshotMetadataEnabled​(boolean releaseSnapshotMetadataEnabled)
        Whether to release the finished snapshot split metadata from the coordinator after entering the stream phase, to reduce JobManager memory pressure (FLINK-40697).

      • build

        public MongoDBSource<T> build()
        Build the MongoDBSource.
        Returns:
        a MongoDBParallelSource with the settings made for this builder.

      • sslEnabled

        public MongoDBSourceBuilder<T> sslEnabled​(boolean sslEnabled)
        Whether the connector will use SSL to connect to MongoDB instances.

      • sslInvalidHostnameAllowed

        public MongoDBSourceBuilder<T> sslInvalidHostnameAllowed​(boolean sslInvalidHostnameAllowed)
        When SSL is enabled, controls whether strict hostname checking is disabled.

      • sslKeyStore

        public MongoDBSourceBuilder<T> sslKeyStore​(String sslKeyStore)
        The location of the key store file for two-way authentication.

      • sslKeyStorePassword

        public MongoDBSourceBuilder<T> sslKeyStorePassword​(String sslKeyStorePassword)
        The password for the key store file.

      • sslTrustStore

        public MongoDBSourceBuilder<T> sslTrustStore​(String sslTrustStore)
        The location of the trust store file for the server certificate verification.

      • sslTrustStorePassword

        public MongoDBSourceBuilder<T> sslTrustStorePassword​(String sslTrustStorePassword)
        The password for the trust store file.