Package org.apache.flink.cdc.connectors.mongodb

Class MongoDBSource.Builder<T>

    • Constructor Detail

      • Builder

        public Builder()

    • Method Detail

      • connectionOptions

        public MongoDBSource.Builder<T> connectionOptions​(String connectionOptions)
        Ampersand (i.e. &) separated MongoDB connection options eg replicaSet=test&connectTimeoutMS=300000 https://docs.mongodb.com/manual/reference/connection-string/#std-label-connections-connection-options

      • databaseList

        public MongoDBSource.Builder<T> databaseList​(String... databaseList)
        Regular expressions list that match database names to be monitored.

      • collectionList

        public MongoDBSource.Builder<T> collectionList​(String... collectionList)
        Regular expressions that match fully-qualified collection identifiers for collections to be monitored. Each identifier is of the form <databaseName>.<collectionName>.

      • batchSize

        public MongoDBSource.Builder<T> batchSize​(int batchSize)
        batch.size

        The cursor batch size. Default: 1024

        The change stream cursor batch size. Specifies the maximum number of change events to return in each batch of the response from the MongoDB cluster. The default is 0 meaning it uses the server's default value. Default: 0

      • pollAwaitTimeMillis

        public MongoDBSource.Builder<T> pollAwaitTimeMillis​(int pollAwaitTimeMillis)
        poll.await.time.ms

        The amount of time to wait before checking for new results on the change stream. Default: 1000

      • pollMaxBatchSize

        public MongoDBSource.Builder<T> pollMaxBatchSize​(int pollMaxBatchSize)
        poll.max.batch.size

        Maximum number of change stream documents to include in a single batch when polling for new data. This setting can be used to limit the amount of data buffered internally in the connector. Default: 1024

      • updateLookup

        public MongoDBSource.Builder<T> updateLookup​(boolean updateLookup)
        change.stream.full.document

        Determines what to return for update operations when using a Change Stream. When set to true, the change stream for partial updates will include both a delta describing the changes to the document and a copy of the entire document that was changed from some time after the change occurred. Default: true

      • scanFullChangelog

        public MongoDBSource.Builder<T> scanFullChangelog​(boolean fullDocumentBeforeChange)
        change.stream.full.document.before.change

        Configures the document pre-image your change stream returns on update operations. The pre-image is not available for source records published while copying existing data, and the pre-image configuration has no effect on copying.

      • copyExisting

        @Deprecated
public MongoDBSource.Builder<T> copyExisting​(boolean copyExisting)
        Deprecated.
        please use startupOptions instead.
        copy.existing

        Copy existing data from source collections and convert them to Change Stream events on their respective topics. Any changes to the data that occur during the copy process are applied once the copy is completed.

      • startupOptions

        public MongoDBSource.Builder<T> startupOptions​(StartupOptions startupOptions)
        scan.startup.mode

        Optional startup mode for MongoDB CDC consumer, valid enumerations are initial, latest-offset, timestamp. Default: initial

      • copyExistingMaxThreads

        @Deprecated
public MongoDBSource.Builder<T> copyExistingMaxThreads​(int copyExistingMaxThreads)
        Deprecated.
        use initialSnapshottingMaxThreads instead.
        copy.existing.max.threads

        The number of threads to use when performing the data copy. Defaults to the number of processors. Default: defaults to the number of processors

      • initialSnapshottingMaxThreads

        public MongoDBSource.Builder<T> initialSnapshottingMaxThreads​(int initialSnapshottingMaxThreads)
        initial.snapshotting.max.threads

        The number of threads to use when performing the data copy. Defaults to the number of processors. Default: defaults to the number of processors

      • copyExistingQueueSize

        @Deprecated
public MongoDBSource.Builder<T> copyExistingQueueSize​(int copyExistingQueueSize)
        Deprecated.
        use initialSnapshottingQueueSize instead.
        copy.existing.queue.size

        The max size of the queue to use when copying data. Default: 10240

      • initialSnapshottingQueueSize

        public MongoDBSource.Builder<T> initialSnapshottingQueueSize​(int initialSnapshottingQueueSize)
        initial.snapshotting.queue.size

        The max size of the queue to use when copying data. Default: 10240

      • copyExistingPipeline

        @Deprecated
public MongoDBSource.Builder<T> copyExistingPipeline​(String copyExistingPipeline)
        Deprecated.
        use initialSnapshottingPipeline instead.
        copy.existing.pipeline eg. [ { "$match": { "closed": "false" } } ]

        An array of JSON objects describing the pipeline operations to run when copying existing data. This can improve the use of indexes by the copying manager and make copying more efficient.

      • initialSnapshottingPipeline

        public MongoDBSource.Builder<T> initialSnapshottingPipeline​(String initialSnapshottingPipeline)
        initial.snapshotting.pipeline eg. [ { "$match": { "closed": "false" } } ]

        An array of JSON objects describing the pipeline operations to run when copying existing data. This can improve the use of indexes by the copying manager and make copying more efficient.

      • heartbeatIntervalMillis

        public MongoDBSource.Builder<T> heartbeatIntervalMillis​(int heartbeatIntervalMillis)
        heartbeat.interval.ms

        The length of time in milliseconds between sending heartbeat messages. Heartbeat messages contain the post batch resume token and are sent when no source records have been published in the specified interval. This improves the resumability of the connector for low volume namespaces. Use 0 to disable.

      • build

        public DebeziumSourceFunction<T> build()
        The properties of mongodb kafka connector. https://docs.mongodb.com/kafka-connector/current/kafka-source