Package org.apache.flink.cdc.connectors.mysql.source

Class MySqlSourceBuilder<T>

  • @PublicEvolving
public class MySqlSourceBuilder<T>
extends Object
    The builder class for MySqlSource to make it easier for the users to construct a MySqlSource. 
    
 MySqlSource
     .<String>builder()
     .hostname("localhost")
     .port(3306)
     .databaseList("mydb")
     .tableList("mydb.users")
     .username(username)
     .password(password)
     .serverId(5400)
     .deserializer(new JsonDebeziumDeserializationSchema())
     .build();

    Check the Java docs of each individual method to learn more about the settings to build a MySqlSource.

    • Constructor Detail

      • MySqlSourceBuilder

        public MySqlSourceBuilder()

    • Method Detail

      • port

        public MySqlSourceBuilder<T> port​(int port)
        Integer port number of the MySQL database server.

      • databaseList

        public MySqlSourceBuilder<T> databaseList​(String... databaseList)
        An required list of regular expressions that match database names to be monitored; any database name not included in the whitelist will be excluded from monitoring.

      • tableList

        public MySqlSourceBuilder<T> tableList​(String... tableList)
        An required list of regular expressions that match fully-qualified table identifiers for tables to be monitored; any table not included in the list will be excluded from monitoring. Each identifier is of the form <databaseName>.<tableName>.

      • username

        public MySqlSourceBuilder<T> username​(String username)
        Name of the MySQL database to use when connecting to the MySQL database server.

      • password

        public MySqlSourceBuilder<T> password​(String password)
        Password to use when connecting to the MySQL database server.

      • serverId

        public MySqlSourceBuilder<T> serverId​(String serverId)
        A numeric ID or a numeric ID range of this database client, The numeric ID syntax is like '5400', the numeric ID range syntax is like '5400-5408', The numeric ID range syntax is required when 'scan.incremental.snapshot.enabled' enabled. Every ID must be unique across all currently-running database processes in the MySQL cluster. This connector joins the MySQL cluster as another server (with this unique ID) so it can read the binlog. By default, a random number is generated between 5400 and 6400, though we recommend setting an explicit value."

      • serverTimeZone

        public MySqlSourceBuilder<T> serverTimeZone​(String timeZone)
        The session time zone in database server, e.g. "America/Los_Angeles". It controls how the TIMESTAMP type in MYSQL converted to STRING. See more https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-temporal-types

      • chunkKeyColumn

        public MySqlSourceBuilder<T> chunkKeyColumn​(org.apache.flink.table.catalog.ObjectPath objectPath,
                                            String chunkKeyColumn)
        The chunk key of table snapshot, captured tables are split into multiple chunks by the chunk key column when read the snapshot of table.

      • splitSize

        public MySqlSourceBuilder<T> splitSize​(int splitSize)
        The split size (number of rows) of table snapshot, captured tables are split into multiple splits when read the snapshot of table.

      • splitMetaGroupSize

        public MySqlSourceBuilder<T> splitMetaGroupSize​(int splitMetaGroupSize)
        The group size of split meta, if the meta size exceeds the group size, the meta will be divided into multiple groups.

      • distributionFactorUpper

        public MySqlSourceBuilder<T> distributionFactorUpper​(double distributionFactorUpper)
        The upper bound of split key evenly distribution factor, the factor is used to determine whether the table is evenly distribution or not.

      • distributionFactorLower

        public MySqlSourceBuilder<T> distributionFactorLower​(double distributionFactorLower)
        The lower bound of split key evenly distribution factor, the factor is used to determine whether the table is evenly distribution or not.

      • fetchSize

        public MySqlSourceBuilder<T> fetchSize​(int fetchSize)
        The maximum fetch size for per poll when read table snapshot.

      • connectTimeout

        public MySqlSourceBuilder<T> connectTimeout​(java.time.Duration connectTimeout)
        The maximum time that the connector should wait after trying to connect to the MySQL database server before timing out.

      • connectMaxRetries

        public MySqlSourceBuilder<T> connectMaxRetries​(int connectMaxRetries)
        The max retry times to get connection.

      • connectionPoolSize

        public MySqlSourceBuilder<T> connectionPoolSize​(int connectionPoolSize)
        The connection pool size.

      • includeSchemaChanges

        public MySqlSourceBuilder<T> includeSchemaChanges​(boolean includeSchemaChanges)
        Whether the MySqlSource should output the schema changes or not.

      • includeTransactionMetadataEvents

        public MySqlSourceBuilder<T> includeTransactionMetadataEvents​(boolean includeTransactionMetadataEvents)
        Whether the MySqlSource should output the transaction metadata events or not.

      • scanNewlyAddedTableEnabled

        public MySqlSourceBuilder<T> scanNewlyAddedTableEnabled​(boolean scanNewlyAddedTableEnabled)
        Whether the MySqlSource should scan the newly added tables or not.

      • releaseSnapshotMetadataEnabled

        public MySqlSourceBuilder<T> releaseSnapshotMetadataEnabled​(boolean releaseSnapshotMetadataEnabled)
        Whether the source coordinator should release the snapshot split metadata after entering the binlog phase to reduce JobManager memory. Incompatible with scanNewlyAddedTableEnabled(boolean); disabled by default.

      • jdbcProperties

        public MySqlSourceBuilder<T> jdbcProperties​(Properties jdbcProperties)
        Custom properties that will overwrite the default JDBC connection URL.

      • debeziumProperties

        public MySqlSourceBuilder<T> debeziumProperties​(Properties properties)
        The Debezium MySQL connector properties. For example, "snapshot.mode".

      • heartbeatInterval

        public MySqlSourceBuilder<T> heartbeatInterval​(java.time.Duration heartbeatInterval)
        The interval of heartbeat event.

      • heartbeatInterval

        public MySqlSourceBuilder<T> heartbeatInterval​(java.time.Duration heartbeatInterval,
                                               boolean includeHeartbeatEvents)
        The interval of heartbeat event and whether to emit heartbeat events.
        Parameters:
        heartbeatInterval - the interval of heartbeat event
        includeHeartbeatEvents - whether to emit heartbeat events

      • skipSnapshotBackfill

        public MySqlSourceBuilder<T> skipSnapshotBackfill​(boolean skipSnapshotBackfill)
        Whether to skip backfill in snapshot reading phase.

        If backfill is skipped, changes on captured tables during snapshot phase will be consumed later in binlog reading phase instead of being merged into the snapshot.

        WARNING: Skipping backfill might lead to data inconsistency because some binlog events happened within the snapshot phase might be replayed (only at-least-once semantic is promised). For example updating an already updated value in snapshot, or deleting an already deleted entry in snapshot. These replayed binlog events should be handled specially.

      • useLegacyJsonFormat

        public MySqlSourceBuilder<T> useLegacyJsonFormat​(boolean useLegacyJsonFormat)
        Whether to use legacy json format. The default value is true, which means there is no whitespace before value and after comma in json format.

      • closeIdleReaders

        public MySqlSourceBuilder<T> closeIdleReaders​(boolean closeIdleReaders)
        Whether to close idle readers at the end of the snapshot phase. This feature depends on FLIP-147: Support Checkpoints After Tasks Finished. The flink version is required to be greater than or equal to 1.14, and the configuration 'execution.checkpointing.checkpoints-after-tasks-finish.enabled' needs to be set to true.

        See more https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished.

      • parseOnLineSchemaChanges

        public MySqlSourceBuilder<T> parseOnLineSchemaChanges​(boolean parseOnLineSchemaChanges)
        Whether to parse gh-ost utility generated schema change events. Defaults to false.

      • assignUnboundedChunkFirst

        public MySqlSourceBuilder<T> assignUnboundedChunkFirst​(boolean assignUnboundedChunkFirst)
        Whether the MySqlSource should assign the unbounded chunks first or not during snapshot reading phase.

      • build

        public MySqlSource<T> build()
        Build the MySqlSource.
        Returns:
        a MySqlParallelSource with the settings made for this builder.