Package io.debezium.connector.mysql

Class MySqlStreamingChangeEventSource

    • Constructor Summary

      Constructors 
      Constructor Description
      MySqlStreamingChangeEventSource​(io.debezium.connector.mysql.MySqlConnectorConfig connectorConfig, MySqlConnection connection, io.debezium.pipeline.EventDispatcher<io.debezium.connector.mysql.MySqlPartition,​io.debezium.relational.TableId> dispatcher, io.debezium.pipeline.ErrorHandler errorHandler, io.debezium.util.Clock clock, io.debezium.connector.mysql.MySqlTaskContext taskContext, io.debezium.connector.mysql.MySqlStreamingChangeEventSourceMetrics metrics)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void execute​(io.debezium.pipeline.source.spi.ChangeEventSource.ChangeEventSourceContext context, io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext)  
      io.debezium.connector.mysql.GtidSet filterGtidSet​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, io.debezium.connector.mysql.GtidSet availableServerGtidSet, io.debezium.connector.mysql.GtidSet purgedServerGtid)
      Apply the include/exclude GTID source filters to the current GTID set and merge them onto the currently available GTID set from a MySQL server.
      protected void handleDelete​(io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Generate source records for the supplied event with an DeleteRowsEventData.
      protected void handleEvent​(io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)  
      protected void handleGtidEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction.
      protected void handleInsert​(io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Generate source records for the supplied event with an WriteRowsEventData.
      protected void handleQueryEvent​(io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.
      protected void handleRotateLogsEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with a RotateEventData that signals the logs are being rotated.
      protected void handleRowsQuery​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with an RowsQueryEventData by recording the original SQL query that generated the event.
      protected void handleServerHeartbeat​(io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive.
      protected void handleServerIncident​(io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event that signals that an out of the ordinary event that occurred on the master.
      protected void handleServerStop​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event that signals that mysqld has stopped.
      protected void handleUpdate​(io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Generate source records for the supplied event with an UpdateRowsEventData.
      protected void handleUpdateTableMetadata​(io.debezium.connector.mysql.MySqlPartition partition, io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle a change in the table metadata.
      protected void ignoreEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)  
      protected void logEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)  
      protected void onEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)  
      protected void prepareTransaction​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle a EventType.XA_PREPARE event.
      protected <T extends com.github.shyiko.mysql.binlog.event.EventData>
      T      		 unwrapData​(com.github.shyiko.mysql.binlog.event.Event event)  
      protected void viewChange​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle a EventType.VIEW_CHANGE event.
      protected io.debezium.DebeziumException wrap​(Throwable error)
      Wraps the specified exception in a DebeziumException, ensuring that all useful state is captured inside the new exception's message.

      • Methods inherited from interface io.debezium.pipeline.source.spi.StreamingChangeEventSource

        commitOffset, executeIteration, init

    • Constructor Detail

      • MySqlStreamingChangeEventSource

        public MySqlStreamingChangeEventSource​(io.debezium.connector.mysql.MySqlConnectorConfig connectorConfig,
                                       MySqlConnection connection,
                                       io.debezium.pipeline.EventDispatcher<io.debezium.connector.mysql.MySqlPartition,​io.debezium.relational.TableId> dispatcher,
                                       io.debezium.pipeline.ErrorHandler errorHandler,
                                       io.debezium.util.Clock clock,
                                       io.debezium.connector.mysql.MySqlTaskContext taskContext,
                                       io.debezium.connector.mysql.MySqlStreamingChangeEventSourceMetrics metrics)

    • Method Detail

      • onEvent

        protected void onEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                       com.github.shyiko.mysql.binlog.event.Event event)

      • ignoreEvent

        protected void ignoreEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                           com.github.shyiko.mysql.binlog.event.Event event)

      • handleEvent

        protected void handleEvent​(io.debezium.connector.mysql.MySqlPartition partition,
                           io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                           com.github.shyiko.mysql.binlog.event.Event event)

      • unwrapData

        protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T unwrapData​(com.github.shyiko.mysql.binlog.event.Event event)

      • handleServerStop

        protected void handleServerStop​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                                com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event that signals that mysqld has stopped.
        Parameters:
        event - the server stopped event to be processed; may not be null

      • handleServerHeartbeat

        protected void handleServerHeartbeat​(io.debezium.connector.mysql.MySqlPartition partition,
                                     io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                                     com.github.shyiko.mysql.binlog.event.Event event)
                              throws InterruptedException
        Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive. Not written to a binary log.
        Parameters:
        event - the server stopped event to be processed; may not be null
        Throws:
        InterruptedException

      • handleServerIncident

        protected void handleServerIncident​(io.debezium.connector.mysql.MySqlPartition partition,
                                    io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                                    com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event that signals that an out of the ordinary event that occurred on the master. It notifies the replica that something happened on the primary that might cause data to be in an inconsistent state.
        Parameters:
        event - the server stopped event to be processed; may not be null

      • handleRotateLogsEvent

        protected void handleRotateLogsEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                                     com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event with a RotateEventData that signals the logs are being rotated. This means that either the server was restarted, or the binlog has transitioned to a new file. In either case, subsequent table numbers will be different than those seen to this point.
        Parameters:
        event - the database change data event to be processed; may not be null

      • handleGtidEvent

        protected void handleGtidEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                               com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction. We don't yet know whether this transaction contains any events we're interested in, but we have to record it so that we know the position of this event and know we've processed the binlog to this point.

        Note that this captures the current GTID and complete GTID set, regardless of whether the connector is filtering the GTID set upon connection. We do this because we actually want to capture all GTID set values found in the binlog, whether or not we process them. However, only when we connect do we actually want to pass to MySQL only those GTID ranges that are applicable per the configuration.

        Parameters:
        event - the GTID event to be processed; may not be null

      • handleRowsQuery

        protected void handleRowsQuery​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                               com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event with an RowsQueryEventData by recording the original SQL query that generated the event.
        Parameters:
        event - the database change data event to be processed; may not be null

      • handleQueryEvent

        protected void handleQueryEvent​(io.debezium.connector.mysql.MySqlPartition partition,
                                io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                                com.github.shyiko.mysql.binlog.event.Event event)
                         throws InterruptedException
        Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.
        Parameters:
        partition - the partition in which the even occurred
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while recording the DDL statements

      • handleUpdateTableMetadata

        protected void handleUpdateTableMetadata​(io.debezium.connector.mysql.MySqlPartition partition,
                                         io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                                         com.github.shyiko.mysql.binlog.event.Event event)
                                  throws InterruptedException
        Handle a change in the table metadata.

        This method should be called whenever we consume a TABLE_MAP event, and every transaction in the log should include one of these for each table affected by the transaction. Each table map event includes a monotonically-increasing numeric identifier, and this identifier is used within subsequent events within the same transaction. This table identifier can change when:

        1. the table structure is modified (e.g., via an ALTER TABLE ... command); or
        2. MySQL rotates to a new binary log file, even if the table structure does not change.
        Parameters:
        event - the update event; never null
        Throws:
        InterruptedException

      • handleInsert

        protected void handleInsert​(io.debezium.connector.mysql.MySqlPartition partition,
                            io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                            com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException
        Generate source records for the supplied event with an WriteRowsEventData.
        Parameters:
        partition - the partition in which the even occurred
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • handleUpdate

        protected void handleUpdate​(io.debezium.connector.mysql.MySqlPartition partition,
                            io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                            com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException
        Generate source records for the supplied event with an UpdateRowsEventData.
        Parameters:
        partition - the partition in which the even occurred
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • handleDelete

        protected void handleDelete​(io.debezium.connector.mysql.MySqlPartition partition,
                            io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                            com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException
        Generate source records for the supplied event with an DeleteRowsEventData.
        Parameters:
        partition - the partition in which the even occurred
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • viewChange

        protected void viewChange​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                          com.github.shyiko.mysql.binlog.event.Event event)
                   throws InterruptedException
        Handle a EventType.VIEW_CHANGE event.
        Parameters:
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • prepareTransaction

        protected void prepareTransaction​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                                  com.github.shyiko.mysql.binlog.event.Event event)
                           throws InterruptedException
        Handle a EventType.XA_PREPARE event.
        Parameters:
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • execute

        public void execute​(io.debezium.pipeline.source.spi.ChangeEventSource.ChangeEventSourceContext context,
                    io.debezium.connector.mysql.MySqlPartition partition,
                    io.debezium.connector.mysql.MySqlOffsetContext offsetContext)
             throws InterruptedException
        Specified by:
        execute in interface io.debezium.pipeline.source.spi.StreamingChangeEventSource<io.debezium.connector.mysql.MySqlPartition,​io.debezium.connector.mysql.MySqlOffsetContext>
        Throws:
        InterruptedException

      • logEvent

        protected void logEvent​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                        com.github.shyiko.mysql.binlog.event.Event event)

      • filterGtidSet

        public io.debezium.connector.mysql.GtidSet filterGtidSet​(io.debezium.connector.mysql.MySqlOffsetContext offsetContext,
                                                         io.debezium.connector.mysql.GtidSet availableServerGtidSet,
                                                         io.debezium.connector.mysql.GtidSet purgedServerGtid)
        Apply the include/exclude GTID source filters to the current GTID set and merge them onto the currently available GTID set from a MySQL server.

        The merging behavior of this method might seem a bit strange at first. It's required in order for Debezium to consume a MySQL binlog that has multi-source replication enabled, if a failover has to occur. In such a case, the server that Debezium is failed over to might have a different set of sources, but still include the sources required for Debezium to continue to function. MySQL does not allow downstream replicas to connect if the GTID set does not contain GTIDs for all channels that the server is replicating from, even if the server does have the data needed by the client. To get around this, we can have Debezium merge its GTID set with whatever is on the server, so that MySQL will allow it to connect. See DBZ-143 for details.

        This method does not mutate any state in the context.

        Parameters:
        availableServerGtidSet - the GTID set currently available in the MySQL server
        purgedServerGtid - the GTID set already purged by the MySQL server
        Returns:
        A GTID set meant for consuming from a MySQL binlog; may return null if the SourceInfo has no GTIDs and therefore none were filtered

      • wrap

        protected io.debezium.DebeziumException wrap​(Throwable error)
        Wraps the specified exception in a DebeziumException, ensuring that all useful state is captured inside the new exception's message.
        Parameters:
        error - the exception; may not be null
        Returns:
        the wrapped Kafka Connect exception