Class MySqlValueConverters

  • All Implemented Interfaces:
    io.debezium.relational.ValueConverterProvider

    @Immutable
    public class MySqlValueConverters
    extends io.debezium.jdbc.JdbcValueConverters
    Copied from Debezium project(1.9.8.Final) to fix FLOAT converted to FLOAT64 type issue. See DBZ-3865, DBZ-5843. Remove it when debezium version is upgraded above 2.0.0.Final.

    Line 240 & 246: add FLOAT type adjustment.

    Line 485 & 499: add method to convert FLOAT value.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static interface  MySqlValueConverters.ParsingErrorHandler  
      • Nested classes/interfaces inherited from class io.debezium.jdbc.JdbcValueConverters

        io.debezium.jdbc.JdbcValueConverters.BigIntUnsignedMode, io.debezium.jdbc.JdbcValueConverters.DecimalMode
    • Field Summary

      • Fields inherited from class io.debezium.jdbc.JdbcValueConverters

        adaptiveTimeMicrosecondsPrecisionMode, adaptiveTimePrecisionMode, bigIntUnsignedMode, binaryMode, decimalMode, logger
    • Constructor Summary

      Constructors 
      Constructor Description
      MySqlValueConverters​(io.debezium.jdbc.JdbcValueConverters.DecimalMode decimalMode, io.debezium.jdbc.TemporalPrecisionMode temporalPrecisionMode, io.debezium.jdbc.JdbcValueConverters.BigIntUnsignedMode bigIntUnsignedMode, io.debezium.config.CommonConnectorConfig.BinaryHandlingMode binaryMode)
      Create a new instance that always uses UTC for the default time zone when_needed converting values without timezone information to values that require timezones.
      MySqlValueConverters​(io.debezium.jdbc.JdbcValueConverters.DecimalMode decimalMode, io.debezium.jdbc.TemporalPrecisionMode temporalPrecisionMode, io.debezium.jdbc.JdbcValueConverters.BigIntUnsignedMode bigIntUnsignedMode, io.debezium.config.CommonConnectorConfig.BinaryHandlingMode binaryMode, java.time.temporal.TemporalAdjuster adjuster, MySqlValueConverters.ParsingErrorHandler parsingErrorHandler)
      Create a new instance that always uses UTC for the default time zone when converting values without timezone information to values that require timezones.
    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static java.time.temporal.Temporal adjustTemporal​(java.time.temporal.Temporal temporal)
      A utility method that adjusts ambiguous 2-digit year values of DATETIME, DATE, and TIMESTAMP types using these MySQL-specific rules: Year values in the range 00-69 are converted to 2000-2069.
      protected ByteOrder byteOrderOfBitType()  
      protected Charset charsetFor​(io.debezium.relational.Column column)
      Return the Charset instance with the MySQL-specific character set name used by the given column.
      static boolean containsZeroValuesInDatePart​(String timestampString, io.debezium.relational.Column column, io.debezium.relational.Table table)  
      protected Object convertBigInt​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)  
      protected Object convertDurationToMicroseconds​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Converts a value object for an expected type of Duration to Long values that represents the time in microseconds.
      protected Object convertEnumToString​(List<String> options, io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Converts a value object for a MySQL ENUM, which is represented in the binlog events as an integer value containing the index of the enum option.
      io.debezium.relational.ValueConverter converter​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn)  
      protected Object convertFloat​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      MySql reports FLOAT(p) values as FLOAT and DOUBLE.
      protected Object convertGeometry​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the a value representing a GEOMETRY byte[] value to a Geometry value used in a SourceRecord.
      protected Object convertInteger​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)  
      protected Object convertJson​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the String byte[] value to a string value used in a SourceRecord.
      protected Object convertPoint​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the a value representing a POINT byte[] value to a Point value used in a SourceRecord.
      protected Object convertSetToString​(List<String> options, io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Converts a value object for a MySQL SET, which is represented in the binlog events contain a long number in which every bit corresponds to a different option.
      protected String convertSetValue​(io.debezium.relational.Column column, long indexes, List<String> options)  
      protected Object convertSmallInt​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)  
      protected Object convertString​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Charset columnCharset, Object data)
      Convert the String or byte[] value to a string value used in a SourceRecord.
      protected Object convertTimestampToLocalDateTime​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)  
      protected Object convertUnsignedBigint​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the a value representing a Unsigned BIGINT value to the correct Unsigned INT representation.
      protected Object convertUnsignedInt​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the a value representing a Unsigned INT value to the correct Unsigned INT representation.
      protected Object convertUnsignedMediumint​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the a value representing a Unsigned MEDIUMINT value to the correct Unsigned SMALLINT representation.
      protected Object convertUnsignedSmallint​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the a value representing a Unsigned SMALLINT value to the correct Unsigned SMALLINT representation.
      protected Object convertUnsignedTinyint​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Convert the a value representing a Unsigned TINYINT value to the correct Unsigned TINYINT representation.
      protected Object convertYearToInt​(io.debezium.relational.Column column, org.apache.kafka.connect.data.Field fieldDefn, Object data)
      Converts a value object for a MySQL YEAR, which appear in the binlog as an integer though returns from the MySQL JDBC driver as either a short or a Date.
      static void defaultParsingErrorHandler​(String message, Exception exception)  
      protected List<String> extractEnumAndSetOptions​(io.debezium.relational.Column column)  
      protected String extractEnumAndSetOptionsAsString​(io.debezium.relational.Column column)  
      protected boolean isGeometryCollection​(String upperCaseTypeName)
      Determine if the uppercase form of a column's type is geometry collection independent of JDBC driver or server version.
      protected boolean matches​(String upperCaseTypeName, String upperCaseMatch)
      Determine if the uppercase form of a column's type exactly matches or begins with the specified prefix.
      protected byte[] normalizeBinaryData​(io.debezium.relational.Column column, byte[] data)  
      org.apache.kafka.connect.data.SchemaBuilder schemaBuilder​(io.debezium.relational.Column column)  
      static java.time.Duration stringToDuration​(String timeString)  
      static java.time.LocalDate stringToLocalDate​(String dateString, io.debezium.relational.Column column, io.debezium.relational.Table table)  
      • Methods inherited from class io.debezium.jdbc.JdbcValueConverters

        convertBinary, convertBinaryToBase64, convertBinaryToBytes, convertBinaryToHex, convertBit, convertBits, convertBits, convertBoolean, convertDateToEpochDays, convertDateToEpochDaysAsDate, convertDecimal, convertDouble, convertNumeric, convertReal, convertRowId, convertString, convertTime, convertTimestampToEpochMicros, convertTimestampToEpochMillis, convertTimestampToEpochMillisAsDate, convertTimestampToEpochNanos, convertTimestampWithZone, convertTimeToMicrosPastMidnight, convertTimeToMillisPastMidnight, convertTimeToMillisPastMidnightAsDate, convertTimeToNanosPastMidnight, convertTimeWithZone, convertTinyInt, convertValue, getTimePrecision, handleUnknownData, padLittleEndian, toBigDecimal, toByteBuffer, unexpectedBinary, withScaleAdjustedIfNeeded
    • Constructor Detail

      • MySqlValueConverters

        public MySqlValueConverters​(io.debezium.jdbc.JdbcValueConverters.DecimalMode decimalMode,
                                    io.debezium.jdbc.TemporalPrecisionMode temporalPrecisionMode,
                                    io.debezium.jdbc.JdbcValueConverters.BigIntUnsignedMode bigIntUnsignedMode,
                                    io.debezium.config.CommonConnectorConfig.BinaryHandlingMode binaryMode)
        Create a new instance that always uses UTC for the default time zone when_needed converting values without timezone information to values that require timezones.

        Parameters:
        decimalMode - how DECIMAL and NUMERIC values should be treated; may be null if JdbcValueConverters.DecimalMode.PRECISE is to be used
        temporalPrecisionMode - temporal precision mode based on TemporalPrecisionMode
        bigIntUnsignedMode - how BIGINT UNSIGNED values should be treated; may be null if JdbcValueConverters.BigIntUnsignedMode.PRECISE is to be used
        binaryMode - how binary columns should be represented
      • MySqlValueConverters

        public MySqlValueConverters​(io.debezium.jdbc.JdbcValueConverters.DecimalMode decimalMode,
                                    io.debezium.jdbc.TemporalPrecisionMode temporalPrecisionMode,
                                    io.debezium.jdbc.JdbcValueConverters.BigIntUnsignedMode bigIntUnsignedMode,
                                    io.debezium.config.CommonConnectorConfig.BinaryHandlingMode binaryMode,
                                    java.time.temporal.TemporalAdjuster adjuster,
                                    MySqlValueConverters.ParsingErrorHandler parsingErrorHandler)
        Create a new instance that always uses UTC for the default time zone when converting values without timezone information to values that require timezones.

        Parameters:
        decimalMode - how DECIMAL and NUMERIC values should be treated; may be null if JdbcValueConverters.DecimalMode.PRECISE is to be used
        temporalPrecisionMode - temporal precision mode based on TemporalPrecisionMode
        bigIntUnsignedMode - how BIGINT UNSIGNED values should be treated; may be null if JdbcValueConverters.BigIntUnsignedMode.PRECISE is to be used
        binaryMode - how binary columns should be represented
        adjuster - a temporal adjuster to make a database specific time modification before conversion
        handler - for errors during postponed binlog parsing
    • Method Detail

      • adjustTemporal

        public static java.time.temporal.Temporal adjustTemporal​(java.time.temporal.Temporal temporal)
        A utility method that adjusts ambiguous 2-digit year values of DATETIME, DATE, and TIMESTAMP types using these MySQL-specific rules:
        • Year values in the range 00-69 are converted to 2000-2069.
        • Year values in the range 70-99 are converted to 1970-1999.
        Parameters:
        temporal - the temporal instance to adjust; may not be null
        Returns:
        the possibly adjusted temporal instance; never null
      • byteOrderOfBitType

        protected ByteOrder byteOrderOfBitType()
        Overrides:
        byteOrderOfBitType in class io.debezium.jdbc.JdbcValueConverters
      • schemaBuilder

        public org.apache.kafka.connect.data.SchemaBuilder schemaBuilder​(io.debezium.relational.Column column)
        Specified by:
        schemaBuilder in interface io.debezium.relational.ValueConverterProvider
        Overrides:
        schemaBuilder in class io.debezium.jdbc.JdbcValueConverters
      • converter

        public io.debezium.relational.ValueConverter converter​(io.debezium.relational.Column column,
                                                               org.apache.kafka.connect.data.Field fieldDefn)
        Specified by:
        converter in interface io.debezium.relational.ValueConverterProvider
        Overrides:
        converter in class io.debezium.jdbc.JdbcValueConverters
      • charsetFor

        protected Charset charsetFor​(io.debezium.relational.Column column)
        Return the Charset instance with the MySQL-specific character set name used by the given column.
        Parameters:
        column - the column in which the character set is used; never null
        Returns:
        the Java Charset, or null if there is no mapping
      • convertJson

        protected Object convertJson​(io.debezium.relational.Column column,
                                     org.apache.kafka.connect.data.Field fieldDefn,
                                     Object data)
        Convert the String byte[] value to a string value used in a SourceRecord.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertSmallInt

        protected Object convertSmallInt​(io.debezium.relational.Column column,
                                         org.apache.kafka.connect.data.Field fieldDefn,
                                         Object data)
        Overrides:
        convertSmallInt in class io.debezium.jdbc.JdbcValueConverters
      • convertInteger

        protected Object convertInteger​(io.debezium.relational.Column column,
                                        org.apache.kafka.connect.data.Field fieldDefn,
                                        Object data)
        Overrides:
        convertInteger in class io.debezium.jdbc.JdbcValueConverters
      • convertBigInt

        protected Object convertBigInt​(io.debezium.relational.Column column,
                                       org.apache.kafka.connect.data.Field fieldDefn,
                                       Object data)
        Overrides:
        convertBigInt in class io.debezium.jdbc.JdbcValueConverters
      • convertFloat

        protected Object convertFloat​(io.debezium.relational.Column column,
                                      org.apache.kafka.connect.data.Field fieldDefn,
                                      Object data)
        MySql reports FLOAT(p) values as FLOAT and DOUBLE. A precision from 0 to 23 results in a 4-byte single-precision FLOAT column. A precision from 24 to 53 results in an 8-byte double-precision DOUBLE column. As of MySQL 8.0.17, the nonstandard FLOAT(M,D) and DOUBLE(M,D) syntax is deprecated, and should expect support for it be removed in a future version of MySQL. Based on this future, we didn't handle the case.
        Overrides:
        convertFloat in class io.debezium.jdbc.JdbcValueConverters
      • convertString

        protected Object convertString​(io.debezium.relational.Column column,
                                       org.apache.kafka.connect.data.Field fieldDefn,
                                       Charset columnCharset,
                                       Object data)
        Convert the String or byte[] value to a string value used in a SourceRecord.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        columnCharset - the Java character set in which column byte[] values are encoded; may not be null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertYearToInt

        protected Object convertYearToInt​(io.debezium.relational.Column column,
                                          org.apache.kafka.connect.data.Field fieldDefn,
                                          Object data)
        Converts a value object for a MySQL YEAR, which appear in the binlog as an integer though returns from the MySQL JDBC driver as either a short or a Date.
        Parameters:
        column - the column definition describing the data value; never null
        fieldDefn - the field definition; never null
        data - the data object to be converted into a year literal integer value; never null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertEnumToString

        protected Object convertEnumToString​(List<String> options,
                                             io.debezium.relational.Column column,
                                             org.apache.kafka.connect.data.Field fieldDefn,
                                             Object data)
        Converts a value object for a MySQL ENUM, which is represented in the binlog events as an integer value containing the index of the enum option. The MySQL JDBC driver returns a string containing the option, so this method calculates the same.
        Parameters:
        options - the characters that appear in the same order as defined in the column; may not be null
        column - the column definition describing the data value; never null
        fieldDefn - the field definition; never null
        data - the data object to be converted into an ENUM literal String value
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertSetToString

        protected Object convertSetToString​(List<String> options,
                                            io.debezium.relational.Column column,
                                            org.apache.kafka.connect.data.Field fieldDefn,
                                            Object data)
        Converts a value object for a MySQL SET, which is represented in the binlog events contain a long number in which every bit corresponds to a different option. The MySQL JDBC driver returns a string containing the comma-separated options, so this method calculates the same.
        Parameters:
        options - the characters that appear in the same order as defined in the column; may not be null
        column - the column definition describing the data value; never null
        fieldDefn - the field definition; never null
        data - the data object to be converted into an SET literal String value; never null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • matches

        protected boolean matches​(String upperCaseTypeName,
                                  String upperCaseMatch)
        Determine if the uppercase form of a column's type exactly matches or begins with the specified prefix. Note that this logic works when the column's type contains the type name followed by parentheses.
        Parameters:
        upperCaseTypeName - the upper case form of the column's type name
        upperCaseMatch - the upper case form of the expected type or prefix of the type; may not be null
        Returns:
        true if the type matches the specified type, or false otherwise
      • isGeometryCollection

        protected boolean isGeometryCollection​(String upperCaseTypeName)
        Determine if the uppercase form of a column's type is geometry collection independent of JDBC driver or server version.
        Parameters:
        upperCaseTypeName - the upper case form of the column's type name
        Returns:
        true if the type is geometry collection
      • extractEnumAndSetOptions

        protected List<String> extractEnumAndSetOptions​(io.debezium.relational.Column column)
      • extractEnumAndSetOptionsAsString

        protected String extractEnumAndSetOptionsAsString​(io.debezium.relational.Column column)
      • convertSetValue

        protected String convertSetValue​(io.debezium.relational.Column column,
                                         long indexes,
                                         List<String> options)
      • convertPoint

        protected Object convertPoint​(io.debezium.relational.Column column,
                                      org.apache.kafka.connect.data.Field fieldDefn,
                                      Object data)
        Convert the a value representing a POINT byte[] value to a Point value used in a SourceRecord.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertGeometry

        protected Object convertGeometry​(io.debezium.relational.Column column,
                                         org.apache.kafka.connect.data.Field fieldDefn,
                                         Object data)
        Convert the a value representing a GEOMETRY byte[] value to a Geometry value used in a SourceRecord.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • normalizeBinaryData

        protected byte[] normalizeBinaryData​(io.debezium.relational.Column column,
                                             byte[] data)
        Overrides:
        normalizeBinaryData in class io.debezium.jdbc.JdbcValueConverters
      • convertUnsignedTinyint

        protected Object convertUnsignedTinyint​(io.debezium.relational.Column column,
                                                org.apache.kafka.connect.data.Field fieldDefn,
                                                Object data)
        Convert the a value representing a Unsigned TINYINT value to the correct Unsigned TINYINT representation.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertUnsignedSmallint

        protected Object convertUnsignedSmallint​(io.debezium.relational.Column column,
                                                 org.apache.kafka.connect.data.Field fieldDefn,
                                                 Object data)
        Convert the a value representing a Unsigned SMALLINT value to the correct Unsigned SMALLINT representation.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertUnsignedMediumint

        protected Object convertUnsignedMediumint​(io.debezium.relational.Column column,
                                                  org.apache.kafka.connect.data.Field fieldDefn,
                                                  Object data)
        Convert the a value representing a Unsigned MEDIUMINT value to the correct Unsigned SMALLINT representation.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertUnsignedInt

        protected Object convertUnsignedInt​(io.debezium.relational.Column column,
                                            org.apache.kafka.connect.data.Field fieldDefn,
                                            Object data)
        Convert the a value representing a Unsigned INT value to the correct Unsigned INT representation.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertUnsignedBigint

        protected Object convertUnsignedBigint​(io.debezium.relational.Column column,
                                               org.apache.kafka.connect.data.Field fieldDefn,
                                               Object data)
        Convert the a value representing a Unsigned BIGINT value to the correct Unsigned INT representation.
        Parameters:
        column - the column in which the value appears
        fieldDefn - the field definition for the SourceRecord's Schema; never null
        data - the data; may be null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertDurationToMicroseconds

        protected Object convertDurationToMicroseconds​(io.debezium.relational.Column column,
                                                       org.apache.kafka.connect.data.Field fieldDefn,
                                                       Object data)
        Converts a value object for an expected type of Duration to Long values that represents the time in microseconds.

        Per the JDBC specification, databases should return Time instances, but that's not working because it can only handle Daytime 00:00:00-23:59:59. We use Duration instead that can handle the range of -838:59:59.000000 to 838:59:59.000000 of a MySQL TIME type and transfer data as signed INT64 which reflects the DB value converted to microseconds.

        Parameters:
        column - the column definition describing the data value; never null
        fieldDefn - the field definition; never null
        data - the data object to be converted into a Duration type; never null
        Returns:
        the converted value, or null if the conversion could not be made and the column allows nulls
        Throws:
        IllegalArgumentException - if the value could not be converted but the column does not allow nulls
      • convertTimestampToLocalDateTime

        protected Object convertTimestampToLocalDateTime​(io.debezium.relational.Column column,
                                                         org.apache.kafka.connect.data.Field fieldDefn,
                                                         Object data)
      • stringToDuration

        public static java.time.Duration stringToDuration​(String timeString)
      • stringToLocalDate

        public static java.time.LocalDate stringToLocalDate​(String dateString,
                                                            io.debezium.relational.Column column,
                                                            io.debezium.relational.Table table)
      • containsZeroValuesInDatePart

        public static boolean containsZeroValuesInDatePart​(String timestampString,
                                                           io.debezium.relational.Column column,
                                                           io.debezium.relational.Table table)
      • defaultParsingErrorHandler

        public static void defaultParsingErrorHandler​(String message,
                                                      Exception exception)