Package org.apache.flink.cdc.common.data.binary

Class BinaryArrayData

  • All Implemented Interfaces:
    ArrayData, BinaryFormat
    public final class BinaryArrayData
extends BinarySection
implements ArrayData
    A binary implementation of ArrayData which is backed by MemorySegments.

    This class provides a way to store array data in a binary format that is compact and efficient. It uses MemorySegments to manage the binary representation of the data, allowing for efficient storage and access.

    The binary layout of BinaryArrayData is structured as follows: 

     [size(int)] + [null bits(4-byte word boundaries)] + [values or offset&length] + [variable length part].
    • size: The first 4 bytes store the number of elements in the array.
    • null bits: A bitmap to track null values, aligned to 4-byte word boundaries. Each bit represents whether an element is null.
    • values or offset&length: The values of the array elements. For fixed-length primitive types, the values are stored directly. For variable-length types (e.g., strings, maps), this part stores the offset and length of the actual data in the variable length part.
    • variable length part: This part of the memory segment stores the actual data for variable-length types (e.g., strings, maps).

    The header size is calculated based on the number of elements in the array, ensuring efficient alignment and access.

    For fields that hold fixed-length primitive types, such as long, double, or int, they are stored compactly in bytes, just like the original Java array.

    The class also provides methods to convert the binary data back into Java primitive arrays, handling various types such as boolean, byte, short, int, long, float, and double.

    • Constructor Detail

      • BinaryArrayData

        public BinaryArrayData()

    • Method Detail

      • calculateHeaderInBytes

        public static int calculateHeaderInBytes​(int numFields)
        Calculates the size of the header in bytes for an array with the specified number of fields.

        The header consists of:

        • 4 bytes to store the size of the array (number of elements).
        • A bitmap to track null values, where each bit represents whether an element is null. This bitmap is aligned to 4-byte word boundaries for efficient memory access and to facilitate the use of bitwise operations.

        The size of the bitmap is determined by the number of elements in the array:

        • Each element requires 1 bit in the bitmap.
        • The total number of bits is rounded up to the nearest multiple of 32 to ensure alignment to 4-byte word boundaries (i.e., a 32-bit integer).

        The formula for calculating the size of the header is: 

           header size = 4 bytes (for array size) + ((numFields + 31) / 32) * 4 bytes (for null bitmap)
        Parameters:
        numFields - the number of elements in the array
        Returns:
        the size of the header in bytes

      • calculateFixLengthPartSize

        public static int calculateFixLengthPartSize​(DataType type)
        It store real value when type is primitive. It store the length and offset of variable-length part when type is string, map, etc.

      • size

        public int size()
        Description copied from interface: ArrayData
        Returns the number of elements in this array.
        Specified by:
        size in interface ArrayData

      • pointTo

        public void pointTo​(org.apache.flink.core.memory.MemorySegment[] segments,
                    int offset,
                    int sizeInBytes)
        Overrides:
        pointTo in class BinarySection

      • isNullAt

        public boolean isNullAt​(int pos)
        Description copied from interface: ArrayData
        Returns true if the element is null at the given position.
        Specified by:
        isNullAt in interface ArrayData

      • setNullAt

        public void setNullAt​(int pos)

      • setNotNullAt

        public void setNotNullAt​(int pos)

      • getLong

        public long getLong​(int pos)
        Description copied from interface: ArrayData
        Returns the long value at the given position.
        Specified by:
        getLong in interface ArrayData

      • setLong

        public void setLong​(int pos,
                    long value)

      • setNullLong

        public void setNullLong​(int pos)

      • getInt

        public int getInt​(int pos)
        Description copied from interface: ArrayData
        Returns the integer value at the given position.
        Specified by:
        getInt in interface ArrayData

      • setInt

        public void setInt​(int pos,
                   int value)

      • setNullInt

        public void setNullInt​(int pos)

      • getString

        public StringData getString​(int pos)
        Description copied from interface: ArrayData
        Returns the string value at the given position.
        Specified by:
        getString in interface ArrayData

      • getDecimal

        public DecimalData getDecimal​(int pos,
                              int precision,
                              int scale)
        Description copied from interface: ArrayData
        Returns the decimal value at the given position.

        The precision and scale are required to determine whether the decimal value was stored in a compact representation (see DecimalData).

        Specified by:
        getDecimal in interface ArrayData

      • getTimestamp

        public TimestampData getTimestamp​(int pos,
                                  int precision)
        Description copied from interface: ArrayData
        Returns the timestamp value at the given position.

        The precision is required to determine whether the timestamp value was stored in a compact representation (see TimestampData).

        Specified by:
        getTimestamp in interface ArrayData

      • getZonedTimestamp

        public ZonedTimestampData getZonedTimestamp​(int pos,
                                            int precision)
        Description copied from interface: ArrayData
        Returns the zoned timestamp value at the given position.

        The precision is required to determine whether the timestamp value was stored in a compact representation (see ZonedTimestampData).

        Specified by:
        getZonedTimestamp in interface ArrayData

      • getBinary

        public byte[] getBinary​(int pos)
        Description copied from interface: ArrayData
        Returns the binary value at the given position.
        Specified by:
        getBinary in interface ArrayData

      • getArray

        public ArrayData getArray​(int pos)
        Description copied from interface: ArrayData
        Returns the array value at the given position.
        Specified by:
        getArray in interface ArrayData

      • getMap

        public MapData getMap​(int pos)
        Description copied from interface: ArrayData
        Returns the map value at the given position.
        Specified by:
        getMap in interface ArrayData

      • getRecord

        public RecordData getRecord​(int pos,
                            int numFields)
        Description copied from interface: ArrayData
        Returns the record value at the given position.

        The number of fields is required to correctly extract the row.

        Specified by:
        getRecord in interface ArrayData

      • getBoolean

        public boolean getBoolean​(int pos)
        Description copied from interface: ArrayData
        Returns the boolean value at the given position.
        Specified by:
        getBoolean in interface ArrayData

      • setBoolean

        public void setBoolean​(int pos,
                       boolean value)

      • setNullBoolean

        public void setNullBoolean​(int pos)

      • getByte

        public byte getByte​(int pos)
        Description copied from interface: ArrayData
        Returns the byte value at the given position.
        Specified by:
        getByte in interface ArrayData

      • setByte

        public void setByte​(int pos,
                    byte value)

      • setNullByte

        public void setNullByte​(int pos)

      • getShort

        public short getShort​(int pos)
        Description copied from interface: ArrayData
        Returns the short value at the given position.
        Specified by:
        getShort in interface ArrayData

      • setShort

        public void setShort​(int pos,
                     short value)

      • setNullShort

        public void setNullShort​(int pos)

      • getFloat

        public float getFloat​(int pos)
        Description copied from interface: ArrayData
        Returns the float value at the given position.
        Specified by:
        getFloat in interface ArrayData

      • setFloat

        public void setFloat​(int pos,
                     float value)

      • setNullFloat

        public void setNullFloat​(int pos)

      • getDouble

        public double getDouble​(int pos)
        Description copied from interface: ArrayData
        Returns the double value at the given position.
        Specified by:
        getDouble in interface ArrayData

      • setDouble

        public void setDouble​(int pos,
                      double value)

      • setNullDouble

        public void setNullDouble​(int pos)

      • setDecimal

        public void setDecimal​(int pos,
                       DecimalData value,
                       int precision)

      • setTimestamp

        public void setTimestamp​(int pos,
                         TimestampData value,
                         int precision)

      • anyNull

        public boolean anyNull()

      • toObjectArray

        public <T> T[] toObjectArray​(DataType elementType)

      • fromPrimitiveArray

        public static BinaryArrayData fromPrimitiveArray​(boolean[] arr)

      • fromPrimitiveArray

        public static BinaryArrayData fromPrimitiveArray​(byte[] arr)

      • fromPrimitiveArray

        public static BinaryArrayData fromPrimitiveArray​(short[] arr)

      • fromPrimitiveArray

        public static BinaryArrayData fromPrimitiveArray​(int[] arr)

      • fromPrimitiveArray

        public static BinaryArrayData fromPrimitiveArray​(long[] arr)

      • fromPrimitiveArray

        public static BinaryArrayData fromPrimitiveArray​(float[] arr)

      • fromPrimitiveArray

        public static BinaryArrayData fromPrimitiveArray​(double[] arr)