Package org.apache.flink.cdc.connectors.hudi.sink.function

Class MultiTableEventStreamWriteFunction

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.ProcessFunction<I,O>

org.apache.hudi.adapter.ProcessFunctionAdapter<I,org.apache.flink.table.data.RowData>

org.apache.hudi.sink.common.AbstractWriteFunction<I>

org.apache.hudi.sink.common.AbstractStreamWriteFunction<Event>

org.apache.flink.cdc.connectors.hudi.sink.function.MultiTableEventStreamWriteFunction

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, EventProcessorFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.operators.BoundedOneInput

public class MultiTableEventStreamWriteFunction
extends org.apache.hudi.sink.common.AbstractStreamWriteFunction<Event>
implements EventProcessorFunction

Multi-table wrapper function that routes events to table-specific EventExtendedBucketStreamWriteFunction instances. This approach maintains table isolation by creating dedicated function instances per table while keeping the core write functions single-table focused.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

org.apache.flink.streaming.api.functions.ProcessFunction.Context, org.apache.flink.streaming.api.functions.ProcessFunction.OnTimerContext

Field Summary

Fields inherited from class org.apache.hudi.sink.common.AbstractStreamWriteFunction

checkpointId, config, correspondent, currentInstant, eventGateway, metaClient, taskID, writeClient, writeStatuses

Constructor Summary

Constructors

Constructor	Description
MultiTableEventStreamWriteFunction(org.apache.flink.configuration.Configuration config)

Method Summary

Modifier and Type	Method	Description
void	close()
void	endInput()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	open(org.apache.flink.api.common.functions.OpenContext openContext)
void	processDataChange(DataChangeEvent event, org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx, org.apache.flink.util.Collector<org.apache.flink.table.data.RowData> out)	Processes change events with context and collector for writing.
void	processElement(Event event, org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx, org.apache.flink.util.Collector<org.apache.flink.table.data.RowData> out)
void	processFlush(FlushEvent event)	Processes flush events for coordinated flushing across table functions.
void	processSchemaChange(SchemaChangeEvent event)	Processes schema events.
void	setSchemaEvolutionClient(SchemaEvolutionClient schemaEvolutionClient)	Sets the SchemaEvolutionClient from the operator level since functions don't have direct access to TaskOperatorEventGateway.
void	snapshotState()
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.hudi.sink.common.AbstractStreamWriteFunction

getCorrespondent, getOperatorEventGateway, handleOperatorEvent, instantToWrite, setCorrespondent, setOperatorEventGateway

Methods inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

onTimer

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiTableEventStreamWriteFunction

public MultiTableEventStreamWriteFunction(org.apache.flink.configuration.Configuration config)

Method Detail

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Overrides:

initializeState in class org.apache.hudi.sink.common.AbstractStreamWriteFunction<Event>

Throws:

Exception

setSchemaEvolutionClient

public void setSchemaEvolutionClient(SchemaEvolutionClient schemaEvolutionClient)

Sets the SchemaEvolutionClient from the operator level since functions don't have direct access to TaskOperatorEventGateway.

open

public void open(org.apache.flink.api.common.functions.OpenContext openContext)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Throws:

Exception

processElement

public void processElement(Event event,
                           org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx,
                           org.apache.flink.util.Collector<org.apache.flink.table.data.RowData> out)
                    throws Exception

Specified by:

processElement in class org.apache.flink.streaming.api.functions.ProcessFunction<Event,org.apache.flink.table.data.RowData>

Throws:

Exception

processSchemaChange

public void processSchemaChange(SchemaChangeEvent event)
                         throws Exception

Processes schema events. For a CreateTableEvent, it ensures that the coordinator is notified and the physical Hudi table is created. For a SchemaChangeEvent, it updates the local schema cache.

Implements EventProcessorFunction.processSchemaChange(SchemaChangeEvent).

Specified by:

processSchemaChange in interface EventProcessorFunction

Parameters:

event - The schema change event

Throws:

Exception - if processing fails

processDataChange

public void processDataChange(DataChangeEvent event,
                              org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx,
                              org.apache.flink.util.Collector<org.apache.flink.table.data.RowData> out)

Processes change events with context and collector for writing. This triggers the actual Hudi write operations as side effects by delegating to table-specific functions.

Specified by:

processDataChange in interface EventProcessorFunction

Parameters:

event - The data change event containing the operation type and data

ctx - The process function context for accessing runtime information

out - The collector for emitting processed RowData records

processFlush

public void processFlush(FlushEvent event)
                  throws Exception

Processes flush events for coordinated flushing across table functions. This handles both table-specific and global flush operations.

Implements EventProcessorFunction.processFlush(FlushEvent).

Specified by:

processFlush in interface EventProcessorFunction

Parameters:

event - The flush event

Throws:

Exception - if processing fails

snapshotState

public void snapshotState()

Specified by:

snapshotState in class org.apache.hudi.sink.common.AbstractStreamWriteFunction<Event>

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Overrides:

snapshotState in class org.apache.hudi.sink.common.AbstractStreamWriteFunction<Event>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

endInput

public void endInput()

Specified by:

endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput

Overrides:

endInput in class org.apache.hudi.sink.common.AbstractStreamWriteFunction<Event>