Package org.apache.flink.cdc.runtime.operators.schema.common

Class SchemaRegistry

  • All Implemented Interfaces:
    AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.operators.coordination.CoordinationRequestHandler, org.apache.flink.runtime.operators.coordination.OperatorCoordinator
    Direct Known Subclasses:
    SchemaCoordinator, SchemaCoordinator
    @Internal
public abstract class SchemaRegistry
extends Object
implements org.apache.flink.runtime.operators.coordination.OperatorCoordinator, org.apache.flink.runtime.operators.coordination.CoordinationRequestHandler
    An abstract centralized schema registry that accepts requests from SchemaEvolutionClient. A legit schema registry should be able to:
    • Handle schema retrieval requests from SchemaEvolutionClients
    • Accept and trace sink writers' registering events
    • Snapshot & Restore its state during checkpoints

    These abilities are done by overriding abstract methods of SchemaRegistry. All methods will run in given ExecutorService asynchronously except SchemaRegistry#restore.

    • Field Detail

      • context

        protected final org.apache.flink.runtime.operators.coordination.OperatorCoordinator.Context context

      • operatorName

        protected final String operatorName

      • rpcTimeout

        protected final java.time.Duration rpcTimeout

      • routeMode

        protected final RouteMode routeMode

      • currentParallelism

        protected transient int currentParallelism

      • activeSinkWriters

        protected transient Set<Integer> activeSinkWriters

    • Method Detail

      • start

        public void start()
           throws Exception
        Specified by:
        start in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator
        Throws:
        Exception

      • expandExistingTableSchemaIfNeeded

        protected boolean expandExistingTableSchemaIfNeeded​(SchemaChangeEvent schemaChangeEvent)
        Handles the initial CreateTableEvent for an existing target table.
        Returns:
        whether the caller should proceed to apply the original CreateTableEvent to the sink; CHECK mode returns false after a successful check.

      • close

        public void close()
           throws Exception
        Specified by:
        close in interface AutoCloseable
        Specified by:
        close in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator
        Throws:
        Exception

      • snapshot

        protected abstract void snapshot​(CompletableFuture<byte[]> resultFuture)
                          throws Exception
        Snapshot current schema registry state in byte array form.
        Throws:
        Exception

      • restore

        protected abstract void restore​(byte[] checkpointData)
                         throws Exception
        Restore schema registry state from byte array.
        Throws:
        Exception

      • shutdown

        protected abstract void shutdown()
                          throws Exception
        Stops schema-change work from the previous coordinator generation. Implementations must block until the previous generation's worker has fully exited, within rpcTimeout.
        Throws:
        Exception

      • initialize

        protected abstract void initialize()
        (Re)initializes transient state, both on start() and after a coordinator reset.

      • handleCustomCoordinationRequest

        protected abstract void handleCustomCoordinationRequest​(org.apache.flink.runtime.operators.coordination.CoordinationRequest request,
                                                        CompletableFuture<org.apache.flink.runtime.operators.coordination.CoordinationResponse> responseFuture)
                                                 throws Exception
        Coordination handler for customized CoordinationRequests.
        Throws:
        Exception

      • handleUnrecoverableError

        protected void handleUnrecoverableError​(String taskDescription,
                                        Throwable t)
        Last chance to execute codes before job fails globally.

      • handleCoordinationRequest

        public final CompletableFuture<org.apache.flink.runtime.operators.coordination.CoordinationResponse> handleCoordinationRequest​(org.apache.flink.runtime.operators.coordination.CoordinationRequest request)
        Specified by:
        handleCoordinationRequest in interface org.apache.flink.runtime.operators.coordination.CoordinationRequestHandler

      • handleEventFromOperator

        public final void handleEventFromOperator​(int subTaskId,
                                          int attemptNumber,
                                          org.apache.flink.runtime.operators.coordination.OperatorEvent event)
        Specified by:
        handleEventFromOperator in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

      • subtaskReset

        public final void subtaskReset​(int subTaskId,
                               long checkpointId)
        Specified by:
        subtaskReset in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

      • executionAttemptFailed

        public void executionAttemptFailed​(int subTaskId,
                                   int attemptNumber,
                                   @Nullable
                                   Throwable reason)
        Specified by:
        executionAttemptFailed in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

      • executionAttemptReady

        public final void executionAttemptReady​(int subTaskId,
                                        int attemptNumber,
                                        org.apache.flink.runtime.operators.coordination.OperatorCoordinator.SubtaskGateway gateway)
        Specified by:
        executionAttemptReady in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

      • checkpointCoordinator

        public final void checkpointCoordinator​(long checkpointId,
                                        CompletableFuture<byte[]> completableFuture)
                                 throws Exception
        Specified by:
        checkpointCoordinator in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator
        Throws:
        Exception

      • notifyCheckpointComplete

        public final void notifyCheckpointComplete​(long checkpointId)
        Specified by:
        notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener
        Specified by:
        notifyCheckpointComplete in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

      • resetToCheckpoint

        public final void resetToCheckpoint​(long checkpointId,
                                    @Nullable
                                    byte[] checkpointData)
                             throws Exception
        Specified by:
        resetToCheckpoint in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator
        Throws:
        Exception

      • runInEventLoop

        protected void runInEventLoop​(org.apache.flink.util.function.ThrowingRunnable<Throwable> action,
                              String actionName,
                              Object... actionNameFormatParameters)
        Run a time-consuming task in given ExecutorService. All overridable functions have been wrapped inside already, so there's no need to call this method again. However, if you're overriding methods from OperatorCoordinator or CoordinationRequestHandler directly, make sure you're running heavy logics inside, or the entire job might hang!

      • loopUntil

        protected void loopUntil​(java.util.function.BooleanSupplier conditionChecker,
                         Runnable message,
                         java.time.Duration timeout,
                         java.time.Duration interval)
                  throws TimeoutException
        Keeps checking if conditionChecker is satisfied. If not, emit a message and retry.
        Throws:
        TimeoutException

      • loopWhen

        protected void loopWhen​(java.util.function.BooleanSupplier conditionChecker,
                        Runnable message,
                        java.time.Duration timeout,
                        java.time.Duration interval)
                 throws TimeoutException
        Keeps checking if conditionChecker is satisfied. Otherwise, emit a message and retry.
        Throws:
        TimeoutException

      • failJob

        protected <T extends Throwable> void failJob​(String taskDescription,
                                             T t)