Examples and Best Practices

This section focuses on practical data ingestion scenarios with Flink CDC YAML pipelines:

• Synchronizing databases into data lakes and message queues.
• Selecting tables, transforming records, and routing tables.
• Enriching data with AI functions during ingestion.
• Tuning and troubleshooting data ingestion pipelines.

Examples

• Real-time MySQL Ingestion into Paimon: merge order tables and synchronize schema and data changes.
• Real-time MySQL Change Distribution to Kafka: publish table changes to separate topics and partition them by primary key.
• Real-time PostgreSQL Ingestion into Iceberg: configure logical replication and synchronize snapshots, row changes, and an added column.
• Table Selection and Adding Tables: exclude temporary tables, capture new tables, and include historical tables through savepoint restore.
• Metadata Columns, Computed Columns, and Soft Deletes: enrich records, compute date partitions, and preserve deletion markers.
• AI Function Configuration and Real-time Text Labeling: configure a model and label, summarize, and mask feedback during ingestion.

For pipeline syntax and supported connectors, see Data Pipeline and Pipeline Connectors. For model configuration and available AI functions, see AI Model.