Paimon Pipeline Connector

The Paimon Pipeline connector can be used as the Data Sink of the pipeline, and write data to Paimon. This document describes how to set up the Paimon Pipeline connector.

What can the connector do?

• Create table automatically if not exist
• Schema change synchronization
• Data synchronization

How to create Pipeline

The pipeline for reading data from MySQL and sink to Paimon can be defined as follows:

source:
  type: mysql
  name: MySQL Source
  hostname: 127.0.0.1
  port: 3306
  username: admin
  password: pass
  tables: adb.\.*, bdb.user_table_[0-9]+, [app|web].order_\.*
  server-id: 5401-5404

sink:
  type: paimon
  name: Paimon Sink
  catalog.properties.metastore: filesystem
  catalog.properties.warehouse: /path/warehouse

pipeline:
  name: MySQL to Paimon Pipeline
  parallelism: 2

Pipeline Connector Options

Option	Required	Default	Type	Description
type	required	(none)	String	Specify what connector to use, here should be 'paimon'.
name	optional	(none)	String	The name of the sink.
catalog.properties.metastore	required	(none)	String	Metastore of paimon catalog, supports filesystem and hive.
catalog.properties.warehouse	optional	(none)	String	The warehouse root path of catalog.
catalog.properties.uri	optional	(none)	String	Uri of metastore server.
commit.user-prefix	optional	"admin"	String	User name for committing data files.
partition.key	optional	(none)	String	Partition keys for each partitioned table, allow setting multiple primary keys for multiTables. Each table are separated by ';', and each partition key are separated by ','. For example, we can set partition.key of two tables by 'testdb.table1:id1,id2;testdb.table2:name'.
catalog.properties.*	optional	(none)	String	Pass options of Paimon catalog to pipeline. See Paimon catalog options.
table.properties.*	optional	(none)	String	Pass options of Paimon table to pipeline. See Paimon table options.

Usage Notes

• Only support Paimon primary key table, so the source table must have primary keys.

• Not support exactly-once. The connector uses at-least-once + primary key table for idempotent writing.

Data Type Mapping

CDC type	Paimon type	NOTE
TINYINT	TINYINT
SMALLINT	SMALLINT
INT	INT
BIGINT	BIGINT
FLOAT	FLOAT
DOUBLE	DOUBLE
DECIMAL(p, s)	DECIMAL(p, s)
BOOLEAN	BOOLEAN
DATE	DATE
TIMESTAMP	TIMESTAMP
TIMESTAMP_LTZ	TIMESTAMP_LTZ
CHAR(n)	CHAR(n)
VARCHAR(n)	VARCHAR(n)
VARIANT	VARIANT