Apache NIFI离线同步MySQL数据

Flow概览

表结构

源表与目标表结构(除了表名都相同)

CREATE TABLE user_model( idint unsigned NOT NULL AUTO_INCREMENT, id_numbervarchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '身份证号', namevarchar(20) NOT NULL, phone_numberchar(11) DEFAULT NULL, addressvarchar(256) DEFAULT NULL, ageint DEFAULT NULL COMMENT '年龄', genderint DEFAULT NULL COMMENT '性别', passwordvarchar(256) DEFAULT NULL, create_timedatetime DEFAULT NULL, update_timedatetime DEFAULT NULL, other_infojson DEFAULT NULL, basic_infosjson DEFAULT NULL, deleted tinyint NOT NULL DEFAULT '0', PRIMARY KEY (id), UNIQUE KEY uni_id_number (id_number) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

CREATE TABLE user_model_target( idint unsigned NOT NULL AUTO_INCREMENT, id_numbervarchar(20) DEFAULT NULL, namevarchar(20) NOT NULL, phone_numberchar(11) DEFAULT NULL, addressvarchar(256) DEFAULT NULL, ageint DEFAULT NULL COMMENT '年龄', genderint DEFAULT NULL COMMENT '性别', passwordvarchar(256) DEFAULT NULL, create_timedatetime DEFAULT NULL, update_timedatetime DEFAULT NULL, other_infojson DEFAULT NULL, basic_infosjson DEFAULT NULL, deleted tinyint NOT NULL DEFAULT '0', PRIMARY KEY (id), UNIQUE KEY uni_id_number (id_number) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

前提条件

• 配置了源和目标的MySQL数据源
• 源表与目标表结构一致

细节说明

GenerateFlowFile 作为流程的触发点,在这里我们设置为每5S触发一次同步.

ExecuteSQL 负责查询源表数据.

PutDatabaseRecord 负责将数据插入或新增到目标表中.