Data syncing in golang for ClickHouse.
based on go-zero
Data workflow of a typical data warehouse architecture
Automatically synchronizing data from MySQL/MongoDB data source to ClickHouse cluster in real time(< 2min).
git clone https://github.com/zeromicro/cds.git cd cds make upAfter the build , pay attention to check if any container exits abnormally.
click http://localhost:3414/cds.html to accessing the web interface.
using username and password below to login
user: [email protected] password: 123456 Chose "create table" tab
1. Click "Target ClickHouse Database Info", 2. Click “connect” 3. Select the schema ("default") synchronized to Clickhouse 4. switch to "Data Source" 5. MySQL input connection string root:root@tcp(mysql:3306)/test_mysql or MongoDB input connection string mongodb://mongo1:30001/test_mongo 6. Click “connect” 7. Select the table, such as the default example_mysql (MySQL) or example (MongoDB) 8. click "Generate create Table SQL" Note: select the partition field as needed, here'PARTITION BY toYYYYMM()' can be deleted, or replaced with'PARTITION BY toYYYYMM(dt)' 9. . Click "send SQL to ClickHouse", and the successful execution will pop up below Chose "full sync" tab
1. Click "+" in the upper right corner 2. MySQL input connection string root:root@tcp(mysql:3306)/test_mysql or MongoDB input connection string mongodb://mongo1:30001/test_mongo 3. Click “connect” 4. Select the table, such as the default example_mysql (MySQL) or example (MongoDB) 5. Click "Target ClickHouse Database Info" 6. Click “connect” 7. Select the schema ("default") synchronized in Clickhouse 10. Click "Add", a pop-up below shows successful execution Refresh the page
Take mysql as an example
chose "Connector" tab
1. Click "+" in the upper right corner 2. MySQL input connection string root:root@tcp(mysql:3306)/test_mysql 3. select table 4. Click "Add" chose "Incremental Sync" tab
1. Click "+" in the upper right corner 2. MySQL input connection string root:root@tcp(mysql:3306)/test_mysql 3. select table 4. Click "Target ClickHouse Database Info" 5. Click “connect” 6. Select the schema ("default") synchronized in Clickhouse 7. Click "Add", a pop-up below shows successful execution refresh page
Execute the initialization database script again, you can insert 100000 rows of data again.
cd sit/docker/ sh ./init.sh Verify the incremental data of mysql in clickhouse：
To clean up all the docker containers started above and restore the initial state, you can ：
cd cds make down only clean
cd cds make docker_clean if you like this project and want to support it，please star 🤝