实践练习四：迁移 MySQL 数据到 OceanBase 集群
- 实验说明
1.1 实验目标
1.使用 mysqldump 将 mysql的表结构和数据同步到 OceanBase 的MySQL 租户中。
2.使用 datax 配置至少一个表的 MySQL 到 OceanBase 的 MySQL 租户的离线同步。
1.2 实验环境信息
本次部署使用环境如下：
- 宿主机Oracle Linux Server release 6.9 48核，320G内存 600G硬盘
- 虚拟机软件vbox VirtualBox-6.0.6
- 虚拟主机4台 8c32g，80G硬盘，/根目录50G
- centos7.6 iso 系统镜像（CentOS-7-x86_64-DVD-1810.iso）
1.3机器角色划分
| Zone | IP | hostname | RPC Port | Connect Port |
| Zone1 | 192.168.56.41 | ob32-2 | 2882 | 2881 |
| Zone2 | 192.168.56.42 | ob32-3 | 2882 | 2881 |
| Zone3 | 192.168.56.43 | ob32-4 | 2882 | 2881 |
| obproxy | 192.168.56.40 | ob32-1 | 2884 | 2883 |
| mysql and obclient客户端 | 192.168.56.40 | ob32-1 |
2.mariadb环境安装准备
2.1配置本地yum源
因为四台LINUX服务器采用最小化安装的，又不能连接外网，需要配置本地ISO的yum源。
cd /etc/yum.repos.d
mkdir bk
mv *.repo bk/
echo "[EL]" > /etc/yum.repos.d/centos-media.repo
echo "name =Linux 7.x DVD" >> /etc/yum.repos.d/centos-media.repo
echo "baseurl=file:///mnt" >> /etc/yum.repos.d/centos-media.repo
echo "gpgcheck=1" >> /etc/yum.repos.d/centos-media.repo
echo "enabled=1" >> /etc/yum.repos.d/centos-media.repo
echo "gpgkey=file:///mnt/RPM-GPG-KEY-CentOS-7" >> /etc/yum.repos.d/centos-media.repo
echo "/dev/sr0 /mnt iso9660 defaults,ro,loop 0 0" >>/etc/fstab
cat /etc/fstab
mount -a
[root@ob32-1 soft]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos_ob32--1-root 81G 12G 69G 15% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 8.7M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop0 4.3G 4.3G 0 100% /mnt
/dev/sda1 1014M 133M 882M 14% /boot
tmpfs 3.2G 0 3.2G 0% /run/user/0
2.2安装必要的mariadb工具包
[root@ob32-1 ~]# yum list|grep mariadb
mariadb-libs.x86_64 1:5.5.60-1.el7_5 @anaconda
mariadb.x86_64 1:5.5.60-1.el7_5 EL
mariadb-bench.x86_64 1:5.5.60-1.el7_5 EL
mariadb-devel.x86_64 1:5.5.60-1.el7_5 EL
mariadb-server.x86_64 1:5.5.60-1.el7_5 EL
mariadb-test.x86_64 1:5.5.60-1.el7_5 EL
yum install -y mariadb mariadb-server mariadb-devel
[root@ob32-1 ~]# mysql -V
mysql Ver 15.1 Distrib 5.5.60-MariaDB, for Linux (x86_64) using readline 5.1
2.3 启动mariadb服务并设置开机启动
systemctl status mariadb
systemctl start mariadb
systemctl enable mariadb
systemctl status mariadb
 |
2.4 mariadb初始化
[root@ob32-1 ~]# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.
Enter current password for root (enter for none):
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.
Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] Y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] Y
... Success!
By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] Y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] Y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!'
主要进行了如下初始化操作，设置mysql root密码为rootroot,删除匿名用户，允许root账号远程登录，删除测试数据库，刷新权限等，可以根据自己的需求个性化设置。
3.tpcc-mysql的安装配置
3.1下载tpcc-mysql安装包
下载地址为：https://github.com/Percona-Lab/tpcc-mysql
下载后解压缩：unzip tpcc-mysql-master.zip
3.2编译tpcc-mysql
cd / tpcc-mysql-master/src
make
编译后会产生两个可执行文件tpcc_load和tpcc_start
编译后的目录如下：
 |
4 源端数据库配置
4.1修改默认字符集
因为maraidb默认创建的数据库字符集为latin1,而OceanBase不支持latin1字符集，后续数据导入时会报错，对象不能创建，所以需要先修改数据库字符集为utf8
编辑/etc/my.cnf
在[mysqld]下增加以下参数
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
重启matiadb数据库
systemctl restart mariadb
再次查看数据库字符集已经修改过来了。
MariaDB [(none)]> show variables like'%character%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
4.2创建数据库建表
[root@ob32-1 tpcc-mysql-master]# mysql -h 127.1 -uroot –prootroot
MySQL [(none)]>create database tpcc1000;
MySQL [(none)]>use tpcc1000;
Database changed
MySQL [tpcc1000]> source create_table.sql
MySQL [tpcc1000]> source add_fkey_idx.sql
MySQL [tpcc1000]>show databases;
 |
5 生成测试数据
[root@ob32-1 tpcc-mysql-master]# ./tpcc_load -h 127.0.0.1 -P 3306 -d tpcc1000 -u root -p rootroot -w 1
*************************************
*** TPCC-mysql Data Loader ***
*************************************
option h with value '127.0.0.1'
option P with value '3306'
option d with value 'tpcc1000'
option u with value 'root'
option p with value 'rootroot'
option w with value '1'
<Parameters>
[server]: 127.0.0.1
[port]: 3306
[DBname]: tpcc1000
[user]: root
[pass]: rootroot
[warehouse]: 1
TPCC Data Load Started...
Loading Item
.................................................. 5000
.................................................. 10000
.................................................. 15000
.................................................. 20000
.................................................. 25000
.................................................. 30000
.................................................. 35000
.................................................. 40000
.................................................. 45000
.................................................. 50000
.................................................. 55000
.................................................. 60000
.................................................. 65000
.................................................. 70000
.................................................. 75000
.................................................. 80000
.................................................. 85000
.................................................. 90000
.................................................. 95000
.................................................. 100000
Item Done.
Loading Warehouse
Loading Stock Wid=1
.................................................. 5000
.................................................. 10000
.................................................. 15000
.................................................. 20000
.................................................. 25000
.................................................. 30000
.................................................. 35000
.................................................. 40000
.................................................. 45000
.................................................. 50000
.................................................. 55000
.................................................. 60000
.................................................. 65000
.................................................. 70000
.................................................. 75000
.................................................. 80000
.................................................. 85000
.................................................. 90000
.................................................. 95000
.................................................. 100000
Stock Done.
Loading District
Loading Customer for DID=1, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=2, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=3, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=4, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=5, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=6, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=7, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=8, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=9, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Customer for DID=10, WID=1
.......... 1000
.......... 2000
.......... 3000
Customer Done.
Loading Orders for D=1, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=2, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=3, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=4, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=5, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=6, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=7, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=8, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=9, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
Loading Orders for D=10, W= 1
.......... 1000
.......... 2000
.......... 3000
Orders Done.
...DATA LOADING COMPLETED SUCCESSFULLY.
MariaDB [tpcc1000]> select count(*) from item;
+----------+
| count(*) |
+----------+
| 100000 |
+----------+
1 row in set (0.02 sec)
MariaDB [tpcc1000]> select count(*) from Warehouse;
ERROR 1146 (42S02): Table 'tpcc1000.Warehouse' doesn't exist'
MariaDB [tpcc1000]> show tables;
+--------------------+
| Tables_in_tpcc1000 |
+--------------------+
| customer |
| district |
| history |
| item |
| new_orders |
| order_line |
| orders |
| stock |
| warehouse |
+--------------------+
9 rows in set (0.00 sec)
MariaDB [tpcc1000]> select count(*) from customer;
+----------+
| count(*) |
+----------+
| 30000 |
+----------+
1 row in set (0.00 sec)
MariaDB [tpcc1000]> select count(*) from history;
+----------+
| count(*) |
+----------+
| 30000 |
+----------+
1 row in set (0.01 sec)
MariaDB [tpcc1000]> select count(*) from stock;
+----------+
| count(*) |
+----------+
| 100000 |
+----------+
1 row in set (0.01 sec)
MariaDB [tpcc1000]> select count(*) from warehouse;
+----------+
| count(*) |
+----------+
| 1 |
+----------+
1 row in set (0.00 sec)
6 mysqldump导出表结构和数据
#导出指定数据库的表结构（不包括数据）
mysqldump -uroot -prootroot -d tpcc1000 --compact > /home/admin/tpcc1000_ddl.sql
#导出指定数据库的表数据（不包括结构）
mysqldump -uroot -prootroot -t tpcc1000 > /home/admin/tpcc1000_data.sql
[admin@ob32-1 ~]$ ll
total 81940
drwxr-xr-x. 8 admin admin 97 Mar 7 15:45 obproxy-3.2.3.5
drwxrwxr-x. 2 admin admin 6 Mar 5 20:17 oceanbase-ce
-rw-r--r--. 1 root root 83896729 Mar 9 13:22 tpcc1000_data.sql
-rw-r--r--. 1 root root 7116 Mar 9 13:22 tpcc1000_ddl.sql
7 目标端建库和数据导入
7.1登录ob集群
su - admin
obclient -h 192.168.56.40 -uroot@obmytest#obdemo -P2883 -prootPWD123 -c -A oceanbase
7.2创建tpcc1000_ob数据库
create database tpcc1000_ob default character set=utf8;
obclient [oceanbase]> use tpcc1000_ob;
7.3 导入表结构和数据
obclient [tpcc1000_ob]> source tpcc1000_ddl.sql;
obclient [tpcc1000_ob]> source tpcc1000_data.sql;
7.4 验证查看数据
select count(*) from item;
+----------+
| count(*) |
+----------+
| 100000 |
+----------+
1 row in set (0.093 sec)
select count(*) from warehouse;
+----------+
| count(*) |
+----------+
| 1 |
+----------+
1 row in set (0.016 sec)
数据已经成功迁移到ob数据库中
8使用datax迁移数据到ob
8.1下载解压datax 编译好的包
下载地址：http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
直接解压即可：
tar -zxvf datax.tar.gz
删除 plugin[reader,writer] 目录下所有 ._ 开头的文件夹/文件，避免mysqlreader加载失败
find ./datax/plugin -name "._*" | xargs rm -f
8.2模板预配置
[root@ob32-1 job]# python ./datax/bin/datax.py -r mysqlreader -w oceanbasev10writer > mysql2ob.json
[admin@ob32-1 job]$ ll
total 8
-rwxr-xr-x. 1 502 games 1587 Nov 24 2017 job.json
-rw-r--r--. 1 root root 1819 Mar 9 15:18 mysql2ob.json
8.3修改mysql2ob.json文件
(注意：删除或注释掉文件头部信息，我这里选择删掉)
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"column": ["*"],
"connection": [
{
"jdbcUrl": ["jdbc:mysql://127.0.0.1:3306/tpcc1000?useUnicode=true&characterEncoding=utf8&useSSL=false"],
"table": ["orders"]
}
],
"password": "rootroot",
"username": "root",
"where": ""
}
},
"writer": {
"name": "oceanbasev10writer",
"parameter": {
"column": ["*"],
"preSql": [
"set ob_trx_timeout=1000000000"
"set foreign_key_checks='off'"
"delete from orders"
],
"connection": [
{
"jdbcUrl": "||_dsc_ob10_dsc_||obdemo:obmytest||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.56.40:2883/tpcc1000_ob?useLocalSessionState=true&useUnicode=true&characterEncoding=utf8",
"table": ["orders"]
}
],
"obWriteMode": "insert",
"password": "rootPWD123",
"username": "root"
}
}
}
],
"setting": {
"speed": {
"channel": "5"
}
}
}
}
8.4安装java环境包
[root@ob32-1 job]# yum install java-1.8.0* -y
8.5清空目标端表数据库
obclient [tpcc1000_ob]> set foreign_key_checks='off';
Query OK, 0 rows affected (0.002 sec)
obclient [tpcc1000_ob]> truncate table orders;
ERROR 1701 (42000): Cannot truncate a table referenced in a foreign key constraint (`tpcc1000_ob`.`order_line`, CONSTRAINT `fkey_order_line_1` FOREIGN KEY (`ol_w_id`, `ol_d_id`, `ol_o_id`) REFERENCES `tpcc1000_ob`.`orders` (`o_w_id`, `o_d_id`, `o_id`))
obclient [tpcc1000_ob]> delete from orders;
Query OK, 30000 rows affected (0.594 sec)
obclient [tpcc1000_ob]> select count(*) from orders;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.031 sec)
8.6启动任务
[root@db01 ~]# python ./datax/bin/datax.py mysql2ob.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-03-09 15:19:34.758 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-03-09 15:19:34.767 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-957.el7.x86_64
cpu num: 8
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
***********************************************************************
8.7数据验证过程
目标端数据同步中：
obclient [tpcc1000_ob]> select count(*) from orders;
+----------+
| count(*) |
+----------+
| 28672 |
+----------+
1 row in set (0.041 sec)
数据同步完成：
obclient [tpcc1000_ob]> select count(*) from orders;
+----------+
| count(*) |
+----------+
| 30000 |
+----------+
1 row in set (0.040 sec)
datax同步总结信息：
任务启动时刻 : 2023-03-09 15:19:34
任务结束时刻 : 2023-03-09 15:21:05
任务总计耗时 : 90s
任务平均流量 : 6.73KB/s
记录写入速度 : 333rec/s
读出记录总数 : 30000
读写失败总数 : 0
9 配置过程中遇到的坑
9.1 tpcc-mysql编译失败
1、tpcc-mysql编译时报错，：
cc -w -O3 -g -I. `mysql_config --include` -c load.c
/bin/sh: mysql_config: command not found
load.c:19:19: fatal error: mysql.h: No such file or directory
#include <mysql.h>
原因是没有安装mariadb-devel 开发包，导致/usr/bin下没有 mysql_config命令。
需要安装 yum install mariadb-devel -y
安装后查看mysql_config命令已经存在了：
[root@ob32-1 ~]# which mysql_config
/usr/bin/mysql_config
9.2 obclient数据表结构导入报错，不能创建
ERROR 1115 (42000): Unknown character set: 'latin1'
Query OK, 0 rows affected (0.001 sec)。
经检查为目标端OB不支持latin1字符集，源端默认创建的数据库字符集为latin1。
 |
需要修改源端数据库字符集，重启数据库再导出导入表结构和字符集
编辑mariadb下的/etc/my.cnf文件，添加以下三行参数
max_connections = 4096
collation-server = utf8_general_ci
character-set-server = utf8
重启matiadb数据库
systemctl restart mariadb
再次查看字符集已经变过来了。
重新创建数据库，生成表数据，导出导入即可。
9.3 obclient数据表结构导入时创建表数量不对。
ERROR 1050 (42S01): Table 'customer' not exists
查看表创建数量：
 |
只有2个表。
查看源端有9张表：
只有2个表。
查看源端有9张表：
 |
原因是因为创建数据时执行了外键创建操作：
source add_fkey_idx.sql;
导致表创建时会检查外键约束等，此时有两种办法，一种是禁用外键约束：
set foreign_key_checks='off';
或者多执行几遍source tpcc1000_ddl.sql;直到所有目标表9张表都创建成功。
 |
9.4 obclient数据表truncate和delete时不成功
obclient [tpcc1000_ob]> truncate table orders;
ERROR 1701 (42000): Cannot truncate a table referenced in a foreign key constraint (`tpcc1000_ob`.`order_line`, CONSTRAINT `fkey_order_line_1` FOREIGN KEY (`ol_w_id`, `ol_d_id`, `ol_o_id`) REFERENCES `tpcc1000_ob`.`orders` (`o_w_id`, `o_d_id`, `o_id`))
obclient [tpcc1000_ob]> delete from orders;
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails
因为外键约束限制，先关闭外键约束set foreign_key_checks='off，再使用delete from orders;
obclient [tpcc1000_ob]> select count(*) from orders;
+----------+
| count(*) |
+----------+
| 30000 |
+----------+
1 row in set (0.034 sec)
obclient [tpcc1000_ob]> set foreign_key_checks='off';
Query OK, 0 rows affected (0.002 sec)
obclient [tpcc1000_ob]> truncate table orders;
ERROR 1701 (42000): Cannot truncate a table referenced in a foreign key constraint (`tpcc1000_ob`.`order_line`, CONSTRAINT `fkey_order_line_1` FOREIGN KEY (`ol_w_id`, `ol_d_id`, `ol_o_id`) REFERENCES `tpcc1000_ob`.`orders` (`o_w_id`, `o_d_id`, `o_id`))
obclient [tpcc1000_ob]> delete from orders;
Query OK, 30000 rows affected (0.594 sec)
obclient [tpcc1000_ob]> select count(*) from orders;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.031 sec).
9.5 datax导入数据时需要依赖java环境
注意：datax需要有 Java 运行环境，且版本不低于 V1.8.0。
[root@ob32-1 job]# python ../bin/datax.py mysql2ob.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
/bin/sh: java: command not found
安装java环境：
[root@ob32-1 job]# yum install java-1.8.0*
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.181-7.b13.el7 will be installed
--> Processing Dependency: xorg-x11-fonts-Type1 for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: fontconfig(x86-64) for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: libgif.so.4()(64bit) for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: libXtst.so.6()(64bit) for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: libXrender.so.1()(64bit) for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: libXi.so.6()(64bit) for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: libXext.so.6()(64bit) for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: libXcomposite.so.1()(64bit) for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: libX11.so.6()(64bit) for package: 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
---> Package java-1.8.0-openjdk-devel.x86_64 1:1.8.0.181-7.b13.el7 will be installed
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.181-7.b13.el7 will be installed
--> Processing Dependency: tzdata-java >= 2015d for package: 1:java-1.8.0-openjdk-headless-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: copy-jdk-configs >= 2.2 for package: 1:java-1.8.0-openjdk-headless-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: lksctp-tools(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.181-7.b13.el7.x86_64
--> Processing Dependency: jpackage-utils for package: 1:java-1.8.0-openjdk-headless-1.8.0.181-7.b13.el7.x86_64
--> Running transaction check
---> Package copy-jdk-configs.noarch 0:3.3-10.el7_5 will be installed
---> Package fontconfig.x86_64 0:2.13.0-4.3.el7 will be installed
--> Processing Dependency: fontpackages-filesystem for package: fontconfig-2.13.0-4.3.el7.x86_64
--> Processing Dependency: dejavu-sans-fonts for package: fontconfig-2.13.0-4.3.el7.x86_64
---> Package giflib.x86_64 0:4.1.6-9.el7 will be installed
--> Processing Dependency: libSM.so.6()(64bit) for package: giflib-4.1.6-9.el7.x86_64
--> Processing Dependency: libICE.so.6()(64bit) for package: giflib-4.1.6-9.el7.x86_64
---> Package javapackages-tools.noarch 0:3.4.1-11.el7 will be installed
--> Processing Dependency: python-javapackages = 3.4.1-11.el7 for package: javapackages-tools-3.4.1-11.el7.noarch
--> Processing Dependency: libxslt for package: javapackages-tools-3.4.1-11.el7.noarch
---> Package libX11.x86_64 0:1.6.5-2.el7 will be installed
--> Processing Dependency: libX11-common >= 1.6.5-2.el7 for package: libX11-1.6.5-2.el7.x86_64
--> Processing Dependency: libxcb.so.1()(64bit) for package: libX11-1.6.5-2.el7.x86_64
---> Package libXcomposite.x86_64 0:0.4.4-4.1.el7 will be installed
---> Package libXext.x86_64 0:1.3.3-3.el7 will be installed
---> Package libXi.x86_64 0:1.7.9-1.el7 will be installed
---> Package libXrender.x86_64 0:0.9.10-1.el7 will be installed
---> Package libXtst.x86_64 0:1.2.3-1.el7 will be installed
---> Package lksctp-tools.x86_64 0:1.0.17-2.el7 will be installed
---> Package tzdata-java.noarch 0:2018e-3.el7 will be installed
---> Package xorg-x11-fonts-Type1.noarch 0:7.5-9.el7 will be installed
--> Processing Dependency: ttmkfdir for package: xorg-x11-fonts-Type1-7.5-9.el7.noarch
--> Processing Dependency: ttmkfdir for package: xorg-x11-fonts-Type1-7.5-9.el7.noarch
--> Processing Dependency: mkfontdir for package: xorg-x11-fonts-Type1-7.5-9.el7.noarch
--> Processing Dependency: mkfontdir for package: xorg-x11-fonts-Type1-7.5-9.el7.noarch
--> Running transaction check
---> Package dejavu-sans-fonts.noarch 0:2.33-6.el7 will be installed
--> Processing Dependency: dejavu-fonts-common = 2.33-6.el7 for package: dejavu-sans-fonts-2.33-6.el7.noarch
---> Package fontpackages-filesystem.noarch 0:1.44-8.el7 will be installed
---> Package libICE.x86_64 0:1.0.9-9.el7 will be installed
---> Package libSM.x86_64 0:1.2.2-2.el7 will be installed
---> Package libX11-common.noarch 0:1.6.5-2.el7 will be installed
---> Package libxcb.x86_64 0:1.13-1.el7 will be installed
--> Processing Dependency: libXau.so.6()(64bit) for package: libxcb-1.13-1.el7.x86_64
---> Package libxslt.x86_64 0:1.1.28-5.el7 will be installed
---> Package python-javapackages.noarch 0:3.4.1-11.el7 will be installed
--> Processing Dependency: python-lxml for package: python-javapackages-3.4.1-11.el7.noarch
---> Package ttmkfdir.x86_64 0:3.0.9-42.el7 will be installed
---> Package xorg-x11-font-utils.x86_64 1:7.5-21.el7 will be installed
--> Processing Dependency: libfontenc.so.1()(64bit) for package: 1:xorg-x11-font-utils-7.5-21.el7.x86_64
--> Running transaction check
---> Package dejavu-fonts-common.noarch 0:2.33-6.el7 will be installed
---> Package libXau.x86_64 0:1.0.8-2.1.el7 will be installed
---> Package libfontenc.x86_64 0:1.1.3-3.el7 will be installed
---> Package python-lxml.x86_64 0:3.2.1-4.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================
Installing:
java-1.8.0-openjdk x86_64 1:1.8.0.181-7.b13.el7 EL 251 k
java-1.8.0-openjdk-devel x86_64 1:1.8.0.181-7.b13.el7 EL 9.8 M
java-1.8.0-openjdk-headless x86_64 1:1.8.0.181-7.b13.el7 EL 32 M
Installing for dependencies:
copy-jdk-configs noarch 3.3-10.el7_5 EL 21 k
dejavu-fonts-common noarch 2.33-6.el7 EL 64 k
dejavu-sans-fonts noarch 2.33-6.el7 EL 1.4 M
fontconfig x86_64 2.13.0-4.3.el7 EL 254 k
fontpackages-filesystem noarch 1.44-8.el7 EL 9.9 k
giflib x86_64 4.1.6-9.el7 EL 40 k
javapackages-tools noarch 3.4.1-11.el7 EL 73 k
libICE x86_64 1.0.9-9.el7 EL 66 k
libSM x86_64 1.2.2-2.el7 EL 39 k
libX11 x86_64 1.6.5-2.el7 EL 606 k
libX11-common noarch 1.6.5-2.el7 EL 164 k
libXau x86_64 1.0.8-2.1.el7 EL 29 k
libXcomposite x86_64 0.4.4-4.1.el7 EL 22 k
libXext x86_64 1.3.3-3.el7 EL 39 k
libXi x86_64 1.7.9-1.el7 EL 40 k
libXrender x86_64 0.9.10-1.el7 EL 26 k
libXtst x86_64 1.2.3-1.el7 EL 20 k
libfontenc x86_64 1.1.3-3.el7 EL 31 k
libxcb x86_64 1.13-1.el7 EL 214 k
libxslt x86_64 1.1.28-5.el7 EL 242 k
lksctp-tools x86_64 1.0.17-2.el7 EL 88 k
python-javapackages noarch 3.4.1-11.el7 EL 31 k
python-lxml x86_64 3.2.1-4.el7 EL 758 k
ttmkfdir x86_64 3.0.9-42.el7 EL 48 k
tzdata-java noarch 2018e-3.el7 EL 185 k
xorg-x11-font-utils x86_64 1:7.5-21.el7 EL 104 k
xorg-x11-fonts-Type1 noarch 7.5-9.el7 EL 521 k
Transaction Summary
=============================================================================================================================================================
Install 3 Packages (+27 Dependent packages)
Total download size: 47 M
Installed size: 160 M
Is this ok [y/d/N]: y
Downloading packages:
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 142 MB/s | 47 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : libxslt-1.1.28-5.el7.x86_64 1/30
Installing : fontpackages-filesystem-1.44-8.el7.noarch 2/30
Installing : libICE-1.0.9-9.el7.x86_64 3/30
Installing : libSM-1.2.2-2.el7.x86_64 4/30
Installing : dejavu-fonts-common-2.33-6.el7.noarch 5/30
Installing : dejavu-sans-fonts-2.33-6.el7.noarch 6/30
Installing : fontconfig-2.13.0-4.3.el7.x86_64 7/30
Installing : python-lxml-3.2.1-4.el7.x86_64 8/30
Installing : python-javapackages-3.4.1-11.el7.noarch 9/30
Installing : javapackages-tools-3.4.1-11.el7.noarch 10/30
Installing : libfontenc-1.1.3-3.el7.x86_64 11/30
Installing : 1:xorg-x11-font-utils-7.5-21.el7.x86_64 12/30
Installing : tzdata-java-2018e-3.el7.noarch 13/30
Installing : libX11-common-1.6.5-2.el7.noarch 14/30
Installing : libXau-1.0.8-2.1.el7.x86_64 15/30
Installing : libxcb-1.13-1.el7.x86_64 16/30
Installing : libX11-1.6.5-2.el7.x86_64 17/30
Installing : libXext-1.3.3-3.el7.x86_64 18/30
Installing : libXi-1.7.9-1.el7.x86_64 19/30
Installing : libXtst-1.2.3-1.el7.x86_64 20/30
Installing : giflib-4.1.6-9.el7.x86_64 21/30
Installing : libXrender-0.9.10-1.el7.x86_64 22/30
Installing : libXcomposite-0.4.4-4.1.el7.x86_64 23/30
Installing : lksctp-tools-1.0.17-2.el7.x86_64 24/30
Installing : copy-jdk-configs-3.3-10.el7_5.noarch 25/30
Installing : 1:java-1.8.0-openjdk-headless-1.8.0.181-7.b13.el7.x86_64 26/30
Installing : ttmkfdir-3.0.9-42.el7.x86_64 27/30
Installing : xorg-x11-fonts-Type1-7.5-9.el7.noarch 28/30
Installing : 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64 29/30
Installing : 1:java-1.8.0-openjdk-devel-1.8.0.181-7.b13.el7.x86_64 30/30
Verifying : libXext-1.3.3-3.el7.x86_64 1/30
Verifying : libXi-1.7.9-1.el7.x86_64 2/30
Verifying : fontconfig-2.13.0-4.3.el7.x86_64 3/30
Verifying : giflib-4.1.6-9.el7.x86_64 4/30
Verifying : libXrender-0.9.10-1.el7.x86_64 5/30
Verifying : 1:xorg-x11-font-utils-7.5-21.el7.x86_64 6/30
Verifying : python-lxml-3.2.1-4.el7.x86_64 7/30
Verifying : libICE-1.0.9-9.el7.x86_64 8/30
Verifying : fontpackages-filesystem-1.44-8.el7.noarch 9/30
Verifying : ttmkfdir-3.0.9-42.el7.x86_64 10/30
Verifying : copy-jdk-configs-3.3-10.el7_5.noarch 11/30
Verifying : python-javapackages-3.4.1-11.el7.noarch 12/30
Verifying : dejavu-fonts-common-2.33-6.el7.noarch 13/30
Verifying : libXcomposite-0.4.4-4.1.el7.x86_64 14/30
Verifying : libXtst-1.2.3-1.el7.x86_64 15/30
Verifying : libxcb-1.13-1.el7.x86_64 16/30
Verifying : lksctp-tools-1.0.17-2.el7.x86_64 17/30
Verifying : 1:java-1.8.0-openjdk-devel-1.8.0.181-7.b13.el7.x86_64 18/30
Verifying : xorg-x11-fonts-Type1-7.5-9.el7.noarch 19/30
Verifying : 1:java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64 20/30
Verifying : libxslt-1.1.28-5.el7.x86_64 21/30
Verifying : libX11-1.6.5-2.el7.x86_64 22/30
Verifying : dejavu-sans-fonts-2.33-6.el7.noarch 23/30
Verifying : javapackages-tools-3.4.1-11.el7.noarch 24/30
Verifying : 1:java-1.8.0-openjdk-headless-1.8.0.181-7.b13.el7.x86_64 25/30
Verifying : libXau-1.0.8-2.1.el7.x86_64 26/30
Verifying : libSM-1.2.2-2.el7.x86_64 27/30
Verifying : libX11-common-1.6.5-2.el7.noarch 28/30
Verifying : tzdata-java-2018e-3.el7.noarch 29/30
Verifying : libfontenc-1.1.3-3.el7.x86_64 30/30
Installed:
java-1.8.0-openjdk.x86_64 1:1.8.0.181-7.b13.el7 java-1.8.0-openjdk-devel.x86_64 1:1.8.0.181-7.b13.el7
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.181-7.b13.el7
Dependency Installed:
copy-jdk-configs.noarch 0:3.3-10.el7_5 dejavu-fonts-common.noarch 0:2.33-6.el7 dejavu-sans-fonts.noarch 0:2.33-6.el7
fontconfig.x86_64 0:2.13.0-4.3.el7 fontpackages-filesystem.noarch 0:1.44-8.el7 giflib.x86_64 0:4.1.6-9.el7
javapackages-tools.noarch 0:3.4.1-11.el7 libICE.x86_64 0:1.0.9-9.el7 libSM.x86_64 0:1.2.2-2.el7
libX11.x86_64 0:1.6.5-2.el7 libX11-common.noarch 0:1.6.5-2.el7 libXau.x86_64 0:1.0.8-2.1.el7
libXcomposite.x86_64 0:0.4.4-4.1.el7 libXext.x86_64 0:1.3.3-3.el7 libXi.x86_64 0:1.7.9-1.el7
libXrender.x86_64 0:0.9.10-1.el7 libXtst.x86_64 0:1.2.3-1.el7 libfontenc.x86_64 0:1.1.3-3.el7
libxcb.x86_64 0:1.13-1.el7 libxslt.x86_64 0:1.1.28-5.el7 lksctp-tools.x86_64 0:1.0.17-2.el7
python-javapackages.noarch 0:3.4.1-11.el7 python-lxml.x86_64 0:3.2.1-4.el7 ttmkfdir.x86_64 0:3.0.9-42.el7
tzdata-java.noarch 0:2018e-3.el7 xorg-x11-font-utils.x86_64 1:7.5-21.el7 xorg-x11-fonts-Type1.noarch 0:7.5-9.el7
Complete!
再次执行dataX数据导入即可。
9.6 DATAX导入数据报错
继续执行python ../bin/datax.py mysql2ob.json时报错：
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-03-09 15:12:01.194 [main] ERROR Engine -
经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Common-00], Describe:[您提供的配置文件存在错误信息，请检查您的作业配置 .] - 配置信息错误. 您提供的配置信息不是合法的JSON格式: syntax error, pos 2 . 请按照标准json格式提供配置信息.
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.alibaba.datax.common.util.Configuration.<init>(Configuration.java:1066)
at com.alibaba.datax.common.util.Configuration.from(Configuration.java:79)
at com.alibaba.datax.core.util.ConfigParser.parseJobConfig(ConfigParser.java:75)
at com.alibaba.datax.core.util.ConfigParser.parse(ConfigParser.java:26)
at com.alibaba.datax.core.Engine.entry(Engine.java:137)
at com.alibaba.datax.core.Engine.main(Engine.java:204)
报错原因是使用python ./datax/bin/datax.py -r mysqlreader -w oceanbasev10writer > mysql2ob.json生成的mysql2ob.json配置模板中表头部分没有注释掉，将表头部分删掉或者加#号注释掉，只留下{}部分：
1026
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
