mysql8导入不同schme_MySQL 8.0 information_schema系统库的改进

information_schema有何用？

通过I_S获取MySQL的一些元数据信息

获取表的数据文件、索引文件的大小、碎片情况、表行数、自增列增长情况等

获取正在运行的事务有那些，是否有阻塞等

获取当前mysql的连接processlist等等

mysql8.0 之前的查询方式

会在查询information_schema 某个表时创建临时表

来自文件的元数据，扫描文件系统获取FRM文件的表定义

存储引擎的详细信息，例如动态表统计信息

来自MySQL服务器中全局数据结构的数据

在表的数量很多时，每次查询I_S会从文件系统中读取每个单独的FRM文件，使用更多的CPU周期来打开表并准备相关的内存数据结构

mysql8.0 开始的查询方式

引入了基于InnoDB的本地数据字典表

表中不在有FRM表定义文件

所有数据库表的元数据都存储在事务数据字典表中

I_S中表被设计为数据字典表上的VIEW(有些还是临时表0_0)

消除了以下成本

查询INFORMATION_SCHEMA时创建的临时表

扫描文件系统目录以查找FRM文件

改进

利用MySQL优化器的全部功能，使用数据字典表上的索引来更好的查询

mysql5.7中表文件

ll test*

Jul 10 10:52 testse.frm

Jul 10 10:52 testse.ibd

mysql8.0中表文件

ll test*

Jul 10 10:25 testse.ibd

mysql5.7 和mysql8.0 I_S 中tables 表的存在方式

mysql5.7.22

show create table information_schema.tables\G

*************************** 1. row ***************************

Table: TABLES

Create Table: CREATE TEMPORARY TABLE `TABLES` (

`TABLE_CATALOG` varchar(512) NOT NULL DEFAULT '',

`TABLE_SCHEMA` varchar(64) NOT NULL DEFAULT '',

`TABLE_NAME` varchar(64) NOT NULL DEFAULT '',

`TABLE_TYPE` varchar(64) NOT NULL DEFAULT '',

`ENGINE` varchar(64) DEFAULT NULL,

mysql8.0.15

show create table information_schema.tables\G

*************************** 1. row ***************************

View: TABLES

Create View: CREATE ALGORITHM=UNDEFINED DEFINER=`mysql.infoschema`@`localhost` SQL SECURITY DEFINER VIEW `

mysql5.7 中I_S中tables表是以临时表的形式存在的(查询该表就会创建临时表，创建的临时表过多，可能会导致mysql占用的内存暴涨，出现OOM)

mysql8.0 中I_S中tables表以视图的形式存在(查询该视图，不会创建临时表，会使用到视图中表的索引)

mysql5.7中获取表大小情况

SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT( FORMAT((data_length + index_length) / 1024 / 1024 / 1024,2), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema = 'test' AND table_name = 'testse';

+------------+-------------+--------------+------------+----------------+------------+---------------------+

| table_name | dbdata_size | dbindex_size | db_size(G) | AVG_ROW_LENGTH | table_rows | update_time |

+------------+-------------+--------------+------------+----------------+------------+---------------------+

| testse | 0.02M | 0.02M | 0.00G | 862 | 19 | 2019-07-10 10:52:02 |

+------------+-------------+--------------+------------+----------------+------------+---------------------+

执行计划中出现了where，未用到索引(存储引擎检索数据后，server层进行过滤)

desc SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT( FORMAT(data_length + index_length / 1024 / 1024 / 1024, 2 ), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema = 'test' AND table_name = 'testse';

+----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+

| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |

+----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+

| 1 | SIMPLE | tables | NULL | ALL | NULL | TABLE_SCHEMA,TABLE_NAME | NULL | NULL | NULL | NULL | Using where; Open_full_table; Scanned 0 databases |

+----+-------------+--------+------------+------+---------------+-------------------------+---------+------+------+----------+---------------------------------------------------+

mysql8.0中获取表大小情况

SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT( FORMAT((data_length + index_length) / 1024 / 1024 / 1024,2), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema = 'test' AND table_name = 'testse';

+------------+-------------+--------------+------------+----------------+------------+---------------------+

| TABLE_NAME | dbdata_size | dbindex_size | db_size(G) | AVG_ROW_LENGTH | TABLE_ROWS | UPDATE_TIME |

+------------+-------------+--------------+------------+----------------+------------+---------------------+

| testse | 0.02M | 0.02M | 0.00G | 862 | 19 | 2019-07-10 10:25:16 |

+------------+-------------+--------------+------------+----------------+------------+---------------------+

能使用到数据字典表的索引

desc SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT( FORMAT((data_length + index_length) / 1024 / 1024 / 1024,2), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema = 'test' AND table_name = 'testse';

+----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+

| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |

+----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+

| 1 | SIMPLE | cat | NULL | index | PRIMARY | name | 194 | NULL | 1 | 100.00 | Using index |

| 1 | SIMPLE | sch | NULL | eq_ref | PRIMARY,catalog_id | catalog_id | 202 | mysql.cat.id,const | 1 | 100.00 | Using index |

| 1 | SIMPLE | tbl | NULL | eq_ref | schema_id | schema_id | 202 | mysql.sch.id,const | 1 | 100.00 | Using where |

| 1 | SIMPLE | stat | NULL | const | PRIMARY | PRIMARY | 388 | const,const | 1 | 100.00 | NULL |

| 1 | SIMPLE | ts | NULL | eq_ref | PRIMARY | PRIMARY | 8 | mysql.tbl.tablespace_id | 1 | 100.00 | NULL |

| 1 | SIMPLE | col | NULL | eq_ref | PRIMARY | PRIMARY | 8 | mysql.tbl.collation_id | 1 | 100.00 | Using index |

+----+-------------+-------+------------+--------+--------------------+------------+---------+-------------------------+------+----------+-------------+

测试5.7和8.0不同版本访问I_S库的性能

机器

cat /etc/RedHat-release | xargs echo '版本 ' && dmidecode -s system-product-name | xargs echo '是否虚拟化 ' && cat /proc/cpuinfo |grep "processor"|wc -l | xargs echo 'cpu核数 '

版本 CentOS Linux release 7.5.1804 (Core)

是否虚拟化 KVM

cpu核数 4

1、分别在mysql5.7和mysql8.0 中创建5个库，每个库中30个表(共1500个表)，每个表记录数为10000

user=admin

passwd=admin

port=57222

host=127.0.0.1

#创建5个库,分别在每个库中创建30个表

for i in {1..5};do

mysql -u$user -p$passwd -P$port -h$host<

create database if not exists test_${i};

EOF

sysbench /usr/local/share/sysbench/oltp_read_write.lua --mysql_storage_engine=innodb --table-size=10000 --tables=30 --mysql-db=test_${i} --mysql-user=$user --mysql-password=$passwd --mysql-port=$port --mysql-host=$host --threads=8 --time=10 --report-interval=1 --events=0 --db-driver=mysql prepare

done;

2、自定义访问I_S.tables表的sysbench脚本

cat tests/mytest.lua

require("oltp_common")

function thread_init(thread_id)

drv=sysbench.sql.driver()

con=drv:connect()

end

local function get_rand_db()

return sysbench.rand.uniform(1, 5)

end

function event(thread_id)

local vid1

local dbprefix

vid1=get_rand_db()

dbprefix = "test_"

-- 生成5个db中的随机一个db,如 test_1 ,test_5

-- vid2=string.format("'%s%s'",test_,vid1)

vid2="'" .. dbprefix .. vid1 .. "'"

con:query("SELECT table_name, CONCAT(FORMAT(data_length / 1024 / 1024, 2), 'M') AS dbdata_size, CONCAT(FORMAT(index_length / 1024 / 1024, 2), 'M') AS dbindex_size, CONCAT( FORMAT((data_length + index_length) / 1024 / 1024 / 1024,2), 'G') AS `db_size(G)`, AVG_ROW_LENGTH, table_rows, update_time FROM information_schema.tables WHERE table_schema =" .. vid2)

end

function thread_done()

con:disconnect()

end

3、脚本通过sysbench测试访问I_S.tables的qps

cat test_I_S_access.sh

user=admin

passwd=admin

host=127.0.0.1

#输入mysql端口参数

port=$1

# port=8015

#输入sysbench测试时间参数多少秒

run_time=$2

function get_create_tmp_tables(){

mysql -u$user -p$passwd -P$port -h$host<

select @@version as 'mysql version';

show global status like '%tmp_table%';

EOF

} 2>/dev/null

function begin_test(){

cd /usr/local/share/sysbench

sysbench ./tests/mytest.lua --mysql-db=test_1 --mysql-host=$host --mysql-port=$port --mysql-user=$user --mysql-password=$passwd --threads=40 --report-interval=10 --rand-type=uniform --time=$run_time run

}

service mysqld${port} restart

get_create_tmp_tables

begin_test

get_create_tmp_tables

4、mysql5.7和mysql8.0 主要配置

mysql5.7.22

5.7.22-log

innodb_buffer_pool_size 128M

innodb_log_buffer_size 64M

innodb_log_file_size 48M

binlog_format ROW

innodb_flush_log_at_trx_commit 1

sync_binlog 1

mysql8.0.15

8.0.15

innodb_buffer_pool_size 128M

innodb_log_buffer_size 64M

innodb_log_file_size 48M

binlog_format ROW

innodb_flush_log_at_trx_commit 1

sync_binlog 1

5、mysql5.7开启测试

bash test_I_S_acess.sh 57222 60

Shutting down MySQL.. SUCCESS!

Starting MySQL.. SUCCESS!

mysql version

5.7.22-log

Variable_name Value

Created_tmp_tables 1

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 40

Report intermediate results every 10 second(s)

Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 10s ] thds: 40 tps: 4765.73 qps: 4765.73 (r/w/o: 4765.73/0.00/0.00) lat (ms,95%): 18.95 err/s: 0.00 reconn/s: 0.00

[ 20s ] thds: 40 tps: 5409.00 qps: 5409.00 (r/w/o: 5409.00/0.00/0.00) lat (ms,95%): 17.95 err/s: 0.00 reconn/s: 0.00

[ 30s ] thds: 40 tps: 5154.45 qps: 5154.45 (r/w/o: 5154.45/0.00/0.00) lat (ms,95%): 18.61 err/s: 0.00 reconn/s: 0.00

[ 40s ] thds: 40 tps: 5383.50 qps: 5383.50 (r/w/o: 5383.50/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s: 0.00

[ 50s ] thds: 40 tps: 5456.11 qps: 5456.11 (r/w/o: 5456.11/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s: 0.00

[ 60s ] thds: 40 tps: 5458.66 qps: 5458.66 (r/w/o: 5458.66/0.00/0.00) lat (ms,95%): 17.63 err/s: 0.00 reconn/s: 0.00

SQL statistics:

queries performed:

read: 316322

write: 0

other: 0

total: 316322

transactions: 316322 (5270.99 per sec.)

queries: 316322 (5270.99 per sec.)

ignored errors: 0 (0.00 per sec.)

reconnects: 0 (0.00 per sec.)

Throughput:

events/s (eps): 5270.9892

time elapsed: 60.0119s

total number of events: 316322

Latency (ms):

min: 0.42

avg: 7.59

max: 858.18

95th percentile: 17.95

sum: 2399738.58

Threads fairness:

events (avg/stddev): 7908.0500/53.41

execution time (avg/stddev): 59.9935/0.00

mysql version

5.7.22-log

Variable_name Value

Created_tmp_tables 316327

cpu使用率接近100%

mysql5.7.22 40个并发线程 压力测试60秒，访问I_S.tables 平均qps为5k，并创建了31w个临时表!

6、mysql8.0开启测试

Shutting down MySQL.. SUCCESS!

Starting MySQL.... SUCCESS!

mysql version

8.0.15

Variable_name Value

Created_tmp_tables 1

sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:

Number of threads: 40

Report intermediate results every 10 second(s)

Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 10s ] thds: 40 tps: 1283.76 qps: 1283.76 (r/w/o: 1283.76/0.00/0.00) lat (ms,95%): 41.10 err/s: 0.00 reconn/s: 0.00

[ 20s ] thds: 40 tps: 1286.43 qps: 1286.43 (r/w/o: 1286.43/0.00/0.00) lat (ms,95%): 39.65 err/s: 0.00 reconn/s: 0.00

[ 30s ] thds: 40 tps: 1333.00 qps: 1333.00 (r/w/o: 1333.00/0.00/0.00) lat (ms,95%): 37.56 err/s: 0.00 reconn/s: 0.00

[ 40s ] thds: 40 tps: 1280.79 qps: 1280.79 (r/w/o: 1280.79/0.00/0.00) lat (ms,95%): 40.37 err/s: 0.00 reconn/s: 0.00

[ 50s ] thds: 40 tps: 1313.57 qps: 1313.57 (r/w/o: 1313.57/0.00/0.00) lat (ms,95%): 38.94 err/s: 0.00 reconn/s: 0.00

[ 60s ] thds: 40 tps: 1222.95 qps: 1222.95 (r/w/o: 1222.95/0.00/0.00) lat (ms,95%): 42.61 err/s: 0.00 reconn/s: 0.00

SQL statistics:

queries performed:

read: 77248

write: 0

other: 0

total: 77248

transactions: 77248 (1286.95 per sec.)

queries: 77248 (1286.95 per sec.)

ignored errors: 0 (0.00 per sec.)

reconnects: 0 (0.00 per sec.)

Throughput:

events/s (eps): 1286.9486

time elapsed: 60.0242s

total number of events: 77248

Latency (ms):

min: 1.88

avg: 31.08

max: 692.98

95th percentile: 40.37

sum: 2400505.33

Threads fairness:

events (avg/stddev): 1931.2000/51.17

execution time (avg/stddev): 60.0126/0.00

mysql version

8.0.15

Variable_name Value

Created_tmp_tables 2

cpu使用率接近100%

mysql8.0.15 40个并发线程，压力测试60秒，访问I_S.tables 平均qps为1.2k，并创建了1个临时表!

结论

mysql8.0开始查询I_S中表不会再从文件系统(FRM, TRG，OPT文件)和myisam、innodb系统表读取元数据信息(8.0开始元数据信息统一存放在数据字典表中)

mysql8.0 访问I_S.tables不会创建临时表，这减少了内存暴涨的可能，但访问I_S.tables的qps大约是mysql5.7.22的1/5，访问速度没有mysql5.7.22的快

mysql8.0 访问I_S.tables 用户空间cpu消耗没有mysql5.7的高(没有创建临时表的缘故吧)，但系统空间cpu消耗高于mysql5.7！