MySql学习 数据批量导入

MySql数据批量导入:批量导入数据时一般会将rewriteBatchedStatements设置为true
对参数rewriteBatchedStatements解释:

MySQL的JDBC连接的url中要加rewriteBatchedStatements参数，并保证5.1.13以上版本的驱动，才能实现高性能的批量插入。
MySQL JDBC驱动在默认情况下会无视executeBatch()语句，把我们期望批量执行的一组sql语句拆散，一条一条地发给MySQL数据库，批量插入实际上是单条插入，直接造成较低的性能。
只有把rewriteBatchedStatements参数置为true, 驱动才会帮你批量执行SQL
另外这个选项对INSERT/UPDATE/DELETE都有效

环境准备:

idea创建maven工程,导入mybatis依赖,单元测试依赖等
pom.xml

<dependency>
 	<groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>5.1.47</version>
</dependency>

<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.mybatis</groupId>
    <artifactId>mybatis</artifactId>
    <version>3.5.6</version>
</dependency>

实体类

@Data
public class Student {
    private int id;
    private String username;
}

方式一:for 循环遍历,单次插入数据

(for循环,每遍历一次,就执行一条插入的SQL语句,性能差,效率低)

@Test
    public void add () throws IOException {
        InputStream is = Resources.getResourceAsStream ( "mybatis-config.xml");
        SqlSessionFactory sqlSessionFactory = new SqlSessionFactoryBuilder ( ).build ( is);
        SqlSession sqlSession = sqlSessionFactory.openSession ( );
        StudentDao mapper = sqlSession.getMapper ( StudentDao.class );
        List<Student> list = new ArrayList<> ( );
        for (int i = 0; i < 5; i++) {
            Student student = new Student ( );
            student.setUsername ( "张三" + i );
            list.add ( student );
        }
        for (int i = 0; i < list.size ( ); i++){
            mapper.addOne ( list.get ( i ) );
        }
        sqlSession.commit ( );//插入数据需要手动提交事务
        sqlSession.close ( );//释放资源
    }

StudentDao.xml

<insert id="addOne" parameterType="student">
     insert into student values (null,#{username})
</insert>

控制台打印执行的SQL语句如下:

Checking to see if class com.itheima.mapper.StudentDao matches criteria [is assignable to Object]
Opening JDBC Connection
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@62fdb4a6]
==>  Preparing: insert into student values (null,?)
==> Parameters: 张三0
<==    Updates: 1
==>  Preparing: insert into student values (null,?)
==> Parameters: 张三1
<==    Updates: 1
==>  Preparing: insert into student values (null,?)
==> Parameters: 张三2
<==    Updates: 1
==>  Preparing: insert into student values (null,?)
==> Parameters: 张三3
<==    Updates: 1
==>  Preparing: insert into student values (null,?)
==> Parameters: 张三4
<==    Updates: 1
Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@62fdb4a6]

Process finished with exit code 0

方式二: 使用foreach标签,批量插入数据

插入多条数据,只执行一次SQL语句,使用foreach标签拼接参数
当参数列表较多时,使用foreach插入,性能会比较差

1. SQL语句过长解析效率低,影响执行效率
2. 数据太多时,可能会报异常或者卡死等问题,MySQL server has gone away.

public void add () throws IOException {
        String resource = "mybatis-config.xml";
        InputStream inputStream = Resources.getResourceAsStream ( resource );
        SqlSessionFactory sqlSessionFactory = new SqlSessionFactoryBuilder ( ).build ( inputStream );
        SqlSession sqlSession = sqlSessionFactory.openSession ( );
        StudentDao mapper = sqlSession.getMapper ( StudentDao.class );
        List<Student> list = new ArrayList<> ( );
        for (int i = 0; i < 5; i++) {
            Student student = new Student ( );
            student.setUsername ( "张三" + i );
            list.add ( student );
        }
        mapper.addList ( list );
        sqlSession.commit ( );
        sqlSession.close ( );
    }

StudentDao.xml中编写addList方法的SQL语句

<insert id="addList" parameterType="list">
    insert into student  values
    <foreach collection="list"  separator=","  item="s" >
         (null,#{s.username})
     </foreach>
</insert>

控制台打印执行的SQL语句如下:

Opening JDBC Connection
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@731f8236]
==>  Preparing: insert into student values (null,?) , (null,?) , (null,?) , (null,?) , (null,?)
==> Parameters: 张三0(String), 张三1(String), 张三2(String), 张三3(String), 张三4(String)
<==    Updates: 5
Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@731f8236]
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@731f8236]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@731f8236]

方式三 开启ExecutorType.BATCH模式

ExecutorType 参数是枚举类型。这个枚举类型定义了三个值:

ExecutorType.SIMPLE：该类型的执行器没有特别的行为。它为每个语句的执行创建一个新的预处理语句。
ExecutorType.REUSE：该类型的执行器会复用预处理语句。
ExecutorType.BATCH：该类型的执行器会批量执行所有更新语句，如果 SELECT 在多个更新中间执行，将在必要时将多条更新语句分隔开来，以方便理解。

注意

1. simple单句模式，该模式下它为每个语句的执行创建一个新的预处理语句，单句提交sql；
2. batch模式重复使用已经预处理的语句，并且批量执行所有语句，大批量模式下性能更优
3. batch模式在Insert操作时事务没有提交之前，是没有办法获取到自增的id

开启ExecutorType.BATCH模式:
在创建连接时,传入参数ExecutorType.BATCH即可开启ExecutorType.BATCH模式
(默认为simple模式)
SqlSession sqlSession = sqlSessionFactory.openSession ( ExecutorType.BATCH);
即使使用for循环单次添加,也会先将数据添加到缓存中,等待commit时再将所有数据保存到数据库中

@Test
    public void add () throws IOException {
        String resource = "mybatis-config.xml";
        InputStream inputStream = Resources.getResourceAsStream ( resource );
        SqlSessionFactory sqlSessionFactory = new SqlSessionFactoryBuilder ( ).build ( inputStream );
        SqlSession sqlSession = sqlSessionFactory.openSession ( ExecutorType.BATCH);
        StudentDao mapper = sqlSession.getMapper ( StudentDao.class );
        List<Student> list = new ArrayList<> ( );
        for (int i = 0; i < 5; i++) {
            Student student = new Student ( );
            student.setUsername ( "张三" + i );
            list.add ( student );
        }
        for (Student student : list) {
            mapper.add ( student );
        }
        sqlSession.commit ( );
        sqlSession.close ( );
    }

Opening JDBC Connection
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@1372ed45]
==>  Preparing: insert into student values (null,?)
==> Parameters: 张三0(String)
==> Parameters: 张三1(String)
==> Parameters: 张三2(String)
==> Parameters: 张三3(String)
==> Parameters: 张三4(String)
Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@1372ed45]
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@1372ed45]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@1372ed45]

对于foreach和数据库批量执行效率:
参考:foreach 和 数据库批量执行 效率比较

1. 开启rewriteBatchedStatements=true 和 ExecutorType.BATCH 效率比 foreach 快,但是在事务没有提交之前，是没有办法获取到自增的id
2. foreach 依旧是 比较合理 批量插入方案，但是需要控制每次插入数据数量