mysql数据库批量插入优化

已于 2024-03-16 15:54:24 修改
阅读量839 收藏 2
点赞数
文章标签： mysql 数据库 java
于 2022-01-27 14:11:07 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xu990128638/article/details/122716911
版权

mysql 数据库插入优化

0. 异常信息

Packet for query is too large (6071393 > 4194304). You can change this value on the server by setting the max_allowed_packet’ variable …

mysql 数据库一般有一个默认的设置，就是每次sql操作的数据不能超过4M。

1.修改配置文件：

修改 my.ini 或 my.cof 加上 max_allowed_packet =67108864，67108864=64M，默认大小4194304 也就是4M

修改完成之后需要重启mysql服务

2.通过命令：

show variables like ‘%max_allowed_packet%’;

set global max_allowed_packet = 67108864；— 实际上就是64M

show variables like ‘%max_allowed_packet%’;

max_allowed_packet	        67108864
slave_max_allowed_packet	1073741824

select 6410241024 from dual; — 64M

3. 优化代码：

这里拿mybatis 举例 ，可以采用多线程，分批次，插入数据。

<insert id="insertAll" useGeneratedKeys="true" keyProperty="id" parameterType="cn.com.taiji.cloud.modules.statistics.networkdevice.domain.Waf">
		insert into t_statistics_waf
		<trim prefix="(" suffix=")" suffixOverrides=",">
			id, proxy_ip, business_ip, business_port, c_t, u_t, is_delete
		</trim>
		values
		<foreach collection="list" item="item" index="index" separator=",">
			(
			#{item.id}, #{item.proxyIp}, #{item.businessIp}, #{item.businessPort}, now(), now(), 0
			)
		</foreach>
	</insert>

void insertAll(List<Waf> resources);


    @Override
    @Transactional(rollbackFor = Exception.class)
    public List<WafDTO> saveUpload(List<Waf> resources) {
        log.info("wafServiceImpl.saveUpload:{}",resources.size());
        // 基于内核 做偏移量 ,基于多线程分批 批量插入数据
        int nThreads = Runtime.getRuntime().availableProcessors() ;
        if (resources.size() > nThreads * 10 ) {
            ExecutorService executorService = Executors.newFixedThreadPool(nThreads * 2);
            List<Future<Integer>> futures = new ArrayList<Future<Integer>>(nThreads);
            int offset = 50;
            for (int i = 0; i < resources.size(); i += offset) {
                if (i + offset > resources.size()) {
                    nThreads = resources.size() - i;
                }
                final List<Waf> wafList  = resources.subList(i, i + offset);
                Callable<Integer> task = () -> {
                    wafRepository.insertAll(wafList);
                    return 1;
                };
                futures.add(executorService.submit(task));
            }
            executorService.shutdown();
            if (!futures.isEmpty() && futures != null) {
                return wafMapper.toDto(resources);
            }
        } else {
            wafRepository.insertAll(resources);
        }
        return wafMapper.toDto(resources);

    }
4. 附录

查看当前数据库的配置参数

show variables ;

auto_generate_certs	ON
auto_increment_increment	1
auto_increment_offset	1
autocommit	ON
automatic_sp_privileges	ON
avoid_temporal_upgrade	OFF
back_log	770
basedir	C:\mysql-5.7.31-winx64\
big_tables	OFF
bind_address	0.0.0.0
binlog_cache_size	32768
binlog_checksum	CRC32
binlog_direct_non_transactional_updates	OFF
binlog_error_action	ABORT_SERVER
binlog_format	MIXED
binlog_group_commit_sync_delay	0
binlog_group_commit_sync_no_delay_count	0
binlog_gtid_simple_recovery	ON
binlog_max_flush_queue_time	0
binlog_order_commits	ON
binlog_row_image	FULL
binlog_rows_query_log_events	OFF
binlog_stmt_cache_size	32768
binlog_transaction_dependency_history_size	25000
binlog_transaction_dependency_tracking	COMMIT_ORDER
block_encryption_mode	aes-128-ecb
bulk_insert_buffer_size	8388608
character_set_client	utf8mb4
character_set_connection	utf8mb4
character_set_database	utf8
character_set_filesystem	binary
character_set_results	utf8mb4
character_set_server	utf8
character_set_system	utf8
character_sets_dir	C:\mysql-5.7.31-winx64\share\charsets\
check_proxy_users	OFF
collation_connection	utf8mb4_general_ci
collation_database	utf8_general_ci
collation_server	utf8_general_ci
completion_type	NO_CHAIN
concurrent_insert	AUTO
connect_timeout	10
core_file	OFF
datadir	C:\mysql-5.7.31-winx64\data\
date_format	%Y-%m-%d
datetime_format	%Y-%m-%d %H:%i:%s
default_authentication_plugin	mysql_native_password
default_password_lifetime	0
default_storage_engine	InnoDB
default_tmp_storage_engine	InnoDB
default_week_format	0
delay_key_write	ON
delayed_insert_limit	100
delayed_insert_timeout	300
delayed_queue_size	1000
disabled_storage_engines	
disconnect_on_expired_password	ON
div_precision_increment	4
end_markers_in_json	OFF
enforce_gtid_consistency	OFF
eq_range_index_dive_limit	200
error_count	0
event_scheduler	OFF
expire_logs_days	0
explicit_defaults_for_timestamp	ON
external_user	
flush	OFF
flush_time	0
foreign_key_checks	ON
ft_boolean_syntax	+ -><()~*:""&|
ft_max_word_len	84
ft_min_word_len	4
ft_query_expansion_limit	20
ft_stopword_file	(built-in)
general_log	OFF
general_log_file	C:\mysql-5.7.31-winx64\data\WIN-OD0Q68NHVKC.log
group_concat_max_len	1024
gtid_executed_compression_period	1000
gtid_mode	OFF
gtid_next	AUTOMATIC
gtid_owned	
gtid_purged	
have_compress	YES
have_crypt	NO
have_dynamic_loading	YES
have_geometry	YES
have_openssl	YES
have_profiling	YES
have_query_cache	YES
have_rtree_keys	YES
have_ssl	YES
have_statement_timeout	YES
have_symlink	YES
host_cache_size	783
hostname	WIN-OD0Q68NHVKC
identity	0
ignore_builtin_innodb	OFF
ignore_db_dirs	
init_connect	
init_file	
init_slave	
innodb_adaptive_flushing	ON
innodb_adaptive_flushing_lwm	10
innodb_adaptive_hash_index	ON
innodb_adaptive_hash_index_parts	8
innodb_adaptive_max_sleep_delay	150000
innodb_api_bk_commit_interval	5
innodb_api_disable_rowlock	OFF
innodb_api_enable_binlog	OFF
innodb_api_enable_mdl	OFF
innodb_api_trx_level	0
innodb_autoextend_increment	64
innodb_autoinc_lock_mode	1
innodb_buffer_pool_chunk_size	134217728
innodb_buffer_pool_dump_at_shutdown	ON
innodb_buffer_pool_dump_now	OFF
innodb_buffer_pool_dump_pct	25
innodb_buffer_pool_filename	ib_buffer_pool
innodb_buffer_pool_instances	1
innodb_buffer_pool_load_abort	OFF
innodb_buffer_pool_load_at_startup	ON
innodb_buffer_pool_load_now	OFF
innodb_buffer_pool_size	32212254720
innodb_change_buffer_max_size	25
innodb_change_buffering	all
innodb_checksum_algorithm	crc32
innodb_checksums	ON
innodb_cmp_per_index_enabled	OFF
innodb_commit_concurrency	0
innodb_compression_failure_threshold_pct	5
innodb_compression_level	6
innodb_compression_pad_pct_max	50
innodb_concurrency_tickets	5000
innodb_data_file_path	ibdata1:12M:autoextend
innodb_data_home_dir	
innodb_deadlock_detect	ON
innodb_default_row_format	dynamic
innodb_disable_sort_file_cache	OFF
innodb_doublewrite	ON
innodb_fast_shutdown	1
innodb_file_format	Barracuda
innodb_file_format_check	ON
innodb_file_format_max	Barracuda
innodb_file_per_table	ON
innodb_fill_factor	100
innodb_flush_log_at_timeout	1
innodb_flush_log_at_trx_commit	1
innodb_flush_method	
innodb_flush_neighbors	1
innodb_flush_sync	ON
innodb_flushing_avg_loops	30
innodb_force_load_corrupted	OFF
innodb_force_recovery	0
innodb_ft_aux_table	
innodb_ft_cache_size	8000000
innodb_ft_enable_diag_print	OFF
innodb_ft_enable_stopword	ON
innodb_ft_max_token_size	84
innodb_ft_min_token_size	3
innodb_ft_num_word_optimize	2000
innodb_ft_result_cache_limit	2000000000
innodb_ft_server_stopword_table	
innodb_ft_sort_pll_degree	2
innodb_ft_total_cache_size	640000000
innodb_ft_user_stopword_table	
innodb_io_capacity	200
innodb_io_capacity_max	2000
innodb_large_prefix	ON
innodb_lock_wait_timeout	50
innodb_locks_unsafe_for_binlog	OFF
innodb_log_buffer_size	16777216
innodb_log_checksums	ON
innodb_log_compressed_pages	ON
innodb_log_file_size	50331648
innodb_log_files_in_group	2
innodb_log_group_home_dir	.\
innodb_log_write_ahead_size	8192
innodb_lru_scan_depth	1024
innodb_max_dirty_pages_pct	75.000000
innodb_max_dirty_pages_pct_lwm	0.000000
innodb_max_purge_lag	0
innodb_max_purge_lag_delay	0
innodb_max_undo_log_size	1073741824
innodb_monitor_disable	
innodb_monitor_enable	
innodb_monitor_reset	
innodb_monitor_reset_all	
innodb_old_blocks_pct	37
innodb_old_blocks_time	1000
innodb_online_alter_log_max_size	134217728
innodb_open_files	2000
innodb_optimize_fulltext_only	OFF
innodb_page_cleaners	1
innodb_page_size	16384
innodb_print_all_deadlocks	OFF
innodb_purge_batch_size	300
innodb_purge_rseg_truncate_frequency	128
innodb_purge_threads	4
innodb_random_read_ahead	OFF
innodb_read_ahead_threshold	56
innodb_read_io_threads	4
innodb_read_only	OFF
innodb_replication_delay	0
innodb_rollback_on_timeout	OFF
innodb_rollback_segments	128
innodb_sort_buffer_size	1048576
innodb_spin_wait_delay	6
innodb_stats_auto_recalc	ON
innodb_stats_include_delete_marked	OFF
innodb_stats_method	nulls_equal
innodb_stats_on_metadata	OFF
innodb_stats_persistent	ON
innodb_stats_persistent_sample_pages	20
innodb_stats_sample_pages	8
innodb_stats_transient_sample_pages	8
innodb_status_output	OFF
innodb_status_output_locks	OFF
innodb_strict_mode	ON
innodb_support_xa	ON
innodb_sync_array_size	1
innodb_sync_spin_loops	30
innodb_table_locks	ON
innodb_temp_data_file_path	ibtmp1:12M:autoextend
innodb_thread_concurrency	0
innodb_thread_sleep_delay	10000
innodb_tmpdir	
innodb_undo_directory	.\
innodb_undo_log_truncate	OFF
innodb_undo_logs	128
innodb_undo_tablespaces	0
innodb_use_native_aio	ON
innodb_version	5.7.31
innodb_write_io_threads	4
insert_id	0
interactive_timeout	240
internal_tmp_disk_storage_engine	InnoDB
join_buffer_size	262144
keep_files_on_create	OFF
key_buffer_size	8388608
key_cache_age_threshold	300
key_cache_block_size	1024
key_cache_division_limit	100
keyring_operations	ON
large_files_support	ON
large_page_size	0
large_pages	OFF
last_insert_id	0
lc_messages	en_US
lc_messages_dir	C:\mysql-5.7.31-winx64\share\
lc_time_names	en_US
license	GPL
local_infile	ON
lock_wait_timeout	31536000
log_bin	ON
log_bin_basename	C:\mysql-5.7.31-winx64\data\mysql-bin
log_bin_index	C:\mysql-5.7.31-winx64\data\mysql-bin.index
log_bin_trust_function_creators	OFF
log_bin_use_v1_row_events	OFF
log_builtin_as_identified_by_password	OFF
log_error	C:\mysql-5.7.31-winx64\data\WIN-OD0Q68NHVKC.err
log_error_verbosity	3
log_output	FILE
log_queries_not_using_indexes	OFF
log_slave_updates	OFF
log_slow_admin_statements	OFF
log_slow_slave_statements	OFF
log_statements_unsafe_for_binlog	ON
log_syslog	ON
log_syslog_tag	
log_throttle_queries_not_using_indexes	0
log_timestamps	UTC
log_warnings	2
long_query_time	10.000000
low_priority_updates	OFF
lower_case_file_system	ON
lower_case_table_names	1
master_info_repository	FILE
master_verify_checksum	OFF
max_allowed_packet	67108864
max_binlog_cache_size	18446744073709547520
max_binlog_size	1073741824
max_binlog_stmt_cache_size	18446744073709547520
max_connect_errors	100
max_connections	3600
max_delayed_threads	20
max_digest_length	1024
max_error_count	64
max_execution_time	0
max_heap_table_size	16777216
max_insert_delayed_threads	20
max_join_size	18446744073709551615
max_length_for_sort_data	1024
max_points_in_geometry	65536
max_prepared_stmt_count	16382
max_relay_log_size	0
max_seeks_for_key	4294967295
max_sort_length	1024
max_sp_recursion_depth	0
max_tmp_tables	32
max_user_connections	0
max_write_lock_count	4294967295
metadata_locks_cache_size	1024
metadata_locks_hash_instances	8
min_examined_row_limit	0
multi_range_count	256
myisam_data_pointer_size	6
myisam_max_sort_file_size	2146435072
myisam_mmap_size	18446744073709551615
myisam_recover_options	OFF
myisam_repair_threads	1
myisam_sort_buffer_size	8388608
myisam_stats_method	nulls_unequal
myisam_use_mmap	OFF
mysql_native_password_proxy_users	OFF
named_pipe	OFF
named_pipe_full_access_group	*everyone*
net_buffer_length	16384
net_read_timeout	30
net_retry_count	10
net_write_timeout	60
new	OFF
ngram_token_size	2
offline_mode	OFF
old	OFF
old_alter_table	OFF
old_passwords	0
open_files_limit	20048
optimizer_prune_level	1
optimizer_search_depth	62
optimizer_switch	index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on,derived_merge=on
optimizer_trace	enabled=off,one_line=off
optimizer_trace_features	greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
optimizer_trace_limit	1
optimizer_trace_max_mem_size	16384
optimizer_trace_offset	-1
parser_max_mem_size	18446744073709551615
performance_schema	ON
performance_schema_accounts_size	-1
performance_schema_digests_size	10000
performance_schema_events_stages_history_long_size	10000
performance_schema_events_stages_history_size	10
performance_schema_events_statements_history_long_size	10000
performance_schema_events_statements_history_size	10
performance_schema_events_transactions_history_long_size	10000
performance_schema_events_transactions_history_size	10
performance_schema_events_waits_history_long_size	10000
performance_schema_events_waits_history_size	10
performance_schema_hosts_size	-1
performance_schema_max_cond_classes	80
performance_schema_max_cond_instances	-1
performance_schema_max_digest_length	1024
performance_schema_max_file_classes	80
performance_schema_max_file_handles	32768
performance_schema_max_file_instances	-1
performance_schema_max_index_stat	-1
performance_schema_max_memory_classes	320
performance_schema_max_metadata_locks	-1
performance_schema_max_mutex_classes	210
performance_schema_max_mutex_instances	-1
performance_schema_max_prepared_statements_instances	-1
performance_schema_max_program_instances	-1
performance_schema_max_rwlock_classes	50
performance_schema_max_rwlock_instances	-1
performance_schema_max_socket_classes	10
performance_schema_max_socket_instances	-1
performance_schema_max_sql_text_length	1024
performance_schema_max_stage_classes	150
performance_schema_max_statement_classes	193
performance_schema_max_statement_stack	10
performance_schema_max_table_handles	-1
performance_schema_max_table_instances	-1
performance_schema_max_table_lock_stat	-1
performance_schema_max_thread_classes	50
performance_schema_max_thread_instances	-1
performance_schema_session_connect_attrs_size	512
performance_schema_setup_actors_size	-1
performance_schema_setup_objects_size	-1
performance_schema_users_size	-1
pid_file	C:\mysql-5.7.31-winx64\data\WIN-OD0Q68NHVKC.pid
plugin_dir	C:\mysql-5.7.31-winx64\lib\plugin\
port	3309
preload_buffer_size	32768
profiling	ON
profiling_history_size	15
protocol_version	10
proxy_user	
pseudo_slave_mode	OFF
pseudo_thread_id	19691
query_alloc_block_size	8192
query_cache_limit	1048576
query_cache_min_res_unit	4096
query_cache_size	1048576
query_cache_type	OFF
query_cache_wlock_invalidate	OFF
query_prealloc_size	8192
rand_seed1	0
rand_seed2	0
range_alloc_block_size	4096
range_optimizer_max_mem_size	8388608
rbr_exec_mode	STRICT
read_buffer_size	131072
read_only	OFF
read_rnd_buffer_size	262144
relay_log	
relay_log_basename	C:\mysql-5.7.31-winx64\data\WIN-OD0Q68NHVKC-relay-bin
relay_log_index	C:\mysql-5.7.31-winx64\data\WIN-OD0Q68NHVKC-relay-bin.index
relay_log_info_file	relay-log.info
relay_log_info_repository	FILE
relay_log_purge	ON
relay_log_recovery	OFF
relay_log_space_limit	0
report_host	
report_password	
report_port	3309
report_user	
require_secure_transport	OFF
rpl_stop_slave_timeout	31536000
secure_auth	ON
secure_file_priv	NULL
server_id	1
server_id_bits	32
server_uuid	206965e9-d4b6-11ea-99d8-286ed4893053
session_track_gtids	OFF
session_track_schema	ON
session_track_state_change	OFF
session_track_system_variables	time_zone,autocommit,character_set_client,character_set_results,character_set_connection
session_track_transaction_info	OFF
sha256_password_auto_generate_rsa_keys	ON
sha256_password_private_key_path	private_key.pem
sha256_password_proxy_users	OFF
sha256_password_public_key_path	public_key.pem
shared_memory	OFF
shared_memory_base_name	MYSQL
show_compatibility_56	OFF
show_create_table_verbosity	OFF
show_old_temporals	OFF
skip_external_locking	ON
skip_name_resolve	OFF
skip_networking	OFF
skip_show_database	OFF
slave_allow_batching	OFF
slave_checkpoint_group	512
slave_checkpoint_period	300
slave_compressed_protocol	OFF
slave_exec_mode	STRICT
slave_load_tmpdir	C:\Windows\TEMP
slave_max_allowed_packet	1073741824
slave_net_timeout	60
slave_parallel_type	DATABASE
slave_parallel_workers	0
slave_pending_jobs_size_max	16777216
slave_preserve_commit_order	OFF
slave_rows_search_algorithms	TABLE_SCAN,INDEX_SCAN
slave_skip_errors	OFF
slave_sql_verify_checksum	ON
slave_transaction_retries	10
slave_type_conversions	
slow_launch_time	2
slow_query_log	OFF
slow_query_log_file	C:\mysql-5.7.31-winx64\data\WIN-OD0Q68NHVKC-slow.log
socket	MySQL
sort_buffer_size	262144
sql_auto_is_null	OFF
sql_big_selects	ON
sql_buffer_result	OFF
sql_log_bin	ON
sql_log_off	OFF
sql_mode	NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
sql_notes	ON
sql_quote_show_create	ON
sql_safe_updates	OFF
sql_select_limit	18446744073709551615
sql_slave_skip_counter	0
sql_warnings	OFF
ssl_ca	ca.pem
ssl_capath	
ssl_cert	server-cert.pem
ssl_cipher	
ssl_crl	
ssl_crlpath	
ssl_key	server-key.pem
stored_program_cache	256
super_read_only	OFF
sync_binlog	1
sync_frm	ON
sync_master_info	10000
sync_relay_log	10000
sync_relay_log_info	10000
system_time_zone	
table_definition_cache	1400
table_open_cache	2000
table_open_cache_instances	16
thread_cache_size	44
thread_handling	one-thread-per-connection
thread_stack	262144
time_format	%H:%i:%s
time_zone	SYSTEM
timestamp	1641953401.960000
tls_version	TLSv1,TLSv1.1,TLSv1.2
tmp_table_size	48234496
tmpdir	C:\Windows\TEMP
transaction_alloc_block_size	8192
transaction_allow_batching	OFF
transaction_isolation	REPEATABLE-READ
transaction_prealloc_size	4096
transaction_read_only	OFF
transaction_write_set_extraction	OFF
tx_isolation	REPEATABLE-READ
tx_read_only	OFF
unique_checks	ON
updatable_views_with_limit	YES
version	5.7.31-log
version_comment	MySQL Community Server (GPL)
version_compile_machine	x86_64
version_compile_os	Win64
wait_timeout	240
warning_count	0
java向mysql数据库批量插入大量数据
m0_67698950的博客
03-31 1万+
首先看下我们的目标：向mysql数据库中批量插入10000条数据 操作环境：Mysql和Java代码都运行在我本地Windows电脑(i7处理器，4核，16G运行内存，64位操作系统 1、JPA单线程执行 代码省略，大概需要39S左右 2、JPA多线程执行 大概需要37S左右，并没有想象中的快很多 原因： 多线程只是大大提高了程序处理数据的时间，并不会提高插入数据库的时间，相反在我这边JPA的框架下，多线程也就意味着多连接，反而更加消耗数据库性能 package com.exampl...
Mysql 插入大批量数据调优方法
Chen_sj的博客
03-02 999
使用PreparedStatement接口允许数据库预编译SQL语句，以后只需传入参数，避免了数据库每次都编译SQL语句，因此性能更好。使用多值插入SQL，SQL语句的总长度减少，即减少了网络IO，同时也降低了连接次数，数据库一次SQL解析，能够插入多条数据。多值插入SQL：INSERT INTO TBL_TEST (id) VALUES (1), (2), (3)从这里可以看出来，真正耗时的不是操作，而是链接，解析的过程。插入index （10% * Index的数目）插入操作 （10% * 词条数目）
MySQL实现批量插入以优化性能的教程
09-10
主要介绍了MySQL实现批量插入以优化性能的教程,文中给出了运行时间来表示性能优化后的对比,需要的朋友可以参考下
Ubuntu用源方式安装MySQL的默认配置信息
weixin_34195142的博客
02-10 218
2019独角兽企业重金招聘Python工程师标准>>> ...
MySQL批量SQL插入性能优化
qq_41587243的博客
10-22 166
 对于一些数据量较大的系统，数据库面临的问题除了查询效率低下，还有就是数据入库时间长。特别像报表系统，可能每天花费在数据导入上的时间就会长达几个小时之久。因此，优化数据库插入性能是很有意义的。        网络上的牛人很多，总会有一些手段可以提高insert效率，大家跟我一起分享一下吧：        1. 一条SQL语句插入多条数据。        我们常用的插入语句大都是一条一个inse...
mysql批量插入优化
touhousonic的博客
09-22 515
Connection conn = DBUtils.getConn(); conn.setAutoCommit(false);//设置自动提交为false，否则无法运行 String sql = “insert into t_detail values(?,?,?,?,?,?)”; int a = 0; int q = 0;//
mysql 批量插入优化
qq_32703495的博客
09-19 230
首先在资源文件jdbc.properties文件中url后添加： ?rewriteBatchedStatements=true 功能为允许MySQL批量插入 测试方法： @Test public void testInsert3() throws SQLException { ConnectionClass connectionClass = new Connection...
Node.js下向MySQL数据库插入批量数据的方法
09-09
在Node.js环境中，当需要向MySQL数据库批量插入数据时，可以显著提高效率并减少与数据库的交互次数。本文将详细讲解如何实现这一功能，并提供一个实际的示例代码。 首先，确保已经安装了`mysql`模块，它是Node.js中...
MYSQL开发性能研究之批量插入数据的优化方法
12-16
本文主要探讨了在MySQL中如何通过批量插入优化性能，对比了几种不同的批量插入方法，并进行了实验验证。 首先，传统的单条插入方式在面对大量数据时效率低下，因为每条插入语句都需要经过解析、优化和执行等多个...
如何优化MySQL的批量插入操作？
最新发布
长风破浪会有时的博客
10-15 332
请注意，在尝试任何优化之前，最好先备份你的数据，并在一个安全的环境中测试这些更改。每个数据库和应用程序都是独特的，因此最佳实践可能因情况而异。在实施任何更改之前，请确保你了解它们对你的特定环境的影响。
mysql大量插入优化_MySQL批量SQL插入性能优化1
weixin_39586353的博客
01-19 140
对于一些数据量较大的系统，数据库面临的问题除了查询效率低下，还有就是数据入库时间长。特别像报表系统，每天花费在数据导入上的时间可能会长达几个小时或十几个小时之久。因此，优化数据库插入性能是很有意义的。经过对MySQL innodb的一些性能测试，发现一些可以提高insert效率的方法，供大家参考参考。1. 一条SQL语句插入多条数据。常用的插入语句如：1234INSERTINTO`insert_t...
MySQL插入大量数据调优
Yuhao的专栏
07-09 2449
1. SQL语句级别调优： > 使用存储过程：(一百万条数据分25万次循环，每次循环插入4条数据）  如果同时从同一个客户端插入很多行，使用含多个VALUE的INSERT语句同时插入几行。这比使用单行INSERT语句快(在某些情况下快几倍)。 DELIMITER $$ CREATE  PROCEDURE Pro_InsertMillionRecords() BEGIN    DECLA
MySQL批量插入优化
feinifi的博客
04-28 2414
前面介绍了MySQL批量插入可以通过存储过程的方式来实现，这里介绍批量插入100W记录，并做一个优化。 建表语句：create_table.sql drop table if exists xx_user; create table xx_user( id int primary key auto_increment, name varchar(20), age int); 直接插入：pr...
MySQL批量插入数据优化
致我那些年搬过的砖
03-20 410
http://www.jb51.net/article/108017.htm
mysql批量插入数据优化
weixin_30341745的博客
04-09 66
一.问题 很早以前做了一个更新功能，就是将A表中的数据全部查找出来，相对B表中改变的数据更新B表，B表中没有的数据插入B表。 最近发现该功能执行速率减慢，有时还跑超时。原来是A表中数据渐渐变多，就有了这个问题。 二.优化知识 1.循环每条插入改为一条语句插入多条。 即使用insert into table () values (),(),()...
c mysql批量插入优化_mysql批量插入优化
weixin_34858011的博客
01-19 129
这段时间一直在参与产品库的设计和实现，中间和mysql的恩恩怨怨给广大喜欢交流学习的网友们，提供一些借鉴的机会。首先从mysql的批量插入开始吧。一直自认为对sql语句的数量使用，完全绝对的低估了现实问题的难度。100w的产品基础数据插入用掉了10个小时的时间。很挫…第一批实验数据100w插入后，让我久久不能释怀，这10个小时让我很纠结。找原因吧，之前先入为主，一颗天真烂漫的心被一篇jdbc批处理...
MySQL批量SQL插入各种性能优化
aodiyi6351的博客
07-04 121
对于一些数据量较大的系统。数据库面临的问题除了查询效率低下，还有就是数据入库时间长。特别像报表系统，每天花费在数据导入上的时间可能会长达几个小时或十几个小时之久。因此。优化数据库插入性能是非常有意义的。 经过对MySQL innodb的一些性能測试，发现一些能够提高insert效率的方法。供大家參考參考。 1、一条SQL语句插入多条数据。 经常使用的插入语句...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

潇凝子潇

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值