elasticsearch集群bulk操作测试

最新推荐文章于 2023-08-04 11:34:07 发布

Chauncy00

最新推荐文章于 2023-08-04 11:34:07 发布

阅读量1.1k

点赞数

分类专栏：闲来无事文章标签： elasticsearch bulk Java

本文链接：https://blog.csdn.net/qq_37707251/article/details/104928931

版权

闲来无事专栏收录该内容

6 篇文章 0 订阅

订阅专栏

服务器信息：

(虚拟机) * 3

cpu:

-------------------------------------------------------

总核数 = 物理CPU个数 X 每颗物理CPU的核数

总逻辑CPU数 = 物理CPU个数 X 每颗物理CPU的核数 X 超线程数

-------------------------------------------------------

物理CPU个数

8

-------------------------------------------------------

查看每个物理CPU中core的个数(即核数)

cpu cores : 1

-------------------------------------------------------

查看逻辑CPU的个数

8

-------------------------------------------------------

查看CPU信息（型号）

8 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz

------------------------------------------------------

free:

total used free shared buff/cache available

Mem: 15G 2.3G 10G 9.0M 3.2G 12G

Swap: 0B 0B 0B

disk:

机械磁盘西部数据紫盘5400

数据来源Kafka

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID

first_floor 3 172554612 268659420 96104808 - - -

first_floor 4 172468166 268664997 96196831 - - -

first_floor 5 170698697 266998946 96300249 - - -

first_floor 6 169190488 265599032 96408544 - - -

first_floor 0 182175112 278532188 96357076 - - -

first_floor 1 43120790 270504406 227383616 - - -

first_floor 2 174732636 270500824 95768188 - - -

测试一：

elasticsearch 存储 2w/s

开始时间	结束时间	历时(s)	总数据	平均值(/s)
		112	2471491	22000

index：

分片数5 无备份 refreshInterval = "10s"

JAVA程序Consumer配置：

@Bean("kafkaListenerEsContainerFactory")
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String,String>> kafkaListenerEsContainerFactory(){
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();

    factory.setConsumerFactory(esConsumerFactory());
    // 并发数量 等于消费者数量
    factory.setConcurrency(7);
    // 批量获取
    factory.setBatchListener(true);
    factory.getContainerProperties().setPollTimeout(3000);
    factory.setAutoStartup(true);
    return factory;

}

public ConsumerFactory<String, String> esConsumerFactory() {
    return new DefaultKafkaConsumerFactory<>(esConsumerConfigs());
}

public Map<String, Object> esConsumerConfigs() {
    Map<String, Object> propsMap = new HashMap<>();
    propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
    propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, enableAutoCommit);
    propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, autoCommitInterval);
    propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class);
    propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, taskGroupId);
    propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetReset);
    //最多批量获取15000个
    propsMap.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,15000);
    return propsMap;
}

@Bean
public KafkaProperties.Listener listener() {
    return new KafkaProperties.Listener();
}

JAVA程序BulkProcessor配置：

@Bean
    public BulkProcessor bulkProcessor() throws UnknownHostException {
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY,new UsernamePasswordCredentials("elastic","Waner123456"));
        Settings settings = Settings.builder()
                .put("cluster.name", "waner-es-cluster")
                .put("client.transport.sniff",true)
                .build();
        TransportClient transportClient = new PreBuiltTransportClient(settings);
//        TransportAddress transportAddress0 = new TransportAddress(Inet4Address.getByName("172.168.1.210"), 9300);
//        TransportAddress transportAddress1 = new TransportAddress(Inet4Address.getByName("172.168.1.211"), 9300);
        TransportAddress transportAddress2 = new TransportAddress(Inet4Address.getByName("172.168.1.212"), 9300);
        TransportAddress transportAddress3 = new TransportAddress(Inet4Address.getByName("172.168.1.213"), 9300);
        TransportAddress transportAddress4 = new TransportAddress(Inet4Address.getByName("172.168.1.214"), 9300);
        transportClient.addTransportAddresses(transportAddress2,transportAddress3,transportAddress4);
        final Long[] start = {0L};
        return BulkProcessor.builder(transportClient, new BulkProcessor.Listener() {
            @Override
            public void beforeBulk(long executionId, BulkRequest request) {
                start[0] = System.currentTimeMillis();
                LOGGER.info("批量执行 [{}] with {} 请求 {}", executionId, executionId,start);
            }
            @Override
            public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
                if(response.hasFailures()){
                    LOGGER.error("Bulk [{}] 执行失败,响应 = {}", executionId, response.buildFailureMessage());
                } else {
                    long end = System.currentTimeMillis();
                    LOGGER.info("Bulk [{}] 请求数据 {} 完成于 {} milliseconds 用时 {}", executionId,request.numberOfActions(), response.getTook().getMillis(),(end-start[0]));
                }
                BulkItemResponse[] responses = response.getItems();

            }
            @Override
            public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
                LOGGER.error("{} 批量数据失败, 原因 :{}", request.numberOfActions(), failure);
            }
        }).setBulkActions(20000)  //每2w条执行一次bulk插入
                .setBulkSize(new ByteSizeValue(17, ByteSizeUnit.MB)) //  数据量达到17M后执行bulk插入
                .setFlushInterval(TimeValue.timeValueSeconds(20)) //无论数据量多少，间隔20s执行一次bulk
                .setConcurrentRequests(10) // 允许并发的bulk请求数
                .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(100), 3))
                .build();
    }

性能监控：

heap偶尔偏高 cpu load一切正常

测试二：

elasticsearch 存储 1.9w/s

开始时间	结束时间	历时(s)	总数据	平均值(/s)
1575341482800	1575342496978	172	3437795	19987.18

index：

分片数5 无备份 refreshInterval = "10s"

JAVA程序Consumer配置：

@Bean("kafkaListenerEsContainerFactory")
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String,String>> kafkaListenerEsContainerFactory(){
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();

    factory.setConsumerFactory(esConsumerFactory());
    // 并发数量 等于消费者数量
    factory.setConcurrency(7);
    // 批量获取
    factory.setBatchListener(true);
    factory.getContainerProperties().setPollTimeout(3000);
    factory.setAutoStartup(true);
    return factory;

}

public ConsumerFactory<String, String> esConsumerFactory() {
    return new DefaultKafkaConsumerFactory<>(esConsumerConfigs());
}

public Map<String, Object> esConsumerConfigs() {
    Map<String, Object> propsMap = new HashMap<>();
    propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
    propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, enableAutoCommit);
    propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, autoCommitInterval);
    propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class);
    propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, taskGroupId);
    propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetReset);
    //最多批量获取20000个
    propsMap.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,20000);
    return propsMap;
}

@Bean
public KafkaProperties.Listener listener() {
    return new KafkaProperties.Listener();
}

JAVA程序BulkProcessor配置：

@Bean
    public BulkProcessor bulkProcessor() throws UnknownHostException {
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY,new UsernamePasswordCredentials("elastic","Waner123456"));
        Settings settings = Settings.builder()
                .put("cluster.name", "waner-es-cluster")
                .put("client.transport.sniff",true)
                .build();
        TransportClient transportClient = new PreBuiltTransportClient(settings);
//        TransportAddress transportAddress0 = new TransportAddress(Inet4Address.getByName("172.168.1.210"), 9300);
//        TransportAddress transportAddress1 = new TransportAddress(Inet4Address.getByName("172.168.1.211"), 9300);
        TransportAddress transportAddress2 = new TransportAddress(Inet4Address.getByName("172.168.1.212"), 9300);
        TransportAddress transportAddress3 = new TransportAddress(Inet4Address.getByName("172.168.1.213"), 9300);
        TransportAddress transportAddress4 = new TransportAddress(Inet4Address.getByName("172.168.1.214"), 9300);
        transportClient.addTransportAddresses(transportAddress2,transportAddress3,transportAddress4);
        final Long[] start = {0L};
        return BulkProcessor.builder(transportClient, new BulkProcessor.Listener() {
            @Override
            public void beforeBulk(long executionId, BulkRequest request) {
                start[0] = System.currentTimeMillis();
                LOGGER.info("批量执行 [{}] with {} 请求 {}", executionId, executionId,start);
            }
            @Override
            public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
                if(response.hasFailures()){
                    LOGGER.error("Bulk [{}] 执行失败,响应 = {}", executionId, response.buildFailureMessage());
                } else {
                    long end = System.currentTimeMillis();
                    LOGGER.info("Bulk [{}] 请求数据 {} 完成于 {} milliseconds 用时 {}", executionId,request.numberOfActions(), response.getTook().getMillis(),(end-start[0]));
                }
                BulkItemResponse[] responses = response.getItems();

            }
            @Override
            public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
                LOGGER.error("{} 批量数据失败, 原因 :{}", request.numberOfActions(), failure);
            }
        }).setBulkActions(30000)  //每2w条执行一次bulk插入
                .setBulkSize(new ByteSizeValue(20, ByteSizeUnit.MB)) //  数据量达到20M后执行bulk插入
                .setFlushInterval(TimeValue.timeValueSeconds(20)) //无论数据量多少，间隔20s执行一次bulk
                .setConcurrentRequests(10) // 允许并发的bulk请求数
                .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(100), 3))
                .build();

以上实际每次插入数据峰值为17300/bulk，可以适当上调bulk_size大小

性能监控：

heap偶尔偏高 cpu load一切正常

Chauncy00

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
elasticsearch集群bulk操作测试

服务器信息：(虚拟机) * 3cpu:-------------------------------------------------------总核数 = 物理CPU个数 X 每颗物理CPU的核数总逻辑CPU数 = 物理CPU个数 X 每颗物理CPU的核数 X 超线程数--------------------------------------------------...
复制链接

扫一扫