Kafka配置SSL加密传输

Kafka配置SSL加密传输

一、证书配置

1、生成证书

如我输入命令如下：依次是 密码—重输密码—名与姓—组织单位—组织名—城市—省份—国家两位代码—密码—重输密码，后面告警不用管，此步骤要注意的是，名与姓这一项必须输入域名，如 “localhost”,切记不可以随意写，我曾尝试使用其他字符串，在后面客户端生成证书认证的时候一直有问题。

keytool -keystore server.keystore.jks -alias localhost -validity 3650 -genkey
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]:  localhost
What is the name of your organizational unit?
[Unknown]:  CH-kafka
What is the name of your organization?
[Unknown]:  kafkadev
What is the name of your City or Locality?
[Unknown]:  shanghai
What is the name of your State or Province?
[Unknown]:  shanghai
What is the two-letter country code for this unit?
[Unknown]:  CH
Is CN=localhost, OU=CH-kafka, O=kafkadev, L=shanghai, ST=shanghai, C=CH correct?
[no]:  yes
Enter key password for <localhost>
	(RETURN if same as keystore password):  
Re-enter new password: 
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore server.keystore.jks -destkeystore server.keystore.jks -deststoretype pkcs12".

完成上面步骤，可使用命令 keytool -list -v -keystore server.keystore.jks 来验证生成证书的内容

2、生成CA

通过第一步，集群中的每台机器都生成一对公私钥，和一个证书来识别机器。但是，证书是未签名的，这意味着攻击者可以创建一个这样的证书来伪装成任何机器。

因此，通过对集群中的每台机器进行签名来防止伪造的证书。证书颁发机构（CA）负责签名证书。CA的工作机制像一个颁发护照的政府。政府印章（标志）每本护照，这样护照很难伪造。其他政府核实护照的印章，以确保护照是真实的。同样，CA签名的证书和加密保证签名证书很难伪造。因此，只要CA是一个真正和值得信赖的权威，client就能有较高的保障连接的是真正的机器。如下，生成的CA是一个简单的公私钥对和证书，用于签名其他的证书，下面为输入命令，依次提示输入为 密码—重输密码—国家两位代码—省份—城市—名与姓—组织名—组织单位—名与姓(域名)—邮箱 ，此输入步骤与上面生成证书世输入步骤相反，输入值要与第一步一致，邮箱可不输入

openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650 Generating a 2048 bit RSA private key .........................................................................+++ ..................+++ writing new private key to 'ca-key' Enter PEM pass phrase: Verifying - Enter PEM pass phrase: ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:CH State or Province Name (full name) []:shanghai Locality Name (eg, city) [Default City]:shanghai Organization Name (eg, company) [Default Company Ltd]:kafkadev Organizational Unit Name (eg, section) []:CH-kafka Common Name (eg, your name or your server's hostname) []:localhost Email Address []: 

将生成的CA添加到**clients' truststore（客户的信任库）**，以便client可以信任这个CA:

keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert 

3、签名证书

用步骤2生成的CA来签名所有步骤1生成的证书，首先，你需要从密钥仓库导出证书：

keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file 

然后用CA签名：{validity}，{ca-password} 两个为参数，

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}

最后,你需要导入CA的证书和已签名的证书到密钥仓库:

keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed 

上文中的各参数解释如下：

keystore: 密钥仓库的位置

ca-cert: CA的证书

ca-key: CA的私钥

ca-password: CA的密码

cert-file: 出口，服务器的未签名证书

cert-signed: 已签名的服务器证书

上面步骤所有执行脚本如下：注意密码修改为自己的密码,以防混淆，所有步骤密码最好设为同一个

#!/bin/bash
#Step 1
keytool -keystore server.keystore.jks -alias localhost -validity 3650 -keyalg RSA -genkey
#Step 2
openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
#Step 3
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3650 -CAcreateserial -passin pass:123456
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
#Step 4
keytool -keystore client.keystore.jks -alias localhost -validity 3650 -keyalg RSA -genkey

4、配置kafka broker（server.properteis）

通过server.properteis 配置，最少监听1个端口，用逗号分隔

如果broker之间通讯未启用SSL（参照下面，启动它），PLAINTEXT和SSL端口是必须要配置，这里我使用了9093为ssl端口，如果想要broker内部通信也使用ssl 要配置 security.inter.broker.protocol=SSL（不建议，kakfa内部通信使用ssl可能影响速度）

ssl.endpoint.identification.algorithm=  设置为空可使证书的主机名与kafka的主机名不用保持一致

listeners=PLAINTEXT://172.17.0.53:9092,SSL://172.17.0.53:9093 ssl.keystore.location=/usr/local/kafka_2.12-2.8.0/ssl/server.keystore.jks ssl.keystore.password=clearwater001! ssl.key.password=clearwater001! ssl.truststore.location=/usr/local/kafka_2.12-2.8.0/ssl/server.truststore.jks ssl.truststore.password=clearwater001! ssl.endpoint.identification.algorithm= 

5、配置Kafka客户端，创建client-ssl.properties 文件，加密传输时需要指定该文件，内容如下

security.protocol=SSL ssl.endpoint.identification.algorithm= group.id=test ssl.truststore.location=/usr/local/kafka_2.12-2.8.0/ssl/client.truststore.jks ssl.truststore.password=clearwater001! 

二、启动命令

1、启动zookeeper

/usr/local/zookeeper/apache-zookeeper-3.5.9-bin/bin/zkServer.sh start 

2、启动kafka-server

./kafka-server-start.sh  -daemon /usr/local/kafka_2.12-2.8.0/config/server.properties  

3、创建topic

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test #查看topic list ./kafka-topics.sh --list --zookeeper localhost:2181 

4、加密前生产消费消息（一般使用新版本命令）

###生产消息###
#老版本
./kafka-console-producer.sh --broker-list localhost:9092 --topic test
#新版本
./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test

###消费消息###
#老版本
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
#新版本
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

5、加密后生产消费消息命令

#生产消息 
./kafka-console-producer.sh --bootstrap-server 172.17.0.53:9093 --topic test --producer.config client-ssl.properties 

#消费消息
./kafka-console-consumer.sh --bootstrap-server 172.17.0.53:9093 --topic test --from-beginning --consumer.config client-ssl.properties

三、SpringBoot集成Kafka

1、导入依赖

<!--引入kafka依赖--> <dependency>     <groupId>org.springframework.kafka</groupId>     <artifactId>spring-kafka</artifactId> </dependency> 

2、配置文件

kafka:
  consumer:
    servers: 172.17.0.53:9093
    topic: test
    enable:
      auto:
        commit: true
    auto:
      commit:
        interval: 100
      offset:
        reset: latest
    group:
      id: test
    concurrency: 3

3、Kafka配置类

/**
 * @describe kafka消费者配置
 * @Author wks
 * @Date 2021/6/30 11:39
*/


@Configuration
@EnableKafka
public class KafkaConsumerConfig {

​    @Value("${kafka.consumer.servers}")
​    private String servers;
​    @Value("${kafka.consumer.enable.auto.commit}")
​    private boolean enableAutoCommit;
​    @Value("${kafka.consumer.auto.commit.interval}")
​    private String autoCommitInterval;
​    @Value("${kafka.consumer.group.id}")
​    private String groupId;
​    @Value("${kafka.consumer.auto.offset.reset}")
​    private String autoOffsetReset;
​    @Value("${kafka.consumer.concurrency}")
​    private int concurrency;

​    @Bean
​    public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
​        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
​        factory.setConsumerFactory(consumerFactory());
​        factory.setConcurrency(concurrency);
​        factory.getContainerProperties().setPollTimeout(1500);
​        return factory;
​    }

​    public ConsumerFactory<String, String> consumerFactory() {
​        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
​    }


​    public Map<String, Object> consumerConfigs() {
​        Map<String, Object> propsMap = new HashMap<>();
​        propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
​        propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, enableAutoCommit);
​        propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, autoCommitInterval);
​        propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
​        propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
​        propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
​        propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetReset);
​        propsMap.put("security.protocol","SSL");
​        propsMap.put("ssl.truststore.location","D:/client.truststore.jks");
​        propsMap.put("ssl.truststore.password","clearwater001!");
​        return propsMap;
​    }
/*
*
​     * kafka监听
​          * @return
*/


​    @Bean
​    public RawDataListener listener() {
​        return new RawDataListener();
​    }


}

4、配置Kafka监听

package com.xlkh.appsec.controller.kafka;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.log4j.Logger;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

import java.io.IOException;

/**
 * @describe kafka监听
 * @Author wks
 * @Date 2021/6/30 11:43

*/

@Component
public class RawDataListener {
    Logger logger=Logger.getLogger(RawDataListener.class);

/**
     * 实时获取kafka数据(生产一条，监听生产topic自动消费一条)
          * @param record
          * @throws IOException

*/

​    @KafkaListener(topics = {"${kafka.consumer.topic}"})
​    public void listen(ConsumerRecord<?, ?> record) throws IOException {
​        String value = (String) record.value();
​        System.out.println(value);
​    }

}