go 通过SSL 双向认证在kafka上生产和消费

最新推荐文章于 2024-04-16 15:58:13 发布

大彩笔_祥哥

最新推荐文章于 2024-04-16 15:58:13 发布

阅读量2.5k

点赞数 8

分类专栏： Go开发文章标签： kafka SSL 通讯

本文链接：https://blog.csdn.net/weixin_36771703/article/details/88762831

版权

Go开发专栏收录该内容

4 篇文章 0 订阅

订阅专栏

一、背景
做项目有个需求：kafka使用SSL加密连接，限制客户端访问，减轻服务端的压力,项目也具有安全性,这就需要给客户端发证书，只允许持有证书的客户端访问。
二、实现思路
1.在实现的时候参考了很多的帖子,java版的实现很多,go实现的目前只找到一篇,或许是其它的每能及时发现,在这个过程中我遇到很多的坑,说多了就是累,希望给有相同需求的人少走点弯路!!
根据其它的帖子实现的思路是这样的:
思路1: 在这里插入图片描述
思路2:

大同小异,我的思路是这样的:
1.验证kafka能正常消费
这个是前提!!!
2.制作证书
a.伪CA
<1>生成秘钥库
<2>根据秘钥库生成证书请求
<3>签名
b.服务端
<1>生成秘钥库
<2>根据秘钥库生成证书请求
<3>签名
<4>ca 签名证书加入信任库
c.客户端
<1>生成秘钥库
<2>根据秘钥库生成证书请求
<3>签名
<4>ca 签名证书加入信任库
2.分发证书修改kafka配置
3.重启kafka验证
4.使用程序连接验证
keystore用来存放key，truststore用来存放受信任的证书，jks是常见证书的一种格式

三、实现方法
3.1 制作证书

#!/usr/bin/bash

set -e

KEYSTORE_FILENAME="kafka.keystore.jks"
VALIDITY_IN_DAYS=3650
DEFAULT_TRUSTSTORE_FILENAME="kafka.truststore.jks"
TRUSTSTORE_WORKING_DIRECTORY="truststore"
KEYSTORE_WORKING_DIRECTORY="keystore"
CA_CERT_FILE="ca-cert"
KEYSTORE_SIGN_REQUEST="cert-file"
KEYSTORE_SIGN_REQUEST_SRL="ca-cert.srl"
KEYSTORE_SIGNED_CERT="cert-signed"

function file_exists_and_exit() {
  echo "'$1' cannot exist. Move or delete it before"
  echo "re-running this script."
  exit 1
}

if [ -e "$KEYSTORE_WORKING_DIRECTORY" ]; then
  file_exists_and_exit $KEYSTORE_WORKING_DIRECTORY
fi

if [ -e "$CA_CERT_FILE" ]; then
  file_exists_and_exit $CA_CERT_FILE
fi

if [ -e "$KEYSTORE_SIGN_REQUEST" ]; then
  file_exists_and_exit $KEYSTORE_SIGN_REQUEST
fi

if [ -e "$KEYSTORE_SIGN_REQUEST_SRL" ]; then
  file_exists_and_exit $KEYSTORE_SIGN_REQUEST_SRL
fi

if [ -e "$KEYSTORE_SIGNED_CERT" ]; then
  file_exists_and_exit $KEYSTORE_SIGNED_CERT
fi

echo
echo "Welcome to the Kafka SSL keystore and trusttore generator script."

echo
echo "First, do you need to generate a trust store and associated private key,"
echo "or do you already have a trust store file and private key?"
echo
echo -n "Do you need to generate a trust store and associated private key? [yn] "
read generate_trust_store

trust_store_file=""
trust_store_private_key_file=""

if [ "$generate_trust_store" == "y" ]; then
  if [ -e "$TRUSTSTORE_WORKING_DIRECTORY" ]; then
    file_exists_and_exit $TRUSTSTORE_WORKING_DIRECTORY
  fi

  mkdir $TRUSTSTORE_WORKING_DIRECTORY
  echo
  echo "OK, we'll generate a trust store and associated private key."
  echo
  echo "First, the private key."
  echo
  echo "You will be prompted for:"
  echo " - A password for the private key. Remember this."
  echo " - Information about you and your company."
  echo " - NOTE that the Common Name (CN) is currently not important."

  openssl req -new -x509 -keyout $TRUSTSTORE_WORKING_DIRECTORY/ca-key \
    -out $TRUSTSTORE_WORKING_DIRECTORY/ca-cert -days $VALIDITY_IN_DAYS

  trust_store_private_key_file="$TRUSTSTORE_WORKING_DIRECTORY/ca-key"

  echo
  echo "Two files were created:"
  echo " - $TRUSTSTORE_WORKING_DIRECTORY/ca-key -- the private key used later to"
  echo "   sign certificates"
  echo " - $TRUSTSTORE_WORKING_DIRECTORY/ca-cert -- the certificate that will be"
  echo "   stored in the trust store in a moment and serve as the certificate"
  echo "   authority (CA). Once this certificate has been stored in the trust"
  echo "   store, it will be deleted. It can be retrieved from the trust store via:"
  echo "   $ keytool -keystore <trust-store-file> -export -alias CARoot -rfc"

  echo
  echo "Now the trust store will be generated from the certificate."
  echo
  echo "You will be prompted for:"
  echo " - the trust store's password (labeled 'keystore'). Remember this"
  echo " - a confirmation that you want to import the certificate"

  keytool -keystore $TRUSTSTORE_WORKING_DIRECTORY/$DEFAULT_TRUSTSTORE_FILENAME \
    -alias CARoot -import -file $TRUSTSTORE_WORKING_DIRECTORY/ca-cert

  trust_store_file="$TRUSTSTORE_WORKING_DIRECTORY/$DEFAULT_TRUSTSTORE_FILENAME"

  echo
  echo "$TRUSTSTORE_WORKING_DIRECTORY/$DEFAULT_TRUSTSTORE_FILENAME was created."

  # don't need the cert because it's in the trust store.
  rm $TRUSTSTORE_WORKING_DIRECTORY/$CA_CERT_FILE
else
  echo
  echo -n "Enter the path of the trust store file. "
  read -e trust_store_file

  if ! [ -f $trust_store_file ]; then
    echo "$trust_store_file isn't a file. Exiting."
    exit 1
  fi

  echo -n "Enter the path of the trust store's private key. "
  read -e trust_store_private_key_file

  if ! [ -f $trust_store_private_key_file ]; then
    echo "$trust_store_private_key_file isn't a file. Exiting."
    exit 1
  fi
fi

echo
echo "Continuing with:"
echo " - trust store file:        $trust_store_file"
echo " - trust store private key: $trust_store_private_key_file"

mkdir $KEYSTORE_WORKING_DIRECTORY

echo
echo "Now, a keystore will be generated. Each broker and logical client needs its own"
echo "keystore. This script will create only one keystore. Run this script multiple"
echo "times for multiple keystores."
echo
echo "You will be prompted for the following:"
echo " - A keystore password. Remember it."
echo " - Personal information, such as your name."
echo "     NOTE: currently in Kafka, the Common Name (CN) does not need to be the FQDN of"
echo "           this host. However, at some point, this may change. As such, make the CN"
echo "           the FQDN. Some operating systems call the CN prompt 'first / last name'"
echo " - A key password, for the key being generated within the keystore. Remember this."

# To learn more about CNs and FQDNs, read:
# https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/X509ExtendedTrustManager.html

keytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME \
  -alias localhost -validity $VALIDITY_IN_DAYS -genkey -keyalg RSA

echo
echo "'$KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME' now contains a key pair and a"
echo "self-signed certificate. Again, this keystore can only be used for one broker or"
echo "one logical client. Other brokers or clients need to generate their own keystores."

echo
echo "Fetching the certificate from the trust store and storing in $CA_CERT_FILE."
echo
echo "You will be prompted for the trust store's password (labeled 'keystore')"

keytool -keystore $trust_store_file -export -alias CARoot -rfc -file $CA_CERT_FILE

echo
echo "Now a certificate signing request will be made to the keystore."
echo
echo "You will be prompted for the keystore's password."
keytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME -alias localhost \
  -certreq -file $KEYSTORE_SIGN_REQUEST

echo
echo "Now the trust store's private key (CA) will sign the keystore's certificate."
echo
echo "You will be prompted for the trust store's private key password."
openssl x509 -req -CA $CA_CERT_FILE -CAkey $trust_store_private_key_file \
  -in $KEYSTORE_SIGN_REQUEST -out $KEYSTORE_SIGNED_CERT \
  -days $VALIDITY_IN_DAYS -CAcreateserial
# creates $KEYSTORE_SIGN_REQUEST_SRL which is never used or needed.

echo
echo "Now the CA will be imported into the keystore."
echo
echo "You will be prompted for the keystore's password and a confirmation that you want to"
echo "import the certificate."
keytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME -alias CARoot \
  -import -file $CA_CERT_FILE
rm $CA_CERT_FILE # delete the trust store cert because it's stored in the trust store.

echo
echo "Now the keystore's signed certificate will be imported back into the keystore."
echo
echo "You will be prompted for the keystore's password."
keytool -keystore $KEYSTORE_WORKING_DIRECTORY/$KEYSTORE_FILENAME -alias localhost -import \
  -file $KEYSTORE_SIGNED_CERT

echo
echo "All done!"
echo
echo "Delete intermediate files? They are:"
echo " - '$KEYSTORE_SIGN_REQUEST_SRL': CA serial number"
echo " - '$KEYSTORE_SIGN_REQUEST': the keystore's certificate signing request"
echo "   (that was fulfilled)"
echo " - '$KEYSTORE_SIGNED_CERT': the keystore's certificate, signed by the CA, and stored back"
echo "    into the keystore"
echo -n "Delete? [yn] "
read delete_intermediate_files

if [ "$delete_intermediate_files" == "y" ]; then
  rm $KEYSTORE_SIGN_REQUEST_SRL
  rm $KEYSTORE_SIGN_REQUEST
  rm $KEYSTORE_SIGNED_CERT
fi
keytool -importkeystore -srckeystore ./truststore/kafka.truststore.jks -destkeystore server.p12 -deststoretype PKCS12
openssl pkcs12 -in server.p12 -nokeys -out server.cer.pem
keytool -importkeystore -srckeystore ./kafka.keystore.jks -destkeystore client.p12 -deststoretype PKCS12
openssl pkcs12 -in client.p12 -nokeys -out client.cer.pem
openssl pkcs12 -in client.p12 -nodes -nocerts -out client.key.pem

这时候会出现几个文件
server.cer.pem client.cer.pem client.key.pem 这几个是客户端要用的
kafka.truststore.jks kafka.keystore.jks 服务端要用

3.2 修改配置
在 config/server.properties 文件里面修改如下项：
listeners如果只支持SSL的话，就需要把 security.inter.broker.protocol 也配置为 SSL，即内部交流方式也为SSL

listeners=SSL://IP:9092
ssl.keystore.location=/opt/kafka/cert/kafka.keystore.jks
ssl.keystore.password=123456
ssl.key.password=123456
ssl.truststore.location=/opt/kafka/cert/kafka.truststore.jks
ssl.truststore.password=123456
ssl.client.auth=required 
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS
security.inter.broker.protocol=SSL

3.3 启动kafka
启动过程中不报错基本就是启动成功
3.4 服务验证
修改配置
producer.properties

security.protocol=SSL
ssl.keystore.location=/opt/kafka/cert/kafka.keystore.jks
ssl.keystore.password=123456
ssl.key.password=123456
ssl.truststore.location=/opt/kafka/cert/kafka.truststore.jks
ssl.truststore.password=123456

启动生成和消费

bin/kafka-console-producer.sh --broker-list  IP:9092 --topic test--producer.config  ./config/producer.properties
bin/kafka-console-consumer.sh --bootstrap-server  IP:9092 --topic test -new-consumer --consumer.config  config/producer.properties

验证成功后,用代码连接
把生成的server.cer.pem client.cer.pem client.key.pem 给客户端使用

package main

import (
"crypto/tls"
"crypto/x509"
"io/ioutil"
"log"
"os"
"os/signal"
"sync"
"github.com/Shopify/sarama"
)

func main() {
	tlsConfig, err := NewTLSConfig("E:/demo/kafka/client.cer.pem",
		"E:/demo/kafka/client.key.pem",
		"E:/demo/kafka/server.cer.pem")
	if err != nil {
		log.Fatal(err)
	}

	//客户端不对服务端验证
	tlsConfig.InsecureSkipVerify = true

	//创建一个配置对象
	consumerConfig := sarama.NewConfig()
	//启动TLS通讯
	consumerConfig.Net.TLS.Enable = true
	consumerConfig.Net.TLS.Config = tlsConfig

	//创建一个客户消费者
	client, err := sarama.NewClient([]string{"172.16.4.165:16667"}, consumerConfig)
	if err != nil {
		log.Fatalf("unable to create kafka client: %q", err)
	}

	consumer, err := sarama.NewConsumerFromClient(client)
	if err != nil {
		log.Fatal(err)
	}
	defer consumer.Close()

	consumerLoop(consumer, "zhang")
}

func NewTLSConfig(clientCertFile, clientKeyFile, caCertFile string) (*tls.Config, error) {
	tlsConfig := tls.Config{}

	// 加载客户端证书
	cert, err := tls.LoadX509KeyPair(clientCertFile, clientKeyFile)
	if err != nil {
		return &tlsConfig, err
	}
	tlsConfig.Certificates = []tls.Certificate{cert}

	// 加载CA证书
	caCert, err := ioutil.ReadFile(caCertFile)
	if err != nil {
		return &tlsConfig, err
	}
	caCertPool := x509.NewCertPool()
	caCertPool.AppendCertsFromPEM(caCert)
	tlsConfig.RootCAs = caCertPool

	tlsConfig.BuildNameToCertificate()
	return &tlsConfig, err
}

func consumerLoop(consumer sarama.Consumer, topic string) {
	partitions, err := consumer.Partitions(topic)
	if err != nil {
		log.Println("unable to fetch partition IDs for the topic", topic, err)
		return
	}

	signals := make(chan os.Signal, 1)
	signal.Notify(signals, os.Interrupt)

	var wg sync.WaitGroup
	for partition := range partitions {
		wg.Add(1)
		go func() {
			consumePartition(consumer, int32(partition), signals)
			wg.Done()
		}()
	}
	wg.Wait()
}

func consumePartition(consumer sarama.Consumer, partition int32, signals chan os.Signal) {
	log.Println("Receving on partition", partition)
	//partitionConsumer, err := consumer.ConsumePartition("zhang", partition, sarama.OffsetNewest)
	partitionConsumer, err := consumer.ConsumePartition("wang", partition, sarama.OffsetOldest)
	if err != nil {
		log.Println(err)
		return
	}
	defer func() {
		if err := partitionConsumer.Close(); err != nil {
			log.Println(err)
		}
	}()

	consumed := 0
ConsumerLoop:
	for {
		select {
		case msg := <-partitionConsumer.Messages():
			log.Printf("Consumed message offset %d\nData: %s\n", msg.Offset, msg.Value)
			consumed++
		case <-signals:
			break ConsumerLoop
		}
	}
	log.Printf("Consumed: %d\n", consumed)
}

a 在测试的时候会报一个错误:
"private key does not match public key"错误
需要修改客户端的证书

Bag Attributes
    friendlyName: caroot
    2.16.840.1.xxx.xxx.1.1: <Unsupported tag 6>
subject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
issuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
-----BEGIN CERTIFICATE-----
MIIDgzCCAmugAwIBAgIJAL95jWSrh9jfMA0GCSqGSIb3DQEBCwUAMFgxCzAJBgNV
BAYTAmNuMQswCQYDVQQIDAJiajELMAkGA1UEBwwCYmoxCzAJBgNVBAoMAnp3MQsw
CQYDVQQLDAJ6dzEVMBMGA1UEAwwMMTkyLjE2OC4yLjMxMB4XDTE4MDMxOTEyMTAx
OVoXDTI4MDMxNjEyMTAxOVowW9....省略n个字符
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: localhost
    localKeyID: 54 69 6D 65 20 31 35 32 31 34 36 31 34 34 36 xx xx xx
subject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
issuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
-----BEGIN CERTIFICATE-----
MIIDLDCCAhQCCQDqwOxGdLTDLjANBgkqhkiG9w0BAQUFADBYMQswCQYDVQQGEwJj
bjELMAkGA1UECAwCYmoxCzAJBgNVBAcMAmJqMQswCQYDVQQKDAJ6dzELMAkGA1UE
CwwCencxFTATBgNVBAMMDDE5Mi4xNjguMi4zMTAeFw0xODAzMTkxMjEwMzBaFw0y
ODAzMTYxMjEwMzBaMFgxCzAJBgNVBAYTAmNuMQswCQYDVQQIEwJiajELMAkGA1UE
BxMCYmoxCzAJBgNVBAoTAnp3MQswCQYDVQQLEwJ6dzEVMBMGA1UEAxMMMTkyLjE2
OC4yLjMxMIIBIjANBgkqhkiG9w0BAQ....省略n个字符
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: CN=192.168.xx.xx,OU=xx,O=xx,L=bj,ST=bj,C=cn
subject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
issuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
-----BEGIN CERTIFICATE-----
MIIDgzCCAmugAwIBAgIJAL95jWSrh9jfMA0GCSqGSIb3DQEBCwUAMFgxCzAJBgNV
BAYTAmNuMQswCQYDVQQIDAJiajELMAkGA1UEBwwCYmoxCzAJBgNVBAoMAnp3MQsw
CQYDVQQLDAJ6dzEVMBMGA1UEAwwMMTkyLjE2OC4yLjMxMB4XDTE4MDMxOTEyMTAx
OVoXDTI4MDMxNjEyMTAxOVowWDELMAkGA1UEBhMCY24xCzAJBgNVBAgMAmJqMQsw
CQYDVQQHDAJiajELMAkGA1UECg....省略n个字符
-----END CERTIFICATE-----

此处查看到有三段-----BEGIN CERTIFICATE----- 和 -----END CERTIFICATE-----，打开client.key.pem看到只有一段friendlyName: localhost，那么找到client.cer.pem（上图为中间一段），删除其余部分，剩余如下：

-----BEGIN CERTIFICATE-----
MIIDLDCCAhQCCQDqwOxGdLTDLjANBgkqhkiG9w0BAQUFADBYMQswCQYDVQQGEwJj
bjELMAkGA1UECAwCYmoxCzAJBgNVBAcMAmJqMQswCQYDVQQKDAJ6dzELMAkGA1UE
CwwCencxFTATBgNVBAMMDDE5Mi4xNjguMi4zMTAeFw0xODAzMTkxMjEwMzBaFw0y
ODAzMTYxMjEwMzBaMFgxCzAJBgNVBAYTAmNuMQswCQYDVQQIEwJiajELMAkGA1UE
BxMCYmoxCzAJBgNVBAoTAnp3MQswCQYDVQQLEwJ6dzEVMBMGA1UEAxMMMTkyLjE2
OC4yLjMxMIIBIjANBgkqhkiG9w0BAQ....省略n个字符
-----END CERTIFICATE-----

在进行验证,就可以了
b 如果,在连接的过程中出现:

WARN Failed to send SSL Close message  (org.apache.kafka.common.network.SslTransportLayer)
java.io.IOException: 断开的管道
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
	at org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:212)
	at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:175)
	at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:703)
	at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:61)
	at org.apache.kafka.common.network.Selector.doClose(Selector.java:739)
	at org.apache.kafka.common.network.Selector.close(Selector.java:727)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:520)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
	at kafka.network.Processor.poll(SocketServer.scala:551)
	at kafka.network.Processor.run(SocketServer.scala:468)
	at java.lang.Thread.run(Thread.java:748)

当服务器的签名证书与客户端密钥库不匹配时，会出现此问题。我们需要生成客户端密钥库并将服务器的签名证书导入客户端密钥库。你按照上面的脚本重新做一次,我已经验证过没有问题

c kafka报错内容：

WARN [Consumer clientId=consumer-1, groupId=console-consumer-950] Connection to node -1 could not be established. Broker may not be available.

这是因为你的配置文件中的PLAINTEXT跟你请求的内容不同。举例来说，我在配置文件里配置的listeners=PLAINTEXT://10.127.96.151:9092，但是我想测试的时候请求的是./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic1 --from-beginning

正确的应该是./kafka-console-consumer.sh --bootstrap-server 10.127.96.151:9092 --topic topic1 --from-beginning

基本上实现到这里就结束了!配置是Centos7.2 kafka 是0.10.1.0

四、结论
写的语无伦次,还是写的少,以后要养成多写博客的习惯,总结下经验,送人玫瑰,手留余香!
写的匆忙,如果有问题请及时指正!

参考:

大彩笔_祥哥

关注

8
点赞
踩
5

收藏

觉得还不错? 一键收藏
0
评论
go 通过SSL 双向认证在kafka上生产和消费

一、背景做项目有个需求：kafka使用SSL加密连接，限制客户端访问，减轻服务端的压力,项目也具有安全性,这就需要给客户端发证书，只允许持有证书的客户端访问。二、实现思路1.在实现的时候参考了很多的帖子,java版的实现很多,go实现的目前只找到一篇,或许是其它的每能及时发现,在这个过程中我遇到很多的坑,说多了就是累,希望给有相同需求的人少走点弯路!!根据其它的帖子实现的思路是这样的:...
复制链接

扫一扫