kafka系列之kafka linux 下的安装部署测试

安装

• 准备工作：kafka需要jdk 和zookeeper。
• 下载安装jdk1.8 这个参考如下博文： https://blog.csdn.net/m0_37948170/article/details/108123864
• 下载安装zookeeper和kafka： 可以参考：https://www.w3cschool.cn/apache_kafka/apache_kafka_installation_steps.html：
• 本次安装为zookeeper 3.6.1 ：https://archive.apache.org/dist/zookeeper/zookeeper-3.6.1/apache-zookeeper-3.6.1-bin.tar.gz
• 启动命令：上传到服务器后进行解压，修改配置文件名称为zoo.cfg：注意：admin.serverPort 端口配置，这里有相关的配置说明：https://zookeeper.apache.org/doc/r3.6.2/zookeeperAdmin.html。
• 因为我这里启动zookeeper报8080已经被被使用，该端口默认为8080 ，所以性需要增加一个配置修改默认端口。

• # The number of milliseconds of each tick
  tickTime=2000
  # The number of ticks that the initial 
  # synchronization phase can take
  initLimit=10
  # The number of ticks that can pass between 
  # sending a request and getting an acknowledgement
  syncLimit=5
  # the directory where the snapshot is stored.
  # do not use /tmp for storage, /tmp here is just 
  # example sakes.
  dataDir=/tt/zookeeper/data
  # the port at which the clients will connect
  clientPort=xxx
  # the maximum number of client connections.
  # increase this if you need to handle more clients
  #maxClientCnxns=60
  #
  # Be sure to read the maintenance section of the 
  # administrator guide before turning on autopurge.
  #
  # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
  #
  # The number of snapshots to retain in dataDir
  #autopurge.snapRetainCount=3
  # Purge task interval in hours
  # Set to "0" to disable auto purge feature
  #autopurge.purgeInterval=1
  admin.serverPort=xxxx

   

• 启动zookeeper：bin/zkServer.sh start
• 下载kafka：  https://archive.apache.org/dist/kafka/2.4.0/kafka_2.11-2.4.0.tgz
• 上传到服务器：解压后修改配置文件：config/server.properties, 配置如下：ip 为你主机的ip地址，需要替换。

• 
  broker.id=0
  
  
  #   FORMAT:
  #     listeners = listener_name://host_name:port
  #   EXAMPLE:
  #     listeners = PLAINTEXT://your.host.name:9092
  listeners=PLAINTEXT://ip:9091
  
  # Hostname and port the broker will advertise to producers and consumers. If not set, 
  # it uses the value for "listeners" if configured.  Otherwise, it will use the value
  # returned from java.net.InetAddress.getCanonicalHostName().
  advertised.listeners=PLAINTEXT://ip:9091
  host.name=ip
  
  # Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
  #listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
  
  # The number of threads that the server uses for receiving requests from the network and sending responses to the network
  num.network.threads=3
  
  # The number of threads that the server uses for processing requests, which may include disk I/O
  num.io.threads=8
  
  # The send buffer (SO_SNDBUF) used by the socket server
  socket.send.buffer.bytes=102400
  
  # The receive buffer (SO_RCVBUF) used by the socket server
  socket.receive.buffer.bytes=102400
  
  # The maximum size of a request that the socket server will accept (protection against OOM)
  socket.request.max.bytes=104857600
  
  
  ############################# Log Basics #############################
  
  # A comma separated list of directories under which to store log files
  log.dirs=/tep/kafka-logs
  
  # The default number of log partitions per topic. More partitions allow greater
  # parallelism for consumption, but this will also result in more files across
  # the brokers.
  num.partitions=1
  
  # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
  # This value is recommended to be increased for installations with data dirs located in RAID array.
  num.recovery.threads.per.data.dir=1
  
  ############################# Internal Topic Settings  #############################
  # The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
  # For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
  offsets.topic.replication.factor=1
  transaction.state.log.replication.factor=1
  transaction.state.log.min.isr=1
  
  ############################# Log Flush Policy #############################
  
  # Messages are immediately written to the filesystem but by default we only fsync() to sync
  # the OS cache lazily. The following configurations control the flush of data to disk.
  # There are a few important trade-offs here:
  #    1. Durability: Unflushed data may be lost if you are not using replication.
  #    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
  #    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
  # The settings below allow one to configure the flush policy to flush data after a period of time or
  # every N messages (or both). This can be done globally and overridden on a per-topic basis.
  
  # The number of messages to accept before forcing a flush of data to disk
  #log.flush.interval.messages=10000
  
  # The maximum amount of time a message can sit in a log before we force a flush
  #log.flush.interval.ms=1000
  
  ############################# Log Retention Policy #############################
  
  # The following configurations control the disposal of log segments. The policy can
  # be set to delete segments after a period of time, or after a given size has accumulated.
  # A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
  # from the end of the log.
  
  # The minimum age of a log file to be eligible for deletion due to age
  log.retention.hours=168
  
  # A size-based retention policy for logs. Segments are pruned from the log unless the remaining
  # segments drop below log.retention.bytes. Functions independently of log.retention.hours.
  #log.retention.bytes=1073741824
  
  # The maximum size of a log segment file. When this size is reached a new log segment will be created.
  log.segment.bytes=1073741824
  
  # The interval at which log segments are checked to see if they can be deleted according
  # to the retention policies
  log.retention.check.interval.ms=300000
  
  ############################# Zookeeper #############################
  
  # Zookeeper connection string (see zookeeper docs for details).
  # This is a comma separated host:port pairs, each corresponding to a zk
  # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
  # You can also append an optional chroot string to the urls to specify the
  # root directory for all kafka znodes.
  zookeeper.connect=localhost:2181
  
  # Timeout in ms for connecting to zookeeper
  zookeeper.connection.timeout.ms=6000
  
  
  ############################# Group Coordinator Settings #############################
  
  # The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
  # The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
  # The default value for this is 3 seconds.
  # We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
  # However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
  group.initial.rebalance.delay.ms=0
  

   

• 配置完毕即可保存：启动：

  bin/kafka-server-stop.sh config/server.properties

• 需要远程访问：则需要打开kafka的端口在防火墙：firewall-cmd --list-all  查看主机对外开放的端口，如果没有则可以通过如下命令添加：永久生效需要执行reload命令，不加永久则临时生效，则不用执行reload。添加完毕使用 --list-all 查看开放的端口即可。

firewall-cmd --add-port=80/tcp(端口和访问类型) --permanent(永久生效)

firewall-cmd --reload

• 创建主题：

  bin/kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --zookeeper localhost:2181

• 创建消费者：

  bin/kafka-console-consumer.sh --bootstrap-server localhost:9091 --topic test--from-beginning

•  

  创建生产者：

  bin/kafka-console-producer.sh --broker-list ip:90921 --topic test

 

• 远程windows系统下验证端口是否开放：telnet  ip  001
• 在本机编写代码验证是否可以成功写入消息：

package kafka.demo.lhk.examples;



import java.util.Properties;

import org.apache.kafka.clients.KafkaClient;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;



/**
 * @Author   kafka-clients-2.5 版本
 * @Time 2020-8-28 9:37:20
 * @Version 1.0 Description:
 */
public class KafkaProducerClientUtils {

    private static final Logger log = LoggerFactory.getLogger(KafkaProducerClientUtils.class);

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "ip:9092");
        props.put("request.timeout.ms", "30000");

        Producer<String, String> producer = new KafkaProducer<String, String>(props, new StringSerializer(),
                new StringSerializer());
//主题为test
        try {
            for (int i = 1; i < 10; i++) {
                producer.send(
                        new ProducerRecord<String, String>("test", Integer.toString(i), "test" + Integer.toString(i)));           }
        } catch (Exception e) {
            // We can't recover from these exceptions, so our only option is to close the
            // producer and exit.
            e.printStackTrace();
            producer.close();
        }
        producer.close();
    }


}

• 验证成功。
• 推荐阅读：《深入理解kafka：核心设计与实践原理》