scrapy-culster集群搭建之kafka安装

本文链接：https://blog.csdn.net/Tilyp/article/details/77034642

环境同上次zookeeper的安装环境一致就不累赘了。我们来下载kafka，这里我下载的是Scala 2.10 - kafka_2.10-0.10.2.0.tgz (asc, md5)版本的

wget https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz

完成后解压：

[root@shulaibao4 ~]# tar zxf kafka_2.11-0.10.2.0.tgz
[root@shulaibao4 ~]# mv kafka_2.11-0.10.2.0  /usr/lib
[root@shulaibao4 ~]# cd /usr/lib/kafka_2.11-0.10.2.0

现在这个单台机器的集群是可以启动起来的，但是我们的目的是搭建三台机器的集群，那我们来开始配置，打开conf文件下的server.properties

[root@shulaibao4 kafka_2.11-0.10.2.0]# vim config/server.properties 

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2                      # kafka的机器编号，
host.name = 172.*.*.13         # 绑定ip
port=9092                        # 默认端口9092，
# Switch to enable topic deletion or not, default value is false
delete.topic.enable=true

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads handling network requests
num.network.threads=3

# The number of threads doing disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs     # kafka的日志目录，话题，偏移量等也存于此处

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=3             # 设置复制数量，默认是1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
#zookeeper.connect=localhost:2181
#zookeeper.connect=workstation1:2181
zookeeper.connect=172.*.*.12:2181,172.*.*.13:2181,172.*.*.14:2181    # 连接zookeeper集群，

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

此配置文件配置好后把kafka文件scp到其他机器上，

[root@shulaibao4 kafka_2.11-0.10.2.0]# cd ..
[root@shulaibao4 lib]# scp -r kafka_2.11-0.10.2.0/ root@172.*.*.14: /usr/lib
......

[root@shulaibao4 lib]# scp -r kafka_2.11-0.10.2.0/ root@172.*.*.12: /usr/lib
......

在这两台机器上只需要修改conf目录下的server.properties文件即可，
第一，修改 broker.id且互相不可重复，
第二，修改 host.name，改为各自机器绑定的IP
每台机器做完这两步，这个小小的集群就搭建起来了，接下来我们让它运行起来
首先启动每台机器的zookeeper，（上篇文章已介绍，这里不在累赘）
然后启动我们的kafka集群，命令如下：

[root@shulaibao4 kafka_2.11-0.10.2.0]# bin/kafka-server-start.sh config/server.properties &
[1] 12608
[root@shulaibao4 kafka_2.11-0.10.2.0]# [2017-08-10 11:56:48,766] INFO KafkaConfig values:  
    … …
    … …
    … …

当我们看到如下输出则证明单台机器启动成功：

       … …
[2017-08-10 11:56:49,930] INFO Kafka version : 0.10.2.0 (org.apache.kafka.common.utils.AppInfoParser)
[2017-08-10 11:56:49,931] INFO Kafka commitId : 576d93a8dc0cf421 (org.apache.kafka.common.utils.AppInfoParser)
[2017-08-10 11:56:49,931] INFO [Kafka Server 2], started (kafka.server.KafkaServer)

接下来我们验证下我们的集群是否可用，首先创建一个 test“topic”，

[root@shulaibao4 kafka_2.11-0.10.2.0]# bin/kafka-topics.sh --create --zookeeper shulaibao3:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
[root@shulaibao4 kafka_2.11-0.10.2.0]#

然后在另一台机器上查看我们刚创建的 topic的详细信息

[root@shulaibao4 kafka_2.11-0.10.2.0]# bin/kafka-topics.sh --describe --zookeeper shulaibao4:2181 --topic test
Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: test Partition: 0    Leader: 3   Replicas: 3 Isr: 3

查看集群中所有的 topic 列表

[root@shulaibao5 kafka_2.11-0.10.2.0]# bin/kafka-topics.sh --list --zookeeper shulaibao4:2181
test
[root@shulaibao5 kafka_2.11-0.10.2.0]#

可以看到这个 topic已经在集群中, 接下来我们利用此 topic 来生产和消费，这里输入完成后敲回车键就可以了，当然也可以读取文件或者数据库将内容发送出去

[root@shulaibao5 kafka_2.11-0.10.2.0]# bin/kafka-console-producer.sh --broker-list shulaibao4:9092 --topic test 
This is a scrapy cluster

接着我们来消费刚才发送的消息：

[root@shulaibao3 kafka_2.11-0.10.2.0]# bin/kafka-console-consumer.sh --zookeeper shulaibao4:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
This is a scrapy cluster