一、安装scala 2.11.4
1、将课程提供的scala-2.11.4.tgz使用WinSCP拷贝到spark1的/usr/local目录下。
2、对scala-2.11.4.tgz进行解压缩:tar -zxvf scala-2.11.4.tgz。
3、对scala目录进行重命名:mv scala-2.11.4 scala
4、配置scala相关的环境变量
vi .bashrc
export SCALA_HOME=/usr/local/scala
export PATH=$SCALA_HOME/bin
source .bashrc
5、查看scala是否安装成功:scala -version
6、按照上述步骤在spark2和spark3机器上都安装好scala。使用scp将scala和.bashrc拷贝到spark2和spark3上即可。
二、 安装Kafka包
1、将kafka_2.9.2-0.8.1.tgz压缩包使用Xshell拷贝到spark1的/usr/local目录下。
2、对kafka_2.9.2-0.8.1.tgz进行解压缩:tar -zxvf kafka_2.9.2-0.8.1.tgz。
3、对kafka目录进行改名:mv kafka_2.9.2-0.8.1 kafka
4、配置kafka
vim /usr/local/kafka/config/server.properties
broker.id:依次增长的整数,0、1、2、3、4,集群中Broker的唯一id
zookeeper.connect=192.168.1.107:2181,192.168.1.108:2181,192.168.1.109:2181
5、安装slf4j
将slf4j-1.7.6.zip压缩包上传到/usr/local目录下
unzip slf4j-1.7.6.zip
把slf4j中的slf4j-nop-1.7.6.jar复制到kafka的libs目录下面
6、按照上述步骤在spark2和spark3分别安装kafka。用scp把kafka拷贝到spark2和spark3行即可。
7、唯一区别的,就是server.properties中的broker.id,要设置为1和2
三、 启动kafka集群
1、在三台机器上分别执行以下命令:
前台启动:
[root@weekend03 bin]./kafka-server-start.sh ../config/server.properties
后台启动:
[root@weekend03 bin]./kafka-server-start.sh ../config/server.properties 1>/dev/null 2>&1 &
./kafka-server-start.sh -daemon ../config/server.properties
指定监听端口
JMX_PORT=2898 ./
kafka-server-start.sh ../config/server.properties 1>/dev/null 2>&1 &
2、解决kafka Unrecognized VM option 'UseCompressedOops'问题
vi bin/kafka-run-class.sh
if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"
fi
去掉-XX:+UseCompressedOops即可
3、使用jps检查启动是否成功
四、 测试kafka集群
1、删除指定topic
kafka-topics.sh --zookeeper localhost:2181 --delete --topic topic-test
2、创建主题:
./kafka-topics.sh --create --zookeeper 192.168.0.128:2181,192.168.0.129:2181,192.168.0.130:2181 --replication-factor 1 --partitions 1 --topic track
3、查看主题列表:
./kafka-topics.sh --list --zookeeper 192.168.0.128:2181,192.168.0.129:2181,192.168.0.130:2181
4、查看主题描述:
./kafka-topics.sh --describe --zookeeper 192.168.0.128:2181,192.168.0.129:2181,192.168.0.130:2181 --topic track
5、创建生产者:
./kafka-console-producer.sh --broker-list 192.168.0.128:9092,192.168.0.129:9092,192.168.0.130:9092 --topic track
6、创建消费者:
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic track
安装搭建
官方安装搭建
Kafka配置文件详解
1. kafka创建topic出现:
Error while executing topic command org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids
原因: 没有在kafka目录下创建zookeeper ,指定myid
解决:
cd /uardata10/chbtmp/package/kafka_2.10-0.8.2.1
mkdir zookeeper
cd zookeeper
touch myid
echo 0 > myid
重新启动kafka就ok
1.1 、创建topic的paration大于可用的boker
创建topic 指定partitions 为5 而 borker只有一个
./bin/kafka-topics.sh -zookeeper idc007128:2181,idc007124:2181,idc007123:2181 -topic test -replication-factor 2 -partitions 5 -create
##问题
Error while executing topic command replication factor: 2 larger than available brokers: 1
解决:
-replication-factor 1 副本数为1
[root@idc007130 kafka_2.10-0.8.2.1]# ./bin/kafka-topics.sh -zookeeper idc007128:2181,idc007124:2181,idc007123:2181 -topic test -replication-factor 1 -partitions 1 -create
Created topic "test".
2、kafka的broker.id 重复导致出错
原因: 复制config/server.properties没有修改主机名和borker.id
[2017-06-14 18:07:55,583] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured brokerId 0 doesn't match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
解决:
第一步: 修改config/server.properties中的boker.id 及主机名
第二步:删除log.dirs=/tmp/kafka-logs目录, 重新启动
3、kafka配置文件错误
[2018-10-12 08:35:04,553] ERROR Error when sending message to topic TestTopic with key: null, value: 12 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
[2018-10-12 08:35:05,335] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-10-12 08:35:06,326] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-10-12 08:35:07,309] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-10-12 08:35:08,444] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
解决:修改server.properties文件
listeners=PLAINTEXT://:9092
4、Kafka登录之后立刻又自动宕掉
kafka登录一段时间之后自动宕掉解决办法,在登录命令加daemon
nohup bin/kafka-server-start.sh -daemon ./config/server.properties &(后台登录方式)
配置好的kafka集群登录后会立刻自动宕掉,经查看kafka日志发现,配置文件中的broker.id与meta.properties文件中的broker.id不相符
config/server.properties配置文件中属性broker.id=0(broker的全局唯一编号,不能重复)与该配置文件中的另一个属性log.dirs=/export/servers/logs/kafka(后面路径查看自己的配置)路径内的meta.properties文件中broker.id值不一致,导致登录失败,修改后成功登录。