flume+kafka+zookeeper+storm实时计算环境搭建(二)

搭建flume+kafka+storm环境

这里，我的flume是采集mysql的数据再存入kafka，我用mysql作为我的source，内存memory作为channels，kafka作为sink，这个要借助一个插件source-ng-sql手机mysql的数据

环境准备：
1.flume1.8.0
2.kafka2.0.0
3.storm1.2.2
4.zookeeper3.4.13

flume
下载：
http://flume.apache.org/download.html
下载
apache-flume-1.8.0-bin.tar.gz
解压
tar -zxvf ~/Desktop/apache-flume-1.8.0-bin.tar.gz
sudo mv apache-flume-1.8.0-bin flume1.8.0
mkdir -p bigdata
sudo mv flume1.8.0 /usr/local/bigdata
改一下拥有者
sudo chown bigdata -R storm：storm
修改flume的conf配置文件
sudo cp flume-conf.properties.template flume.conf
sudo vim flume.conf

a1.sources=sql
a1.channels=mem
a1.sinks=kafka

1.sources.sql.type=org.keedio.flume.source.SQLSource
a1.sources.sql.hibernate.connection.url=jdbc:mysql://数据库地址

a1.sources.sql.hibernate.connection.user=root
a1.sources.sql.hibernate.connection.password=123456
a1.sources.sql.hibernate.connection.autocommit=true
a1.sources.sql.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
a1.sources.sql.hibernate.connection.driver_class=com.mysql.jdbc.Driver
a1.sources.sql.run.query.delay=5000
a1.sources.sql.status.file.path=sql状态文件保存路径
a1.sources.sql.status.file.name=sql状态文件名
# Custom query
a1.sources.sql.start.from=0
a1.sources.sql.custom.query=数据库查询语句
a1.sources.sql.batch.size=1000
a1.sources.sql.max.rows=1000
a1.sources.sql.hibernate.connection.provider_class=org.hibernate.connection.C3P0ConnectionProvider
a1.sources.sql.hibernate.c3p0.min_size=1
a1.sources.sql.hibernate.c3p0.max_size=10
# use memory as the channels
a1.channels.mem.type=memory
a1.channels.mem.capacity=1000
a1.channels.mem.transactionCapacity=1000
a1.channels.mem.byteCapacityBufferPercentage=20
a1.channels.mem.byteCapacity=800000

a1.sinks.kafka.type=org.apache.flume.sink.kafka.KafkaSink
a1.sinks.kafka.topic=创建的kafka topic名
a1.sinks.kafka.brokerList=master:9092,slave1:9092,slave2:9092
a1.sinks.kafka.requireAcks=1
a1.sinks.kafka.batchSize=20
a1.sinks.kafka.partitionIdHeader=0

a1.sources.sql.channels=mem
a1.sinks.kafka.channel=mem

配置环境变量
sudo vim /etc/profile

flume-ng-sql-source插件
下载
https://github.com/xiaomoo/oeasy
在flume的目录下创建目录plugins.d

目录结构如上
lib放编译好的flume-ng-sql-source.jar
libext放mysql的jar

zookeeper
下载：
http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.13/
解压：
tar -zxvf zookeeper-3.4.13.tar.gz
sudo mv zookeeper-3.4.13 zookeeper3.4.13
sudo zookeeper3.4.13 /usr/local/bigdata
修改配置文件
sudo vim zoo.cfg
单节点配置

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataLogDir=/usr/local/bigdata/zookeeper3.4.13/logs
dataDir=/usr/local/bigdata/zookeeper3.4.13/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

然后新建data目录，单节点为standalone模式

zookeeper集群配置

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataLogDir=/usr/local/bigdata/zookeeper3.4.13/logs
dataDir=/usr/local/bigdata/zookeeper3.4.13/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=0.0.0.0:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888

新建data目录
mkdir -p data
cat 1 > myid
通过ssh将zookeeper目录发送给slave1，slave2
scp -t zookeeper3.4.13 storm@slave1:/usr/local/bigdata/
scp -t zookeeper3.4.13 storm@slave2:/usr/local/bigdata/
分别将slave1，slave2新建data目录
slave1：
cat 2 > myid
修改zoo.cfg

server.1=master:2888:3888
server.2=0.0.0.0:2888:3888
server.3=slave2:2888:3888

slave2：
cat 3 > myid
修改zoo.cfg

server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=0.0.0.0:2888:3888

zookeeper集群会选择一个leader，其他为follower
配置环境变量

通过bin/zkServer.sh start启动zookeeper
bin/zkServer.sh stop关闭zookeeper
bin/zkServer.sh status查看集群状态

kafka
下载：
http://kafka.apache.org/downloads
解压：
tar -zxvf kafka_2.11-2.0.0.tgz
sudo mv kafka_2.11-2.0.0 kafka2.0.0
sudo mv kafka2.0.0 /usr/local/bigdata
cd /usr/local/bigdata/kafka2.0.0/config
修改kafka配置server.properties
sudo vim server.properties

broker.id=0
host.name=本机的ip地址

listeners=PLAINTEXT://192.168.5.211:9092
log.dirs=/usr/local/bigdata/kafka2.0.0/logs

num.network.threads=3
num.io.threads=8

socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600
delete.topic.enable=true

log.dirs=/usr/local/bigdata/kafka2.0.0/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

zookeeper.connect=你的zookeeper地址，可以使单节点也可以是集群
zookeeper.connection.timeout.ms=6000
roup.initial.rebalance.delay.ms=0

打开zookeeper.properties
sudo vim zookeeper.properties

dataDir=你的zookeeper的data目录路径
clientPort=2181

将kafka发送到slave1，slave2
scp -t kafka2.0.0 storm@slave1:/usr/local/bigdata/
scp -t kafka2.0.0 storm@slave2:/usr/local/bigdata/
分别修改slave1，slave2的kafka中config的server.properties
slave1：
将broker.id改为1
slave2：
将broker.id改为2

配置环境变量

storm
下载
http://storm.apache.org/downloads.html
解压
tar -zxvf apache-storm-1.2.2.tar.gz
sudo mv apache-storm-1.2.2 storm1.2.2
sudo mv storm1.2.2 /usr/local/bigdata

修改配置文件,在conf目录下
sudo vim storm.yaml

storm.local.dir: "/usr/local/bigdata/storm1.2.2/localdir"
storm.zookeeper.servers:
     - zookeeper地址
     
nimbus.seeds: ["master"]
ui.host: 0.0.0.0
ui.port: 8093
supervisor.slots.ports:
     - 6700
     - 6701
     - 6702
     - 6703

drpc.servers:
     - "master"
     - "slave1"
     - "slave2"

将storm发送给slave1，slave2
scp -t storm1.2.2 storm@slave1:/usr/local/bigdata/
scp -t storm1.2.2 storm@slave2:/usr/local/bigdata/

配置环境变量

测试环境
首先启动zookeeper
$ZK_HOME/bin/zkServer.sh start(我用的单节点)
$ZK_HOME/bin/zkServer.sh status查看状态

启动kafka集群
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

启动storm
master：
storm nimbus &
storm ui &

slave1:
storm supervisor &

slave2:
storm supervisor &

jps命令查看启动进程
master：

slave1：

slave2：

至此，环境就搭建完成了。