提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档
组件对应进程jps
命令 | 组件 | 进程名node1 | 进程名node2 | 进程名node3 | 端口号 |
---|---|---|---|---|---|
zkServer.sh | zookeeper | QuorumPeerMain | QuorumPeerMain | QuorumPeerMain | |
start-dfs.sh | hadoop | NameNode、 DataNode、journalnode(通信)、DFSZKFailoverController | NameNode、 DataNode、journalnode(通信)、DFSZKFailoverController | DataNode、journalnode | 50070 |
start-all.sh | hadoop on yarn (ha) | NameNode、 DataNode、journalnode、DFSZKFailoverController、ResourceManager、NodeManager | NameNode、 DataNode、journalnode、DFSZKFailoverController、NodeManager | DataNode、journalnode、NodeManager | 8088 |
start-cluster.sh | flink(单机) | StandaloneSessionClusterEntrypoint、TaskManagerRunner | StandaloneSessionClusterEntrypoint | StandaloneSessionClusterEntrypoint | 8081/nc -lk 9001 |
kafka-server-start.sh config/server.properties | kafka | kafka | kafka | kafka | |
flume-ng agent -n a1 -c /opt/dtc/software/flume/conf/ -f kafka_spool.conf -Dflume.root.logger=INFO,console | flume | ||||
一、zookeeper
启动:
单机启动:
cd /opt/dtc/software/zooleeper/bin
./zkServer.sh start
查看状态:
bin/zkServer.sh status
脚本多集群启动:
集群开启gedit zkStart-all.sh
#chmod -R 777 zkStart-all.sh
#!/bin/bash
echo "start 动感光郭zk启动中......."
for i in 6 4
do
ssh PC$i "source /etc/profile;/usr/local/zookeeper/bin/zkServer.sh start"
done
echo "zkServer started!"
集群关闭gedit zkStop-all.sh
#chmod -R 777 zkStop-all.sh
#!/bin/bash
echo "stop 动感光郭zk关闭中......."
for i in 6 4
do
ssh PC$i "source /etc/profile;/usr/local/zookeeper/bin/zkServer.sh stop"
done
echo "zkServer stoped!"
二、hdfs
启动:
包括hdfs+yarn
start-all.sh
启动NameNode:
hadoop-daemon.sh start namenode
停止NameNode:
hadoop-daemon.sh stop namenode
hdfs haadmin -getServiceState nn1//查看那么node状态
yarn-daemon.sh start resourcemanager//启动rm
yarn-daemon.sh stop resourcemanager//关闭
yarn rmadmin -getAllServiceState//获取所有rm状态
yarn rmadmin -getServiceState rm1//查看rm1状态
三、kafka(先启动zk)
启动:
其中a为ip地址
cd /opt/dtc/software/kafka
bin/kafka-server-start.sh config/server.properties#显启动
/opt/dtc/software/kafka/bin/kafka-server-start.sh
nohup /opt/dtc/software/kafka/config/server.properties > /dev/null 2>&1 &#隐启动
#创建一个topic,名字为topic-demo,该topic包含1个partition,副本因子=1
kafka-topics.sh --zookeeper a:2181 --create --topic topic-demo1 --partitions 1 --replication-factor 1
#创建成功后,可以通过如下命令查看当前Kafka集群中的所有topic:
kafka-topics.sh --zookeeper a:2181 --list
#生产者
kafka-console-producer.sh --broker-list a:9092 --topic t1
#开启一个新的终端,启动一个Consumer进程,消费topic-demo主题中的消息,并输出到控制台:
#消费者
kafka-console-consumer.sh --bootstrap-server a:9092 --topic t1 --from-beginning
#删除topic,此操作需要在server.properties中将设置delete.topic.enable=true,否则就只是标记删除,而不会真正删除
bin/kafka-topics.sh --zookeeper a:2181 --delete --topic t1
四、flink
启动:
cd /opt/dtc/software/flink
bin/start-cluster.sh
nc -lk 9001
停止:
bin/stop-cluster.sh
运行自带窗口程序
./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9001
在nc窗口输入
在web控制台上可以看到统计的当前窗口的词频统计结果:
maven打成jar包放到集群运行
flink run -c com.dtc.bigdata.realtime.Flink.SoketWindowWordCount dtc-realtime-1.0-SNAPSHOT.jar
四、flink on yarn
cd /jars
flink run -m yarn-cluster -ynm socket-word-count -yjm 2g -ytm 2g -c com.dtc.bigdata.realtime.Flink.SoketWindowWordCount dtc-realtime-1.0-SNAPSHOT.jar
如果flink程序提交到yarn上就挂掉,则查看8088progress的大小,如果为100%,则调大-yjm、-ytm
五、启动Flume
cd /opt/dtc/software/flume/conf/
flume-ng agent -n a1 -c ../conf/ -f kafka_spool.conf -Dflume.root.logger=INFO,console
参数:
总结
久病成医