大数据的一些基本命令汇总

一，集群启动命令

CDH 启动:
/opt/cm-5.1.3/etc/init.d/cloudera-scm-server start

/opt/cm-5.1.3/etc/init.d/cloudera-scm-agent start

CDH 关闭:
/opt/cm-5.1.3/etc/init.d/cloudera-scm-server stop

/opt/cm-5.1.3/etc/init.d/cloudera-scm-agent stop

启动mysql
/usr/local/mysql/bin/mysqld_safe --user=mysql &  （默认是启动的）
登录 
/usr/local/mysql/bin/mysql --user=root -p123 

1，全部节点启动懂zookeeper 
zkServer.sh start 

2，在node1启动 hadoop 
start-dfs.sh  
3，在node1启动yarn
start-yarn.sh
4,node 3 node4启动 
yarn-daemon.sh start resourcemanager

//查看端口是否被占用
netstat -anp |grep 9998
启动：

zkServer.sh start   //启动zk

//启动kafka
./bin/kafka-server-start.sh -daemon ./config/server.properties 

//后台启动kafka-manager 
nohup bin/kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=9998 & 

 
start-dfs.sh 
start-yarn.sh
yarn-daemon.sh start resourcemanager

//启动spark
/home/spark-2.2.0-bin-hadoop2.6/sbin/start-all.sh

启动Hbase
/home/hbase-1.2.6/bin/start-hbase.sh

关闭：

yarn-daemon.sh stop resourcemanager
stop-yarn.sh
stop-dfs.sh   
zkServer.sh stop

//关闭spark
./sbin/stop-all.sh

//关闭kafka
./bin/kafka-server-stop.sh

关闭Hbase
/home/hbase-1.2.6/bin/stop-hbase.sh

HDFS Web页面：
50070

yarn Web页面：
8088

beeline -u jdbchive2://localhost:10000 /

spark:
7077
export SPARK_MASTER_IP=node1 
export SPARK_MASTER_PORT=7077  
export SPARK_WORKER_CORES=1   
export SPARK_WORKER_INSTANCES=1  
export SPARK_WORKER_MEMORY=512m

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local lib/spark-examples-1.0.0-hadoop2.2.0.jar 
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://node1:7077 ./lib/spark-examples-1.6.1-hadoop2.6.0.jar  100

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local ./lib/spark-examples-1.6.1-hadoop2.6.0.jar  100

 

 

二，端口排查

 

1，一个进程有多个端口，一个端口只能被一个进程占用

lsof -i tcp:9092  -- 查看9092端口被占用的是哪些程序

查看端口是否被占用：
lsof -i:8082

2，显示网络相关信息：
sudo netstat -anp|grep 8080

有数据代表被占用

3,Linux中怎么通过PID号找到对应的进程名及所在目录方法
ps -aux |grep -v grep|grep 28990

4，通过进程名字 查看进程信息
    ps -ef | grep 'nginx'
    
    
5, jps -m | grep 'nginx'

 

 

三，kafka的一些命令

 
查看topic某分区偏移量最大（小）值
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic hive-mdatabase-hostsltable  --time -1 --broker-list node86:9092 --partitions 0

bin/kafka-console-producer.sh --broker-list node2:9091,node3:9091 --topic fcy_test
bin/kafka-console-producer.sh --broker-list node4:9092,node5:9092,node6:9092 --topic dianyou_wxgz_test

bin/kafka-console-consumer.sh --bootstrap-server node4:9092,node5:9092,node6:9092 --topic dianyou_wxgz_test2 --from-beginning

###展示命令### 
/zywa/kafka/kafka_2.11-1.1.0/bin/kafka-topics.sh --list --zookeeper node1:2181,node2:2181,node3:2181

------------------------------------------------------------------------------------------------------------------------------------------------------
###创建topic###
/zywa/kafka/kafka_2.11-1.1.0/bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 2 --partitions 1 --topic  wx_test2

/zywa/kafka/kafka_2.11-1.1.0/bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 2 --partitions 2 --topic  dianyou_topic_test
------------------------------------------------------------------------------------------------------------------------------------------------------

 

生产者命令：

/zywa/kafka/kafka_2.11-1.1.0/bin/kafka-console-producer.sh --broker-list node2:9091,node3:9091  --topic dianyou_filter_test
/zywa/kafka/kafka_2.11-1.1.0/bin/kafka-console-producer.sh --broker-list node2:9091,node3:9091  --topic dianyou_topic_test

/zywa/kafka/kafka_2.11-1.1.0/bin/kafka-console-consumer.sh --bootstrap-server node4:9092,node5:9092,node6:9092 --topic dianyou_wxgz_test2

/zywa/kafka/kafka_2.11-1.1.0/bin/kafka-console-consumer.sh --bootstrap-server node4:9092,node5:9092,node6:9092 --topic dianyou_filter 

 

/zywa/kafka/kafka_2.11-0.10.2.0/bin/kafka-console-producer.sh --broker-list node2.hadoop:9092 --topic aa
/zywa/kafka/kafka_2.11-0.10.2.0/bin/kafka-console-consumer.sh --zookeeper node2.hadoop:2181 --topic aa
/zywa/kafka/kafka_2.11-0.10.2.0/bin/kafka-console-consumer.sh --zookeeper node2.hadoop:2181 --topic aa 

 

四，