数据源相关命令整理

数据源相关命令整理

1.hbase

查询数据

count ‘hbase_gp_batch_0122’,INTERVAL => 5000000,CACHE => 5000000

2.kafka

./kafka-console-consumer.sh --bootstrap-server
service-kafka-bigdata-broker-2:9092 --topic kafka_mongo_test
–from-beginning

展示所有topic：

kafka-topics.sh --zookeeper hdhost01:24002/kafka --list

创建topic：

kafka-topics.sh --zookeeper
hdhost01:24002,hdhost02:24002,hdhost03:24002/kafka --create --topic
mq_hbase_kerberos_mongo_0313 --partitions 6 --replication-factor 2

查看topic相关信息：

kafka-topics.sh --zookeeper
hdhost01:24002,hdhost02:24002,hdhost03:24002/kafka --describe --topic
BIGDATA_TEST_TOPIC

生产数据：

kafka-console-producer.sh --broker-list
hdhost01:21007,hdhost02:21007,hdhost03:21007 --topic
BIGDATA_TEST_TOPIC --producer.config
/opt/hadoopclient/Kafka/kafka/config/producer.properties

消费数据：

kafka-console-consumer.sh --topic BIGDATA_TEST_TOPIC
–bootstrap-server hdhost01:21007,hdhost02:21007,hdhost03:21007 --from-beginning --consumer.config /opt/hadoopclient/Kafka/kafka/config/consumer.properties

3.hive

执行hive，进入hive窗口
查看所有的database

show databases

查看table在hdfs上的存储路径

show create table M_BD_T_GAS_ORDER_INFO_H

创建分区表

create EXTERNAL table ddlpartition3(f1 string,f2 bigint,f3 string)
PARTITIONED BY(year int,month int,day int) STORED AS PARQUET location ‘hdfs://bigdata-batch:8020/tmp/hive/ddlpartition3’