1业务需求分析
(1)、捕获数据日志或数据库数据信息
(2)、实时分析前当前数据内容
(3)、实时统计当前数据量
(4)、根据业务需求新增统计规划
2、平台组件
hadoop2.8.4
spark2.3.1
hive2.3.3
kafka2.12
zookeeper3.4.12
Hbase
flume
sqoop
3、宏观构架图
4、集群资源规划
| 机器1 | 机器2 | 机器3 | 机器4 | 机器5 |
HDFS | NAMENODE | NAMENODE | DATANONE | DATANODE | DATANODE |
YARN | RESOURCEMANAGER | RESOURCEMANAGER | NONEMANAGER | NONEMANAGER | NONEMANAGER |
ZOOKEEPER | zookeeper | zookeeper | zookeeper |
|
|
kafka |
|
| kafka | kafka | kafka |
HBASE | master | master | regionSERver | regionSERver | regionSERver |
flume | flume |
|
| flume | flume |
hive |
| hive |
|
|
|
mysql |
| mysql |
|
|
|
spark | spark |
|
|
|
|