系统简介
本系统负责将Hive处理后的数据导出到MySQL服务器上,采用主/从架构。zeus2将待导出的数据信息放到zookeeper上,Zookeeper将该信息发送给master。master通过JobWatcher接收待处理的表信息,将这些信息转换为任务,并分发给client处理。当client处理完成时,会更新MySQL上数据处理表,表示该部分任务已经处理完成
模块简介
master
简介
为服务,会一直运行。包含接收zookeeper上传来的待导出数据的元信息、任务导出、任务超时处理、任务执行完成后的元数据更新、告警等功能
流程
启动master服务
通过JobWatcher线程获取待处理的任务,并更新到队列MasterContext.finishedTZ中
MasterContext中加入一个线程,扫描队邻喎�"http://www.it165.net/pro/pkqt/" target="_blank" class="keylink">QTWFzdGVyQ29udGV4dC5maW5pc2hlZFRao6zI57n709DIzs7xo6zU8r+qyry94s72yM7O8bKivfjQ0MjOzvG31reiPGxpPsjOzvG31reitcTKsbryo6y9q7fWt6LIzs7xvNPI67W9ZXhlY3V0b3KjrNa00NDN6rPJtcTKsbryo6y1w7W9t7W72MLro6yyorj5vt23tbvYwuujrL340NDP4NOmtcS0psDtPGxpPk1hc3RlckNvbnRleHTW0LzTyOvSu7j2z9+zzKOs08PT2rzgv9izrMqxtcRjbGllbnSjrMjnufuzrMqxo6zU8r2ruMNjbGllbnS808jrtb2zrMqxwdCx7dbQo6zS1Lrzsru9+NDQt9a3ogo8bGk+udi8/LXjCgo8bGk+TXlTUUy92rXjtcTRodTxCgo8bGk+PHN0cm9uZz7Q6MfzPC9zdHJvbmc+PGJyPgrOqsHLuvPQ+LzGy+O1xLe9seOjrNDo0qq9q8TctOa0otTa0ru49k15U1FMt/7O8cb3yc+1xMr9vt3Iq7K/tbyz9rW90ru49r3atePJz6OswP3I58nPuPbUwtW+teNBtbyz9rW9TXlTUUwxyc+jrNXiuPbUwqOsu7m1w72r1b6140G1xMr9vt21vLP2tb292rXjTXlTUUwxyc+hozxsaT48c3Ryb25nPr3ivva3vbC4PC9zdHJvbmc+PGJyPgq1vLP2yrGjrM6qwcuxo9akw7+0zsr9vt22vLW8s/a1vdK7uPa92rXjyc/IpaOs0OjSqs6ss9bSu9XFc2l0ZbrNaG9zdLzkttTTprXEudjPtbHtoaO2+LK/t9bVvrXjtcTK/b7dt8ezo7Tzo6y74bOsuf1NeVNRTLf+zvHG97XEtaWx7ePQJiMyMDU0MDujrNXi0fmyv7fW1b6147XEyv2+3dDo0qq31reitb2yu82stcS92rXjyc/IpaGjc2l0ZbrNvdq149auvOS1xLnYz7Wyu8rH0rvSu7bU06a1xKGjtvi089W+tePWu8rH08O7p9bQtcTSu7K/t9ajrLu5tObU2tK70KnQodW+teOjrNK7uPZNeVNRTLf+zvHG97/JxNy05rfFyv249tW+teO1xMr9vt2ho86qwcvTprbU1eLQqcz01b2jrM7Sw8e9q9W+teO31s6qyP3W1lNJVEVfTEVWRUyjulNNQUxMX1NJVEWhokJJR19TSVRFoaJIVUdfU0lURaOssqK31rHwssnTw7K7zay1xLW8s/ay38LUoaM8YnI+CjxzdHJvbmc+U01BTExfU0lURTwvc3Ryb25nPiDN+NW+tcTK/b7dwb+9z9Cho6zSu7j21b6149a7tOa3xdTa0ru49k15U1FMt/7O8cb3yc/IpaGjy/nT0LXEyv2+3ba8u+G1vLP2tb3Su8yoTVlTUUy3/s7xxvfJz8iloaO1scr9vt2zrLn9TVlTUUy3/s7xxve1pbHtz97WxrXEyrG68qOsu+G9q8r9vt21vLP2tb24utTY1+7QobXETXlTUUy3/s7xxvfJz8iloaPEs3NpdGW63LOkyrG85NLUwLTSu9axyrnTw87Sw8e1xLf+zvHG98qxo6y/ycTcu+Gz9s/W1eLW1sfpv/ahozxicj4KPHN0cm9uZz5CSUdfU0lURTwvc3Ryb25nPiC1vLP2tcSy38LUus1TTUFMTF9TSVRF0rvR+aOstavKx7vxyKFNeVNRTLf+zvHG97XEt723qLrNU01BTExfU0lURbK70rvR+aOsQklHX1NJVEWwtNXVwtbRr7XEt73Kvb2rSElWRcnPtcTK/b7dtbyz9rW9TVlTUUzW0Milo6y8tL3xzOy1xMr9vt21vLP2tb1NWVNRTDHJz6Osw/fM7LXEyv2+3b/JxNy1vLP2tb1NeVNRTDLJz6GjtvhTTUFMTF9TSVRFtcTK/b7dvvm1vLP2tb3Su8yoTVlTUUy3/s7xxvfJz6GjPGJyPgo8c3Ryb25nPkhVR19TSVRFPC9zdHJvbmc+IL2r1b6148O/zOy1xLfDzsrQxc+it9a3orW9srvNrLXETVlTUUy3/s7xxvfJz8ilCjxsaT7Xor3iCgo8bGk+SERGU8K3vrY8YnI+Ci91c2VyL2hpdmUvd2FyZWhvdXNlL3B0bWluZF9kYXRhLmRiLyR7dGFibGVOYW1lfV8ke3RhYmxlVHlwZX0vc2l0ZXR6PSR7dGltZXpvbmV9L3BhcnRkdD0ke2RhdGV9L3BhcnRzaWQ9JHtzaWR9PGJyPgrI5y91c2VyL2hpdmUvd2FyZWhvdXNlL3B0bWluZF9kYXRhLmRiL3N1bV9wYWdlX3Zpc2l0c19zdGF0c19vbGFwX2Qvc2l0ZXR6PUUwODAwL3BhcnRkdD0yMDE0LTA2LTAyL3BhcnRzaWQ9NTZmYmNlNGU8bGk+dGFibGVUeXBlPGJyPgrD98+4se21xMDg0M3OqngsxuTL+7Ht1N3Ksda71qez1szsZAo8cD48L3A+CjxwcmUgY2xhc3M9"brush:java;">private String getTabType(String tableName) {
if (tableName.equals(Constant.TB_1)) {
return x;
}
else {
return d;
}
}
client
简介
部署在MySQL服务器上
执行HDFS2MySQL的导出任务
流程
通过clientBootstrap监控消息
当监控到任务时,执行HDFS2MySQL的导出任务
2.1 通过shell脚本,从HDFS上下载数据
2.2 将元数据更新到MySQL服务器中
2.3 删除本地文件
2.4 根据表中插入行的数目判断数据是否导出成功
2.5 将执行情况返回给master
定时向master发送心跳信息
注解
存储的本地路径:/tmp/ptbalancer/data/${tableName}_${tableType}_${date}_${当前时间戳}
节点间通信
中间件 netty
master ServerBootstrap
client ClientBootstrap
传输数据 PB
相比XML,PB有更好的传输效率、压缩率更高、解析速度更快