mysql2tb_Hive2MySQL初步架构

系统简介

本系统负责将Hive处理后的数据导出到MySQL服务器上，采用主/从架构。zeus2将待导出的数据信息放到zookeeper上，Zookeeper将该信息发送给master。master通过JobWatcher接收待处理的表信息，将这些信息转换为任务，并分发给client处理。当client处理完成时，会更新MySQL上数据处理表，表示该部分任务已经处理完成

模块简介

master

简介

为服务，会一直运行。包含接收zookeeper上传来的待导出数据的元信息、任务导出、任务超时处理、任务执行完成后的元数据更新、告警等功能

流程

启动master服务

通过JobWatcher线程获取待处理的任务，并更新到队列MasterContext.finishedTZ中

MasterContext中加入一个线程，扫描队邻喎�"http://www.it165.net/pro/pkqt/" target="_blank" class="keylink">QTWFzdGVyQ29udGV4dC5maW5pc2hlZFRao6zI57n709DIzs7xo6zU8r+qyry94s72yM7O8bKivfjQ0MjOzvG31reiPGxpPsjOzvG31reitcTKsbryo6y9q7fWt6LIzs7xvNPI67W9ZXhlY3V0b3KjrNa00NDN6rPJtcTKsbryo6y1w7W9t7W72MLro6yyorj5vt23tbvYwuujrL340NDP4NOmtcS0psDtPGxpPk1hc3RlckNvbnRleHTW0LzTyOvSu7j2z9+zzKOs08PT2rzgv9izrMqxtcRjbGllbnSjrMjnufuzrMqxo6zU8r2ruMNjbGllbnS808jrtb2zrMqxwdCx7dbQo6zS1Lrzsru9+NDQt9a3ogo8bGk+udi8/LXjCgo8bGk+TXlTUUy92rXjtcTRodTxCgo8bGk+PHN0cm9uZz7Q6MfzPC9zdHJvbmc+PGJyPgrOqsHLuvPQ+LzGy+O1xLe9seOjrNDo0qq9q8TctOa0otTa0ru49k15U1FMt/7O8cb3yc+1xMr9vt3Iq7K/tbyz9rW90ru49r3atePJz6OswP3I58nPuPbUwtW+teNBtbyz9rW9TXlTUUwxyc+jrNXiuPbUwqOsu7m1w72r1b6140G1xMr9vt21vLP2tb292rXjTXlTUUwxyc+hozxsaT48c3Ryb25nPr3ivva3vbC4PC9zdHJvbmc+PGJyPgq1vLP2yrGjrM6qwcuxo9akw7+0zsr9vt22vLW8s/a1vdK7uPa92rXjyc/IpaOs0OjSqs6ss9bSu9XFc2l0ZbrNaG9zdLzkttTTprXEudjPtbHtoaO2+LK/t9bVvrXjtcTK/b7dt8ezo7Tzo6y74bOsuf1NeVNRTLf+zvHG97XEtaWx7ePQJiMyMDU0MDujrNXi0fmyv7fW1b6147XEyv2+3dDo0qq31reitb2yu82stcS92rXjyc/IpaGjc2l0ZbrNvdq149auvOS1xLnYz7Wyu8rH0rvSu7bU06a1xKGjtvi089W+tePWu8rH08O7p9bQtcTSu7K/t9ajrLu5tObU2tK70KnQodW+teOjrNK7uPZNeVNRTLf+zvHG97/JxNy05rfFyv249tW+teO1xMr9vt2ho86qwcvTprbU1eLQqcz01b2jrM7Sw8e9q9W+teO31s6qyP3W1lNJVEVfTEVWRUyjulNNQUxMX1NJVEWhokJJR19TSVRFoaJIVUdfU0lURaOssqK31rHwssnTw7K7zay1xLW8s/ay38LUoaM8YnI+CjxzdHJvbmc+U01BTExfU0lURTwvc3Ryb25nPiDN+NW+tcTK/b7dwb+9z9Cho6zSu7j21b6149a7tOa3xdTa0ru49k15U1FMt/7O8cb3yc/IpaGjy/nT0LXEyv2+3ba8u+G1vLP2tb3Su8yoTVlTUUy3/s7xxvfJz8iloaO1scr9vt2zrLn9TVlTUUy3/s7xxve1pbHtz97WxrXEyrG68qOsu+G9q8r9vt21vLP2tb24utTY1+7QobXETXlTUUy3/s7xxvfJz8iloaPEs3NpdGW63LOkyrG85NLUwLTSu9axyrnTw87Sw8e1xLf+zvHG98qxo6y/ycTcu+Gz9s/W1eLW1sfpv/ahozxicj4KPHN0cm9uZz5CSUdfU0lURTwvc3Ryb25nPiC1vLP2tcSy38LUus1TTUFMTF9TSVRF0rvR+aOstavKx7vxyKFNeVNRTLf+zvHG97XEt723qLrNU01BTExfU0lURbK70rvR+aOsQklHX1NJVEWwtNXVwtbRr7XEt73Kvb2rSElWRcnPtcTK/b7dtbyz9rW9TVlTUUzW0Milo6y8tL3xzOy1xMr9vt21vLP2tb1NWVNRTDHJz6Osw/fM7LXEyv2+3b/JxNy1vLP2tb1NeVNRTDLJz6GjtvhTTUFMTF9TSVRFtcTK/b7dvvm1vLP2tb3Su8yoTVlTUUy3/s7xxvfJz6GjPGJyPgo8c3Ryb25nPkhVR19TSVRFPC9zdHJvbmc+IL2r1b6148O/zOy1xLfDzsrQxc+it9a3orW9srvNrLXETVlTUUy3/s7xxvfJz8ilCjxsaT7Xor3iCgo8bGk+SERGU8K3vrY8YnI+Ci91c2VyL2hpdmUvd2FyZWhvdXNlL3B0bWluZF9kYXRhLmRiLyR7dGFibGVOYW1lfV8ke3RhYmxlVHlwZX0vc2l0ZXR6PSR7dGltZXpvbmV9L3BhcnRkdD0ke2RhdGV9L3BhcnRzaWQ9JHtzaWR9PGJyPgrI5y91c2VyL2hpdmUvd2FyZWhvdXNlL3B0bWluZF9kYXRhLmRiL3N1bV9wYWdlX3Zpc2l0c19zdGF0c19vbGFwX2Qvc2l0ZXR6PUUwODAwL3BhcnRkdD0yMDE0LTA2LTAyL3BhcnRzaWQ9NTZmYmNlNGU8bGk+dGFibGVUeXBlPGJyPgrD98+4se21xMDg0M3OqngsxuTL+7Ht1N3Ksda71qez1szsZAo8cD48L3A+CjxwcmUgY2xhc3M9"brush:java;">private String getTabType(String tableName) {

if (tableName.equals(Constant.TB_1)) {

return x;

}

else {

return d;

}

}

client

简介

部署在MySQL服务器上

执行HDFS2MySQL的导出任务

流程

通过clientBootstrap监控消息

当监控到任务时，执行HDFS2MySQL的导出任务

2.1 通过shell脚本，从HDFS上下载数据

2.2 将元数据更新到MySQL服务器中

2.3 删除本地文件

2.4 根据表中插入行的数目判断数据是否导出成功

2.5 将执行情况返回给master

定时向master发送心跳信息

注解

存储的本地路径：/tmp/ptbalancer/data/${tableName}_${tableType}_${date}_${当前时间戳}

节点间通信

中间件 netty

master ServerBootstrap

client ClientBootstrap

传输数据 PB

相比XML，PB有更好的传输效率、压缩率更高、解析速度更快