文章目录
概要
问题描述： hiveServer2跑一段时间后, 一般在凌晨就会宕机（凌晨有很多定时任务），宕机比较有规律，就是跑几天就宕一次，分析一下原因。作业没有特别大的任务。
分析方法：jstat, dump
分析结论：hive2的hiveServer本身内存泄漏造成
解决方法：不解决或升级。
一、进行heap dump
1.1 查HiveServer2的进程id，得到219045
root@tbds-10-232-2-37:/data/var> ps -ef|grep HiveServer2
hive 219045 1 92 Feb17 ? 3-13:55:08 /usr/jdk64/jdk1.8.0_111/bin/java -Xmx16384m -Dhdp.version=2.2.0.0-2041 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Dhadoop.log.dir=/data/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/data/bigdata/tbds/usr/hdp/2.2.0.0-2041/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/data/bigdata/tbds/usr/hdp/2.2.0.0-2041/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx16384m -Djava.security.auth.login.config=/etc/hive/conf.server/kafka_client_for_ranger_jaas.conf -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=70 -XX:+PrintGCDateStamps -XX:GCLogFileSize=512M -verbose:gc -XX:+PrintGCDetails -Xloggc:/data/var/log/hive/hive_gc.log -Dlog4j.configurationFile=/etc/hive/conf/hive-log4j2.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.2.0.0-2041/hive/lib/hive-service-2.2.0.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-pig-adapter-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-pig-adapter.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-server-extensions-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-server-extensions.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-streaming-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-streaming.jar -hiveconf hive.metastore.uris=
1.2 根据进程号，查jstat的gc信息
root@tbds-10-x-xx-x:/data/var>jstat -gcutil 219045 2000
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 0.00 0.00 88.12 73.03 36.52 14741 3097.156 3 94.687 3191.843
0.00 0.00 0.00 88.12 73.03 36.52 14741 3097.156 3 94.687 3191.843
0.00 0.00 0.00 88.12 73.03 36.52 14741 3097.156 3 94.687 3191.843
0.00 0.00 0.00 88.12 73.03 36.52 14741 3097.156 3 94.687 3191.843
发现fullGC后，老年代，还是很满(FullGC的次数由2变到3)。
艰难的fullGC,
经过28s的fullGC，顺利完成gc,但老年代92.19%，按照目前趋势 应该明天凌晨就跑挂，看看预言成真不。。。
二、heap dump分析
2.1 把hiveServer2的heap dump回来
经过FullGC后进行的heap dump.
-dump:live 只要存活的对象
hive@tbds-x-x-x-x:/data/var/hiveDump>jmap -dump:live,format=b,file=/data/var/hiveDump/hiveserver2.afterFgc.dump.bin 219045
Dumping heap to /data/var/hiveDump/hiveserver2.dump.bin ...
Heap dump file created
dump回来，发现文件大小17G。
2.2 使用eclipse memory analyzer进行分析
修改EMA的MemoryAnalyzer.ini，把Xmx改为28G, -Xmx28672m.(我的电脑32G)
2.3 leak suspects
发现问题：
问题1： 7627个org.apache.hadoop.hive.ql.exec.UDFClassLoader 占用4.1G内存
问题2：16585个log4jd的RandomAccessFile占用 4.1G内存
问题3：java.util.jar.JarFile占用2.9G
这三个对象把都实例放到HashMap上进行缓存，导致实例永远在FullGC时都可达，导致无法进行回收。
Shallow 与 Retained的区别
Shallow Size
Shallow Size是对象本身占据的内存的大小，不包含其引用的对象。对于常规对象（非数组）的Shallow Size由其成员变量的数量和类型来定，而数组的ShallowSize由数组类型和数组长度来决定，它为数组元素大小的总和。
Retained Size
Retained Size=当前对象大小+当前对象可直接或间接引用到的对象的大小总和。(间接引用的含义：A->B->C,C就是间接引用) ，并且排除被GC Roots直接或者间接引用的对象
换句话说，Retained Size就是当前对象被GC后，从Heap上总共能释放掉的内存。
不过，释放的时候还要排除被GC Roots直接或间接引用的对象。他们暂时不会被被当做Garbage。
最大的retain heap
有busy monitor thread
2.4 dominatorTree对象占用空间比例
下图中，大的对象，无法回收
第一个MedatadataProvider一看就是里面用了享元模式（缓存），数据是无法回收的。其他系统类无法回收，最后的UDFClassLoader占用589k，看上去不大，但有7627个（2.3的问题1中分析出这个数字）。7627个*589k/1024/1024=4.28G。恐怖，全部无法回收。
2.5 整个jstat后台脚本，记录gc情况
脚本
root@tbds-x-x-x-37:/data/var>s nohup jstat -gcutil 219045 2000 > /data/var/hiveserver.gcutil.20220221.log 2>&1 &
第二天上班，发现的确凌晨挂了。
挂前的jstat信息：
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 0.00 0.00 99.99 71.97 35.45 21507 5736.954 547 16373.339 22110.293
0.00 0.00 62.14 99.99 71.97 35.45 21508 5736.954 547 16403.868 22140.822
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.986
0.00 0.00 61.76 100.00 71.97 35.46 21511 5738.118 548 16436.419 22174.538
0.00 0.00 0.00 100.00 71.98 35.46 21516 5740.012 548 16436.419 22176.431
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.149
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.833
0.00 0.00 62.00 99.98 71.98 35.46 21519 5741.179 550 16496.641 22237.821
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.592
0.00 0.00 0.00 99.98 71.99 35.46 21524 5742.970 551 16526.578 22269.549
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.672
0.00 0.00 0.00 99.99 71.99 35.47 21531 5745.369 552 16555.798 22301.168
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.263
0.00 0.00 17.35 99.99 71.99 35.46 21534 5746.465 553 16586.625 22333.090
0.00 0.00 0.00 99.99 71.99 35.47 21540 5748.518 553 16586.625 22335.143
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.881
0.00 0.00 0.00 99.98 71.99 35.46 21545 5750.336 554 16616.478 22366.813
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.500
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.533
0.00 0.00 0.00 99.99 71.98 35.46 21553 5753.008 556 16679.451 22432.459
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.711
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.262
0.00 0.00 59.14 99.98 71.98 35.46 21561 5755.845 558 16736.086 22491.931
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.602
0.00 100.00 52.22 100.00 71.98 35.46 21563 5756.977 559 16762.848 22519.825
0.00 0.00 0.00 99.99 71.97 35.44 21569 5758.853 559 16762.848 22521.700
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.816
0.00 0.00 0.00 99.99 71.97 35.44 21573 5760.294 560 16791.153 22551.447
0.00 0.00 0.00 99.99 71.97 35.44 21577 5762.010 560 16791.153 22553.163
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.808
0.00 0.00 58.51 100.00 71.97 35.44 21579 5762.655 561 16820.409 22583.064
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.183
0.00 0.00 9.57 99.98 71.98 35.44 21584 5764.774 562 16852.386 22617.160
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.946
0.00 0.00 50.54 99.98 71.96 35.45 21586 5765.560 563 16881.137 22646.697
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.328 563 16881.137 22648.465
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.878
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.533
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.521
0.00 0.00 0.00 99.99 71.97 35.45 21599 5769.848 566 16968.008 22737.856
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.600 566 16968.008 22739.608
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.886
0.00 0.00 57.61 100.00 71.97 35.45 21605 5771.878 567 17001.663 22773.541
0.00 0.00 0.00 100.00 71.97 35.45 21610 5773.741 567 17001.663 22775.404
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.503
0.00 0.00 0.00 100.00 71.97 35.45 21619 5776.975 568 17032.080 22809.055
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.788
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.581
0.00 0.00 0.00 99.99 71.96 35.45 21626 5779.559 570 17097.383 22876.941
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.801
0.00 0.00 0.00 99.98 71.96 35.45 21632 5781.952 571 17132.676 22914.628
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.987
0.00 0.00 0.00 99.98 71.96 35.45 21637 5783.815 572 17162.441 22946.256
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.622
0.00 0.00 0.00 99.99 71.96 35.45 21642 5785.868 573 17194.331 22980.199
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.569
0.00 0.00 22.99 100.00 71.96 35.45 21642 5786.238 574 17223.926 23010.164
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.942
0.00 0.00 0.00 99.98 71.96 35.44 21649 5788.379 575 17253.060 23041.439
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.341
0.00 0.00 0.00 99.98 71.96 35.45 21655 5790.690 576 17284.518 23075.208
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.841
0.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17313.260 23105.583
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.396
0.00 0.00 0.00 99.99 71.97 35.45 21662 5793.492 578 17341.110 23134.601
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
0.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705
最后频繁的几十次FullGC都无法回收1%的老年代内存，最终OOM.
3.结论与解决方法
经过分析，随着程序运行，不可回收的对象肯定会原来越多，最终必然导致OOM。
如何解决？
**解决方法：**不解决，延缓问题，增大jvm，延长宕机间隔。升级版本（查查新版中是否解决了本问题）
目前用的hive版本是2.2版
如何查版本？
看lib的后缀：
root@tbds-x-x-x-x:/usr/hdp/2.2.0.0-2041/hive/lib>ls hive*
hive-accumulo-handler-2.2.0.jar hive-common-2.2.0.jar hive-exec-2.2.0.jar.bak.20200616 hive-hplsql-2.2.0.jar hive-llap-common-2.2.0.jar hive-metastore-2.2.0.jar hive-service-rpc-2.2.0.jar hive-shims-scheduler-2.2.0.jar
hive-ant-2.2.0.jar hive-contrib-2.2.0.jar hive-hbase-handler-2.2.0.jar hive-hwi-2.2.0.jar hive-llap-ext-client-2.2.0.jar hive-orc-2.2.0.jar hive-shims-0.23-2.2.0.jar hive-storage-api-2.3.1.jar
hive-beeline-2.2.0.jar hive-druid-handler-2.2.0.jar hive-hcatalog-core-2.2.0.jar hive-jdbc-2.2.0.jar hive-llap-server-2.2.0.jar hive-serde-2.2.0.jar hive-shims-2.2.0.jar hive-testutils-2.2.0.jar
hive-cli-2.2.0.jar hive-exec-2.2.0.jar hive-hcatalog-server-extensions-2.2.0.jar hive-llap-client-2.2.0.jar hive-llap-tez-2.2.0.jar hive-service-2.2.0.jar hive-shims-common-2.2.0.jar
总结
经过分析，对hiveServer 以一定频率宕机的根本原因，得以知晓。
1409
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
