基于hbase2.1.3编译phoenix 5.0.0蹚坑记录

应公司要求，组件版本升级，hadoop3.1.0，hbase2.1.3，背景见《hadoop3.0升级可研》。
因为我们的产品监控是基于phoenix实现，hbase版本升级首先带来的就是phoenix版本升级，目前官方最高版本为基于hbase2.0的phoenix5.0版本，没有提供基于hbase2.1版本，故只能自己编译。

先下载phoenix5.0.0-hbase2.0 tag代码, 打包编译没有问题。
将pom.xml中的hbase版本从2.0.0改到2.0.2，先进行小版本升级实验，mvn打包报错如下：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project phoenix-core: Compilation failure: Compilation failure:
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/util/csv/CsvUpsertExecutor.java:[30,36] 找不到符号
[ERROR] 符号:   类 Base64
[ERROR] 位置: 程序包 org.apache.hadoop.hbase.util
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/util/PhoenixConfigurationUtil.java:[38,36] 找不到符号
[ERROR] 符号:   类 Base64
[ERROR] 位置: 程序包 org.apache.hadoop.hbase.util
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/util/json/JsonUpsertExecutor.java:[31,36] 找不到符号
[ERROR] 符号:   类 Base64
[ERROR] 位置: 程序包 org.apache.hadoop.hbase.util
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/schema/types/PVarbinary.java:[23,36] 找不到符号
[ERROR] 符号:   类 Base64
[ERROR] 位置: 程序包 org.apache.hadoop.hbase.util
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/CsvBulkImportUtil.java:[22,36] 找不到符号
[ERROR] 符号:   类 Base64
[ERROR] 位置: 程序包 org.apache.hadoop.hbase.util
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/util/csv/CsvUpsertExecutor.java:[192,34] 找不到符号
[ERROR] 符号:   变量 Base64
[ERROR] 位置: 类 org.apache.phoenix.util.csv.CsvUpsertExecutor.SimpleDatatypeConversionFunction
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/covered/data/IndexMemStore.java:[84,20] <匿名org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1>中的compare(org.apache.hadoop.hbase.Cell,org.apache.hadoop.hbase.Cell)无法覆盖org.apache.hadoop.hbase.CellComparatorImpl中的compare(org.apache.hadoop.hbase.Cell,org.apache.hadoop.hbase.Cell)
[ERROR] 被覆盖的方法为final
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/util/PhoenixConfigurationUtil.java:[509,16] 找不到符号
[ERROR] 符号:   变量 Base64
[ERROR] 位置: 类 org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/util/PhoenixConfigurationUtil.java:[516,46] 找不到符号
[ERROR] 符号:   变量 Base64
[ERROR] 位置: 类 org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/util/json/JsonUpsertExecutor.java:[215,30] 找不到符号
[ERROR] 符号:   变量 Base64
[ERROR] 位置: 类 org.apache.phoenix.util.json.JsonUpsertExecutor.SimpleDatatypeConversionFunction
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/schema/types/PVarbinary.java:[134,25] 找不到符号
[ERROR] 符号:   变量 Base64
[ERROR] 位置: 类 org.apache.phoenix.schema.types.PVarbinary
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/CsvBulkImportUtil.java:[71,27] 找不到符号
[ERROR] 符号:   变量 Base64
[ERROR] 位置: 类 org.apache.phoenix.mapreduce.CsvBulkImportUtil
[ERROR] /Users/hadoop/Work/opensource/apache/phoenix/phoenix-core/src/main/java/org/apache/phoenix/mapreduce/CsvBulkImportUtil.java:[80,27] 找不到符号
[ERROR] 符号:   变量 Base64
[ERROR] 位置: 类 org.apache.phoenix.mapreduce.CsvBulkImportUtil
[ERROR] -> [Help 1]

发现全是Base64的类的报错，查看相关文件源代码，在2.0.2版本 hbase将该类统一替换为了java.util.Base64。
具体设计替换方法如下：

原方法：Base64.decode(encoded)
替换后方法：Base64.getDecoder().decode(encoded)

原方法：Base64.encodeBytes(endKey)
替换后方法：Bytes.toString(Base64.getEncoder().encode(endKey))

期间通过idea加载项目时，遇到项目老是导入失败的问题，后来发现是maven3.6.0的一个bug，
参考链接https://blog.csdn.net/Atimynyc/article/details/85174381，具体idea log如下

下午3:44	Unable to import maven project: See logs for details
2019-03-20 15:44:21,023 [426266910]   WARN - ution.rmi.RemoteProcessSupport - [RMI TCP Connection(3)-127.0.0.1] [1;36mDEBUG[m org.eclipse.aether.internal.impl.DefaultLocalRepositoryProvider - Using manager EnhancedLocalRepositoryManager with priority 10.0 for /Users/hadoop/Work/software/maven/repository 
2019-03-20 15:44:21,036 [426266923]  Unable to import maven project -      #org.jetbrains.idea.maven - org.eclipse.aether.internal.impl.DefaultArtifactResolver.setLoggerFactory(Lorg/eclipse/aether/spi/log/LoggerFactory;)Lorg/eclipse/aether/internal/impl/DefaultArtifactResolver; 
java.lang.NoSuchMethodError: org.eclipse.aether.internal.impl.DefaultArtifactResolver.setLoggerFactory(Lorg/eclipse/aether/spi/log/LoggerFactory;)Lorg/eclipse/aether/internal/impl/DefaultArtifactResolver;
	at org.jetbrains.idea.maven.server.Maven3ServerEmbedderImpl.resolve(Maven3ServerEmbedderImpl.java:1223)
	at org.jetbrains.idea.maven.server.Maven3ServerEmbedderImpl.doResolve(Maven3ServerEmbedderImpl.java:1178)
	at org.jetbrains.idea.maven.server.Maven3ServerEmbedderImpl.doResolve(Maven3ServerEmbedderImpl.java:1172)
	at org.jetbrains.idea.maven.server.Maven3ServerEmbedderImpl.resolve(Maven3ServerEmbedderImpl.java:1041)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2019-03-20 15:44:21,037 [426266924]  ERROR -      #org.jetbrains.idea.maven - IntelliJ IDEA 2017.3.4  Build #IU-173.4548.28 
2019-03-20 15:44:21,037 [426266924]  ERROR -      #org.jetbrains.idea.maven - JDK: 1.8.0_152-release 
2019-03-20 15:44:21,037 [426266924]  ERROR -      #org.jetbrains.idea.maven - VM: OpenJDK 64-Bit Server VM 
2019-03-20 15:44:21,037 [426266924]  ERROR -      #org.jetbrains.idea.maven - Vendor: JetBrains s.r.o 
2019-03-20 15:44:21,037 [426266924]  ERROR -      #org.jetbrains.idea.maven - OS: Mac OS X 
2019-03-20 15:44:21,037 [426266924]  ERROR -      #org.jetbrains.idea.maven - Last Action: Maven.Reimport

修改Base64问题后，idea build没有报错。

将hbase版本由2.0.2改为2.1.3，build提示CellComparatorImp构造方法参数问题，原来的compare方法用的是两参数方法，现在原来的方法设置了final属性，我这里改为三参数方法，但里面的默认值不变。修改后如下

public IndexMemStore() {
  this(new CellComparatorImpl(){
      @Override
      public int compare(Cell a, Cell b, boolean ignoreSequenceid) {
          return super.compare(a, b, true);
      }
  });
}

idea build没有报错，比想象中的要顺利，原来设想要遇到大量报错。
mvn package打包，得到bin目录，Server和Client包，将其上传到服务器，将Server jar包放到hbase/lib下，然后重启hbase。

通过bin/sqline.py 连接失败，regionserver全挂。查看regionserver日志，发现如下报错。

2019-03-20 17:45:39,705 ERROR [RS_OPEN_PRIORITY_REGION-regionserver/hadoop001:60020-1] coprocessor.CoprocessorHost: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw java.lang.NoClassDefFoundError: org/apache/htrace/Sampler
java.lang.NoClassDefFoundError: org/apache/htrace/Sampler
	at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:520)
	at org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:72)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:263)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:226)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:275)
	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:826)
	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:715)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:7002)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7199)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7172)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7130)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7081)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Sampler
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 24 more

报错显示缺少htrace对应jar包，仔细查了pom文件，发现是整体打包的方式，没有单独的lib依赖，觉得很奇怪。反复编译了好几遍，发现server jar没有变化。追踪这段代码，发现在core工程，把core对应jar包也拷到hbase/lib，还是不行，期间hbase clean --cleanAll也没用，最后没办法只能翻墙。

google了一下，发现hbase 2.1.0版本引入htrace，缺这个包，需要自己拷贝到hbase/lib，真坑。

HBase 2.1.0 release uses HTrace, that is an incubating Apache Foundation project.
There is a folder for 3rd-party libraries in HBase lib folder, client-facing-thirdparty. You need to copy htrace-core-3.1.0-incubating.jar from there to the HBase lib directory. (see reference)
There is also another solution at Cloudera Community that changes a configuration instead of adding the library manually.
https://stackoverflow.com/questions/50585216/noclassdeffounderror-org-cloudera-htrace-sampler-in-apache-phoenix-jdbc
https://stackoverflow.com/questions/53106952/starting-hbase-java-lang-classnotfoundexception-org-apache-htrace-samplerbuild

拷包后问题解决，测试监控功能逻辑正常，后续还需要进行稳定性性能测试。