升级Phoenix4.7到4.12遇到的问题

最近由于业务需要使用Phoenix的Spark模块，但是Phoenix4.10以下的版本和Spark2.1.1不兼容，所以打算升级Phoenix。
下载地址：
http://archive.apache.org/dist/phoenix/apache-phoenix-4.12.0-HBase-1.1/bin/apache-phoenix-4.12.0-HBase-1.1-bin.tar.gz
解压后将之前的Phoenix的软连接修改到4.12版的文件夹，然后将文件夹下的phoenix-4.12.0-HBase-1.1-server.jar 粘贴到各节点的HBase路径下的lib目录，重启HBase。按道理说接下来就万事大吉了，可是事情没有想象的那么简单，先是Phoenix的sqlline报can’t find locations异常，然后启动HBase的shell，执行命令出现

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
        at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2177)
        at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:821)
        at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:48468)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2112)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
        at java.lang.Thread.run(Thread.java:745)

接下来就是排查的过程了，在监控工具上看到HMaster的状态是健康的，但是所有的RegionServer节点的健康状态都是红色，且16020和16030端口都不可访问，于是去看了一眼RegionServer的日志，发现报出了

java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.LocalIndexMerger
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:155)
        at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.<init>(RegionServerCoprocessorHost.java:68)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893)
        at java.lang.Thread.run(Thread.java:745)

java.lang.RuntimeException: HRegionServer Aborted
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

在网上查找相关问题后，在官网的Release Note上看到

Phoenix-4.8.0 Release Notes
PHOENIX-3164 is a relatively serious bug that affects the Query Server deployed with “security enabled” (Kerberos or Active Directory). Due to another late-game change in the 4.8.0 release as well as an issue with the use of Hadoop’s UserGroupInformation class, every “client session” to the Phoenix Query Server with security enabled will result in a new instance of the Phoenix JDBC driver PhoenixConnection (and other related classes). This ultimately results in a new connection to ZooKeeper for each “client session”.

Within a short amount of time of active use with the Phoenix Query Server creating a new ZooKeeper connection for each “client session”, the number of ZooKeeper connections will have grown rapidly likely triggering ZooKeeper’s built-in denial of service protection (maxClientCnxns). This will cause all future connections to ZooKeeper by the host running the Phoenix Query Server to be dropped. This would prevent all HBase client API calls which need to access ZooKeeper from completing.
As part of PHOENIX-1734 we have changed the local index implementation to store index data in the separate column families in the same data table. So while upgrading the phoenix at server we need to remove below local index related configurations from hbase-site.xml and run upgrade steps mentioned here

<property>
  <name>hbase.master.loadbalancer.class</name>
  <value>org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer</value>
</property>
<property>
  <name>hbase.coprocessor.master.classes</name>
  <value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
</property>
<property>
  <name>hbase.coprocessor.regionserver.classes</name>
  <value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
</property>

在4.8版本之后，安装Phoenix需要把hbase-site.xml文件中的这三个配置项给删掉，因为这三个配置项在4.7及之前的版本会导致在”security enabled”的情况下，每次建立和phoenix server的会话时都会建立一个新的连接实例，短时间内创建大量连接会触发Zookeeper内置的保护机制，拒绝连接。

简单来说，就是HBase在重启的时候，之前赋值到hbase lib目录下的phoenix-4.12.0-HBase-1.1-server.jar包会读取hbase-site.xml文件，结果发现了这三个过期的配置项，然后引起了异常，把这三个配置项删掉就好。