EsgynDB Troubleshooting-Problem binding to /0.0.0.0:60020 : Address already in use.

现象

EsgynDB所在的集群，HBase有4台RegionServer，因某种原因有一个RegionServer下线。
从CDH Manager中手动启动此RegionServer无法启动，报错信息如下，

2019-08-03 10:40:48,501 ERROR org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegionServer
        at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2706)
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64)
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2721)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2704)
        ... 5 more
Caused by: java.io.IOException: Problem binding to /0.0.0.0:60020 : Address already in use. To switch ports use the 'hbase.regionserver.port' configuration proper
ty.
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:946)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.createRpcServices(HRegionServer.java:665)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:541)
        ... 10 more
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.apache.hadoop.hbase.ipc.RpcServer.bind(RpcServer.java:2571)
        at org.apache.hadoop.hbase.ipc.RpcServer$Listener.<init>(RpcServer.java:577)
        at org.apache.hadoop.hbase.ipc.RpcServer.<init>(RpcServer.java:2040)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:939)
        ... 12 more

解决

根据报错信息，发现是60020端口被占用了，通过命令"netstat -tapn | grep 60020"检查被占用的端口，

tcp        0     42 172.31.234.7:37590      172.31.234.8:60020      ESTABLISHED 64793/java          
tcp6       0      0 172.31.234.7:48390      172.31.234.8:60020      ESTABLISHED 169495/tm           
tcp6       0      0 172.31.234.7:44362      172.31.234.10:60020     ESTABLISHED 357207/mxosrvr      
tcp6       0      0 172.31.234.7:38362      172.31.234.10:60020     ESTABLISHED 357668/mxosrvr      
tcp6       0      0 172.31.234.7:47100      172.31.234.10:60020     ESTABLISHED 169495/tm           
tcp6       0      0 172.31.234.7:54084      172.31.234.9:60020      ESTABLISHED 357668/mxosrvr      
tcp6       0      0 172.31.234.7:47380      172.31.234.8:60020      TIME_WAIT   -                   
tcp6       0      0 172.31.234.7:54064      172.31.234.9:60020      ESTABLISHED 359387/mxosrvr      
tcp6       0      0 172.31.234.7:55884      172.31.234.10:60020     ESTABLISHED 358441/mxosrvr      
tcp6       0      0 172.31.234.7:55282      172.31.234.9:60020      ESTABLISHED 357207/mxosrvr      
tcp6       0      0 172.31.234.7:44243      172.31.234.10:60020     ESTABLISHED 355979/mxosrvr      
tcp6       0      0 172.31.234.7:60020      172.31.234.9:2181       ESTABLISHED 358163/mxosrvr      
tcp6       0      0 172.31.234.7:33496      172.31.234.9:60020      ESTABLISHED 360563/mxosrvr      
tcp6       0      0 172.31.234.7:48068      172.31.234.8:60020      ESTABLISHED 357668/mxosrvr      
tcp6       0      0 172.31.234.7:47348      172.31.234.8:60020      ESTABLISHED 359387/mxosrvr      
tcp6       0      0 172.31.234.7:58994      172.31.234.10:60020     ESTABLISHED 359033/mxosrvr      
tcp6       0      0 172.31.234.7:53806      172.31.234.9:60020      ESTABLISHED 169495/tm           
tcp6       0      0 172.31.234.7:56668      172.31.234.10:60020     ESTABLISHED 357641/mxosrvr      
tcp6       0      0 172.31.234.7:46186      172.31.234.8:60020      ESTABLISHED 357207/mxosrvr      
tcp6       0      0 172.31.234.7:43650      172.31.234.8:60020      ESTABLISHED 169495/tm           
tcp6       0      0 172.31.234.7:33582      172.31.234.9:60020      ESTABLISHED 361781/mxosrvr      
tcp6       0      0 172.31.234.7:42910      172.31.234.10:60020     ESTABLISHED 359387/mxosrvr      
tcp6       0      0 172.31.234.7:49102      172.31.234.10:60020     ESTABLISHED 359462/mxosrvr

我们发现有一条“tcp6 0 0 172.31.234.7:60020 172.31.234.9:2181 ESTABLISHED 358163/mxosrvr”，此mxosrvr进程正占用着60020端口，导致RegionServer无法启动。
手动杀掉此进程号，问题解决。

kill -9 358163