HDP2.5更换集群IP

场景：
linux centos6.9 Ambari + HDP + Kerberos
目前集群节点有3个，运行一切正常。由于客户ip发生变化，需要统一将原先的ip全部替换。

注：首先将dataNode目录下的数据进行备份

1、通过Ambari界面将所有服务停了

2、修改hosts(win/linux)

（1）修改linux 之hosts（所有节点都得修改）
[root@hdp39 network-scripts]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.22.103 hdp39
192.168.22.40 hdp40
192.168.22.41 hdp41
192.168.22.103 hdp39.bigdata.com
192.168.22.40 hdp40.bigdata.com
192.168.22.41 hdp41.bigdata.com

注：只修改ip不修改映射名，原因是在原先环境中有很多地方用的是映射名，如果映射名没改那些地方便不需要再进行修改，如下为原先的hosts文件内容
（2）修改window 之host

      C:\Windows\System32\drivers\etc\hosts

（3）修改当前ip映射的信息

        /etc/sysconfig/network-scripts/ifcfg-Auto_eth0

查找当前映射信息配置文件的命令
[root@hdp39 network-scripts]# cd /etc/sysconfig/network-scripts
[root@hdp39 network-scripts]# grep -rn 192.168.22.39           
ifcfg-Auto_eth0_bak:4:IPADDR=192.168.22.39
Binary file .ifcfg-Auto_eth0.swp matches

（4）重启网络服务

 命令：
  service network restart或/etc/init.d/network restart

[root@hdp39 network-scripts]# service network restart
Shutting down interface Auto_eth0:  Device state: 3 (disconnected)
                                                           [  OK  ]
Shutting down interface Auto_eth0_bak:                     [  OK  ]
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface Auto_eth0:  Active connection state: activating
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/2
state: activated
Connection activated
                                                           [  OK  ]
Bringing up interface Auto_eth0_bak:  Active connection state: activated
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/3
                                                           [  OK  ]
Bringing up interface eth0:  Error: No suitable device found: no device found for connection 'System eth0'.
                                                           [FAILED] 

解决：

cd /etc/udev/rules.d
rm 70-persistent-cd.rules

验证网络访问是否正常

ping baidu.com

（5）重启服务器

 init 6 或reboot

（6）检查防火墙是否开启

[root@hdp39 ~]# service iptables status
iptables: Firewall is not running.

（7）重启 Ldap

[root@hdp39 ~]# find / -name 'ldap.sh'
    /usr/hdp/2.5.3.0-37/knox/bin/ldap.sh
[root@hdp39 bin]# ./ldap.sh start
    Starting LDAP succeeded with PID 5330. 

（8）开启Ambari并登录Ambari

    启动Ambari
     ambari-server start 

    通过界面访问ambari
     http://<your.ambari.server>:8080  
由于Ambari之前配置了knox单点登录，因此会自动跳转到knox单点登录对应的 

为了让其不跳转到knox中，因此在跳转之前快速停止浏览器的跳转，
也就是直接跳转到ambari主页，其url 为http://<your.ambari.server>:8080  

（9）通过Abmari 界面修改有写了固定 IP 地址的地方。

 修改 yarn、hdfs、mapreduce、zookeeper 等组件中有写了ip地址的配置文件。

遇到的坑：

一、修改完之后ambari 后与hosts 后，重启服务器发现登录不进ambari 主界面
原因一：重启服务器后防火墙被重新打开了
原因二：knox 对应的Ldap 服务没有开户

参考文档: knox官方文档
http://knox.apache.org/books/knox-0-12-0/user-guide.html

[root@hdp39 bin]# ./ldap.sh start
Starting LDAP succeeded with PID 46528.

解决方法：在集群正常运行的情况下先将knox 停了，修改knox主机ip

---

二、Ambari Metrics 启动失败
报的错如下：

 org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.query.DefaultPhoenixDataSource:
 Unable to connect to HBase store using Phoenix.
org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG.

This is usually due to AMS Data being corrupt.

1、Shut down Ambari Monitors, and Collector via Ambari
2、Cleared out the /var/lib/ambari-metrics-collector dir for fresh restart
3、From Ambari -> Ambari Metrics -> Config -> Advanced ams-hbase-site get the hbase.rootdir and hbase-tmp directory

4、Delete or Move the hbase-tmp and hbase.rootdir directories to an archive folder

Started AMS.
All services will came online and graphs started to display, after a few minutes

参考文档：
https://community.hortonworks.com/articles/11805/how-to-solve-ambari-metrics-corrupted-data.html

---

三、dfs.data.dir 配置导致datanode无法启动

报错如下所示：

查看日志如下：（日志为：less /var/log/hadoop/hdfs/hadoop-hdfs-datanode-hdp40.log）

2017-12-20 12:51:21,201 WARN  datanode.DataNode (DataNode.java:checkStorageLocations(2524)) - Invalid dfs.datanode.data.dir /bigdata/hadoop/hdfs/data : 
java.io.FileNotFoundException: File file:/bigdata/hadoop/hdfs/data does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:624)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:850)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:614)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:422)
        at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:139)
        at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:156)
        at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2479)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2521)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2503)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2395)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2442)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2623)
        at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
2017-12-20 12:51:21,203 ERROR datanode.DataNode (DataNode.java:secureMain(2630)) - Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/bigdata/hadoop/hdfs/data" 
        at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2530)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2503)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2395)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2442)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2623)
        at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
2017-12-20 12:51:21,204 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2017-12-20 12:51:21,208 INFO  datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: 

解决：
（1）确认在更换了Ip 之后，该机器确实不存在该目录了

[root@hdp40 hdfs]# cd /bigdata/hadoop/hdfs/data
-bash: cd: /bigdata/hadoop/hdfs/data: No such file or directory

（2）手动创建对应的目录

     mkdir -p hdfs/data

（3）改变指定目录以及其子目录下的所有文件的拥有者和群组

[root@hdp40 hdfs]# chown -R -v hdfs:hadoop data
changed ownership of `data' to hdfs:hadoop
[root@hdp40 hdfs]# ll
total 4
drwxr-xr-x 2 hdfs hadoop 4096 Dec 20 16:07 data

如果权限不是755则给他授权
[root@hdp40 hdfs]# chmod -R 755 data/

注：上述操作需要在所有做为datanode节点的机器上进行操作。

执行完上述步骤后：
重启dataNode，执行正常（笔者第一次重启的时候依然是报错，但是有点奇怪的是第二重启便可以正常启动了，只是启动的时间久了点）

四、Ranger启动失败

报错如下所示：

Watcher org.apache.solr.common.cloud.ConnectionManager@7b1083f6 name:ZooKeeperConnection Watcher:hdp39:2181,hdp40:2181,hdp41:2181/infra-solr got event WatchedEvent state:AuthFailed type:None path:null path:null type:None
zkClient received AuthFailed
makePath: /configs/ranger_audits/managed-schema
Error uploading file /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf/managed-schema to zookeeper path /configs/ranger_audits/managed-schema
java.io.IOException: Error uploading file /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf/managed-schema to zookeeper path /configs/ranger_audits/managed-schema
    at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:69)
    at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:59)
    at java.nio.file.Files.walkFileTree(Files.java:2670)
    at java.nio.file.Files.walkFileTree(Files.java:2742)
    at org.apache.solr.common.cloud.ZkConfigManager.uploadToZK(ZkConfigManager.java:59)
    at org.apache.solr.common.cloud.ZkConfigManager.uploadConfigDir(ZkConfigManager.java:121)
    at org.apache.ambari.logsearch.solr.commands.UploadConfigZkCommand.executeZkConfigCommand(UploadConfigZkCommand.java:39)
    at org.apache.ambari.logsearch.solr.commands.UploadConfigZkCommand.executeZkConfigCommand(UploadConfigZkCommand.java:29)
    at org.apache.ambari.logsearch.solr.commands.AbstractZookeeperConfigCommand.executeZkCommand(AbstractZookeeperConfigCommand.java:38)
    at org.apache.ambari.logsearch.solr.commands.AbstractZookeeperRetryCommand.createAndProcessRequest(AbstractZookeeperRetryCommand.java:38)
    at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
    at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
    at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.uploadConfiguration(AmbariSolrCloudClient.java:218)
    at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:469)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /configs
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:294)
    at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:291)
    at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
    at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:291)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:486)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:408)
    at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:67)
    ... 13 more

解决方式：

• 将各服务器的时间同步便解决了。观察上述日志发现有关Kerberos认证的错误，因而想到的第一点可能是服务器时间不同步所引起的。

五、有些服务启动成功后又立刻失败

• 当遇到那些启动成功后又很快就自动停止的，首先是确认该服务是否已有一个启动了，但是还没有关闭或者看该服务端口是否被占用了。
• 如：hive 的metastore 启动成功又立马自动停止。查看对应的端口发现已有对应的服务启动了，这种情况kill掉，重启便可。