Directory /data/hadoop/hdfs/data became unmounted from / . Current mount point: /data HDFS,Kafka启动不了

生产环境使用ambari搭建集群，一台机器突然hdfs和kafka启动不了，查看错误日志如下（部分）：

stderr:
2019-10-31 14:39:05,949 -
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
Directory /data/hadoop/hdfs/data became unmounted from / . Current mount point: /data . Please ensure that mounts are healthy. If the mount change was intentional, you can update the contents of /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist.
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****

Traceback (most recent call last):
File “/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py”, line 161, in
DataNode().execute()
File “/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py”, line 329, in execute
method(env)
File “/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py”, line 67, in start
datanode(action=“start”)
File “/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py”, line 89, in thunk
return fn(*args, **kwargs)
File “/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py”, line 68, in datanode
create_log_dir=True
File “/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py”, line 275, in service
Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
File “/usr/lib/python2.6/site-packages/resource_management/core/base.py”, line 166, in init
self.env.run()
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 160, in run
self.run_action(resource, action)
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 124, in run_action
provider_action()
File “/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py”, line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File “/usr/lib/python2.6/site-packages/resource_management/core/shell.py”, line 72, in inner
result = function(command, **kwargs)
File “/usr/lib/python2.6/site-packages/resource_management/core/shell.py”, line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File “/usr/lib/python2.6/site-packages/resource_management/core/shell.py”, line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File “/usr/lib/python2.6/site-packages/resource_management/core/shell.py”, line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of ‘ambari-sudo.sh su hdfs -l -s /bin/bash -c ‘ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode’’ returned 1. starting datanode, logging to /data/log/hadoop/hdfs/hadoop-hdfs-datanode-siger-slave01.shunfeng.com.out
…

网上看了各种办法，比如

/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist
该文件存储每个hdfs文件夹的最后一个安装点。在您的情况下，您似乎正在尝试将HDFS文件夹挂载在不同的路径上，因此datanode不会启动以防止数据丢失。修复文件以指向新的安装点并启动datanode。

然而查看后三个文件都一样，没什么问题
继续回头看日志，一行

/usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh: 第 183 行:echo: 写错误: 设备上没有空间

立马查看磁盘空间

查看以后是zookeeper的out.log占用大量空间
/var/log/zookeeper/zookeeper-zookeeper-server-xxx.out