Impala create file /tmp/impala-scratch 报错

最新推荐文章于 2022-11-28 17:10:19 发布

Sin_Geek

最新推荐文章于 2022-11-28 17:10:19 发布

阅读量1.3k

点赞数

分类专栏：大数据技术文章标签： impala scratch

本文链接：https://blog.csdn.net/sin_geek/article/details/86128450

版权

在处理大数据量时，Impala遇到错误，原因是其默认使用/tmp/impala-scratch目录存储中间文件。当该目录不存在或权限不足时，Impala无法运行。解决方案是手动在每个impalad节点上创建该目录并设置读写权限。

摘要由CSDN通过智能技术生成

背景

使用impala对大数据量进行处理时出现如下错误

Create file /tmp/impala-scratch/XXX failed with errno=2 description=Error(2): No such file or directory

原因

查资料发现impala在大数据量处理时会用到磁盘保存中间数据

By default, intermediate files used during large sort, join, aggregation, or analytic function operations are stored in the directory /tmp/impala-scratch. These files are removed when the operation finishes. (Multiple concurrent queries can perform operations that use the “spill to disk” technique, without any name conflicts for these temporary files.) You can specify a different location by starting the impalad daemon with the --scratch_dirs=“path_to_directory” configuration option or the equivalent configuration option in the Cloudera Manager user interface. You can specify a single directory, or a comma-separated list of directories. The scratch directories must be on the local filesystem, not in HDFS. You might sp