本地运行mapred问题

1. 问题描述

在hadoop集群上跑mapreduce代码很慢，可以通过设置

conf.set("mapreduce.framework.name", "local");

使mapreduce代码能在本地运行。

运行时报以下错误：

2020-03-14 20:24:30,241 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-03-14 20:24:31,625 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1129)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2020-03-14 20:24:31,626 INFO  [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2020-03-14 20:24:31,924 WARN  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2020-03-14 20:24:31,948 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(250)) - Cleaning up the staging area file:/var/sxt/hadoop/ha/mapred/staging/wangji241520974/.staging/job_local241520974_0001
ExitCodeException exitCode=1: chmod: /private/var/sxt/hadoop/ha/mapred/staging/wangji241520974/.staging/job_local241520974_0001: No such file or directory

2. 解决方法

在/private/var目录下创建sxt目录,并把权限改为777

sudo mkdir sxt
sudo chmod 777 sxt