spark sql 本地调试_调试本地spark sql环境

本帖最后由 abc50319 于 2018-8-14 15:37 编辑

1.准备hadoop环境

目录为D:\software\hadoop-2.7.4

1.2 配置hadoop环境变量

先添加HADOOP_HOME

再添加path

1.3 下载hadoop.dll-and-winutils.exe-for-hadoop2.7.3-on-windows_X64-master，

将D:\software\hadoop.dll-and-winutils.exe-for-hadoop2.7.3-on-windows_X64-master\bin目录下的全部文件copy到

D:\software\hadoop-2.7.3\bin下。

1.4 本地调试hadoop代码还要找到winutills.exe的CLASSPATH

1.5 还要把D:\software\hadoop.dll-and-winutils.exe-for-hadoop2.7.3-on-windows_X64-master\bin的hadoop.dll

copy 到C:\Windows\System32，解决这个报错的问题。

Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode

2.将/app/spark-2.1.1-bin-hadoop2.6/conf目录下的hdfs-site.xml和hive-site.xml配置文件放到工程的resour