解决此问题之前要首先检查hdfs连接是否有问题,如果如下图所示位置的连接标错说明windows下hadoop的环境变量未配置:
此时在黑窗口执行 hadoop version如果出现以下界面则hadoop环境变量配置成功
如果此时解析运行该转换会出现以下错误:
2024/09/12 13:25:53 - Hadoop file output.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : Unexpected error
2024/09/12 13:25:53 - Hadoop file output.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : org.pentaho.di.core.exception.KettleStepException:
2024/09/12 13:25:53 - Hadoop file output.0 - File name field [null] couldn't be found in the input stream!
2024/09/12 13:25:53 - Hadoop file output.0 -
2024/09/12 13:25:53 - Hadoop file output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.getOutputFileName(TextFileOutput.java:265)
2024/09/12 13:25:53 - Hadoop file output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.writeRowToFile(TextFileOutput.java:352)
2024/09/12 13:25:53 - Hadoop file output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.writeRowTo(TextFileOutput.java:468)
2024/09/12 13:25:53 - Hadoop file output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.processRow(TextFileOutput.java:461)
2024/09/12 13:25:53 - Hadoop file output.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2024/09/12 13:25:53 - Hadoop file output.0 - at java.lang.Thread.run(Thread.java:750)
2024/09/12 13:25:53 - Hadoop file output.0 - 完成处理 (I=0, O=0, R=1, W=0, U=0, E=1)
2024/09/12 13:25:53 - 表输入.0 - Finished reading query, closing connection.
2024/09/12 13:25:53 - hadoop-driver - 转换被检测
2024/09/12 13:25:53 - hadoop-driver - 转换正在杀死其他步骤!
2024/09/12 13:25:53 - 表输入.0 - 完成处理 (I=34, O=0, R=0, W=32, U=0, E=0)
2024/09/12 13:25:53 - hadoop-driver - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : 错误被检测到!
2024/09/12 13:25:53 - Spoon - 转换完成!!
2024/09/12 13:25:53 - hadoop-driver - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : 错误被检测到!
2024/09/12 13:25:53 - hadoop-driver - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : 错误被检测到!
然后我们重启kettle再次在来南京Hdfs的页面点击测试图一所示的位置连接成功则hdfs就连接成功了。然后我们运行创建好的转换(mysql表输入和hadoopoutput输出)此时发现出现了以下错误:
2024/09/12 18:18:06 - Hadoop file output.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : Unexpected error
2024/09/12 18:18:06 - Hadoop file output.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : org.pentaho.di.core.exception.KettleException:
2024/09/12 18:18:06 - Hadoop file output.0 - Error opening new file : java.lang.RuntimeException: com.ctc.wstx.exc.WstxIOException: Invalid UTF-8 middle byte 0x3f (at char #1450, byte #20)
2024/09/12 18:18:06 - Hadoop file output.0 -
显而易见就是我们文件的格式除了问题,此时我们回到kettle的启动文件下:
在如下图所示的位置加入我们解析utf-8的代码(位置不是特别好找建议索引查找一下):
"-Dfile.encoding=UTF-8"
此时再运行我们的转换即可