一、实现任务
1、准备数据文件
- 在
/home
目录创建words.txt
hello scala world
hello spark world
scala is very concise
spark is very powerful
let us learn scala and spark
we can learn them well
- 上传单词文件到HDFS指定目录
hdfs dfs -mkdir p /wordcount/input
hdfs dfs -put words.txt /wordcount/input
2、创建Maven项目
- 创建Maven项目 -
SparkSQLWordCount
3、修改源程序目录
4、添加依赖和设置源程序目录
- 在pom.xml文件里添加依赖和设置源程序目录
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>net.huawei.sql</groupId>
<artifactId>SparkSQLWordCount</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency