Spark SQL典型案例

一、实现任务

1、准备数据文件

• 在/home目录创建words.txt
  

hello scala world
hello spark world
scala is very concise
spark is very powerful
let us learn scala and spark
we can learn them well

• 上传单词文件到HDFS指定目录

hdfs dfs -mkdir p /wordcount/input
hdfs dfs -put words.txt /wordcount/input

2、创建Maven项目

• 创建Maven项目 - SparkSQLWordCount
  

3、修改源程序目录

4、添加依赖和设置源程序目录

• 在pom.xml文件里添加依赖和设置源程序目录

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>net.huawei.sql</groupId>
    <artifactId>SparkSQLWordCount</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency