download
http://mirror.bit.edu.cn/apache/spark/spark-3.0.1/
https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.0.1/
初尝试
cd software/spark-3.0.1-bin-hadoop2.7
- 启动 master
./sbin/start-master.sh
访问 http://localhost:8080 得到 spark://MacBook-Pro-2.local:7077
- 启动 Worker
./bin/spark-class org.apache.spark.deploy.worker.Worker spark://MacBook-Pro-2.local:7077
访问 http://localhost:8081 可查看提交的作业
- 提交作业
./bin/spark-submit --master spark://MacBook-Pro-2.local:7077 --class morning.cat.SparkTest /Users/morningcat/Documents/java-project/spark-demo/out/artifacts/spark_demo_jar/spark-demo.jar
package morning.cat;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.rdd.RDD;
public class SparkTest {
public static void main(String[] args) {
SparkConf sparkConf = new SparkConf().setAppName("SparkTest").setMaster("spark://MacBook-Pro-2.local:7077");
SparkContext sparkContext = new SparkContext(sparkConf);
RDD<String> rdd = sparkContext.textFile("/Users/morningcat/Documents/settings.xml", 1);
long lineCount = rdd.count();
System.out.println(lineCount);
}
}
IDEA 打包流程
File -> Project Structure -> Artifaces -> + -> Jar -> From Modules… -> copy to the output… ->
Build -> Build Artifaces… -> xxx:jar -> Build
工程下生成 out 目录