sjmz30071360的专栏

操作系统：Linux（CenOS7）Spark集群至少需要有JDK、Hadoop、Scala、Spark一、安装Spark依赖的Scala（每个节点都安装）1.1 下载Scala（2.11.12）wget "https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz"1.2 解压、配置环境变量vi /etc...

submit_spark.sh#!/bin/bash################################################### @date 2018/11/03# @author sjmz# @desc 启动spark作业示例##################################################HOME="/ho...

一、新建项目输入项目名称：SparkDemo_WordCount，点击“完成”项目创建成功，如下所示："C:\Program Files\Java\jdk1.8.0_45\bin\java" -Dmaven.multiModuleProjectDirectory=C:\Users\Administrator\AppData\Local\Temp\archetype1t...

一、Spark读取Linux本地文件val textFile = spark.read.textFile("file:///usr/spark-2.3.1-bin-hadoop2.7/README.md")[root@master spark-2.3.1-bin-hadoop2.7]# ./bin/spark-shell2019-01-06 21:48:02 WARN NativeC...

参看网上查阅的一些资料，结合自己实践，整理如下：方法一、程序中设置一个开关来停止服务（推荐）当检查到HDFS中存在"/user/root/stop_sparkStreaming"目录时，则优雅地停止服务package com.sjmz.sparkdemoimport org.apache.hadoop.conf.Configurationimport org.apache.ha...

Printing elements of an RDDAnother common idiom is attempting to print out the elements of an RDD using rdd.foreach(println) or rdd.map(println). On a single machine, this will generate the expected...