spark 2.2.0 各个计算因子的使用方法 java版

最新推荐文章于 2021-09-22 16:17:42 发布
最新推荐文章于 2021-09-22 16:17:42 发布
阅读量686 收藏
点赞数
文章标签： spark 计算因子 Transformation
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u011654843/article/details/79046572
版权

map:

	private static void map(){
		SparkConf conf =new SparkConf().setAppName("map").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Integer> numbers = Arrays.asList(1,2,3,4,5);
	    JavaRDD<Integer> numberRDD = sc.parallelize(numbers);
	    JavaRDD<Integer> multipleNumberRDD = numberRDD.map(new Function<Integer,Integer>(){

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public Integer call(Integer v1) throws Exception {
				// TODO Auto-generated method stub
				return v1*2;
			}
	    });
	    multipleNumberRDD.foreach(new VoidFunction<Integer>() {
			
			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Integer t) throws Exception {
				System.out.println(t);
				
			}
		});
		sc.close();
	}

filter

private static void filter(){
		SparkConf conf =new SparkConf().setAppName("map").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Integer> numbers =Arrays.asList(1,2,3,4,5,6,7,8,9,10);
		JavaRDD<Integer> numberRDD =sc.parallelize(numbers);
		JavaRDD<Integer> evennumberRDD= numberRDD.filter(new Function<Integer, Boolean>() {

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public Boolean call(Integer v1) throws Exception {
				// TODO Auto-generated method stub
				return v1%2==0;
			}
		});
		evennumberRDD.foreach(new VoidFunction<Integer>() {

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Integer t) throws Exception {
				System.out.println(t);
				
			}
			
		});
		sc.close();
	}

flatmap:

private static void flatMap(){
		SparkConf conf =new SparkConf().setAppName("flatMap").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<String> linelist = Arrays.asList("hello you","hello me","hello world");
		JavaRDD<String> lines = sc.parallelize(linelist);
		JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String,String>() {

			@Override
			public Iterator<String> call(String t) throws Exception {
				
				return Arrays.asList(t.split(" ")).iterator();
			}
		});
		words.foreach(new VoidFunction<String>() {

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public void call(String t) throws Exception {
				System.out.println(t);
				
			}
			
		});
		sc.close();
	}
groupByKey: 
private static void groupByKey(){
		SparkConf conf =new SparkConf().setAppName("groupByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<String, Integer>> scoreList = Arrays.asList(
				new Tuple2<String, Integer>("class1",80),
				new Tuple2<String, Integer>("class2",75),
				new Tuple2<String, Integer>("class1",90),
				new Tuple2<String, Integer>("class2",65));
		JavaPairRDD<String, Integer> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<String, Iterable<Integer>> groupScorese=scores.groupByKey();
		groupScorese.foreach(new VoidFunction<Tuple2<String,Iterable<Integer>>>() {

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Tuple2<String, Iterable<Integer>> t)
					throws Exception {
				System.out.println("class: " + t._1);
				Iterator<Integer> ite =t._2.iterator();
				while(ite.hasNext()){
					System.out.println(ite.next());
				}
				System.out.println("=====================================");
				
				
			}
			
		});
		
		sc.close();
	}
reduceByKey: 

private static void reduceByKey(){
		SparkConf conf =new SparkConf().setAppName("reduceByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<String, Integer>> scoreList = Arrays.asList(
				new Tuple2<String, Integer>("class1",80),
				new Tuple2<String, Integer>("class2",75),
				new Tuple2<String, Integer>("class1",90),
				new Tuple2<String, Integer>("class2",65));
		JavaPairRDD<String, Integer> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<String, Integer> reduceScorese=scores.reduceByKey(new Function2<Integer, Integer, Integer>(){

			@Override
			public Integer call(Integer arg0, Integer arg1) throws Exception {
				// TODO Auto-generated method stub
				return arg0+arg1;
			}});
		    reduceScorese.foreach(new VoidFunction<Tuple2<String,Integer>>() {

				@Override
				public void call(Tuple2<String, Integer> t) throws Exception {
					System.out.println(t._1+":"+t._2);
					
				}
			});
		
		sc.close();
	}
sortByKey: 

private static void sortByKey(){
		SparkConf conf =new SparkConf().setAppName("reduceByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<Integer, String>> scoreList = Arrays.asList(
				new Tuple2<Integer, String>(65,"ieo"),
				new Tuple2<Integer, String>(50,"tom"),
				new Tuple2<Integer, String>(100,"marry"),
				new Tuple2<Integer, String>(80,"jack"));

		JavaPairRDD<Integer, String> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<Integer,String> sortScores =scores.sortByKey(false);
		sortScores.foreach(new VoidFunction<Tuple2<Integer,String>>(){

			@Override
			public void call(Tuple2<Integer, String> t) throws Exception {
				System.out.println(t._1+ ": "+t._2);
				
			}
			
		});
		sc.close();
	}
join: 

private static void join(){
		SparkConf conf =new SparkConf().setAppName("reduceByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<Integer,String>> studentList =Arrays.asList(
				new Tuple2<Integer,String>(1, "leo"),
				new Tuple2<Integer,String>(2, "jack"),
				new Tuple2<Integer,String>(3, "tom"));
		List<Tuple2<Integer,Integer>> scoreList =Arrays.asList(
				new Tuple2<Integer,Integer>(1, 100),
				new Tuple2<Integer,Integer>(2, 90),
				new Tuple2<Integer,Integer>(3, 60));
		JavaPairRDD<Integer, String> students = sc.parallelizePairs(studentList);
		JavaPairRDD<Integer, Integer> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<Integer, Tuple2<String, Integer>> studentScores = students.join(scores);
		studentScores.foreach(new VoidFunction<Tuple2<Integer,Tuple2<String,Integer>>>() {

			@Override
			public void call(Tuple2<Integer, Tuple2<String, Integer>> t)
					throws Exception {
				System.out.println(t._1+":"+t._2._1+":"+t._2._2);
				
			}
		});
		
		sc.close();
	}
cogroup: 
private static void cogroup(){
		SparkConf conf =new SparkConf().setAppName("reduceByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<Integer,String>> studentList =Arrays.asList(
				new Tuple2<Integer,String>(1, "leo"),
				new Tuple2<Integer,String>(2, "jack"),
				new Tuple2<Integer,String>(3, "tom"));
		List<Tuple2<Integer,Integer>> scoreList =Arrays.asList(
				new Tuple2<Integer,Integer>(1, 100),
				new Tuple2<Integer,Integer>(2, 90),
				new Tuple2<Integer,Integer>(3, 60),
				new Tuple2<Integer,Integer>(1, 70),
				new Tuple2<Integer,Integer>(2, 80),
				new Tuple2<Integer,Integer>(3, 50));
		JavaPairRDD<Integer, String> students = sc.parallelizePairs(studentList);
		JavaPairRDD<Integer, Integer> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<Integer, Tuple2<Iterable<String>, Iterable<Integer>>> studentScores = students.cogroup(scores);
		studentScores.foreach(new VoidFunction<Tuple2<Integer,Tuple2<Iterable<String>,Iterable<Integer>>>>() {

			@Override
			public void call(
					Tuple2<Integer, Tuple2<Iterable<String>, Iterable<Integer>>> t)
					throws Exception {
				System.out.println(t._1+":"+t._2._1+":"+t._2._2);
				
			}
		}

			
		);
		
		sc.close();
	}
全文件： 

package cn.spark.study.core;

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Arrays;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

public class TransformationOperation {
	public static void main(String[] args) {
	//	map();
	//	filter();
	//	flatMap();
	//	groupByKey();
	//	reduceByKey();
	//	sortByKey();
	//	join();
		cogroup();
	}
	
	private static void map(){
		SparkConf conf =new SparkConf().setAppName("map").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Integer> numbers = Arrays.asList(1,2,3,4,5);
	    JavaRDD<Integer> numberRDD = sc.parallelize(numbers);
	    JavaRDD<Integer> multipleNumberRDD = numberRDD.map(new Function<Integer,Integer>(){

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public Integer call(Integer v1) throws Exception {
				// TODO Auto-generated method stub
				return v1*2;
			}
	    });
	    multipleNumberRDD.foreach(new VoidFunction<Integer>() {
			
			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Integer t) throws Exception {
				System.out.println(t);
				
			}
		});
		sc.close();
	}
	private static void filter(){
		SparkConf conf =new SparkConf().setAppName("map").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Integer> numbers =Arrays.asList(1,2,3,4,5,6,7,8,9,10);
		JavaRDD<Integer> numberRDD =sc.parallelize(numbers);
		JavaRDD<Integer> evennumberRDD= numberRDD.filter(new Function<Integer, Boolean>() {

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public Boolean call(Integer v1) throws Exception {
				// TODO Auto-generated method stub
				return v1%2==0;
			}
		});
		evennumberRDD.foreach(new VoidFunction<Integer>() {

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Integer t) throws Exception {
				System.out.println(t);
				
			}
			
		});
		sc.close();
	}
	private static void flatMap(){
		SparkConf conf =new SparkConf().setAppName("flatMap").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<String> linelist = Arrays.asList("hello you","hello me","hello world");
		JavaRDD<String> lines = sc.parallelize(linelist);
		JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String,String>() {

			@Override
			public Iterator<String> call(String t) throws Exception {
				
				return Arrays.asList(t.split(" ")).iterator();
			}
		});
		words.foreach(new VoidFunction<String>() {

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public void call(String t) throws Exception {
				System.out.println(t);
				
			}
			
		});
		sc.close();
	}
	private static void groupByKey(){
		SparkConf conf =new SparkConf().setAppName("groupByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<String, Integer>> scoreList = Arrays.asList(
				new Tuple2<String, Integer>("class1",80),
				new Tuple2<String, Integer>("class2",75),
				new Tuple2<String, Integer>("class1",90),
				new Tuple2<String, Integer>("class2",65));
		JavaPairRDD<String, Integer> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<String, Iterable<Integer>> groupScorese=scores.groupByKey();
		groupScorese.foreach(new VoidFunction<Tuple2<String,Iterable<Integer>>>() {

			/**
			 * 
			 */
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Tuple2<String, Iterable<Integer>> t)
					throws Exception {
				System.out.println("class: " + t._1);
				Iterator<Integer> ite =t._2.iterator();
				while(ite.hasNext()){
					System.out.println(ite.next());
				}
				System.out.println("=====================================");
				
				
			}
			
		});
		
		sc.close();
	}
	private static void reduceByKey(){
		SparkConf conf =new SparkConf().setAppName("reduceByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<String, Integer>> scoreList = Arrays.asList(
				new Tuple2<String, Integer>("class1",80),
				new Tuple2<String, Integer>("class2",75),
				new Tuple2<String, Integer>("class1",90),
				new Tuple2<String, Integer>("class2",65));
		JavaPairRDD<String, Integer> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<String, Integer> reduceScorese=scores.reduceByKey(new Function2<Integer, Integer, Integer>(){

			@Override
			public Integer call(Integer arg0, Integer arg1) throws Exception {
				// TODO Auto-generated method stub
				return arg0+arg1;
			}});
		    reduceScorese.foreach(new VoidFunction<Tuple2<String,Integer>>() {

				@Override
				public void call(Tuple2<String, Integer> t) throws Exception {
					System.out.println(t._1+":"+t._2);
					
				}
			});
		
		sc.close();
	}
	private static void sortByKey(){
		SparkConf conf =new SparkConf().setAppName("reduceByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<Integer, String>> scoreList = Arrays.asList(
				new Tuple2<Integer, String>(65,"ieo"),
				new Tuple2<Integer, String>(50,"tom"),
				new Tuple2<Integer, String>(100,"marry"),
				new Tuple2<Integer, String>(80,"jack"));

		JavaPairRDD<Integer, String> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<Integer,String> sortScores =scores.sortByKey(false);
		sortScores.foreach(new VoidFunction<Tuple2<Integer,String>>(){

			@Override
			public void call(Tuple2<Integer, String> t) throws Exception {
				System.out.println(t._1+ ": "+t._2);
				
			}
			
		});
		sc.close();
	}
	private static void join(){
		SparkConf conf =new SparkConf().setAppName("reduceByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<Integer,String>> studentList =Arrays.asList(
				new Tuple2<Integer,String>(1, "leo"),
				new Tuple2<Integer,String>(2, "jack"),
				new Tuple2<Integer,String>(3, "tom"));
		List<Tuple2<Integer,Integer>> scoreList =Arrays.asList(
				new Tuple2<Integer,Integer>(1, 100),
				new Tuple2<Integer,Integer>(2, 90),
				new Tuple2<Integer,Integer>(3, 60));
		JavaPairRDD<Integer, String> students = sc.parallelizePairs(studentList);
		JavaPairRDD<Integer, Integer> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<Integer, Tuple2<String, Integer>> studentScores = students.join(scores);
		studentScores.foreach(new VoidFunction<Tuple2<Integer,Tuple2<String,Integer>>>() {

			@Override
			public void call(Tuple2<Integer, Tuple2<String, Integer>> t)
					throws Exception {
				System.out.println(t._1+":"+t._2._1+":"+t._2._2);
				
			}
		});
		
		sc.close();
	}
	private static void cogroup(){
		SparkConf conf =new SparkConf().setAppName("reduceByKey").setMaster("local");
		JavaSparkContext sc =new JavaSparkContext(conf);
		List<Tuple2<Integer,String>> studentList =Arrays.asList(
				new Tuple2<Integer,String>(1, "leo"),
				new Tuple2<Integer,String>(2, "jack"),
				new Tuple2<Integer,String>(3, "tom"));
		List<Tuple2<Integer,Integer>> scoreList =Arrays.asList(
				new Tuple2<Integer,Integer>(1, 100),
				new Tuple2<Integer,Integer>(2, 90),
				new Tuple2<Integer,Integer>(3, 60),
				new Tuple2<Integer,Integer>(1, 70),
				new Tuple2<Integer,Integer>(2, 80),
				new Tuple2<Integer,Integer>(3, 50));
		JavaPairRDD<Integer, String> students = sc.parallelizePairs(studentList);
		JavaPairRDD<Integer, Integer> scores = sc.parallelizePairs(scoreList);
		JavaPairRDD<Integer, Tuple2<Iterable<String>, Iterable<Integer>>> studentScores = students.cogroup(scores);
		studentScores.foreach(new VoidFunction<Tuple2<Integer,Tuple2<Iterable<String>,Iterable<Integer>>>>() {

			@Override
			public void call(
					Tuple2<Integer, Tuple2<Iterable<String>, Iterable<Integer>>> t)
					throws Exception {
				System.out.println(t._1+":"+t._2._1+":"+t._2._2);
				
			}
		}

			
		);
		
		sc.close();
	}
}


豆筱豆1993
关注
[Spark版本升级]-- spark-2.2.0发行说明
欢迎来到我的博客，一起探索代码里的世界！
07-18 6611
Spark-2.2.0版本发行时间：2017-7-11   一、Jira说明： https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&amp;version=12338275   子任务 [ SPARK-1267 ] - 添加PySpark的pip安装程序 [ SPARK-3249 ] - ...
[Spark版本更新]--2.3.0发行说明
热门推荐
欢迎来到我的博客，一起探索代码里的世界！
03-03 1万+
自从2017年12月1日发布spark-2.2.1以来，已有3个月时间。2018年2月28日，spark官方发布了一个大版本Spark-2.3.0，解决了1399个大大小小的问题。一、DataBricks做了相关说明今天，我们很高兴地宣布Databricks上的Apache Spark 2.3.0作为其Databricks Runtime 4.0的一部分。我们要感谢Apache Spark社区为S...
spark2.1.0 JAVA API
02-06
spark 通过java操作的API详细说明，chm格式，可搜索自己想要的API，内部有详细的用法说明
spark 2.2.0 各个计算因子的使用方法 python版
豆筱豆的博客
01-12 490
map:def map(): sc = SparkContext("spark://node0:7077", "map") list=[1,2,3,4,5] listRdd=sc.parallelize(list) listmap =listRdd.map(lambda s:s*2) print listmap.collect() sc.stop()
spark 2.2.0 action操作 java版
豆筱豆的博客
01-14 421
package cn.spark.study.core; import java.util.List; import java.util.Arrays; import java.util.Map; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaPairRDD; import org.apache.s
Java-Spark系列2-quick-start
只是甲的博客
09-22 1万+
文章目录一.idea搭建maven工程二.Maven编译Java应用程序2.1 Java代码三.导出maven工程四.运行jar程序参考: 一.idea搭建maven工程 前面已经有博客介绍idea下搭建maven工程了，这里略过，主要介绍下pom.xml的配置。 <project> <groupId>edu.berkeley</groupId> <artifactId>simple-project</artifactId> <
foreachrdd mysql_spark - mysql
weixin_28991715的博客
01-26 157
在spark的数据源中，只支持Append, Overwrite, ErrorIfExists, Ignore,这几种模式，但是我们在线上的业务几乎全是需要upsert功能的，就是已存在的数据肯定不能覆盖，在mysql中实现就是采用：ON DUPLICATE KEY UPDATE，有没有这样一种实现？官方：不好意思，不提供，dounine：我这有呀，你来用吧。哈哈，为了方便大家的使用我已经把项目打...
Spark弹性分布式数据集编程向导(Java)
qqq83150li的博客
02-28 232
概览 1、大体上说，每一个Spark应用都是一个驱动程序，包含两部分，一部分是运行用户的主函数，另一部分是在集群中执行多种并行任务。 2、Spark提供了（抽象出来了）RDD弹性分布式数据集，是一个在集群中跨节点的分区的元素集合，可以进行并行操作。 3、RDD的创建有两种方式，第一种是由hadoop文件系统中的文件创建，也就是hdfs文件，也可以是hadoop支持的其它文件系统。第二种是由驱动程序...
spark的使用
pengfeijiuwanli的博客
05-16 1725
Apache Spark 什么是Spark？ Spark是Lightning-fast unified analytics engine- 快如闪电的统一的分析引擎（不参与数据持久化）。 快 （1）Spark基于内存的计算引擎，相比于MapReduce磁盘计算，速度自然快- 大众认知 （2）Spark使用先进的DAG（矢量计算）计算模型，将一个复杂的任务拆分成若干个stage（阶段）， 这样复杂的任务Spark只需要一个job即可完成。（如果使用MapReduce计算模型可能需要串连若干个Job） （3）
Apache Spark 基础知识总结及应用示例
击水三千里的专栏
07-09 1825
对比与FlinkSpark 基于微批量处理，Spark 对 SQL 支持更好，Spark 的社区更加活跃，Spark 对机器学习的支持很好
Apache Spark 2.x for Java Developers
10-08
Apache Spark 2.x for Java Developers,使用Java开发Spark
基于Spark的主成分分析和因子分析并行化的研究与实现.zip
01-11
基于Spark的主成分分析和因子分析并行化的研究与实现.zip
Spark 计算框架
ziwenCSDN的博客
11-03 2332
Apache Spark™是用于大规模数据处理的快速和通用引擎.
Spark常用转换因子介绍
梦里蓝天
09-20 1497
Map(f): 作用：使RDD数据集合中的每一个数据与另一个数据组合，返回最终组合后的新的RDD； var res = Array(10,30,40,60,80,90,56,5,9); var rdd = res.map((_,1)); 结果：(10,1)(30,1)(40,1)(60,1)(80,1)(90,1)(56,1)(5,1)(9,1)
spark2.1.0配置windows本地基于java语言的Eclipse开发环境
ASN_forever的博客
12-03 1226
前面配置过在windows本地通过Eclipse开发hadoop程序然后打成jar包在Linux集群中运行的经历，因此一方面是基于同样的需求，另一方面是因为我是spark小白，想通过这种方式快速了解和学习spark开发。 （注意：此文配置的是基于java语言开发spark的，如果要用scala开发，则需要在Eclipse中添加scala插件并做进一步配置。如果要用python开发，则最好用pyc...
Java Spark2.1.0 读取文本写入MySQL
向阳争渡
09-13 798
import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.function.Function; import org.apache.spark.sql.Dataset; import org.apache.spark.sql.RowFactory; import org.apache.spark.sql.S...
因子回测效果框架
kangzuoning的博客
08-03 1187
输入：开始日期、结束日期、因子名、调仓间隔、滑点、手续费、基准、分组个数、资金、股票池输出：因子效果图、对数图转化
spark2.3.0 java版本_CentOS7.5之spark2.3.0安装
weixin_39815345的博客
02-28 241
一简介1.1 Spark概述Apache Spark是一个快速且通用的集群计算系统。它提供Java，Scala，Python和R中的高级API以及支持通用执行图的优化引擎。是UC Berkeley AMP lab (加州大学伯克利分校的AMP实验室)所开源的类Hadoop MapReduce的通用并行计算框架，Spark拥有Hadoop MapReduce所具有的优点；但不同于MapReduce的...
Spark每日半小时（2）——初识Spark基于Java的开发
DK_ing的博客
05-29 774
初识Spark的Java 这节主要带大家更好的步入Spark大堂，所以用一个，Spark官网提供的一个简单例子，给大家演示Spark关于java的开发。 这里我们用到的Maven工程，管理Spark的依赖： <dependency> <groupId>org.apache.spark</groupId> <artifactId&gt...
CM5.12.1安装Spark2.2.0 Cloudera步骤详解：附详细图文教程
3. **JDK版本**: 必须使用JDK 8进行Spark 2.2.0的安装，因为Cloudera Distribution对Spark 2.2.0的要求是JDK 8。 接下来，开始具体的安装步骤： **一、准备工作** 1. **下载资源**：从官方文档...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值