交集,并集,差集的算子

union 并集

		SparkConf conf = new SparkConf();
		conf.setMaster("local").setAppName("union");
		JavaSparkContext sc = new JavaSparkContext(conf);
		JavaRDD<Integer> rdd1 = sc.parallelize(Arrays.asList(1,2,3),3);
		JavaRDD<Integer> rdd2 = sc.parallelize(Arrays.asList(4,5,6),2);
		JavaRDD<Integer> union = rdd1.union(rdd2);
		System.out.println("union.partitions().size()---"+union.partitions().size());
		union.foreach(new VoidFunction<Integer>() {
			private static final long serialVersionUID = 1L;
			@Override
			public void call(Integer t) throws Exception {
				System.out.println(t);
			}
		});
		sc.stop();

结果

union.partitions().size()---5
1
2
3
4
5

intersection交集

注意RDD间的格式要一致

	SparkConf conf = new SparkConf();
		conf.setMaster("local").setAppName("intersection");
		JavaSparkContext sc = new JavaSparkContext(conf);
		JavaRDD<String> rdd1 = sc.parallelize(Arrays.asList("a","b","c"));
		JavaRDD<String> rdd2 = sc.parallelize(Arrays.asList("a","e","f"));
        JavaRDD<String> intersection = rdd1.intersection(rdd2);
        System.out.println(intersection.partitions().size());
		intersection.foreach(new VoidFunction<String>() {
			private static final long serialVersionUID = 1L;

			@Override
			public void call(String t) throws Exception {
				System.out.println(t);
			}
		});
		sc.stop();
		

subtract 差集

	SparkConf conf = new SparkConf();
		conf.setMaster("local").setAppName("subtract");
		JavaSparkContext sc = new JavaSparkContext(conf);
		JavaRDD<String> rdd1 = sc.parallelize(Arrays.asList("a","b","c"));
		JavaRDD<String> rdd2 = sc.parallelize(Arrays.asList("a","e","f"));
		//subtract取差集,两个RDD的类型要一致。
//		JavaRDD<String> subtract = rdd1.subtract(rdd2);
		JavaRDD<String> subtract = rdd2.subtract(rdd1);
		subtract.foreach(new VoidFunction<String>() {
			private static final long serialVersionUID = 1L;
			@Override
			public void call(String t) throws Exception {
				System.out.println(t);
			}
		});
		sc.stop();

结果

第一个结果
bc
第二个结果
ef
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值