用spark自带的示例SparkPi测试scala和spark集群

最新推荐文章于 2024-03-13 18:13:28 发布

即将起飞的笨鸟

最新推荐文章于 2024-03-13 18:13:28 发布

阅读量6.1k

点赞数 1

分类专栏： Hadoop Spark 文章标签： spark 自带的示例SparkPi 测试scala和spark集群 hadoop

本文链接：https://blog.csdn.net/chenyuangege/article/details/50824808

版权

Hadoop 同时被 2 个专栏收录

17 篇文章 0 订阅

订阅专栏

Spark

5 篇文章 0 订阅

订阅专栏

在按照王家林的文档安装完scala，spark集群和idea-IC开发工具后，用spark自带的示例SparkPi测试scala和spark集群

1、按照王家林文档中的方法把spark自带的SparkPi示例的代码复制到idea-IC开发工具中创建的项目中后，把SparkPi代码的修改为：

/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import scala.math.random
import org.apache.spark._

/** Computes an approximation to pi */
object SparkPi {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Spark Pi").setMaster("spark://192.168.91.128:7077")//改成自己master主机的ip地址
val spark = new SparkContext(conf)
spark.addJar("/home/cy/workspace/untitled/out/artifacts/sparkpi_jar/untitled.jar")//把写的代码打包成jar包，存放打包后jar包的路径
val slices = if (args.length > 0) args(0).toInt else 2
val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
val count = spark.parallelize(1 until n, slices).map { i =>
val x = random * 2 - 1
val y = random * 2 - 1
if (x*x + y*y < 1) 1 else 0
}.reduce(_ + _)
println("Pi is roughly " + 4.0 * count / n)
spark.stop()
}
}