JavaPairRDD的countByValue方法讲解
官方文档
/**
* Return the count of each unique value in this RDD as a map of (value, count) pairs. The final
* combine step happens locally on the master, equivalent to running a single reduce task.
*/
说明
返回RDD中每个值的计数,作为(value,count)对的映射。
返回的是map
函数原型
// java
public static java.util.Map<T,Long> countByValue()
// scala
def countByValue(): Map[(K, V), Long]
示例
public class CountByValue {
public static void main(String[] args) {
System.setProperty("hadoop.home.dir", "E:\\hadoop-2.7.1");
SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("Spark_DEMO");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
JavaPairRDD<String, String> javaPairRDD1 = sc.parallelizePairs(Lists.newArrayList(
new Tuple2<String, String>("cat", "11"), new Tuple2<String, String>("dog", "22"),
new Tuple2<String, String>("cat", "11"), new Tuple2<String, String>("pig", "44"),
new Tuple2<String, String>("duck", "55"), new Tuple2<String, String>("cat", "66")), 3);
Map<Tuple2<String, String>, Long> value = javaPairRDD1.countByValue();
for (Map.Entry<Tuple2<String, String>, Long> entry : value.entrySet()){
System.out.println(entry.getKey()+"->"+entry.getValue());
}
}
}
结果
19/03/20 17:15:31 INFO DAGScheduler: Job 0 finished: countByValue at CountByValue.java:23, took 1.093040 s
19/03/20 17:15:31 INFO SparkContext: Invoking stop() from shutdown hook
(duck,55)->1
(dog,22)->1
(pig,44)->1
(cat,66)->1
(cat,11)->2
19/03/20 17:15:31 INFO SparkUI: Stopped Spark web UI at http://10.124.209.6:4040
JavaPairRDD的countByValueApprox方法讲解
官方文档
/**
* Approximate version of countByValue().
*
* The confidence is the probability that the error bounds of the result will
* contain the true value. That is, if countApprox were called repeatedly
* with confidence 0.9, we would expect 90% of the results to contain the
* true count. The confidence must be in the range [0,1] or an exception will
* be thrown.
*
* @param timeout maximum time to wait for the job, in milliseconds
* @param confidence the desired statistical confidence in the result
* @return a potentially incomplete result, with error bounds
*/
说明
CountByValue()的近似版本。
置信度必须在[0,1]范围内,否则异常将被扔掉。
*@参数超时等待作业的最长时间(毫秒)
*@参数置信度结果中所需的统计置信度
*@返回一个可能不完整的结果,带有错误界限
函数原型
// java
public static PartialResult<java.util.Map<T,BoundedDouble>> countByValueApprox(long timeout)
public static PartialResult<java.util.Map<T,BoundedDouble>> countByValueApprox(long timeout,
double confidence)
// scala
def countByValueApprox(timeout: Long): PartialResult[Map[(K, V), BoundedDouble]]
def countByValueApprox(timeout: Long, confidence: Double): PartialResult[Map[(K, V), BoundedDouble]]