MapReduce Algorithms for Big Data Analysis

原创 2013年12月02日 15:53:58
1. MapReduce Algorithms for Big Data Analysis; A. Madaan, S. Kikuchi, and S. Bhalla (EDS.): DNIS 2013, LNCS 7813, PP. 44-48, 2013.
2. Abstract. As there is an increasing trend of applications being expected to deal with big data that usually do not fit in the main memory of a single machine, analyzing big data is a challenging problem today. For such data-intensive applications, the MapReduce framework has recently attracted considerable attention and started to be investigated as a cost effective option to implement scalable parallel algorithms for big data analysis which can handle petabytes of data for millions of users. MapReduce is a programming model that allows easy development of scalable parallel applications to process big data on large clusters of commodity machines. Google’s MapReduce or its open-source equivalent Hadoop is a powerful tool for building such applications.
    In this tutorial, we will introduce the MapReduce framework based on Hadoop and present the state-of-the-art in MapReduce algorithms for query processing, data analysis and data mining. The intended audience of this tutorial is professionals who plan to design and develop MapReduce algorithms and researchers who should be aware of the state-of-theart in MapReduce algorithms available today for big data analysis.
3. MapReduce Framework
     -map funciton
     -reduce function
     -word counting and building inverted indexes
     -combine function which can improve the performance significantly
4. Join Processing: 
      -Join algorithms
      -n-way theta joins
      -similarity joins, include Jaccard similarity, Ruzicka similarity, Cosine similarity, Minkowski distance
      -top-k similarity join algorithm using MapReduce
5. Data Mining: 
      -EM clustering algorithm for learning probabilistic model parameters can be parallelized using MapReduce
      -hierarchical clustering, density-based clustering and co-clustering
      -frequent pattern mining and classification with tree model learning
      -parallel graph mining algorithms
      -The covered paralled algorithms include Probabilistic Latent Semantic Index (PLSI) ,TWITOBI, Latent Dirichlet Allocation (LDA) and Hidden Markov model
      -Potpourri: parallel wavelet construction algorithms and nonnegative matrix factorization algorithms


Coursera--Big Data Analysis with Scala and Spark

package wikipedia import org.apache.spark.SparkConf import org.apache.spark.SparkContext import org...

Learning Spark: Lightning-Fast Big Data Analysis 中文翻译

 Learning Spark: Lightning-Fast Big Data Analysis 中文翻译行为纯属个人对于Spark的兴趣,仅供学习。 如果我的翻译行为侵犯您的版权,请您告知...

MapReduce and MPP: Two sides of the Big Data coin?

Summary: To many, Big Data goes hand-in-hand with Hadoop + MapReduce. But MPP (Massively Parallel ...
您举报文章:MapReduce Algorithms for Big Data Analysis