MapReduce Algorithms for Big Data Analysis

1. MapReduce Algorithms for Big Data Analysis; A. Madaan, S. Kikuchi, and S. Bhalla (EDS.): DNIS 2013, LNCS 7813, PP. 44-48, 2013.

2. Abstract. As there is an increasing trend of applications being expected to deal with big data that usually do not fit in the main memory of a single machine, analyzing big data is a challenging problem today. For such data-intensive applications, the MapReduce framework has recently attracted considerable attention and started to be investigated as a cost effective option to implement scalable parallel algorithms for big data analysis which can handle petabytes of data for millions of users. MapReduce is a programming model that allows easy development of scalable parallel applications to process big data on large clusters of commodity machines. Google’s MapReduce or its open-source equivalent Hadoop is a powerful tool for building such applications.

    In this tutorial, we will introduce the MapReduce framework based on Hadoop and present the state-of-the-art in MapReduce algorithms for query processing, data analysis and data mining. The intended audience of this tutorial is professionals who plan to design and develop MapReduce algorithms and researchers who should be aware of the state-of-theart in MapReduce algorithms available today for big data analysis.

3. MapReduce Framework

     -map funciton

     -reduce function

     -word counting and building inverted indexes

     -combine function which can improve the performance significantly

4. Join Processing: 

      -Join algorithms

      -n-way theta joins

      -similarity joins, include Jaccard similarity, Ruzicka similarity, Cosine similarity, Minkowski distance

      -top-k similarity join algorithm using MapReduce

5. Data Mining: 

      -K-means

      -EM clustering algorithm for learning probabilistic model parameters can be parallelized using MapReduce

      -hierarchical clustering, density-based clustering and co-clustering

      -frequent pattern mining and classification with tree model learning

      -parallel graph mining algorithms

      -The covered paralled algorithms include Probabilistic Latent Semantic Index (PLSI) ,TWITOBI, Latent Dirichlet Allocation (LDA) and Hidden Markov model

      -Potpourri: parallel wavelet construction algorithms and nonnegative matrix factorization algorithms