spark系列之一

1 spark简介

       Spark是基于内存计算的大数据分布式计算框架，提高了大数据环境下数据处理的实时性。

       分布式计算

       内存计算

       容错

      多计算范式

2  安装部署 spark on yarn

          master 192.168.100.200

          slave1: 192.168.100.201

          slave2:192.168.100.202

          hadoop环境

     2.1 安装scala环境

                   tar -zxvf scala-2.10.6.tgz

                   vi /etc/profile

                                    SCALA_HOME=/usr/local/scala-2.10.6

                                    PATH=$SCALA_HOME/bin:$PATH

                                    export SCAlA_HOME

                                    export PATH

                    source /etc/profile

                   在命令行输入:scala测试安装成功

       2.2 安装spark

                   tar -zxvf spark-2.1.0-bin-hadoop2.7.tgz

                   cd spark-2.1.0-bin-hadoop2.7/conf/

                   cp spark-env.sh.template spark-env.sh

                   vi spark-env.sh

                                   export SPARK_HOME=/usr/local/spark-2.1.0-bin-hadoop2.7/
                                   export SCALA_HOME=/usr/local/scala-2.10.6/
                                   export JAVA_HOME=/usr/local/jdk1.7.0_79/
                                   export HADOOP_HOME=/usr/local/hadoop-2.7.1/
                                   export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin
                                   export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
                                   export YARN_CONF_DIR=$YARN_HOME/etc/hadoop
                                   export SPARK_MASTER_IP=master
                                   SPARK_LOCAL_DIRS=/usr/local/spark-2.1.0-bin-hadoop2.7/spadata
                                   SPARK_DRIVER_MEMORY=1G
                                   export SPARK_LIBARY_PATH