Spark

-- Spark SQLSELECT * FROM dp_fk_tmp.t_aaronyu_test_0215 LIMIT 11;SELECT IF(Fdata IS NULL OR Fdata = '', (SELECT Fdata FROM dp_fk_tmp.t_aaronyu_test_0215 WHERE Fdata IS NOT NULL AND length(Fdata) >= 1 LIMIT 1), Fdata) result FROM dp_fk_tmp.t_aaronyu_.

#Spark三种部署方式1、Standalone模式2、YARN模式3、High Availability如果Spark的部署方式选择Standalone，一个采用Master/Slaves的典型架构，那么Master是有SPOF（单点故障，Single Point of Failure）。Spark可以选用ZooKeeper来实现HA。Spark采用的是Zookeep

参考urlhttp://www.infoq.com/cn/news/2016/01/spark-streaming-what-is-it-and-w

参考urlhttp://www.cnblogs.com/payton/p/4227770.html

一、安装Scala下载scala-2.9.3.zip，并配置环境变量二、配置IDEA下载Scala插件

1.1、Spark SQL CLI配置 Spark SQL访问Hive，Hive服务器必须先启动Hive Metastore（Spark会读取$SPARK_HOME/conf/hive-site.xml）Spark SQL is designed to be compatible with the Hive Metastore, SerDes and UDFs. Currently Spa

编译命令mvn -Pyarn-alpha -Phadoop-2.0.6-alpha -Phive -Phive-thriftserver -Dhadoop.version=2.0.6-alpha -DskipTests clean package./make-distribution.sh --tgz --name 2.0.6-alpha -Pyarn-alpha -Phadoop-2

一、问题背景执行最后一行时出错： ./spark-shell MASTER=spark://bigdata0:7077scala> val file = sc.textFile("hdfs://bigdata1:9000/wordcount/wc_in/test1.txt")scala> val count = file.flatMap(line => line.split

进入Spark SQL：spark-shell MASTER=spark://bigdata0:7077 --executor-memory 3gval sqlContext=new org.apache.spark.sql.SQLContext(sc)import sqlContext.createSchemaRDD1、测试case class Person(

需求：Flume接收HTTP请求，数据流向Spark Streaming。Spark Streaming接收Flume数据：bin/run-example org.apache.spark.examples.streaming.FlumeEventCount bigdata3 6667运行后会在bigdata3:6667上开启一个Avro Source Server

1、NetworkWordCount1.1、开启发送数据的服务器cd /home/jianxin/spark java -jar LoggerSimulation.jar 9999 10java -jar NetworkWordCount.jar localhost 9999netstat -nalp | grep 9999lsof -i:99991.2、

1、val rdd = sc.textFile("/home/jianxin/spark/sourceDir/wordcount.txt")rdd.foreach(println)hello   youhello   me2、rdd.map(_.split(' ')).firstres74: Array[String] = Array(hello, "", "", yo

一、环境说明scala-2.9.3spark-0.8.1-incubating-bin-hadoop2二、安装配置2.1、mkdir -p /opt/sparktar -zxvf scala-2.9.3.tgzln -s scala-2.9.3 scala修改/etc/profileexport SCALA_HOME=/opt/spark/scala

本文详细总结Spark分布式集群的安装步骤，帮助想要学习Spark的技术爱好者快速搭建Spark的学习研究环境。