![](https://img-blog.csdnimg.cn/20201014180756925.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
spark
文章平均质量分 82
blackproof
这个作者很懒,什么都没留下…
展开
-
spark编译报错
jars/EmarOlap-0.0.1-SNAPSHOT.jar with timestamp 1428464475056Exception in thread "main" java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnProtos$PriorityProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFie原创 2015-06-06 14:05:00 · 198 阅读 · 0 评论 -
spark eclipse写wordcount
原创 2015-02-03 10:18:01 · 123 阅读 · 0 评论 -
转 spark简单实例
8)groupByKey (9)sortByKey1、准备文件?1wget http://statweb.stanford.edu/~tibs/ElemStatLearn/datasets/spam.data 2、加载文件?1scala> val inFile = sc.textFile("/home/scipio/spam.data")原创 2015-02-02 15:09:01 · 123 阅读 · 0 评论 -
spark集群搭建
需要和hadoop版本对应,不然会有protocal的错)解压配置spark-env.shexport JAVA_HOME=/home/hadoop-cdh/java/jdk1.7.0_06export SCALA_HOME=/home/hadoop-cdh/app/test/scala-2.11.4export HADOOP_HOME=/home/hadoop-cdh/app/hadoop-2.3.0-cdh5.1.0export YARN_CONF_DIR=$HADOOP_H原创 2015-02-02 12:04:05 · 86 阅读 · 0 评论 -
spark编译报错
spark成功之后运行例子报错 问题一: spark.SparkContext: Added JAR file:/home/hadoop-cdh/app/test/sparktest/EmarOlap-0.0.1-SNAPSHOT.jar at http://192.168.5.143:32252/jars/EmarOlap-0.0.1-SNAPSHOT.jar w...原创 2015-06-06 14:05:00 · 142 阅读 · 0 评论 -
spark tachyon 搭建 配置
mvn -Dhadoop.version=2.3.0 -DskipTests clean package spark-env.sh因为需要访问hdfs,hive,所以需要压缩lzo,和mysqlexport SPARK_CLASSPATH=$SPARK_CLASSPATH:/data/hadoop/hadoop-2.3.0-cdh5.1.0/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jarexport SPARK_LIBRARY_P原创 2015-11-19 18:20:30 · 352 阅读 · 0 评论