spark
文章平均质量分 76
sxyqhyt
这个作者很懒,什么都没留下…
展开
-
spark集群中运行SparkPi的示例
1、SparkPi.scala源码(官网例子) import scala.math.randomimport org.apache.spark._/** Computes an approximation to pi */object SparkPi { def main(args: Array[String]) { val conf...原创 2014-12-31 15:42:09 · 764 阅读 · 0 评论 -
spark sql自定义函数udf
def visitview(vtimes : Iterable[String]): Long ={ var times = 0L if(vtimes.size == 0){ }else{ val lb = scala.collection.mutable.ListBuffer.empty[String] for(vtime <-...原创 2015-04-23 15:29:15 · 567 阅读 · 0 评论 -
Spark把执行结果写入到mysql中
sc.parallelize(List((stattime,"pv",1L), (stattime,"ip",2L), (stattime,"uv",3L), (stattime,"newuser",4L), (stattime,"beakrate",5L), (stattime,"visittimes",6L),原创 2015-04-24 18:05:34 · 194 阅读 · 0 评论 -
spark查看历史任务的信息
1、在spark-default.conf中增加:spark.eventLog.enabled truespark.eventLog.dir hdfs://10.0.0.37:9000/sparklogsspark.eventLog.compress true 2、在sp...原创 2015-05-18 15:56:26 · 965 阅读 · 0 评论 -
spark streaming使用数据源方式插入mysql数据
import java.sql.{Connection, ResultSet}import com.jolbox.bonecp.{BoneCP, BoneCPConfig}import org.slf4j.LoggerFactoryobject ConnectionPool { val logger = LoggerFactory.getLogger(this.get...原创 2015-06-03 17:13:18 · 190 阅读 · 0 评论 -
spark streaming准实时计算demo
package com.chexun.statisticimport java.sql.{Connection, DriverManager}import java.util.Dateimport com.chexun.statistic.RealTimeAdv._import kafka.serializer.StringDecoderimport o...原创 2015-08-13 17:44:50 · 319 阅读 · 0 评论 -
spark读取hbase数据
package ceshiimport java.io.IOExceptionimport java.utilimport java.util.{Random, UUID}import org.apache.hadoop.conf.Configurationimport org.apache.hadoop.hbase.HBaseConfigurationimpor...原创 2017-05-24 16:34:12 · 265 阅读 · 0 评论