![](https://img-blog.csdnimg.cn/20201014180756757.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
Spark-Streaming
LJ2415
这个作者很懒,什么都没留下…
展开
-
模拟生成单词,消费单词
package kafka;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.common.serialization.StringSerializer;import...原创 2018-12-26 15:19:51 · 307 阅读 · 0 评论 -
Streaming-kafka-mysql (scalikejdbc)
package kafkaimport java.sql.DriverManagerimport com.typesafe.config.ConfigFactoryimport org.apache.kafka.common.TopicPartitionimport org.apache.spark.streaming.kafka010.OffsetRangeimport scal...原创 2018-12-26 11:53:24 · 275 阅读 · 0 评论 -
SparkStreaming整合kafka入门
package kafkaimport com.typesafe.config.ConfigFactoryimport org.apache.kafka.clients.consumer.ConsumerRecordimport org.apache.kafka.common.serialization.StringDeserializerimport org.apache.spark...原创 2018-12-25 22:26:12 · 168 阅读 · 0 评论 -
SparkStreaming窗口函数的应用
package windonimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.{Seconds, StreamingContext}/** * 背景描述 * 在社交网(微博),电子商务(淘宝),搜索引擎(百度),股票...原创 2018-12-24 16:04:57 · 742 阅读 · 0 评论 -
Kafka-Straming-redis(offset)
package kafka2import java.utilimport org.apache.kafka.common.TopicPartitionimport utils.Jpoolsimport scala.collection.mutable._/** * 获取redis里面存储的偏移量数据 */object RedisOffset { def apply...原创 2018-12-27 20:30:17 · 549 阅读 · 0 评论 -
scalikeJDBC的使用
# MySQL exampledb.default.driver="com.mysql.jdbc.Driver"db.default.url="jdbc:mysql://hadoop01:3306/test?characterEncoding=UTF-8"db.default.user="root"db.default.password="123456"package scalik...原创 2018-12-27 19:41:50 · 598 阅读 · 0 评论 -
SparkStreaming集成Spark Sql
package testimport java.sql.DriverManagerimport com.typesafe.config.ConfigFactoryimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.rdd.RDDimport ...原创 2018-12-22 11:25:58 · 298 阅读 · 0 评论 -
将Streaming拉取的数据存入redis中
启动redis: ./redis-cli -h hadoop01package utilsimport org.apache.commons.pool2.impl.GenericObjectPoolConfigimport redis.clients.jedis.{Jedis, JedisPool}/** * 创建jedis连接池 */object Jpools {...原创 2018-12-21 23:00:23 · 662 阅读 · 0 评论 -
初识SparkStreaming
写一个wordcount: linux命令: nc -lk 8888package day01import org.apache.spark.streaming.dstream.DStreamimport org.apache.spark.streaming.{Seconds, StreamingContext}import org.apache.spark.{SparkC...原创 2018-12-16 19:01:11 · 179 阅读 · 0 评论 -
Streaming-Kafka-redis
package utilsimport org.apache.commons.pool2.impl.GenericObjectPoolConfigimport redis.clients.jedis.{Jedis, JedisPool}/** * 创建jedis连接池 */object Jpools { private val poolConfig = new Gene...原创 2018-12-26 16:19:16 · 141 阅读 · 0 评论