- 博客(13)
- 收藏
- 关注
RSS订阅原创 记一次利用Redis手动维护kafka的Consumer消费的offest
package com.yh.scala.musicproject.eds.testimport java.utilimport com.yh.scala.musicproject.common.ConfigUtilsimport org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}import org.apache.kafka.common.TopicPartitionimport org.apache.kaf.
2022-03-29 14:41:39
2154
原创 利用sqoop从MySQL全量/增量导入数据到hive脚本
全量导入#! /bin/bashdate=`date -d today +"%Y-%m-%d"`if [ x"$1" = x ]; then echo "====使用自动生成的今天日期===="else date=$1fi echo "日期为：$date"sqoop import \ -D mapred.job.queue.name=hive \ -D mapred.task.timeout=0 \ --connect jd...
2022-03-23 11:19:17
2458
原创 记一次Azkaban操作
config: to_date: ""nodes: - name: jobA type: command config: command: /opt/shell/project/to_ods.sh ${to_date} - name: jobB type: command dependsOn: - jobA config: command: /opt/shell/project/to_dw...
2022-03-14 11:58:35
50
原创 spark输出到MySQL、Redis、kafka
MySQLobject streaming_mysql { def main(args: Array[String]): Unit = { //初始化spark配置信息 val conf = new SparkConf().setMaster("local[*]").setAppName(this.getClass.getName) //初始化SparkStreamingContext val ssc = new StreamingContext(conf, Se
2022-03-14 11:53:43
3023
3
原创 记一次Flume-KAFKA、kafka-hdfs操作
TAILDIR-KAFKAa1.sources = r1a1.channels = c1 c2a1.sources.r1.type = TAILDIRa1.sources.r1.filegroups = f1a1.sources.r1.filegroups.f1 = /tmp/logs/app-.*loga1.sources.r1.positionFile = /opt/installs/flume1.9/logs/taildir_position.jsona1.sources.r1..
2022-03-14 11:50:29
2397
空空如也
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人