drl_blogs-CSDN博客

原创 Vue的路由实现：hash模式和 history模式

hash模式：在浏览器中符号“#”，#以及#后面的字符称之为hash，用window.location.hash读取；特点：hash虽然在URL中，但不被包括在HTTP请求中；用来指导浏览器动作，对服务端安全无用，hash不会重加载页面。hash 模式下，仅 hash 符号之前的内容会被包含在请求中，如 http://www.xxx.com，因此对于后端来说，即使没有做到对路由的全覆盖，也不会...

2019-10-18 20:11:45 376

原创 mysql隔离级别

事务的四种隔离级别隔离级别脏读（Dirty Read）不可重复读（NonRepeatable Read）幻读（Phantom Read）未提交读（Read uncommitted）可能可能可能已提交读（Read committed）不可能可能可能可重复读（Repeatable read）不可能不可能可能可串行化（Serializable...

2019-10-18 19:52:16 190

原创 nginx-1.16.0安装

上传 nginx-1.16.0.tar.gz解压 tar -zxvf nginx-1.16.0.tar.gz -C /usr/local/src/进入解压的目录cd /usr/local/src/nginx-1.16.0安装插件yum install -y gcc openssl openssl-devel pcre-devel预编译./configure...

2019-07-13 14:57:46 935

原创 MyUtil：二分查找法（Scala)

object BinarySearch { def main(args: Array[String]): Unit = { val arr = Array(1, 2, 3, 10, 100, 19999, -1998, 9, 234, 234, 9, 43) val index = Binarysearch(arr, 100) println("the found...

2019-07-13 10:54:55 208

原创 MyUtil:保存数据到Mysql

import java.sql.DriverManagerimport org.apache.kafka.common.TopicPartitionimport org.apache.spark.rdd.RDDimport org.apache.spark.streaming.kafka010.OffsetRangeimport scala.collection.mutableobjec...

2019-07-13 08:51:44 251

原创 MyUtil：时间戳转成时间字符串

import java.text.SimpleDateFormatimport java.util.Dateobject tranTimeToString { def main(args: Array[String]): Unit = { val tm = "1502036122000" val a = tranTimeToString(tm) println...

2019-07-12 10:42:49 157

原创 MyUtil：时间字符串转时间戳

package utilimport java.text.SimpleDateFormatobject tranTimeToLong { def main(args: Array[String]): Unit = { val tm = "20170801164432123" val a = tranTimeToLong(tm) println(a) }...

2019-07-12 09:34:06 106

原创 MyUtil：保存数据到redis

import org.apache.spark.rdd.RDDimport org.apache.spark.streaming.kafka010.OffsetRangeimport redis.Jpoolsobject DataToRedis { //保存数据到redis def saveDataOffset(result1: RDD[((String,String,String...

2019-07-10 17:13:01 312

原创 flume常用配置文件

文章目录kafka-hdfs.confflume-kafka.confflume-hdfs.confflume_pull_streaming.confflume_push_streaming.confkafka-hdfs.conf#定义agent名， source、channel、sink的名称a1.channels = c1a1.sinks = k1a1.channels.c1.ty...

2019-07-09 14:24:13 649

原创安装 flink-1.8.1-bin-scala_2.11

1. 上传文件 flink-1.8.1-bin-scala_2.11.tgz2. 解压文件 tar -zxvf flink-1.8.1-bin-scala_2.11.tgz -C /usr/local/3. 配置环境变量vim /etc/profile修改内容如下#flinkexport FLINK_HOME=/usr/local/flink-1.8.1export PAT...

2019-07-08 19:18:16 1651

原创示例：Spark Streming+Kafka整合(spark-streaming-kafka-0-10_2.11)

import org.apache.kafka.common.serialization.StringDeserializerimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.kafka010.ConsumerStrategies....

2019-07-05 11:01:03 2867

原创示例：python模拟日志生成+Flume+Kafka+Spark Streaming

生成模拟数据编写 generate_log.py#coding=UTF-8import randomimport timeurl_paths=[ "class/112.html", "class/128.html", "class/145.html", "class/130.html", "class/146.html", "cla...

2019-07-02 16:44:44 999

原创【error】SparkUI端口被占用

ERROR ui.SparkUI: Failed to bind SparkUIjava.net.BindException: Address already in use: bind: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropri...

2019-07-02 14:35:00 1795 1

原创示例：Log4j日志模拟数据整合 Flume+Kafka+Spark Streaming

flume_kafka.confagent1.sources = avro-sourceagent1.channels = logger-channelagent1.sinks = kafka-sink# define sourceagent1.sources.avro-source.type = avroagent1.sources.avro-source.bind = 0....

2019-07-01 21:22:06 277

原创示例：Spark Streming+Kafka整合 (spark-streaming-kafka-0-8_2.11)

文章目录ReceiverDirectReceiverimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.kafka.KafkaUtilsimport org.apache.spark.streaming.{Seconds, Str...

2019-07-01 17:24:20 2295

原创示例：Spark Streaming+Flume整合

文章目录pushpullpushimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.dstream.ReceiverInputDStreamimport org.apache.spark.streaming.flume.{Flum...

2019-07-01 14:49:50 257

原创 Spark：【error】无法解析重载方法“agg”

错误信息：Cannot resolve overloaded method 'agg'解决方法：导入包import org.apache.spark.sql.functions._

2019-06-29 13:13:50 2835

原创示例：根据IP解析位置

下载公共开源项目：https://github.com/wzhe06/ipdatabase打JAR包上传到maven示例：https://blog.csdn.net/drl_blogs/article/details/94011565引入jar包和依赖 <dependency> <groupId>com.ggstar</groupI...

2019-06-28 17:19:26 604

原创示例：redis输入/输出

创建连接池：https://blog.csdn.net/drl_blogs/article/details/93979067写入redis//df为已经处理好的DataFramedf.foreachPartition(filter=>{ val jedis = Jpools.getJedis filter.foreach(tp=>{ ...

2019-06-28 12:22:23 339

原创 MyUtil：创建jedis连接池

package MyUtilimport org.apache.commons.pool2.impl.GenericObjectPoolConfigimport redis.clients.jedis.{Jedis, JedisPool}/** * 创建jedis连接池 */object Jpools { //获取连接对象 private val config = n...

2019-06-28 11:19:34 177

原创 MyUtil：将[dd/MMM/yyyy:HH:mm:ss Z]类型时间转换为yyyy-MM-dd hh:mm:ss格式

package MyUtilimport java.util.{Date, Locale}import org.apache.commons.lang3.time.FastDateFormatobject DateUtils { //日期时间格式 [10/Nov/2016:00:01:02 +0800] val YYYYMMDDHHMM_TIME_FORMAT = FastDa...

2019-06-27 23:50:57 997

原创 flume-1.9.0安装

上传 apache-flume-1.9.0-bin.tar.gz解压tar -zxvf apache-flume-1.9.0-bin.tar.gz -C /usr/local/配置环境变量vim /etc/profile添加内容export FLUME_HOME=/usr/local/apache-flume-1.9.0-binexport PATH=$PATH:$FL...

2019-06-26 15:14:29 1014

原创示例：Kafka Stream 数据清洗

接受数据后的处理逻辑import org.apache.kafka.streams.processor.Processor;import org.apache.kafka.streams.processor.ProcessorContext;public class LogProcessor implements Processor<byte[], byte[]> { p...

2019-06-25 16:07:44 1168

原创示例：kafka拦截器

拦截内容在内容前加时间戳import java.util.Map;import org.apache.kafka.clients.producer.ProducerInterceptor;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.clients.producer.Re...

2019-06-25 14:58:48 313

原创示例：统计IP地址对应的省份，并把结果存入到mysql

数据ip.txtaccess.logimport java.io.{BufferedReader, FileInputStream, InputStreamReader}import java.sql.{Connection, DriverManager, PreparedStatement}import org.apache.spark.{SparkConf, SparkCont...

2019-06-25 10:20:58 889 3

原创 kafka_2.11-0.11.0.0 安装配置

1.上传kafka_2.11-0.11.0.0.tgz2.解压tar -zxvf kafka_2.11-0.11.0.0.tgz -C /usr/local/3.修改server.propertiesvim server.properties修改内容如#broker的全局唯一编号，不能重复broker.id=0#删除topic功能使能delete.topic.enable=t...

2019-06-24 10:32:18 779

原创示例：Spark SQL自定义函数（UDF/UDAF）

文章目录UDF函数UDAF函数UDF函数scala> val df=spark.read.json("people.json")df: org.apache.spark.sql.DataFrame = [age: bigint, name: string]scala> df.show+---+------+|age| name|+---+------+| 30| ...

2019-06-20 17:12:00 1547

原创 Spark：【error】DataFrames转DataSet失败

Error:(45, 63) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for seria...

2019-06-20 17:01:13 310

原创 Spark ：【error】System memory 259522560 must be at least 471859200

java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.。。。。。。。...

2019-06-20 15:36:29 448

原创 Spark:常用JAR包集成（pom.xml）

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ma...

2019-06-19 23:17:11 686

原创 Spark SQL：RDD、DataFrames、DataSet之间的转换

文章目录RDD转DataFramesRDD转DataSetDataFrame/Dataset转RDDDataFrame转DatasetDataset转DataFramepeople.txtMichael,29Andy,30Justin,19RDD转DataFramesscala> val rdd=sc.textFile("people.txt")rdd: org.apache...

2019-06-19 16:55:28 256

原创 redis:编译时报错 /bin/sh: cc: command not found

[root@hadoop01 redis-3.0.0]# makecd src && make allmake[1]: Entering directory `/usr/local/redis-3.0.0/src'rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-dump redi...

2019-06-19 16:54:52 1811

原创 Spark Core:数据输入输出

文本文件输入输出读取文本文件scala> sc.textFile("./wc.txt")res4: org.apache.spark.rdd.RDD[String] = ./wc.txt MapPartitionsRDD[5] at textFile at <console>:25保存文本文件scala> res4.saveAsTextFile("./test"...

2019-06-19 16:54:20 487

原创 Spark Core:RDD编程Action

Action目录reduce(f: (T, T) => T): Tcollect(): Array[T]count(): Longfirst(): Ttake(num: Int): Array[T]takeOrdered(num: Int)(implicit ord: Ordering[T])aggregate[U: ClassTag](zeroValue: U)(seqOp: (U, T)...

2019-06-19 16:53:59 186

原创 Spark Core:RDD编程Transformation

文章目录创建RDD操作map[U: ClassTag](f: T => U): RDD[U]filter(f: T => Boolean): RDD[T]flatMap[U: ClassTag](f: T => TraversableOnce[U]): RDD[U]mapPartition[U: ClassTag]( f: Iterator[T] => Iterator[U...

2019-06-19 16:53:34 669

原创 Spark Core:创建RDD

#方式一：把数据并行化分片到节点sc.parallelize(Array(1,2,3,4))#方式二：把数据并行化分片到节点sc.makeRDD(Array(1,2,3))#方式三：可以指定RDD的存放位置#创建一个list集合val list1=List((1,List("Hello","Word","spark")),(2,List("at","as")))#将List集合放入R...

2019-06-17 16:35:16 185

原创 Spark：【error】远程调试 root:supergroup:drwxr-xr-x

Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=xxxx, access=WRITE, inode="/test/out/_temporary/0":root:supergroup:drwxr-xr-x.....................

2019-06-15 16:27:46 825

原创 spark-2.2.2-bin-hadoop2.7 HA 配置

安装spark-2.2.2-bin-hadoop2.7：https://blog.csdn.net/drl_blogs/article/details/919483941.编辑主节点conf/spark-env.shexport JAVA_HOME=/usr/local/jdk1.8.0_211# export SPARK_MASTER_HOST=hadoop01# export ...

2019-06-14 11:08:28 311

原创 spark-2.2.2-bin-hadoop2.7 安装

1.上传spark-2.2.2-bin-hadoop2.7.tgz2.解压文件tar -zxvf spark-2.2.2-bin-hadoop2.7.tgz -C /usr/local/3.进入conf/下把spark-env.sh.template改为spark-env.shcd /usr/local/spark-2.2.2-bin-hadoop2.7/conf/mv s...

2019-06-14 09:23:37 3464

原创示例：Scala读取xml文件

test.xml内容如下<?xml version="1.0"?><Tests> <Test TestId="0001" TestType="CMD"> <Name>Convert number to string</Name> <CommandLine>Examp1.EXE<...

2019-06-13 15:10:47 2119 1

空空如也

空空如也