- 博客(66)
- 资源 (38)
- 收藏
- 关注
RSS订阅原创 Storm 整合kafka
1.依赖<dependency> <groupId>org.apache.curator</groupId> <artifactId>curator-framework</artifactId> <version>${curator.version}</vers...
2018-06-30 20:30:32
644
1
原创 Storm 整合 Hbase
1.依赖<dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-hbase</artifactId> <version>1.1.1</version> <exclusions&
2018-06-30 20:29:55
2338
1
原创 storm 整合hdfs
1.依赖<dependency><groupId>org.apache.storm</groupId><artifactId>storm-hdfs</artifactId><version>${storm.version}</version><type>jar</t
2018-06-30 20:29:17
503
原创 Storm 整合jdbc wordcount
<dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-jdbc</artifactId> <version>1.1.1</version> <type>jar
2018-06-30 20:28:39
1391
原创 Storm 把结果存到redis
Maven 依赖添加 <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-redis</artifactId> <version>${storm.version}</version> &
2018-06-30 20:28:04
1049
原创 storm 分组策略
Stream groupingsPart of defining a topology is specifying for each bolt which streams it should receive as input. A stream grouping defines how that stream should be partitioned among the bolt's tasks...
2018-06-30 20:27:22
459
原创 leetcode 反转链表 python
反转一个单链表。示例: 输入: 1->2->3->4->5->NULL 输出: 5->4->3->2->1->NULL 进阶: 你可以迭代或递归地反转链表。你能否用两种方法解决这道题？# Definition for singly-linked list.# class ListNode:# def __init__(self...
2018-06-30 20:26:10
805
原创 node.js 使用Sequelize mysql ORM 来搞RESTful API
1.安装sequelizenpm install sequelize --save2. 先看mysql 数据库表结构 3. node.js 目录结构4. config.json{ "development": { "username": "root", "password": "root", "database": "baidusong",
2018-06-30 20:25:28
2975
1
原创 node.js supertest 单元测试 RESTful API
npm install supertest should --saveconst app = require('../app')const request = require('supertest')(app)require('should');request .get('/api') .expect(200) .end((err, res) => { ...
2018-06-30 20:24:02
4085
原创 node.js Express RESTful API 认证
接上一篇文章《node.js Express 创建RESTful API》1.安装依赖node-token-jwt，morgannpm install jsonwebtoken morgan --savejsonwebtoken 用来创建和验证json token2. 项目结构增加了一个user model，一个setup 用来创建一个测试用户，config里记录一个key用来给jsonwebt...
2018-06-30 20:22:56
1535
原创 Leetcode 有效的括号字符串 python
class Stack: """模拟栈""" def __init__(self): self.items = [] def isEmpty(self): return len(self.items)==0 def push(self, item): s...
2018-06-24 13:35:59
869
原创 两数相加 leetcode Python
给定两个非空链表来表示两个非负整数。位数按照逆序方式存储，它们的每个节点只存储单个数字。将两数相加返回一个新的链表。你可以假设除了数字 0 之外，这两个数字都不会以零开头。示例：输入：(2 -> 4 -> 3) + (5 -> 6 -> 4)输出：7 -> 0 -> 8原因：342 + 465 = 807# Definition for singly-lin...
2018-06-24 13:35:21
1641
原创 node.js Express 创建RESTful API
0.项目机构1.package.js 安装mongoose依赖和body-parser依赖{ "name": "test", "version": "0.0.0", "private": true, "scripts": { "start": "node ./bin/www" }, "dependencies
2018-06-24 13:31:55
3822
原创 Storm 本机运行 提交StormSubmitter
1. Storm 启动#启动Zookeepernohup sh storm dev-zookeeper &#启动主节点nohup sh storm nimbus &#启动从节点nohup sh storm supervisor &#启动UI界面nohup sh storm ui &#启动日志查看服务nohup sh storm logviewer ...
2018-06-24 13:30:41
1097
原创 spark window 的使用
1. window 用在rank 中的使用看这样一个需求，求出每个销售人员的按照销售金额大小的orderidpackage com.waitingfyimport org.apache.spark.sql.SparkSessionimport org.apache.spark.sql.expressions.Windowimport org.apache.spark.sql.functio...
2018-06-24 13:29:38
7369
原创 Spark 可视化实战 分析日志的搜索引擎爬虫来源
1.日志部分内容：66.249.79.35 - - [14/Jun/2018:06:45:24 +0000] "GET /img/20180504/702434-20180302101540805-554506523.jpg HTTP/1.1" 200 10013 "-" "Googlebot-Image/1.0"66.249.79.35 - - [14/Jun/2018:06:45:25 +0...
2018-06-22 19:20:12
1449
原创 Storm wordcount
package com.waiting;import org.apache.storm.Config;import org.apache.storm.LocalCluster;import org.apache.storm.shade.org.apache.commons.io.FileUtils;import org.apache.storm.spout.SpoutOutputCol...
2018-06-22 19:19:07
321
2
原创 storm 入门例子
TopologiesThe logic for a realtime application is packaged into a Storm topology. A Storm topology is analogous to a MapReduce job. One key difference is that a MapReduce job eventually finishes, wher...
2018-06-22 19:17:56
711
1
原创 spark foreachPartition 把df 数据插入到mysql
package com.waitingfyimport java.sql.{Connection, DriverManager, PreparedStatement}import org.apache.spark.sql.SparkSessionimport org.apache.spark.sql.functions._import scala.collection.mutable...
2018-06-22 19:17:18
7218
原创 spark 读取jdbc mysql
1.加mysql jarspark-shell --master local[2] --jars /Users/walle/app/mysql-connector-java-8.0.11.jar2. 连接jdbcval mysqlDF = spark.read.format("jdbc").option("url", "jdbc:mysql://localhost:3306").option("d...
2018-06-22 19:16:29
1840
原创 spark 读取hive 数据
1.启动spark-shell 需要加上mysql jar位置spark-shell --master local[2] --jars /Users/walle/app/mysql-connector-java-8.0.11.jar2. 简单sqlspark.sql("show databases").showspark.sql("show tables").showspark.sql("us...
2018-06-22 19:15:45
6849
原创 spark 读取csv 到dataframe
val df = spark.read.format("csv").option("header", "true").load("/usr/local/Cellar/spark-2.3.0/examples/src/main/resources/people.csv")df.show《spark 编程方式指定dataframe的 Schema》 如果csv没有第一行，就只能指定schema了，参...
2018-06-22 19:13:44
6775
1
原创 spark 读取parquet
列式存储布局（比如 Parquet）可以加速查询，因为它只检查所有需要的列并对它们的值执行计算，因此只读取一个数据文件或表的小部分数据。Parquet 还支持灵活的压缩选项，因此可以显著减少磁盘上的存储。如果您在 HDFS 上拥有基于文本的数据文件或表，而且正在使用 Spark SQL 对它们执行查询，那么强烈推荐将文本数据文件转换为 Parquet 数据文件，以实现性能和存储收益。当然，转换需要...
2018-06-22 19:13:05
18493
1
原创 spark 编程方式指定dataframe的 Schema
package com.immooc.sparkimport com.immooc.spark.ReflectionTest.Personimport org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType}import org.apache.spark.{SparkConf, SparkCo...
2018-06-19 18:24:03
4330
1
原创 spark dataframe 转成 dataset
package com.immooc.sparkimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.sql.{Encoders, SparkSession}object DataSetTest { case class Person(name:String, age:Long) d...
2018-06-19 18:23:22
6330
原创 flume 从log4j 收集日志 到kafka
1. flume 配置# Define a memory channel called ch1 on agent1agent1.channels.ch1.type = memoryagent1.channels.ch1.capacity = 1000agent1.channels.ch1.transactionCapacity = 100 agent1.sources.avro-sou...
2018-06-19 18:21:47
958
原创 spark streaming kafka 整合
package com.test.sparkimport org.apache.kafka.clients.consumer.ConsumerRecordimport org.apache.kafka.common.serialization.StringDeserializerimport org.apache.spark.SparkConfimport org.apache.spark...
2018-06-19 18:20:47
304
原创 kafka 简单 java 生产消费API 1.1
1. KafkaPropertiespackage com.immooc.spark.kafka;public class KafkaProperties { public static final String ZK = "localhost:2181"; public static final String TOPIC = "test"; public stat...
2018-06-19 18:19:27
649
2
原创 spark streaming 中使用 spark sql
package com.immooc.sparkimport org.apache.spark.{SparkConf, rdd}import org.apache.spark.rdd.RDDimport org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming.{Seconds, StreamingCon...
2018-06-19 18:06:25
2299
1
原创 spark streaming foreachRDD 使用
package com.immooc.sparkimport java.sql.DriverManagerimport org.apache.spark.SparkConfimport org.apache.spark.streaming.{Seconds, StreamingContext}object ForeachRDDApp { def main(args: Array[...
2018-06-19 17:56:16
585
原创 spark streaming 文件系统
package com.immooc.sparkimport org.apache.spark.SparkConfimport org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming.{Seconds, StreamingContext}object FileWordCount { def mai...
2018-06-19 17:54:47
283
原创 spark streaming networkwordcount
package com.immooc.sparkimport org.apache.spark.SparkConfimport org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming.{Seconds, StreamingContext}object NetworkWordCount { de...
2018-06-19 17:52:43
303
原创 kafka 简单 java 生产消费API
1. KafkaProperties package com.immooc.spark.kafka;public class KafkaProperties { public static final String ZK = "localhost:2181"; public static final String TOPIC = "test"; public sta...
2018-06-19 17:50:06
265
原创 flume 收集日志到 kafka 整合
就是服务器B的Sink要换成kafka 的sink即可服务器A还是不变：# Define a memory channel called ch1 on agent1agent1.channels.ch1.type = memoryagent1.channels.ch1.capacity = 1000agent1.channels.ch1.transactionCapacity = 100 ...
2018-06-19 17:48:14
354
原创 kafka 单机环境搭建
1. 安装zookeeperkafka要先安装zookepper1.1下载解压zookeeper1.2 conf/zoo_sample.cfg 复制为zoo.cfgdataDir=/Users/walle/data/zookeeper 修改下dataDir 目录即可1.3 运行zookeeperzkServer.sh start 2. 下载解压kafka3. 修改 config/server....
2018-06-16 19:11:11
807
原创 flume 搜集A 服务器 的日志到B 服务器
就是服务器A的Sink 类型是AVRO， 而 服务器 B的Source 是AVRO 1.服务器A的配置# Define a memory channel called ch1 on agent1agent1.channels.ch1.type = memoryagent1.channels.ch1.capacity = 1000agent1.channels.ch1.transaction...
2018-06-16 19:09:23
533
原创 flume exec 监听log文件修改
# Define a memory channel called ch1 on agent1agent1.channels.ch1.type = memoryagent1.channels.ch1.capacity = 1000agent1.channels.ch1.transactionCapacity = 100 # Define an Avro source called avro...
2018-06-16 19:08:37
1088
原创 flume 环境部署
1. 下载解压略2. 设置环境变量略3. 修改配置conf/flume-env.sh.template 复制一份flume-env.sh，然后修改下JAVA_HOME即可4. 例子# Define a memory channel called ch1 on agent1agent1.channels.ch1.type = memoryagent1.channels.ch1.capacit...
2018-06-16 19:07:42
279
原创 Scala 语法入门
1. for 循环，while 循环 for(i <- 1 to 3; j <- 1 to 3) println(i * j) for(i<-1 to 3;j<-1 to 3 if i != j) print(f"${10*i+j}%3d") 2. 函数//参数后面跟类型def sum(a : Int, b : Int) : Int = {return a + b}...
2018-06-16 19:03:50
287
原创 spark 编译包含第三方jar
1. 加插件<plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <archive> <ma...
2018-06-16 18:56:44
441
native-audio.rar
2012-04-20
三星Calendar
2013-08-26
Air Hockey.7z
2013-08-10
boost xpressive
2013-07-24
MFC 多线程
2013-07-22
MFC 数据库操作
2013-07-22
Android Opengles 例子
2013-06-22
号码归属地.rar
2013-05-28
CursorLoadSample.rar
2013-05-17
HLSL Pixel Shader
2012-12-18
YawPitchRoll.rar
2012-12-07
RotationArbitrary.rar
2012-12-04
3ds max 2009 导出x文件插件(panda)
2012-10-15
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人