sunbin11220904的博客

DataFrame 的函数Action 操作1、 collect() ,返回值是一个数组，返回dataframe集合所有的行2、 collectAsList() 返回值是一个java类型的数组，返回dataframe集合所有的行3、 count() 返回一个number类型的，返回dataframe集合的行数4、 describe(cols: String*) 返回一个通过数学计算的类表值(count, mean, stddev, min, and max)，这个可以传多个参数，中间用逗号分隔，如

Flume下载flume1.81:将下载的flume包，解压到/opt目录中cd /opttar -zxvf apache-flume-1.8.0-bin.tar.gz -C /optmv apache-flume-1.8.0-bin flume2:修改conf下的 flume-env.sh 配置文件,主要是JAVA_HOME变量设置cp复制flume-env.sh.template为flume-env.shexport JAVA_HOME=/opt/jdk3)验证是否安装成功./bin

一 数据的获取来源 我们这里直接从数据库中获取在mysql生成数据给定source sql数据结构如下我们数仓的分层分层 ：ods-ded-dws-dmdw(dwd,dwds)将dw层拆成了dwd，dwdsdwd（data warehouse detail）:数据明细层dws:数据服务层（轻度的汇总，dws层一般都是跨表的）dm（data market）：数据集市层（高度汇总）二使用sqoop导入 按实际条件情况导入全量导入:snbap_ods.ods user全量导入:snb

全量抽取customer表sqoop import \--connect jdbc:mysql://localhost:3306/sales_source \--driver com.mysql.jdbc.Driver \--username root \--password ok \--table customer \--hive-import \--hive-table sales_rds.customer \--hive-overwrite \--target-dir tmp \

在mysql中生成数据source /root/snbap_ods.sql;Hive 分层: ODS->DWD->DWS->DM将DW层拆成了DWD,DWDSDWD(data warehouse detail): 数据明细层DWS(data warehouse service): 数据服务层(轻度的汇总,DWS层一般是跨表的)DM(data market):数据集市层(高度汇总)全量导入: snbap_ods.ods_user全量导入: snbap_ods.ods_user

poll方式：package test01import org.apache.spark.SparkConfimport org.apache.spark.streaming.dstream.ReceiverInputDStreamimport org.apache.spark.streaming.flume.{FlumeUtils, SparkFlumeEvent}import org.apache.spark.streaming.{Seconds, StreamingContext}ob

使用Spark Streaming统计hdfs文件单词代码package test01import org.apache.spark.SparkConfimport org.apache.spark.streaming.dstream.DStreamimport org.apache.spark.streaming.{Seconds, StreamingContext}import test01.Demo01.sscobject HDFSInputStreamDemo extends Ap

代码如下package steamingimport org.apache.spark.SparkConfimport org.apache.spark.streaming.dstream.{DStream, ReceiverInputDStream}import org.apache.spark.streaming.kafka.KafkaUtilsimport org.apache.spark.streaming.{Seconds, StreamingContext}object Stre

以train.csv 为例 .conf文件如下train.sources = trainSourcetrain.channels = kafkaChannel hdfsChanneltrain.sinks = kafkaSink hdfsSinktrain.sources.trainSource.type = spooldirtrain.sources.trainSource.spoolDir = /opt/kb07file/flumeFile/traintrain.source

用flume 读取users.csv 传出到kafka读取 .conf 文件如下users.sources=usersSourceusers.channels =usersChannelusers.sinks =usersSinkusers.sources.usersSource.type = spooldirusers.sources.usersSource.spoolDir=/opt/kb07file/flumeFile/usersusers.sources.usersSo

kafka-2.11-2.0.0[root@lijia1 config]# vi ./server.propertiesbroker.id=0advertised.listeners=PLAINTEXT://192.168.153.141:9092log.dirs=/opt/bigdata/kafka211/kafka-logszookeeper.connect=192.168.153.141:2181delete.topic.enable=true// 配置环境变量export KAFK

过滤器Java代码：import org.apache.flume.Context;import org.apache.flume.Event;import org.apache.flume.interceptor.Interceptor;import java.util.ArrayList;import java.util.List;import java.util.Map;public class InterceptorDemo implements Interceptor {

taildir.confa1.sources = r1a1.sinks = k1a1.channels = c1a1.sources.r1.type = TAILDIRa1.sources.r1.filegroups=f1 f2a1.sources.r1.filegroups.f1=/root/data/tail_1/example.loga1.sources.r1.filegroups.f2=/root/data/tail_2/.*log.*a1.sources.r1.position

logger输出csv文件events.confevents.sources =eventsSourceevents.channels =eventsChannelevents.sinks= eventsSinkevents.sinks.eventsSink.type= loggerevents.sources.eventsSource.type =spooldirevents.sources.eventsSource.spoolDir = /opt/kb07file/flumeFile/

user_friends.sources = userFriendsSourceuser_friends.channels = userFriendsChanneluser_friends.sinks = userFriendsSinkuser_friends.sources.userFriendsSource.type = spooldiruser_friends.sources.userFriendsSource.spoolDir = /opt/kb07file/flumeFile/user_

hbase-env.sh#The java implementation to use. Java 1.7+ required.export JAVA_HOME=/opt/jdk#Tell HBase whether it should manage it's own instance of Zookeeper or not.export HBASE_MANAGES_ZK=falsehbase-site.xml<property> <name>hb

Hive建表语句创建内部表：– 创建一个内部表create table if not exists student(id int, name string)row format delimited fields terminated by ‘\t’stored as textfilelocation ‘/home/hadoop/hive/warehouse/student’;– 查询表的类型desc formatted student;Hive建表语句解析：Hive建表高阶语句

HDFS特点HDFS优点：支持处理超大文件可运行在廉价机器上高容错性流式文件写入HDFS缺点：不适合低延时数据访问场景不适合小文件存取场景不适合并发写入，文件随机修改场景简单的一些l命令创建存放数据文件的目录：hdfs dfs -mkdir /hdfs/shellhdfs dfs -ls /hdfs/shell将通讯数据上传到HDFS并查看hdfs dfs -put /home/hadoop/data/mobile.txt /hdfs/shellhdfs dfs -text

创建maven工程new project -Maven quickstart配置prm.xml<dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</

这里说复制虚拟机：vi /hadoop/hdfs-site.xmlvi etc/hadoop/slaves :hadoop04hadoop05hadoop06在传到其他两个格式化HDFShadoop namenode -format启动hadoopstart-all.sh（jps查看进程）安装ZooKeeper修改zookeepr/conf/zoo.cfg）（修改完后改名）配置里面的server是zookeeper服务器的主机名。# The number of

一、配置虚拟机（配置好jdk）配置生成秘钥： ssh-keygen -t rsa -P “”cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys配置网卡cd /etc/sysconfig/network-scripts/修改：ifcfg-enp0s3BOOTPROTO=staticONBOOT=yesIPADDR=192.168.56.137ifcfg-enp0s8BOOTPROTO=dhcpONBOOT=yes

put /索引/类型put /索引/类型/1{json串}POST /索引/类型/1/_update{“doc”：{列名：值//进准修改其中某个列}}DELETE /索引POST /_bulk ---批量操作增删改--必须写在同一行{“create”：{“_index”:索引，“_type”：类型，“_id”：编号}}{json串}{“update”：{“index”：索引，“_type”：类型，“_id”：编号}}“doc”：{json串}{“delete

一、入门编写规范：代码规范：#!/bin/bash [指定告知系统当前这个脚本要使用的shell解释器]shell 相关指令文件名规范:文件名.sh .sh是linux下bash shell 的默认后缀使用流程：1创建.sh文件 touch/vim2编写shell代码3执行shell脚本 脚本必须得有执行权限demo1：创建test.sh，实现第一个shell脚本程序，输出赋权：chmod +x test.sh 运行注意：输出的内容如果包含字母和符号