大数据
关注
分享
扫一扫
文章平均质量分 82
大数据相关知识整理及环境搭建
lql_h
这个作者很懒，什么都没留下…
展开
-
kylin
配置分发hive并建立软链接cdh1:deploy.shhive-1.1.0-cdh5.10.0/root/app/slavecdh2,cdh3:ln-shive-1.1.0-cdh5.10.0hive环境变量[所有节点]vi/etc/profileexportHADOOP_HOME=/root/app/hadoopexportP...原创 2020-04-20 14:56:59 · 382 阅读 · 0 评论 -
sparkstructuredstreaming
官网实例http://spark.apache.org/docs/2.3.0/structured-streaming-programming-guide.html#quick-example实例代码Completeimportorg.apache.spark.sql.functions._importorg.apache.spark.sql.SparkSession...原创 2020-04-17 15:55:23 · 277 阅读 · 0 评论 -
spark集成
Hive配置将hive的配置文件hive-site.xml拷贝到sparkconf目录cp/root/app/hive/conf/hive-site.xml/root/app/spark-alone/conf/vihive-site.xml<property><name>hive.metastore.uris</name&...原创 2020-04-17 15:51:31 · 322 阅读 · 0 评论 -
sparksql
pom<dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql_2.11</artifactId><version>2.3.0</version><scope>provided</scope&g...原创 2020-04-17 15:38:29 · 194 阅读 · 0 评论 -
spark实时计算
第一个实例安装ncyuminstall-ync实例官网http://spark.apache.org/docs/2.3.0/streaming-programming-guide.html#a-quick-example启动服务nc-lk9999./run-examplestreaming.NetworkWordCountlocalhost999...原创 2020-04-17 15:31:39 · 452 阅读 · 0 评论 -
spark集群搭建
Scala安装及环境变量配置环境变量JAVA_HOME=/root/app/jdkSCALA_HOME=/root/app/scalaCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarPATH=$JAVA_HOME/bin:$SCALA_HOME/bin:/root/tools:/home/hadoop...原创 2020-04-17 15:20:48 · 160 阅读 · 0 评论 -
spark编程模型
创建RDD从集合创建RDD创建方式一：parallelizescala> val paraRDD = sc.parallelize(arr)paraRDD: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:26scala> val...原创 2020-04-17 15:14:10 · 129 阅读 · 0 评论 -
spark快速入门
基本安装官网http://spark.apache.org/docs/2.3.0/quick-start.html解压并创建软连接tar-xzvfspark-2.3.0-bin-hadoop2.6.tgzln-sspark-2.3.0-bin-hadoop2.6sparksparkshell[root@cdh1bin]#./spark-shell...原创 2020-04-17 15:08:12 · 306 阅读 · 0 评论 -
kafka
kafka配置配置文件zookeeper.propertiesdataDir=/root/data/zookeeper/zkdataconsumer.propertieszookeeper.connect=cdh1:2181,cdh2:2181,cdh3:2181producer.propertiesbootstrap.servers=cdh1:9092,cdh2:90...原创 2020-04-08 11:13:24 · 205 阅读 · 0 评论 -
flume
flume配置与启动官方文档http://flume.apache.org/配置flume-conf.properties.templateagent.sources=seqGenSrcagent.channels=memoryChannelagent.sinks=loggerSink#Foreachoneofthesources,th...原创 2020-04-08 11:10:00 · 143 阅读 · 0 评论 -
sqoop
sqoop导入hdfs官方文档http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html修改sqoop-env.sh#Setpathtowherebin/hadoopisavailableexportHADOOP_COMMON_HOME=/root/app/hadoop#Setpathto...原创 2020-04-08 11:02:25 · 175 阅读 · 0 评论 -
HIVE-java api
pom <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> ...原创 2020-04-01 21:56:44 · 175 阅读 · 0 评论 -
HBASE-Java api
快捷键Alt + / Alt +shift+ lpom <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> &...原创 2020-04-01 21:53:03 · 130 阅读 · 0 评论 -
HIVE环境搭建
HIVE安装解压并创建软连接tar-xzvfhive-1.1.0-cdh5.10.0.tar.gzln-shive-1.1.0-cdh5.10.0hive配置文件hive-log4j.propertieshive.log.dir=/root/data/hive/logshive-env.shexportHADOOP_HOME=/root/app/h...原创 2020-04-01 21:43:08 · 267 阅读 · 0 评论 -
HBASE环境搭建
HBASE安装解压并创建软连接tar-xzvfhbase-1.2.0-cdh5.10.0.tar.gzlnhbase-1.2.0-cdh5.10.0hbase配置文件hbase-site.xml<configuration><property><name>hbase.zookeeper.quorum</name&...原创 2020-04-01 21:32:14 · 233 阅读 · 0 评论 -
HDFS的api
获取文件系统句柄 public static FileSystem getFileSystem() throws Exception{ Configuration conf = new Configuration(); URI uri = new URI("hdfs://cdh2:9000"); //active,不能是stand...原创 2020-03-30 11:55:42 · 121 阅读 · 0 评论 -
hadoop运行程序
配置pomMaven地址https://mvnrepository.com/tags/maven<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-common</artifactId><versio...原创 2020-03-30 11:32:15 · 413 阅读 · 0 评论 -
hadoop集群安装部署
HDFS解压 Hadooptar -xzvf hadoop-2.6.0-cdh5.10.0.tar.gz创建软连接ln -s hadoop-2.6.0-cdh5.10.0 hadoop修改配置文件路径/root/app/hadoop-2.6.0-cdh5.10.0/etc/hadoopcore-site.xml<configuration>...原创 2020-03-26 09:45:04 · 262 阅读 · 0 评论 -
zookeeper
JDK安装解压缩tar-xzvfjdk-8u51-linux-x64.tar.gz创建软连接ln-sjdk1.8.0_51jdk修改环境配置变量vi/etc/profileJAVA_HOME=/root/app/jdkCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarPATH=...原创 2020-03-25 15:32:25 · 138 阅读 · 0 评论