diwu20060107-CSDN博客

转载 brew: command not found

sudo vim .bash_profileadd : export PATH=/usr/local/bin:$PATHsave : source .bash_profile to make change work。/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/mas...

2017-11-23 21:49:00 164

转载 rest get post 请求工具_restclient.jar

下载：restclient.jar启动命令： java -jar restclient.jar界面如图所示：转载于:https://www.cnblogs.com/liupuLearning/p/7245659.html

2017-07-27 16:40:00 164

转载判断文件是否存在 local/hdfs

在Linux文件系统中，我们可以使用下面的Shell脚本判断某个文件是否存在：# 这里的-f参数判断$file是否存在 if [ ! -f "$file" ]; thenecho "文件不存在!"fi但是我们想判断HDFS上某个文件是否存在咋办呢？别急，Hadoop内置提供了判断某个文件是否存在的命令：$ hadoop fs -help......-test -[defsz] &lt...

2017-07-17 11:11:00 213

转载 Batch the files in the directory

#!/bin/bash#sourceFolder = /home/bigdatagfts/pl62716/refdata#targetFolder = /home/bigdatagfts/pl62716/refdata_targetsourceFolder=$1targetFolder=$2if [ $# != 2 ] ; then ech...

2017-06-19 16:54:00 131

转载 jasper_excel_sheet tab color

<property name="net.sf.jasperreports.export.xls.sheet.tab.color" value="#00FF00"/>转载于:https://www.cnblogs.com/liupuLearning/p/6971884.html

2017-06-09 14:57:00 104

转载 hive_hiveserver2 hive-site.xml config and start

hive-site.xml# vi hive-site.xml <configuration><property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://sd-9c1f-2eac:...

2017-05-05 17:21:00 126

转载 Caused by: MetaException(message:Hive Schema version 2.1.0 does not match metastore's schema version...

Caused by: MetaException(message:Hive Schema version 2.1.0 does not match metastore's schema version 1.2.0 Metastore is not upgraded or corrupt)[Dev root @ sd-9c1f-2eac /usr/local/src/apach...

2017-05-05 14:29:00 160

转载 Hive MetaStore Upgrade

# cd $HIVE_HOME/scripts/metastore/upgrade/mysql [Dev root @ sd-9c1f-2eac /usr/local/src/apache-hive-2.1.1-bin/scripts/metastore/upgrade/mysql]# ls001-HIVE-972.mysql.sql 027-HIVE-12819.mysql.sq...

2017-05-05 11:51:00 572

转载 Error: Duplicate key name 'PCS_STATS_IDX' (state=42000,code=1061)

以下异常说明mysql已经启动。应先关掉先前启动的mysql.再执行初始化schema操作。$service mysql stop;# $HIVE_HOME/bin/schematool -dbType mysql -initSchemawhich: no hbase in (/usr/lib64/qt-3.3/bin:/usr/local/sbin:/bin:/usr/...

2017-05-03 16:47:00 484

转载 Caused by: MetaException(message:Hive Schema version 2.1.0 does not match metastore's schema version...

解决方案汇总：(1)删除HDFS上的hive数据与hive数据库 hadoop fs -rm -r -f /tmp/hive hadoop fs -rm -r -f /user/hive(2)删除MySQL上的hive的元数据信息 mysql -uroot -p drop database hive(3)初始化hive, 将mys...

2017-03-24 10:53:00 298

转载 Scala_Load csv data to hive via spark2.1_via pass parameters_HiveAllType

prepare CSV dataNT,col_SMALLINT,col_BIGINT,col_INT,col_FLOAT,col_DOUBLE,col_DECIMAL,col_TIMESTAMP,col_DATE,col_INTERVAL,col_STRING,col_VARCHAR,col_CHAR,col_BOOLEAN,col_BINARY100Y,100S,100L,...

2017-03-17 18:30:00 106

转载 Scala_Load csv data to hive via spark2.1_via pass parameters

prepare csv:# vim /tmp/cars.csv year,make,model,comment,blank"2012","Tesla","S","No comment","1997","Ford,E350","Go get one now they are going fast","2015","Chevy","Volt"scala ...

2017-03-17 16:25:00 101

转载 Scala_Load csv data to hive via spark2.1

code:package com.liupuimport org.apache.spark.{ SparkContext, SparkConf }import org.apache.spark.sql.hive.HiveContextimport org.apache.spark.sql.hive.orc._object HiveContextLoadCsv {...

2017-03-17 16:18:00 103

转载 Hadoop_配置Hadoop开发环境（Eclipse）

通常我们可以用Eclipse作为Hadoop程序的开发平台。1）下载Eclipse下载地址：http://www.eclipse.org/downloads/根据操作系统类型，选择合适的版本下载并安装。2）下载并编译Hadoop的Eclipse插件网上可直接下载到Hadoop 1.x 的Eclipse插件，但是该插件和Hadoop2.2 不兼容，所以无法使用。...

2017-03-15 18:24:00 85

转载 spark_运行spark-shell报错_javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the ...

error:# ./spark-shellCaused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:mysql://sd-9c1f-2eac:3306/hive?createDatabase...

2017-03-15 11:33:00 413

转载 spark_load csv to hive via hivecontext

//prepare csvyear,make,model,comment,blank"2012","Tesla","S","No comment","1997","Ford,E350","Go get one now they are going fast","2015","Chevy","Volt"//Processing and inserting ...

2017-03-15 11:18:00 94

转载 spark_spark连接hive config

1 环境hadoop2.7.3apache-hive-2.1.1-binspark-2.1.0-bin-hadoop2.6jdk1.82 配置文件在hive-site.xml中配置mysql数据库连接。cpapache-hive-2.1.1-bin/conf/hive-site.xml ./spark-2.1.0-bin-hadoop2.6/conf...

2017-03-14 17:52:00 197

转载 Hive_hdfs csv导入hive demo

1 create csv file.student.csv4,Rose,M,78,77,765,Mike,F,99,98,982 put it to hdfs.# hdfs dfs -put student.csv /input3 create table in hive.create table student_csv(sid int, ...

2017-03-07 16:52:00 126

转载 Maven_setting.xml

<?xml version="1.0" encoding="UTF-8"?><!--Licensed to the Apache Software Foundation (ASF) under oneor more contributor license agreements. See the NOTICE filedistributed wi...

2017-03-03 13:55:00 115

转载 scala 基础

// ::链接字符串链接字符和listscala> val a = List(1,2,3)a: List[Int] = List(1, 2, 3)scala> val b = 0::ab: List[Int] = List(0, 1, 2, 3)scala> val c = "x"::"y"::"z" :: Nilc: List[St...

2017-03-01 15:55:00 80

转载 scala worksheet demo

object worksheet_lp { println("Welcome to the Scala worksheet") //> Welcome to the Scala worksheet var x = 1 //> x : Int = 1 println(...

2017-02-28 15:44:00 125

转载 Linux Maven install

1 下载　　maven : http://maven.apache.org/download.cgi2 解压　　tar -xvf apache-maven-3.3.9-bin.tar.gz3 移到所需目录　　mv -rf apache-maven-3.3.9 /usr/local/4 编辑profile文件　　vi /etc/profileMAVEN_HOME=/usr/lo...

2017-02-27 10:12:00 131

转载 build spark

Error :Failed to find Spark jars directory (/home/pl62716/spark-2.2.0-SNAPSHOT/assembly/target/scala-2.10/jars).You need to build Spark with the target "package" before running this program....

2017-02-27 09:41:00 586

转载 eclipse plugin

快速查看目录org.sf.easyexplore_1.0.4.jarmongo DBnet.jumperz.app.MMonjaDB_1.0.16.jarjasper reportjasperreports-6.1.1.jar转载于:https://www.cnblogs.com/liupuLearning/p/6406626.html...

2017-02-16 17:10:00 68

转载 Python入门_汇总_未完待续

if/elseif/else for while break continue 多重循环list []duple()dict {}set {[]}函数help(abs) 查看abs函数的帮助信息abs(x)cmp(x, y) 比较函数,需要两个参数,如果 x<y，返回 -1，如果 x==y，返回 0，如果 x>y，返回 1...

2017-02-13 14:32:00 64

转载 Hive进阶_汇总

===========================================================================第2章 Hive数据的导入使用Load语句执行数据的导入：LOAD 导入语句到数据仓库的表当中。load data [local] inpath 'filepath' [overwrite]into table ...

2017-01-25 18:02:00 75

转载 Hive进阶_开发Hive的自定义函数

Hive中的自定义函数简介(1) 在类中创建自定义函数。自定义UDF需要继承'org.apache.hadoop.hive.ql.exec.UDF',实现evaluate函数,evaluate函数支持重载。(2) 将该类所在的包导出成jar包，放入linux目录下。(3) 进入hive客户端，删除旧的jar包hive> delete jar /dir/.jar;...

2017-01-25 16:28:00 119

转载 Hive进阶_Hive的客户端操作

启动远程客户端 # hive --service hiveserver2获取连接-〉创建运行环境-〉执行HQL-〉处理结果-〉释放资源工具类 1 package demo.utils; 2 3 import java.sql.Connection; 4 import java.sql.DriverManager; 5 import java.sql.Re...

2017-01-24 18:09:00 73

转载 java.lang.ClassNotFoundException: org.slf4j.LoggerFactory

缺少slf4j-api.jar和slf4j-log4j12.jar这两个jar包导致的错误。转载于:https://www.cnblogs.com/liupuLearning/p/6347585.html

2017-01-24 17:48:00 966

转载 java.sql.SQLException: No suitable driver found for jdbc:hive://localhost:10000/default

error:java.sql.SQLException: No suitable driver found for jdbc:hive://localhost:10000/default at java.sql.DriverManager.getConnection(DriverManager.java:596) at java.sql.DriverManag...

2017-01-24 17:17:00 741

转载 Hive进阶_Hive的子查询

- 集合中如果含null数据，不可使用not in, 可以使用in- hive只支持where和from子句中的子查询- 主查询和自查询可以不是同一张表select e.enamefrom emp ewhere e.deptno in(select d.deptno from dept d where d.dname='SALES' or d.dname='ACC...

2017-01-24 14:50:00 272

转载 Hive进阶_Hive的表连接

等值连接select e.empno, d.deptnofrom emp e, dept dwhere e.deptno=d.deptno;不等值连接select e.empno, e.ename, e.sal, s.gradefrom emp e, salgrade swhere e.sal between s.losal and s.hisal...

2017-01-24 14:15:00 69

转载 Hive进阶_内置函数

Hive数学函数round : 四舍五入ceil : 向下取整floor : 向上取整select round(45.926,2),round(45.926,1),round(45.926,0),round(45.926,-1),round(45.926,-2);Hive字符函数select lower('Hello World'), u...

2017-01-23 16:00:00 67

转载 Hive进阶_Hive数据查询

简单查询和fetch task简单查询：简单查询的 fetch task 功能，从HDFS拉取，不用map reduce. 前两种配置，当前session有效。修改hive-site.xml永久有效。查询中使用过滤在查询中使用排序排序使用别名时要开启使用别名模式：set hive.groupby.orderby.position.al...

2017-01-23 15:26:00 84

转载 Hive进阶_Hive数据的导入

使用Load语句执行数据的导入语法：load data [local] inpath 'filepath' [overwrite]into table tablename [partition (partcoll=val1, partcal2=val2...)]explain: local是从本地导入，无local则默认从HDFS导入,overwrit...

2017-01-23 14:54:00 116

转载 Hive_Hive的数据模型_汇总

体系结构：元数据 /HQL的执行安装：嵌入 /远程 /本地管理： CLI /web界面 /远程服务数据类型：基本 /复杂 /时间数据模型：数据存储 /内部表 /分区表 /外部表 /桶表 /视图=========================================================================================...

2017-01-20 16:00:00 206

转载 Hive_Hive的数据模型_视图

- 视图是一种虚表，是一个逻辑概念；可以跨越多张表- 视图建立在已有表的基础上，视图赖以建立的这些表称为基表。- 视图可以简化复杂的查询。创建视图create view viewNameasselect data from table where condition;查看视图结构desc viewName;查询视图se...

2017-01-20 15:57:00 88

转载 Hive_Hive的数据模型_桶表

对数据进行HASH运算，放在不同文件中，降低热块，提高查询速度。例如：根据sname进行hash运算存入5个桶中。create table bucket_table(sid int, sname string, age int)clustered by (sname) into 5 buckets;hash值相同的数据放入同一个桶中。转载于:https:...

2017-01-20 15:48:00 103

转载 Hive_Hive的数据模型_外部表

Hive的数据模型之外部表外部表(External Table)- 指向已经在HDFS中存在的数据，可以创建Partition- 它和内部表在元数据的组织上是相同的，而实际数据的存储则有较大的差异。- 外部表侄有一个过程，加载数据和创建表同时完成，并不会移动到数据仓库目录中，只是与外部数据建立一个链接。当删除一个外部表时，仅删除该链接。1、准备几张相同数据结构的数据txt文件...

2017-01-20 15:31:00 98

转载 Hive_Hive的数据模型_分区表

Hive的数据模型之分区表准备数据表：create table sampledata(sid int, sname string, gender string, language int, math int, english int)row format delimited fields terminated by ',' stored as textfile;...

2017-01-20 13:30:00 88

空空如也

空空如也