docsz的博客

使用kettle将Oracle数据生产json文本

Sqoop处理Oracle大字段BLOB

ambar2.7配置hive_aux_jars_path

hive复制表结构

1、脚本{ "job": { "content": [{ "reader": { "name": "hdfsreader", "parameter": { "column": [ { "index": 0, "type": "STRING" }, { "index": 1, "type": "STRING" }, { "index": 2,

1、脚本{ "core": { "transport": { "channel": { "speed": { "byte": 10485760, "record": 100000 } } } }, "job": { "content": [{ "reader": { "name": "oraclereader", "parameter": { "column": [ "BJBH", "

1、Sqoop从Hive导入Oraclesqoop export \--connect "jdbc:oracle:thin:@//IP:1521/orcl" \--username "root" \--password "123456" \--table STG_ZFW_WANGGEXINXI \--export-dir /user/hive/warehouse/mask_data.db/stg_zfw_wanggexinxi/* \--columns XM,ZJHM,WANGGE,BUILD

1、Hive建表CREATE TABLE `rcdn_ads.ads_rcdn_zc_xq`( `qybm` varchar(25) COMMENT '区域编码', `csbm` varchar(25) COMMENT '城市编码', `shengbm` varchar(25) COMMENT '省份编码', `cybm` varchar(25) COMMENT '产业编码', `zclxbm` varchar(25) COMMENT '政策类型编码', `zcztbm`

1、脚本{ "setting": {}, "job": { "setting": { "speed": { "channel": 2 } }, "content": [ { "reader": { "name": "mysqlreader", "para

1、脚本{ "job": { "setting": { "speed": { "channel": 3 } }, "content": [ { "reader": { "name": "hdfsreader", "parameter":

1、脚本{ "job": { "setting": { "speed": { "channel": 3 } }, "content": [ { "reader": { "name": "hdfsreader", "parameter":

sqoop import query

sqoop Import of TINYINT

sqoop mysql导入hive parquet表timestamp,decimal

sqoop import decimal

Oracle批量导出Hive建表语句

1、json脚本{ "job": { "setting": { "speed": { "channel": 3 }, "cipher": { "sfjm": "Y", "instance": "AES128" }, "errorLimit": { "record": "-1" } }, "content": [{ "reader":

1、mysql-to-hive.json{ "setting": {}, "job": { "setting": { "speed": { "channel": 2 } }, "content": [ { "reader": { "name": "mysqlreader", "parameter": { "username": "root", "password": "123456", "connec

OLAP系统建模方法:典型的数据仓库建模方法有ER模型、维度模型、Data Value、Anchor

数据仓库VS数据库

drop database if exists `ods`;create database `ods`;use ods; create table `dept`(`deptno` int, `dname` varchar(14),`loc` varchar(13)) row format delimited fields terminated by '\t' stored as textfile; insert into dept values (10,'accounting','n

Statistics使用1、表，分区级别的统计信息对于新创建的表或者分区，默认情况下，如果通过INSERT OVERWRITE的方式插入数据，那么Hive会自动将该表或分区的统计信息更新到元数据。有一个参数来控制是否自动统计，hive.stats.autogather，默认为true.对于已经存在表或分区我们可以通过ANALYZE命令手动更新其Statistics信息：ANALYZE TABLE Table1 COMPUTE STATISTICS;//统计全表的所有分区的信息ANALYZE TA

待完善，比如何删除建表语句的倒数第2到13行；欢迎指正

create table t_shop_dm.t_tmp_txtrow format delimited fields terminated by ‘\t’ lines terminated by ‘\n’stored as textfile as select * from t_shop_dm.t_tmp;

1、修改pom文件 <spark.version>3.1.1</spark.version> <scala.binary.version>2.12</scala.binary.version> <scala.version>2.12.10</scala.version> <hadoop.version>3.2.2</hadoop.version> <guava.version&g

1、pom依赖 <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-jdbc</artifactId> <version>${hive.version}</version> </dependency> <dependen

Hive批量删除表分区#!/usr/bin/bash#获取当前脚本所在路径cur_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"cd ${cur_dir}#导出所有hive数据库名hive -e "show databases;" > ${cur_dir}/all_database.txt#获取以src开头的库名cat all_database.txt | awk '{print $2}' | awk '/

-----------创建表CREATE TABLE dept(deptno INT(2) NOT NULL,dname VARCHAR(14),loc VARCHAR(13),CONSTRAINT pk_dept PRIMARY KEY(deptno)) ENGINE=InnoDB DEFAULT CHARSET=utf8;CREATE TABLE emp (empno int(4) NOT NULL PRIMARY KEY,ename VARCHAR(10),job VARCHAR(

1：切换到root用户chmod -R 777 /usr/hdp/3.1.4.0-315/hive/lib/hive-contrib-3.1.0.3.1.4.0-315.jar2：切换到hive用户：su - hive,进入hive;3：hive授权：set role admin;4：add jar /usr/hdp/3.1.4.0-315/hive/lib/hive-contrib-3...