![](https://img-blog.csdnimg.cn/20201014180756925.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
hive
独狐游清湖
学海无涯
展开
-
pyspark到mysql
1 先上整体代码import syssys.path.append('/data/gtair/tools/DWH')import pandas as pdfrom sqlalchemy import create_engineengine = create_engine('mysql+pymysql://edu_ws:QbvkrBHKXO@chb2gc-pd-edu-dws.mysql.rds.aliyuncs.com:3306/edu_platform')df_spark =ss.sql("原创 2021-09-28 15:58:48 · 249 阅读 · 0 评论 -
ambari+hdp下mysql元数据用户ambari、hive、root、azkaban用户密码修改
写在前面ambari+hdp 安装集群的时候密码修改太简单,出于安全考虑,做各用户的密码修改。1 mysql 用户管理表字段CREATE TABLE `user` ( `Host` char(60) COLLATE utf8_bin NOT NULL DEFAULT '', `User` char(32) COLLATE utf8_bin NOT NULL DEFAULT '', `Select_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL原创 2021-09-07 17:14:49 · 1486 阅读 · 1 评论 -
hive函数之collect_set,collect_list,concat_ws,array_contains 联合使用
hive函数之collect_set/collect_list,concat_ws,array_contains 联合使用1 表数据select id,act_status from ods.ods_ta_task limit 6;+----------------------+-------------+| id | act_status |+----------------------+-------------+| 34999455991526037原创 2021-08-30 16:37:33 · 906 阅读 · 0 评论 -
hive合并小文件
hive合并小文件创建备用表 create table new_table like old_table;设置合并参数,再将原表数据插入到备用表SET hive.merge.mapfiles = true;SET hive.merge.mapredfiles = true;SET hive.merge.size.per.task = 256000000;SET hive.merge.smallfiles.avgsize = 134217728;SET hive.exec.compr原创 2021-08-26 14:30:52 · 224 阅读 · 0 评论 -
canal+hbase+hive实时数据同步方案
MySQL数据实时同步到HBase实时方案MySQL数据实时同步HBase分两步:第一步先数据初试化,即:将历史数据同步到HBase;第二步将MySQL增量数据实时同步到HBase;一、数据初试化,即:历史数据同步1、方案2、DataX环境搭建2.1、前置条件LinuxJDK(1.8以上,推荐1.8)Python(推荐Python2.6.X)2.2、下载搭建DataX环境cd /optmkdir dataxcd dataxwget http://datax-opensource原创 2021-04-23 10:28:19 · 1487 阅读 · 0 评论 -
hive-e直接计算数据结果到mysql的便捷写法
2 hive直接计算到mysql的shell写法#!bin/bash#source /etc/profileDAY=$1time=$(date "+%Y-%m-%d %H:%M:%S")base_dir=$(cd `dirname ../../`; pwd)if [ ! -n "$DAY" ]; thenDATE=`date +%Y-%m-%d`elseDATE=`date -d "$DAY" +%Y-%m-%d`fiUSER=***PASSWD=***ADDRESS=*原创 2021-02-24 16:59:21 · 213 阅读 · 0 评论 -
hive报错积累
1 distinctselectcast(a.teach_plan_id as bigint) as teach_plan_id,cast(nvl(teach_activity_num ,0) as int) as teach_activity_num,cast(nvl(knowledge_num,0) as int) as knowledge_numfrom(select distinct id as teach_plan_id from ods.ods_teachingplan where原创 2021-02-20 18:01:01 · 510 阅读 · 0 评论