JavaBigData的博客

【代码】hive升级HiveException java. lang. RuntimeException。

yarn

hdfs 50070无法访问，原因是未配置hdfs-site.xml<property> <name>dfs.namenode.http-address</name> <value>hadoop102:50070</value></property>其他节点未启动 NameNode NodeManager，在/home/baiyun/my_soft/hadoop-3.2.1/etc/hadoop/work

hadoop

192.168.138.102 hadoop102192.168.138.103 hadoop103192.168.138.104 hadoop104

【代码】Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpat。

grafana监控

supervisor离线安装。

flink on yarn提交样例，控制1个taskmanger 拥有的slot和cpu个数参数配置

【代码】python保存网页截图。

python钉钉机器人发图片

git指定版本安装

在idea中使用package命令带包，不带依赖flink_cdc-1.0-SNAPSHOT.jar1.添加第三方包到lib目录下，所有从节点也需要mv test/flink-connector-mysql-cdc-1.1.1.jar /opt/module/flink/lib/xsync /opt/module/flink/lib/flink-connector-mysql-cdc-1.1.1.jar2.启动集群start-cluster.shlocalhost:8081 webui3.

解压后重命名keytool -keystore keystore -alias jetty -genkey -keyalg RSA在 azkaban目录下生成keystore文件，然后mv到/lib/azkaban/azkaban-web/lib/azkaban/azkaban-web/conf/azkaban.properties#Azkaban Personalization Settingsazkaban.name=Testazkaban.label=My Local Az.

mail.sender=xxx@163.commail.host=smtp.163.commail.user=xxx@163.commail.password=邮箱授权码job.failure.email=xxx@163.comjob.success.email=xxx@163.com

flink-standalone 安装配置好后启动flink集群/opt/module/flink/bin/start-cluster.shhttp://192.168.9.101:8081/#/task-manager编写样例import org.apache.flink.api.scala.ExecutionEnvironmentimport org.apache.flink.streaming.api.scala._object FlinkTest { def ma

sqoop export \--connect jdbc:mysql://hd101:3306/company \--username root \--password 123456 \--table staff2 \--num-mappers 1 \--export-dir /company \--input-fields-terminated-by "\t"

1.JobHistoryServer得启动2.修改windows的hosts文件 C:\Windows\System32\drivers\etc\hosts,因为在服务器上可能配置的ip的别名，到Windows浏览器不识别3.vim /opt/module/hadoop/etc/hadoop/mapred-site.xml 中添加以下日志参数...

import javax.swing.*;import java.io.BufferedReader;import java.io.IOException;import java.io.InputStreamReader;import java.util.ArrayList;import java.util.Arrays;import java.util.Comparator;import java.util.TreeSet;public class Main9 { public

import java.io.BufferedReader;import java.io.IOException;import java.io.InputStreamReader;import java.util.ArrayList;import java.util.Collections;// afv1234ad 4public class Main10 { public static void main(String[] args) throws IOException { .

查看vmnet8无法访问打开window的 VMware NET Service

[baiyun@hd102 bin]$ hivels: 无法访问/opt/module/spark/jars/spark-*.jar: 没有那个文件或目录解决hive启动报错提示 修改/opt/module/hive/bin/hiveif [[ -n “$SPARK_HOME” ]]thensparkAssemblyPath=ls ${SPARK_HOME}/lib/spark-assembly-*.jarCLASSPATH="CLASSPATH:{CLASSPATH}:CLASSPATH:{s

zookeeper安装后报错查询zookeeper.out日志java.net.NoRouteToHostException: 没有到主机的路由 (Host unreachable)查看防火墙xcall sudo service iptables status发现有个机器未关闭关闭防火墙sudo service iptables stop重新启动zookeeper集群xcall zkServer.sh status -> OK...

opt/module/hive/bin/hiveserver2需要将以下两个包提前放进去，不然下载很慢同时需要开启

只能保留 host=%的，其他都需要删掉Caused by: java.sql.SQLException: Access denied for user 'root'@'hd101' (using password: YES) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1078) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237) at co

hdfs上传文件只有部分节点有副本部分节点防火墙关闭，部分未关闭

hadoop:50070和yarn8088 web网页打不开检查namenode和resourcemanager机器的防火墙sudo service iptables status-若运行中，则需要关闭sudo service iptables stop

[baiyun@hadoop103 yanagishima-18.0]$ nohub bin/yanagishima-start.sh >y.log 2>&1 &jetty.port=7080presto.datasources=baiyun-prestopresto.coordinator.server.baiyun-presto=http://hadoop103:8881catalog.atiguigu-presto=hiveschema.atiguigu-pres

from pyhive import hiveconn = hive.Connection(host='hadoop104',port=10000,username='baiyun' ,database='gmall')cursor = conn.cursor()cursor.execute('show tables')cursor.execute('show tables')cursor.execute('create table create_by_pyhive as select 1')..

使用 pip install sasl-0.2.1-cp36-cp36m-win_amd64.whl 离线安装https://www.lfd.uci.edu/~gohlke/pythonlibs/#saslLooking in indexes: https://pypi.tuna.tsinghua.edu.cn/simpleCollecting saslUsing cached https://pypi.tuna.tsinghua.edu.cn/packages/8e/2c/45dae93d666a

用户名为linux的用户名称，密码不需要

hive cast('' as decimail(10,8))强转空字符串输出null

with a as (select code,name from tb_name group by code,name),b as (select code ,count(*) cn from a)select * from b where cn >1;

if(flag rlike '\^[0-9a-zA-Z]',flag ,null)