大数据实战
文章平均质量分 68
木生火18624
多练内功
展开
-
升级对HDFS的依赖
org.apache.flink.runtime.client.JobExecutionException: Could not retrieve JobResult. at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:643) at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.jav...原创 2020-07-12 10:52:05 · 486 阅读 · 0 评论 -
org.apache.hadoop.hive.shims.Hadoop23Shims
xception in thread "main" java.lang.ExceptionInInitializerError at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:108) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) ...原创 2020-07-11 10:36:17 · 974 阅读 · 1 评论 -
org/apache/hadoop/hive/shims/ShimLoader
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoader at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:371) at org.apache.hadoop.hive.conf.HiveConf.<clinit>(Hiv...原创 2020-07-11 10:27:33 · 1152 阅读 · 0 评论 -
Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
20/07/10 21:51:48 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.20/07/10 21:51:48 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apach原创 2020-07-11 09:54:14 · 1947 阅读 · 0 评论 -
Running job时空间不足,一直卡着不动
java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1249) at java.lang.Thread.join(Thread.java:1323) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeRespond...原创 2020-07-10 23:28:56 · 588 阅读 · 0 评论 -
开启mysql的远程连接
java.sql.SQLException: null, message from server: "Host 'USER-20200511CB' is not allowed to connect to this MySQL server"因为用户名是root。所以需要用root 进行登录 。然后进行授权对hive库下的所有表 运行flink用户的任何ip 进行登录。 flink用户的密码是000000grant all PRIVILEGES on hive.* to 'fl...原创 2020-07-09 21:15:27 · 221 阅读 · 1 评论 -
hive中放弃保留关键件的使用
FailedPredicateException(identifier,{useSQL11ReservedKeywordsForIdentifier()}?) at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10924) at org.apache.hadoop.hive.ql.parse.HiveParse...原创 2020-07-09 19:08:53 · 384 阅读 · 0 评论 -
hive的编码问题
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)创建表的时候报错后面将生成的hive表删除。重启启动hive修改hive字符集alter database原创 2020-07-09 13:12:08 · 806 阅读 · 0 评论 -
Linux下centos6.8上的root与非root上安装Mysql5.6
0、查看之前是否有安装过mysql1、先查看是否有mysql / mariadb的安装依赖,如果有则卸载需要切换到root用户下进行卸载2、解压,拷贝到root用户下3、安装服务端,客户端4、启动,设置开机自启5、查看密码,登录6、grant all on hive.* to 'root'@'%' identified by '000000';...原创 2020-07-06 22:42:14 · 243 阅读 · 0 评论 -
java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are un
java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:528) at org.apache.hadoop....原创 2020-07-05 12:05:10 · 11362 阅读 · 0 评论 -
Redis在Linux下安装准备接受Flink上传的数据
将前面转换好后上传的数据,存到Redis中1、先下载一个Redis的包https://www.redis.net.cn/download/下载3.2.11版本(需要哪个版本直接改版本号),http://download.redis.io/releases/redis-3.2.11.tar.gz1、先安装gcc(因为:Redis是C实现的,需要gcc来进行编译)yum -y install gcc gcc-c++ libstdc++-devel如果遇到gcc安装不了显示...原创 2020-06-29 20:22:51 · 168 阅读 · 0 评论 -
Caused by: java.lang.ClassNotFoundExceptionorg/apache/flink/shaded/jackson2/com/fasterxml/jackson/co
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/shaded/jackson2/com/fasterxml/jackson/core/JsonParseException at org.apache.flink.runtime.webmonitor.WebMonitorEndpoint.initializeHandlers(WebMonitorEndpoint.java:212) at o...原创 2020-06-26 11:13:12 · 4409 阅读 · 1 评论 -
搭建kafka环境
查看就可以得到已经解压好的安装包进到配置文件,对zoo.cfg进行配置zookeeper下面创建一个zk_data,然后编辑zoo.cfg将其路径拷贝给dataDir然后在配置环境中进行配置zk_home然后接着安装kafka先到kafka下面创建一个ka_data作为日志存放地方进到kafka的配置文件中,编辑vi server.propertieskafka环境搭建后,测试kafka是否好用再打开一个窗口 创建kafka的主题...原创 2020-06-25 16:56:15 · 176 阅读 · 0 评论 -
01实现WordCount(批处理+流处理)快速上手——Flink
搭建 maven 工程 FlinkTutorial然后在pom.xml中添加依赖,和引入插件<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="htt...原创 2020-07-12 13:26:20 · 182 阅读 · 0 评论 -
02为实现Flink的部署做前期准备——Flink
官网中,有两种版本,一种是有hadoop支持的版本,一种是没有hadoop支持的版本解压后重点有两个目录,一个是bin,一个是conf原创 2020-07-12 13:27:08 · 116 阅读 · 0 评论 -
Linux的centos7在重启时一直黑屏
启动Linux时候一直是黑屏(依据网上经验)systemd-fsck[546]:fsck:error 2(No such file or directory) while executing fsck.ext2 for /dev/mapper/c1-roothost smbus controller not enabled但是参照好多方法都不行,后面问了群主。说可能是节点坏...原创 2020-01-09 16:59:36 · 7832 阅读 · 0 评论 -
通过yum安装资源失败
Loaded plugins: fastestmirror, langpacksOne of the configured repositories failed (Unknown),and yum doesn't have enough cached data to continue. At this point the onlysafe thing yum can do is fa...原创 2020-01-09 10:29:34 · 770 阅读 · 0 评论 -
SSH免密登录失败
不晓得是什么情况,可能是在删除操作的时候多删除东西了。导致免密登录失败。后面参考了: ssh: connect to host slave1 port 22: Connection refused下面依据节点查找原因。免密登录失败节点查不出东西(ssh服务安装出问题了)参考了:https://blog.csdn.net/YlanHds/article/details/801...原创 2020-01-09 10:46:38 · 380 阅读 · 0 评论 -
关于xampp集成环境的官网下载
在xampp的官网上选择对应系统的版本进行下载,官网地址:https://www.apachefriends.org/zh_cn/index.html找到这个正在下载我先将其上传至普通用户下。再通过命令移动到root用户下。可以参考linux应用之xampp集成环境的安装及配置(centos)http://www.mamicode.com/i...原创 2020-01-08 15:31:06 · 1945 阅读 · 0 评论