bigdata_异常处理
乔治大哥
脚踏实地,仰望星空。
展开
-
Exception in thread “main“ java.lang.NoSuchMethodError: scala.collection.mut
scala版本不一致,采用maven中spark自带版本即可。原创 2022-02-15 17:38:10 · 1814 阅读 · 0 评论 -
win10下cmd ping不通虚拟机,xshell连不上虚拟机【如果实在想不出来原因,就看看吧】
请关掉在家办公的你的VPN原创 2020-04-22 22:38:15 · 969 阅读 · 2 评论 -
VMware Workstation虚拟机“”繁忙——解决方案
帮小老弟解决问题找了一阵错误,看到网上有朋友说是兼容性的问题,win10呀。。。然后解决方案:1、进入vm的安装目录下2、兼容性疑难解答...原创 2020-02-18 21:53:10 · 5621 阅读 · 1 评论 -
Unsupported Hardware Detected
意思呢?就是,不支持的硬件检测,可能因为新电脑的某些内部配置问题,但是我直接点了OK,并没什么问题,正常启动引导!有理解透彻的,可以留言说一下!...原创 2020-02-16 17:02:59 · 2253 阅读 · 0 评论 -
kibana启动出错 error [warning][stats-collection] [index_not_found_exception] no such index [.kibana], wi
error [12:25:22.371] [warning][stats-collection] [index_not_found_exception] no such index [.kibana], with { resource.type="index_or_alias" & resource.id=".kibana" & index_uuid="_na_" & ...原创 2020-02-12 21:02:52 · 6599 阅读 · 1 评论 -
hive运行任务org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error parsing application ID: applic
HTTP ERROR 500Problem accessing /proxy/applicat%20ion_1575880064293_0002/. Reason: Error parsing application ID: applicat ion_1575880064293_0002Caused by:org.apache.hadoop.yarn.except...原创 2019-12-16 14:19:28 · 961 阅读 · 0 评论 -
SparkStreaming Exception in thread "main" java.lang.IllegalArgumentException xxx is not a valid
Exception in thread "main" java.lang.IllegalArgumentException xxx is not a valid DFS filename.原因分析:代码无法识别文件的所在处,是win还是linux由于我在项目里的resources下放了hadoop的配置文件。core hdfs 并且系统高可用。我这边的解决方案是:将...原创 2019-12-12 16:57:28 · 987 阅读 · 0 评论 -
java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing
java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute解决方法:tranformation最后一步产生的那个RDD必须有相应Action操作,例如massages.print()等...原创 2019-12-12 16:49:54 · 609 阅读 · 0 评论 -
高可用的hadoop集群下,idea读取hive文件,Exception in thread "main" java.lang.IllegalArgumentException: java.net.U
Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) at o...原创 2019-12-09 14:35:52 · 874 阅读 · 0 评论 -
Hive创表异常,FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:hdfs://mycluster/employees is not a directory or unable to create one)我这边的原因是因为我的上传表文件,此...原创 2019-12-09 14:20:00 · 823 阅读 · 0 评论 -
解决创建maven项目后,不能创建scala
问题:解决:右键项目ok!!原创 2019-12-09 10:56:11 · 395 阅读 · 0 评论 -
Spark SQL读数据库时不支持某些数据类型的问题(Timestamp with local Timezone)
在大数据平台中,经常需要做数据的ETL,从传统关系型数据库RDBMS中抽取数据到HDFS中。之前开发数据湖新版本时使用Spark SQL来完成ETL的工作,但是遇到了 Spark SQL不支持某些数据类型(比如ORACLE中的Timestamp with local Timezone)的问题。一、系统环境 Spark 版本:2.1.0.cloudera1 JDK 版本:Jav...原创 2019-11-28 19:27:35 · 1543 阅读 · 0 评论 -
Kafka的常见错误分析
1、UnknownTopicOrPartitionExceptionorg.apache.kafka.common.errors.UnknownTopicOrPartitionException:This server does not host this topic-partition报错内容:分区数据不在原因分析:producer向不存在的topic发送消息,用户可以检查to...原创 2019-11-24 21:37:11 · 8861 阅读 · 0 评论 -
Kafka删除topic Note: This will have no impact if delete.topic.enable is not【另外强烈推荐一个kafka小工具】
vi /opt/kafka/config/server.properties需要server.properties中设置delete.topic.enable=true否则只是标记删除或者直接重启。另外推荐小工具:Kafka Tool查阅,添加,删除,等等才做均可...原创 2019-11-21 16:00:50 · 4398 阅读 · 0 评论 -
Hive提交任务FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
Query ID = root_20191114010024_e70138f9-544a-4cd7-86a1-58fd1054cd6aTotal jobs = 2Launching Job 1 out of 2Number of reduce tasks not specified. Estimated from input data size: 1In order to change ...原创 2019-11-20 20:17:35 · 1527 阅读 · 2 评论 -
Error: Could not open client transport with JDBC Uri: jdbc:hive2://henu2:10000: java.n et.ConnectExc
Error: Could not open client transport with JDBC Uri: jdbc:hive2://henu2:10000: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)启动beeline时,报错,查看hiveserver2是否启...原创 2019-11-16 10:47:42 · 2275 阅读 · 0 评论 -
启动hiveException in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Una ble to
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.se...原创 2019-11-16 10:22:07 · 1619 阅读 · 0 评论 -
启动spark集群,JAVA_HOME is not set
就睡了个觉,他就死了。。。JAVA_HOME is not set嗯那就在配置文件中加上。[root@henu2 sbin]# vi /opt/spark/sbin/spark-config.sh再次运行:解决!!!...原创 2019-11-16 09:44:37 · 254 阅读 · 0 评论 -
idea提交spark任务,内存不足,指定JVM内存的解决方法
-server -Xms512M -Xmx1024M -XX:PermSize=256M -XX:MaxNewSize=512M -XX:MaxPermSize=512M在run即可!!!!!!成功!!!原创 2019-11-15 21:29:17 · 1927 阅读 · 0 评论 -
hadoop高可用集群下,然后启动spark集群,提交任务时读取hdfs失败。Caused by: java.net.UnknownHostException: mycluster
java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) at org.apache.hadoop.hdfs.NameNodeP...原创 2019-11-15 15:52:29 · 3044 阅读 · 0 评论 -
-bash: jps: command not found
使用rpm安装java后,直接输入java命令,成功但是输入jps,却出现-bash: jps: command not found我还以为安装rpm不需要配置环境变量呢。我错了。。。export JAVA_HOME=/usr/java/jdk1.7.0_67PATH=$PATH:$JAVA_HOME/bin解决!!!...原创 2019-11-13 08:16:39 · 3875 阅读 · 0 评论 -
hive提交任务Ended Job = job_1573218357825_0009 with errors Error during job, obtaining debugging inform
Query ID = root_20191109201329_35facb5a-75c2-4ab9-b493-abc369c92cc6Total jobs = 1Launching Job 1 out of 1Number of reduce tasks determined at compile time: 1In order to change the average load...原创 2019-11-09 21:57:26 · 5234 阅读 · 4 评论 -
在hive的beeline下,Error: org.apache.thrift.transport.TTransportException: java.net.SocketException: Bro
Error: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)解决方案:、、、看看是不是自己的hiveserver2掉了。。。...原创 2019-11-09 17:18:25 · 4055 阅读 · 0 评论 -
kafka启动后会挂掉的原因
在三台节点上看:cd /opt/kafka/logs/more meta.properties看在meta.properties中的broker.id和你在/opt/kafka/config/server.properties中的broker.id是否一致。henu4 节点上的 /opt/kafka/config/server.properties 配置 其他就不说...原创 2019-11-08 20:33:30 · 4597 阅读 · 2 评论 -
kafka命令行操作 socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
WARN Session 0x0 for server null, unexpected error, closingsocket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)java.net.ConnectException: Connection refused at sun.nio.c...原创 2019-11-08 19:43:20 · 3090 阅读 · 0 评论 -
启动Kafka失败Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 107
报错信息:Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000,1073741824, 0) failed; error='Cannot allocate memory' (errno=12)## There is insufficient memory for the...原创 2019-11-08 18:57:04 · 863 阅读 · 0 评论 -
SparkShell中提交任务java.net.ConnectException: Call From henu4/192.168.248.244 to henu2:9000 failed on co
java.net.ConnectException: Call From henu4/192.168.248.244 to henu2:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/h...原创 2019-11-06 21:04:58 · 876 阅读 · 0 评论 -
Hive belline提交命令Error: org.apache.thrift.transport.TTransportException: java.net.SocketException:
Error: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0)检查hiveserver2的服务是不是被终止了,重新启动hiveserver2并重新连接hive belline即可:...原创 2019-11-05 19:11:45 · 13165 阅读 · 0 评论 -
hive提交命令org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error parsing application ID:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error parsing application ID: 怎么说?我说我重新提交一次行了,你信不信?我的分析:因为在之前的hive中我设置了reducetask的数量,再根据其yarn的错误,我认为是资源调度方面的问题。本来想在提一下内存,然后嗯,重新进...原创 2019-11-05 16:39:20 · 1968 阅读 · 0 评论 -
-bash: xxx: command not found
找到命令所在的文件,大概率在于命令没有在我们所配置的path中,所以将其配置,或许使用ln创建链接原创 2019-11-04 13:12:46 · 518 阅读 · 0 评论 -
hive启动mapreduce任务后,被killed
如果本机搭建集群,当启动过多节点时,难免占用你所运行的内存空间,与此同时,当你在启动mapreduce任务时,运行中的任务虚拟机内存分配不足,则终止任务。所以右键虚拟机设置:【将内存设置大点】再次启动hive命令。即可。...原创 2019-10-31 15:36:18 · 2684 阅读 · 0 评论 -
hive 运行命令后FAILED: Execution Error, return code 137 from org.apache.hadoop.hive.ql.ex ec.mr.MapredLoc
FAILED: Execution Error, return code 137 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTaskvi hive-site.xml <property> <name >mapred.child.java.opts</name> ...原创 2019-10-31 14:27:36 · 1660 阅读 · 0 评论 -
hive 启动 报错Exception in thread "main" java.lang.IncompatibleClassChangeError: Found c interface was e
[ERROR] Terminal initialization failed; falling back to unsupportedjava.lang.IncompatibleClassChangeError: Found class jline.Terminal, but in at jline.TerminalFactory.create(TerminalFactory.java:...原创 2019-10-30 22:09:29 · 1780 阅读 · 0 评论 -
Hive更改表名FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Una ble
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. Unable to access old location hdfs://mycluster/user/sjk/xx for table db1.xx【注】注意:重命...原创 2019-10-30 19:16:58 · 5422 阅读 · 0 评论 -
hive插入数据:FAILED: ParseException line 1:12 missing TABLE at 'student' near '<EOF>'
修改:hive> insert into table student values(100,"george4"); 运行过程:Query ID = root_20191030202828_a1df1d18-762e-4fec-8f02-2ab18dde6810Total jobs = 3Launching Job 1 out of 3Number of reduc...原创 2019-10-30 13:42:50 · 9814 阅读 · 1 评论 -
安装hive阶段,Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException;
Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/opt/hive/conf/hive-site.xml; lineNumber: 2; columnNumber: 6; The processing instruction target ma...原创 2019-10-30 13:34:48 · 6172 阅读 · 0 评论 -
hadoop HA启动时 两个namenode节点都是standby,解决办法
记住先启动你的Zookeeper,因为配置ha 需要zookeeper。zkServer.sh start其他的就是配置问题了,原创 2019-10-28 16:43:33 · 1656 阅读 · 0 评论 -
Error: java.lang.RuntimeException: java.lang.NoSuchMethodException: com.henu.WC $WCMapper.<init>()
运行MapReduce jar包时的错误。解决方法:”原创 2019-10-28 16:41:18 · 777 阅读 · 1 评论 -
Spark启动集群——JAVA_HOME is not set
starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-henu1.outhenu3: starting org.apache.spark.deploy.worker.Worker, logging ...原创 2019-10-20 20:27:05 · 1219 阅读 · 0 评论 -
异常处理——namenode启动成功但是没有namenode进程
别乱找,记住先看日志:你会问日志在哪?我会不说?然后根据你的错误去百度。。。。。。。如果是这个错误,嗯重新格式化就好了。2019-09-29 19:24:30,527 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem:Encountered exception loading fsimageorg.apache...原创 2019-09-29 13:23:20 · 2633 阅读 · 4 评论