Java调用hdfs出现java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto异常

最新推荐文章于 2024-06-09 10:07:49 发布
最新推荐文章于 2024-06-09 10:07:49 发布
阅读量3.2k 收藏
点赞数
文章标签： hdfs maven
本文介绍了如何解决因protobuf-java版本不一致导致的问题。具体来说，当Hadoop-HDFS依赖于特定版本的protobuf-java（例如2.5.0），而项目中存在其他不同版本时，可能会出现版本冲突的情况。为避免此类问题，需要确保项目依赖中只使用指定版本的protobuf-java。

http://blog.csdn.net/judy1017/article/details/49496213

原因：protobuf-java包版本冲突。hadoop-hdfs包依赖protobuf-java包，且要求protobuf-java包的版本号为2.5.0，因此必须保证该程序pom所有依赖中，没有其他低版本的protobuf-java，否则会有上方的protobuf-java包版本冲突。

江山醉
关注
Hadoop程序中的ClassCastException错误
CodeWWWCode的博客
09-23 620
在上面的示例中，我们定义了一个Mapper类，它将输入文本拆分为单词，并将每个单词作为键，计数为值进行输出。例如，如果您的Mapper输出的键是Text类型，而Reducer期望的键是IntWritable类型，那么在Reducer阶段可能会发生ClassCastException错误。请检查您的程序的键/值类型定义并进行修正。如果您在程序中使用了自定义的数据类型，并且没有正确实现序列化接口，那么在MapReduce过程中进行数据传递时可能会发生ClassCastException错误。
Hadoop 集成 hive 碰到的版本问题
11-14 493
最近在随大流搞下 hadoop， 太多坑了。前辈们估计跟我差不太多。 把今天的一个版本冲突的问题写到博客里，如果碰到一样的问题。不用走弯路吧。 hadoop 2.2.0 与 hive 0.12  集成时碰到这种异常   org.springframework.beans.factory.BeanCreationException: Error creating bean with nam...
Exception in thread “main“ java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.Clien
m0_74043854的博客
01-03 626
Exception in thread "main" java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetListingRequestProto cannot be cast to com.google.protobuf.Message这个报错要怎么解决?
hadoop+javaWeb的开发中遇到包冲突问题(java.lang.VerifyError)
weixin_33781606的博客
09-29 747
  1.HDFS + WEB 项目 报java.lang.VerifyError... 异常 抛异常： Exception in thread "main" java.lang.VerifyError: (class: com/google/common/collect/Interners, method: newWeakInterner signature: ()Lcom/google/c...
StarRocks需要通过HiveCatalog查HDFS-3.2.2的数据报错 StarRocks-3.3.13能正常查询CHDFS的数据，查不了HDFS的数据，会报错ERROR 1064 (HY000): Failed to get remote files, msg: com.starrocks.connector.exception.StarRocksConnectorException: Failed to get hive remote file's metadata on path: RemotePathKey{path='hdfs://HDFS1015025/user/hive/warehouse/tpcds_sf1000_orc_hdfs.db/call_center', isRecursive=true}. msg: org.apache.hadoop.thirdparty.protobuf.ServiceException: java.lang.VerifyError: Bad type on operand stack Exception Details: Location: org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$LocatedBlockProto$Builder.mergeBlockToken(Lorg/apache
最新发布
09-30
`StarRocks-3.3.13` 通过 `HiveCatalog` 查询 `HDFS-3.2.2` 数据时出现 `ERROR 1064 (HY000): Failed to get remote files` 且包含 `java.lang.VerifyError: Bad type on operand stack` 错误，通常是由于字节码验证...
大数据集群各种报错及解决方案
qq_38220334的博客
06-09 898
这个错误信息是Hadoop分布式文件系统（HDFS）中常见的问题，当你尝试在Hive或其他Hadoop应用程序中创建目录时遇到。具体来说，错误原因是NameNode（HDFS的元数据存储和管理器）正处于安全模式（Safe Mode）。安全模式是一种HDFS启动期间或在某些情况下为了维护数据完整性而执行的保护机制。安全模式下，NameNode不会接受新的写入请求，包括创建新的目录或文件，以防止数据损坏。在执行任何操作之前，建议查阅集群的官方文档或联系系统管理员，以确保操作的正确性和安全性。
Hive 源码
zhixingheyi_tian的博客
01-05 885
Hive
hadoop Caused by: java.lang.ClassCastException
qq_42668255的博客
06-29 1682
使用hadoop进行排序时报如下错误：Caused by: java.lang.ClassCastException 原因是因为在你在自己定义的 XXXWritable类中实现的是 Writable 接口，而如果要排序，则需要实现WritableComparable接口 将key作为排序键 再在类中重写如下方法即可： @Override public int compareT...
java调用Hbase API 出现org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Result overrides final 错误
热门推荐
IT_xhf的专栏
07-19 1万+
java.io.IOException: java.lang.reflect.InvocationTargetException     at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:389)     at org.apache.hadoop.hbase
hadoop源码分析系列（四）——org.apache.hadoop.hdfs包之协议篇
luyee2010的专栏
02-28 1457
摘要: hdfs包是hadoop HDFS的主要实现，首先分析下协议包，这个包定义了hdfs在不同节点中的通信协议，对于协议的分析有助于后面的章节对于hdfs服务端、客户端通信的深入理解，按照惯例，首先看一下这个包中几个孤立的类： ... hdfs包是hadoop HDFS的主要实现，首先分析下协议包，这个包定义了hdfs在不同节点中的通信协议，对于协议的分析有助于后面的章节对于hdfs服务
java.lang.ClassCastException: org.apache.hadoop.conf.Configuration cannot be cast to org.apache.hado
Mrknowledge
03-24 2162
Exception in thread "main" java.lang.ClassCastException: org.apache.hadoop.conf.Configuration cannot be cast to org.apache.hadoop.yarn.conf.YarnConfiguration at org.apache.spark.deploy.yarn.Applicat...
Tomcat下整合Spring、CXF报错 Failed to start component [StandardEngine[Tomcat].StandardHost[localhost]]...
weixin_30478923的博客
09-14 367
执行tomcat7:run时出现以下错误： java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Tomcat].StandardHost[localhost].StandardContext[]] a...
java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.FileSplit cannot be cast to...
weixin_43672764的博客
04-29 1800
Error: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.FileSplit cannot be cast to org.apache.hadoop.mapred.FileSplit hadoop 执行报错，类别： java.lang.ClassCastException 说明类的类型转换错误。详细原因是：...
Hadoop Jar 运行HBase类的报错，解决方法
大数据开发
09-30 1826
问题如下： [root@qf01 ~]# hadoop jar testHBase-1.0-SNAPSHOT.jar com.qf.hbase.hbasewordcount.HBaseWordCountApp /tmp/hbase-root/hbase/data/default/wordcount /outdata/hbasewordcount Exception in thread "main...
HBASE 异常 java.lang.ClassCastException: org.apache.hadoop.hbase.client.Put cannot be cast to org.apac
大萝卜博客
12-19 2613
java.lang.ClassCastException: org.apache.hadoop.hbase.client.Put cannot be cast to org.apache.hadoop.hbase.KeyValue 下面我们来看报错信息 WARN mapred.LocalJobRunner: job_local244116549_0001 java.lang.ClassCa...
java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.TaggedInput
qq_25529925的博客
07-21 2523
java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.TaggedInputSplit cannot be cast to org.apache.hadoop.mapreduce.lib.input.FileSplit at org.apache.hadoop.mapred
java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot be cast
weixin_42390253的博客
07-28 3494
错误异常 java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot be cast to org.apache.hadoop.io.ByteWritable java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot be cast to org.apach
hadoop源码解析-HDFS通讯协议(上- ClientProtocol 和 ClientDataNodeProtocol)
胡俊辉的博客
04-12 2413
NameNode 的内存中除了保存文件系统的namespace外，还保存了文件系统中所有的数据块与DataNode节点的对应关系，因此，NameNode 的内存将会成为HDFS的横向扩展的瓶颈，而当数据块过多，也会引起NameNode频繁的GC，影响系统的稳定性。当客户端读取一个HDFS文件时，需要获取这个文件对应的所有数据块的长度，用于简历数据块的输入流，然后进行数据读取，但是NameNode 的元数据中文件最后一个数据块的长度可能与实际的不一致，所以需要调用这个方法获取真实长度。
SQL 错误 [1] [08S01]: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Permission denied: user=anonymous, access=EXECUTE, inode="/tmp":root:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:311) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:238) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:189) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1753) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1737) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1682) at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:64) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1781) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:830) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:469) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2606) at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257) at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=anonymous, access=EXECUTE, inode="/tmp":root:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:311) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:238) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:189) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1753) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1737) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1682) at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:64) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1781) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:830) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:469) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2606) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.setPermission(DFSClient.java:1987) at org.apache.hadoop.hdfs.DistributedFileSystem$35.doCall(DistributedFileSystem.java:1690) at org.apache.hadoop.hdfs.DistributedFileSystem$35.doCall(DistributedFileSystem.java:1687) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.setPermission(DistributedFileSystem.java:1687) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:720) at org.apache.hadoop.mapreduce.JobResourceUploader.mkdirs(JobResourceUploader.java:648) at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:167) at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:128) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:101) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:414) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1232) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:255) ... 11 more Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=anonymous, access=EXECUTE, inode="/tmp":root:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:311) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:238) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:189) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1753) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1737) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1682) at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:64) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1781) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:830) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:469) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2606) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1507) at org.apache.hadoop.ipc.Client.call(Client.java:1453) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy30.setPermission(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setPermission(ClientNamenodeProtocolTranslatorPB.java:365) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy31.setPermission(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.setPermission(DFSClient.java:1985) ... 44 more
06-18
可以临时放宽HDFS上/tmp目录的权限（注意：这可能有安全风险，仅用于测试环境）：```bash#为/tmp目录添加其他用户(o)的执行权限hdfsdfs-chmodo+x/tmp#如果问题在/tmp的子目录（如/tmp/hadoop-yarn），则同样需要修改...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值