Unrecognized Hadoop major version number: 3.0.0-cdh6.3.2

于 2023-08-18 10:24:58 发布
阅读量916 收藏
点赞数
分类专栏： 大数据 文章标签： spark 大数据 分布式
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/anguoan/article/details/132356102
版权

 一.环境描述

spark提交job到yarn报错，业务代码比较简单，通过接口调用获取数据，将数据通过sparksql将数据写入hive中，尝试各种替换hadoop版本，最后拿下

1.hadoop环境

2.项目 pom.xml

spark-submit \
--name GridCorrelationMain \
--master yarn \
--deploy-mode cluster \
--executor-cores 2 \
--executor-memory 4G \
--num-executors 5 \
--driver-memory 2G \
--class cn.zd.maincode.wangge.GridCorrelationMain \
/home/boeadm/zwj/iot/cp-etl-spark-data/target/cp_zhengda_spark_utils-1.0-SNAPSHOT.jar




eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2OTI0MzU5NjgsImlhdCI6MTY5MjM0OTU2Mywic3ViIjo1MjB9.rCmnhF2EhdzH62T7lP3nmxQSxh17PotscxEcZkjL5hk



    <dependencies>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-configuration2</artifactId>
            <version>2.9.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.3.3</version>
            <exclusions>
                <exclusion>
                    <artifactId>hadoop-client</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>

        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.3.3</version>
            <!--<scope>provided</scope>-->
           <!-- <exclusions>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
            </exclusions>-->
        </dependency>

<!--
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>15.0</version>
        </dependency>
-->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>commons-codec</groupId>
                    <artifactId>commons-codec</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>commons-httpclient</groupId>
                    <artifactId>commons-httpclient</artifactId>
                </exclusion>

                <!--          <exclusion>
                              <groupId>com.google.guava</groupId>
                              <artifactId>guava</artifactId>
                          </exclusion>-->

            </exclusions>
            <!--<scope>provided</scope>-->
        </dependency>


        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>2.3.2</version>
            <exclusions>
                <exclusion>
                    <artifactId>hive-exec</artifactId>
                    <groupId>org.spark-project.hive</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hive-metastore</artifactId>
                    <groupId>org.spark-project.hive</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>${hadoop.version}</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.eclipse.jetty.aggregate</groupId>
                    <artifactId>jetty-all</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-shims</artifactId>
                </exclusion>
                <exclusion>
                    <artifactId>hbase-mapreduce</artifactId>
                    <groupId>org.apache.hbase</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hbase-server</artifactId>
                    <groupId>org.apache.hbase</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>log4j-slf4j-impl</artifactId>
                    <groupId>org.apache.logging.log4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
            <version>2.1.1</version>
        </dependency>




        <!--服务验证相关依赖-->
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.13</version>
             <exclusions>
                <exclusion>
                    <groupId>commons-codec</groupId>
                    <artifactId>commons-codec</artifactId>
                </exclusion>
            </exclusions>
            <!--<scope>provided</scope>-->
        </dependency>

        <!--本地跑的话 需要这个jar-->
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.15</version>
            <!--<scope>provided</scope>-->
        </dependency>






        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.3.1</version>

        </dependency>

        <!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.62</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>${fastjson.version}</version>
        </dependency>



        <!-- https://mvnrepository.com/artifact/org.json/json -->
        <dependency>
            <groupId>org.json</groupId>
            <artifactId>json</artifactId>
            <version>20160810</version>
        </dependency>

        <dependency>
            <groupId>com.github.qlone</groupId>
            <artifactId>retrofit-crawler</artifactId>
            <version>1.0.0</version>
        </dependency>


        <dependency>
            <groupId>com.oracle.database.jdbc</groupId>
            <artifactId>ojdbc8</artifactId>
            <version>12.2.0.1</version>
        </dependency>

        <!--mysql连接-->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.40</version>

        </dependency>

        <dependency>
            <groupId>javax.mail</groupId>
            <artifactId>javax.mail-api</artifactId>
            <version>1.5.6</version>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-email</artifactId>
            <version>1.4</version>
        </dependency>

    </dependencies>

3.项目集群提交报错


        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:696)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:730)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:685)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:715)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:708)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:89)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:86)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.resolveOperatorsUp(AnalysisHelper.scala:86)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$1.apply(AnalysisHelper.scala:87)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$1.apply(AnalysisHelper.scala:87)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:87)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:86)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.resolveOperatorsUp(AnalysisHelper.scala:86)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$1.apply(AnalysisHelper.scala:87)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$1.apply(AnalysisHelper.scala:87)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:87)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:86)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.resolveOperatorsUp(AnalysisHelper.scala:86)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:708)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:654)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
        at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:127)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:121)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:106)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:651)
        at cn.zd.maincode.wangge.GridCorrelationMain$.createDataFrameAndTempView(GridCorrelationMain.scala:264)
        at cn.zd.maincode.wangge.GridCorrelationMain$.horecaGridInfo(GridCorrelationMain.scala:148)
        at cn.zd.maincode.wangge.GridCorrelationMain$.main(GridCorrelationMain.scala:110)
        at cn.zd.maincode.wangge.GridCorrelationMain.main(GridCorrelationMain.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
Caused by: java.lang.ExceptionInInitializerError
        at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
        at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:153)
        at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:118)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:292)
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:395)
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:284)
        at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:68)
        at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:67)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:217)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:217)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:217)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
        ... 72 more
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-cdh6.3.2
        at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:169)
        at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:134)
        at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:95)
        at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:354)
        ... 88 more

End of LogType:stderr

4.最终解决方式

 将相关依赖不打进包中

   <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.eclipse.jetty.aggregate</groupId>
                    <artifactId>jetty-all</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-shims</artifactId>
                </exclusion>
                <exclusion>
                    <artifactId>hbase-mapreduce</artifactId>
                    <groupId>org.apache.hbase</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hbase-server</artifactId>
                    <groupId>org.apache.hbase</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>log4j-slf4j-impl</artifactId>
                    <groupId>org.apache.logging.log4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
            </exclusions>
            <version>2.1.1</version>
        </dependency>




        <!--服务验证相关依赖-->
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.13</version>
             <exclusions>
                <exclusion>
                    <groupId>commons-codec</groupId>
                    <artifactId>commons-codec</artifactId>
                </exclusion>
            </exclusions>
            <!--<scope>provided</scope>-->
        </dependency>

        <!--本地跑的话 需要这个jar-->
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.15</version>
            <!--<scope>provided</scope>-->
        </dependency>






        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.3.1</version>

        </dependency>

        <!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.62</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>${fastjson.version}</version>
        </dependency>



        <!-- https://mvnrepository.com/artifact/org.json/json -->
        <dependency>
            <groupId>org.json</groupId>
            <artifactId>json</artifactId>
            <version>20160810</version>
        </dependency>

        <dependency>
            <groupId>com.github.qlone</groupId>
            <artifactId>retrofit-crawler</artifactId>
            <version>1.0.0</version>
        </dependency>


        <dependency>
            <groupId>com.oracle.database.jdbc</groupId>
            <artifactId>ojdbc8</artifactId>
            <version>12.2.0.1</version>
        </dependency>

        <!--mysql连接-->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.40</version>

        </dependency>

<!--10月31日 新取消-->
<!--        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>28.0-jre</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/org.apache.directory.studio/org.apache.commons.codec -->


        <!-- https://mvnrepository.com/artifact/org.apache.commons/org.apache.commons.codec -->


        <!--邮件发送依赖-->

        <dependency>
            <groupId>javax.mail</groupId>
            <artifactId>javax.mail-api</artifactId>
            <version>1.5.6</version>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-email</artifactId>
            <version>1.4</version>
        </dependency>



        <!--
                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                    <version>2.11.2</version>
                </dependency>


                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-reflect</artifactId>
                    <version>2.11.2</version>
                </dependency>


                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-compiler</artifactId>
                    <version>2.11.2</version>
                </dependency>-->



<!--        <dependency>-->
<!--            <groupId>com.starrocks</groupId>-->
<!--            <artifactId>starrocks-spark2_2.11</artifactId>-->
<!--            <version>1.0.1</version>-->
<!--        </dependency>-->


    </dependencies>

远方有海，小样不乖
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
【hadoop-3.0.0版本】winutils.exe和hadoop.dll
01-22
hadoop3.0.0 windows平台需要的文件 .
解决python脚本中error: unrecognized arguments: True错误
09-17
主要介绍了解决python脚本中error: unrecognized arguments: True错误，具有很好的参考价值，希望对大家有所帮助。一起跟随小编过来看看吧
解决maven仓库默认不支持cdh JAVAJDBC连接hive
我就是隔壁老袁
06-28 6918
前面的话： 目前maven仓库中没有支持cdh的相关依赖。cloudera自己建立了一个相关的仓库。要想利用maven添加相关依赖，则必须单独添加cloudera仓库。一、项目pom.xml, 添加仓库配置 &lt;repositories&gt; &lt;repository&gt; &lt;id&gt;cloudera&lt;/id&gt; &lt;url&g...
spark写hive报错 （Unrecognized Hadoop major version number: 3.0.0-cdh6.2.1）
TaiYangb的博客
11-26 3515
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-cdh6.2.1 业务场景：spark 抽取ck数据进Hive 结论：CDH 自带的hive 版本和hadoop支持的hive 版本对应不上 导致报错 办法： 1：去掉pom文件中的hive 依赖(去掉依赖的话 本地执行会报错 但是线上不会报错) <!-- <dependency>--&
Unrecognized Hadoop major version number: 3.0.0
小爷欣欣
03-27 9976
出现这种情况，别费劲想怎么解决了。直接删掉原先的hadoop，安装cdh版本。hadoop hive hbase都安装成cdh版本的
基于3.0.0-cdh6.3.2版本编译Flink1.14.4
永青技术博客
05-25 1194
java.lang.Runtimelxception:java,lang.IllegalArgumentException:UnrecoonizedHadoopmajorversionnumber:3.0.0-cdh6.2.1
spark 集群启动出现 Unrecognized Hadoop major version number: 3.0.0-cdh6.3.2
前方的路在刚开始
08-06 1621
spark使用hive中有比较多的坑，尤其是版本问题引起的jar包冲突，比较好的方式是使用与CDH匹配的hive和hadoop版本，这样可以减少很多的jar冲突问题，但是在IDEA调试过程中还是难免会碰到jar包冲突问题。...
Unrecognized Hadoop major version number: 3.0.0 版本冲突解决
m0_49945160的博客
07-05 5299
hive与hadoop的兼容性问题 1. 问题出现 工作需要，在本地安装hive进行相关测试工作，版本选的是hadoop-3.0.0，hive-2.1.1。这个版本是按照CDH集群版本配的，本来以为没有冲突 #执行初始化hive元数据库时报错 schematool -initSchema -dbType mysql #报错信息 Exception in thread "main" java.lang.IllegalArgumentException: Unrecognized Hadoop major
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0
zhu_zhu_zhu_zhu_zhu的博客
09-17 5087
Spark本地使用IntelliJ IDEA连接Hive关于“Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0”的报错解决 对于这种报错，可能的一种原因是由于pom.xml文件中的hadoop版本与本地环境中的hadoop版本不同造成。 这种报错原因的解决方案：将pom.xml文件中的hadoop版本与本地环境中的hadoop版本统一即可。 ...
Unrecognized Hadoop major version number:3.0.0报错解决办法
qq_41387313的博客
03-29 6401
如文字标题， 这个问题困扰了我三天，最终还是排查解决了。 这个是运用spark.env.sh里面的hadoop_home_dir路径指向空然后就好了。 原创不易，转发把我文贴上。 ...
Pytest使用allure提示unrecognized arguments
12-21
unrecognized arguments: –alluredir=reports 在pycharm的虚拟环境汇总安装好pytest，allure-pytest后， 命令行执行pytest “test.py” –alluredir=report 报错如下: 查找问题原因 直接通过错误信息看，是因为...
certbot-dns-aliyun:certbot dns插件，可使用阿里云获取证书
05-04
适用于Certbot的Aliyun DNS Authenticator插件certbot dns插件，用于使用阿里云获取证书。获取阿里云RAM AccessKey 并确保您的RAM帐户具有AliyunDNSFullAccess权限。安装pip install certbot-dns-aliyun 对于快照： ...
certbot-dns-aliyun：阿里云DNS的certbot插件，使用解决阿里云DNS不能自动为通配符证书续期的问题
03-03
certbot-dns-aliyun解决阿里云DNS不能自动为通配符证书续期的问题原理每个certbot申请的证书有效期为3个月，虽然certbot提供了贴心的自动续期命令，但是当我们把自己续期命令配置为定时时，我们需要手动添加TXT记录...
lv_font_conv:将TTFWOFF字体转换为紧凑的位图格式
05-01
lv_font_conv-字体转换器转换为紧凑位图格式 将TTF / WOFF / OTF字体转换为，适用于小型嵌入式系统。 主要特点是： 允许双色调和抗锯齿字形（每个像素1-4位）。 保留字距调整信息。 压缩。 ...
cdh6.3.2 spark Unrecognized Hadoop major version number: 3.0.0-cdh6.3.2报错问题
qq_34444616的博客
04-16 2237
cdh6.3.2 spark Unrecognized Hadoop major version number: 3.0.0-cdh6.3.2报错问题将scala spark提交到yarn环境报错解决办法 将scala spark提交到yarn环境报错 找了半天没有找到是为什么通过报错信息找到报错的类ShimLoader找到报错的行数。 从这里看到他去拿了一个version-info.properties的文件 在第一个类里面它去case了 值这里对应的是3（3.0.0-cdh6.3.2）所以报错了
MapReduce任务报错：Unrecognized Hadoop major version number: 3.0.0-cdh6.2.1
u013771019的专栏
11-29 1万+
问题描述： 之前写了一个读取RCFile的MapReduce任务，现在放到测试集群上报错了 错误信息： Error: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-cdh6.2.1 at org.apache.hadoop.hive.shims.ShimL...
搭建Hive on Spark on YARN常见问题及解决方案
逝无痕——kaidy
03-02 1万+
文章目录1、在Hive cli中往表中插入记录，报错信息如下错误信息a错误信息b 1、在Hive cli中往表中插入记录，报错信息如下 错误信息a Unrecognized Hadoop major version number: 3.2.0 当时环境版本信息： hadoop版本：3.2.0 spark版本：2.4.0 hive版本：3.1.1 解决方案： 版本兼容性问题，通过查看hive源码根...
解决 dependency org.apache.hadoop:hadoop-client:2.6.0-cdh5.15.1 not found
热门推荐
qq_21277357的博客
09-15 1万+
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.6.0-cdh5.5.1</version> </depen...
error: unrecognized arguments: --permanentsudo firewall-cmd
最新发布
06-03
非常抱歉，我的回答中有一个小错误。正确的命令应该是： ``` sudo firewall-cmd --zone=public --add-port=3000/tcp --permanent sudo firewall-cmd --reload ``` 其中，`--permanent` 是用来将规则永久保存的参数，而不是 `--permanentsudo`。这个错误可能是因为我在回答时输入有误。 再次抱歉给您带来困惑，希望我的纠正能够帮到您。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

远方有海，小样不乖

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值