大数据学习15：报错 FAILED: SemanticException Unable to determine if

最新推荐文章于 2024-01-10 11:46:34 发布

月魂_123123

最新推荐文章于 2024-01-10 11:46:34 发布

阅读量7.7k

点赞数

分类专栏：大数据文章标签：大数据 hive

本文链接：https://blog.csdn.net/ggwxk1990/article/details/78067803

版权

大数据学习中遇到Hive报错FAILED: SemanticException，原因是修改了HDFS端口后，Hive无法确定文件加密状态。解决方案在于更新MySQL中的元数据，通过REPLACE函数替换原有HDFS端口为新的端口，确保元数据与配置一致。

摘要由CSDN通过智能技术生成

大数据学习15：报错 FAILED: SemanticException Unable to determine if

报错：由于修改了hdfs 的端口，导致报错，这里是因为使用UDF自定义函数，hive要将hfds上的jar传入分布式缓存中，并且mysql中元数据未修改导致。
hive (default)> select ename ,hello(ename) from emp;
converting to local hdfs://hadoop002:8020/lib/hive_train-1.0.jar
Added [/tmp/03626a44-b1f4-4a3a-b60d-676754ec373c_resources/hive_train-1.0.jar] to class path
Added resources: [hdfs://hadoop002:8020/lib/hive_train-1.0.jar]
FAILED: SemanticException Unable to determine if hdfs://hadoop002:9000/user/hive/warehouse/emp is encrypted: java.lang.IllegalArgumentException: Wrong FS: hdfs://hadoop002:9000/user/hive/warehouse/emp, expected: hdfs://hadoop002:8020

解决办法：
原因，是mysql中dbs 和sds 有 hive 的元数据信息，修改了hdfs的端口，吗么也需要去mysql中也进行修改。
update 表名 set 字段名=REPLACE (字段名,’原来的值’,’要修改的值’)
范例：

update DBS set DB_LOCATION_URI=REPLACE (DB_LOCATION_URI,'bdc240.hexun.com:8020','nameservice1');  
update SDS set LOCATION=REPLACE (LOCATION,'bdc240.hexun.com:8020','nameservice1');

实际操作：

update DBS set DB_LOCATION_URI=REPLACE (DB_LOCATION_URI,'hadoop002:9000','hadoop002:8020');
update SDS set LOCATION=REPLACE (LOCATION,'hadoop002:9000','hadoop002:8020');

结果
输入前：

mysql> select * from dbs;
+-------+-----------------------+--------------------------------------------------+---------+------------+------------+
| DB_ID | DESC                  | DB_LOCATION_URI                                  | NAME    | OWNER_NAME | OWNER_TYPE |
+-------+-----------------------+--------------------------------------------------+---------+------------+------------+
|     1 | Default Hive database | hdfs://hadoop002:9000/user/hive/warehouse        | default | public     | ROLE       |
|     6 | NULL                  | hdfs://hadoop002:9000/user/hive/warehouse/wxk.db | wxk     | root       | USER       |
+-------+-----------------------+--------------------------------------------------+---------+------------+------------+
mysql> select * from sds;
+-------+-------+------------------------------------------+---------------+---------------------------+-------------------------------------------------------+-------------+------------------------------------------------------------+----------+
| SD_ID | CD_ID | INPUT_FORMAT                             | IS_COMPRESSED | IS_STOREDASSUBDIRECTORIES | LOCATION                                              | NUM_BUCKETS | OUTPUT_FORMAT                                              | SERDE_ID |
+-------+-------+------------------------------------------+---------------+---------------------------+-------------------------------------------------------+-------------+------------------------------------------------------------+----------+
|    10 |    10 | org.apache