iceberg集成hive，insert失败问题排查与解决

背景：

创建iceberg表成功

CREATE TABLE iceberg_test1 (i int)
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler';

insert数据失败

INSERT INTO iceberg_test1 values(1);

hive的hive.execution.engine为mr

报错：

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-2: Map: 1   Cumulative CPU: 10.77 sec   HDFS Read: 170682 HDFS Write: 2654 FAIL
Total MapReduce CPU Time Spent: 10 seconds 770 msec

查看hive.log无异常
查看hadoop log，resourcemanager的日志：

org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore\n	
at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:84)\n	
at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:34)\n	
at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:125)\n	
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:56)\n	
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)\n	
at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122)\n	
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:158)\n	
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97)\n	
at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:80)\n	
at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:47)\n	at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:124)\n	
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:111)\n	
at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.commitTable(HiveIcebergOutputCommitter.java:320)\n	
at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.lambda$commitJob$2(HiveIcebergOutputCommitter.java:214)\n
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)\n	
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)\n	
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)\n	
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)\n	
at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.commitJob(HiveIcebergOutputCommitter.java:207)\n	
at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)\n	
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:286)\n	
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:238)\n	
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n	
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n	
at java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface\n	at java.lang.ClassLoader.defineClass1(Native Method)\n	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)\n	
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)\n	
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)\n	
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)\n	
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)\n	
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)\n	
at java.security.AccessController.doPrivileged(Native Method)\n	
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)\n	
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n	
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)\n	
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n	
at java.lang.Class.forName0(Native Method)\n	at java.lang.Class.forName(Class.java:348)\n	at org.apache.hadoop.hive.metastore.utils.JavaUtils.getClass(JavaUtils.java:52)\n	
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:146)\n	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)\n	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:112)\n	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n	
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n	
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n	
at java.lang.reflect.Method.invoke(Method.java:498)\n	
at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:60)\n	
at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:72)\n	
at org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:185)\n	
at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:63)\n	... 24 more\nCaused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface\n	
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)\n	
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n	
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)\n	
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n	... 50 more\n\n,totalAllocatedContainers=2

其中

at java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface\n 

格外刺眼

想起来在部署hive的时候在lib中加了libfb303-0.9.3.jar，那么 是不是在hadoop中也要加上

解决：

随便找了个hadoop的lib文件夹 ～/hadoop-3.2.3/share/hadoop/common/lib，把libfb303-0.9.3.jar加进去

ps.前提是在mapred-site.xml的mapreduce.application.classpath配置过相关路径。

再次执行，成功