Hive3.1.2 on MapReduce,Reduce的时候出错

已解决，参考
https://stackoverflow.com/questions/60811684/nosuchfieldexception-parentoffset-hive-on-spark?r=SearchResults

检查master和worker节点是不是用的都是jdk8，如果其中某个节点版本更高，就会报这个
NoSuchfieldException:parentOffset 的错。
还是外国的月亮圆，国内咋都没这个问题（

原
来
的
问
题
：

用的Hadoop3.3.0 jdk8

记录一下，真的很奇怪，网上根本找不到解决方法

随便写一段sql，

hive> select a.moviename ,(select b.movieid from t_rating b) as movie_id from t_movie a;

运行过程：

Warning: Map Join MAPJOIN[32][bigTable=?] in task 'Stage-5:MAPRED' is a cross product
Warning: Shuffle Join JOIN[17][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product
Warning: Map Join MAPJOIN[33][bigTable=?] in task 'Stage-7:MAPRED' is a cross product
Query ID = hadoop_20200902124158_ac107116-4a4e-41a0-a773-0da5e06f711b
Total jobs = 4
Launching Job 1 out of 4
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1599017699586_0003, Tracking URL = http://master:8088/proxy/application_1599017699586_0003/
Kill Command = /home/hadoop/app/hadoop/bin/mapred job  -kill job_1599017699586_0003
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
2020-09-02 12:42:11,737 Stage-3 map = 0%,  reduce = 0%
2020-09-02 12:42:40,880 Stage-3 map = 100%,  reduce = 100%
Ended Job = job_1599017699586_0003 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1599017699586_0003_m_000000 (and more) from job job_1599017699586_0003

Task with the most failures(4): 
-----
Task ID:
  task_1599017699586_0003_m_000000

URL:
  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1599017699586_0003&tipid=task_1599017699586_0003_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.<init>(SerializationUtilities.java:388)
	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:234)
	at org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:51)
	at org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:278)
	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:413)
	at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:335)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:435)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:881)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:874)
	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:716)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:175)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:444)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:425)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
Caused by: java.lang.NoSuchFieldException: parentOffset
	at java.base/java.lang.Class.getDeclaredField(Class.java:2489)
	at org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.<init>(SerializationUtilities.java:382)
	... 17 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-3: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

反编译找到对应的代码，就是这段：

class SerializationUtilities$ArrayListSubListSerializer
  extends Serializer<List<?>>
{
  private Field _parentField;
  private Field _parentOffsetField;
  private Field _sizeField;
  
  public SerializationUtilities$ArrayListSubListSerializer()
  {
    try
    {
      Class<?> clazz = Class.forName("java.util.ArrayList$SubList");
      this._parentField = clazz.getDeclaredField("parent");
      this._parentOffsetField = clazz.getDeclaredField("parentOffset");
      this._sizeField = clazz.getDeclaredField("size");
      this._parentField.setAccessible(true);
      this._parentOffsetField.setAccessible(true);
      this._sizeField.setAccessible(true);
    }
    catch (Exception e)
    {
      throw new RuntimeException(e);
    }
  }
  ...
  

报反射错误，说这个找不到。
java.lang.NoSuchFieldException: parentOffset
我寻思怎么找不到呢，这个东西干嘛使得？？
日后再思考吧