未解决问题2_Unknown rpc kind RPC_WRITABLE

在CLI中使用hive 执行引起起 MapReduce 任务的查询时报错，因为还么解决，贴出完整异常：

hive> select count(*) from table;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unknown rpc kind RPC_WRITABLE
	at org.apache.hadoop.ipc.Client.call(Client.java:1225)
	at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
	at org.apache.hadoop.mapred.$Proxy16.getStagingAreaDir(Unknown Source)
	at org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1324)
	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:919)
	at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:448)
	at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1374)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1160)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:973)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:893)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(Unknown rpc kind RPC_WRITABLE)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

就代码本身来看是因为 客户端与服务端RPC通信时使用了不一致的RPC类型。

客户端使用了Hadoop 最开始用的 Writable,而Server端使用了Protobuffer  。

Hadoop中的RPC协议有三种：

 public enum RpcKind {
    RPC_BUILTIN ((short) 1),         // Used for built in calls by tests
    RPC_WRITABLE ((short) 2),        // Use WritableRpcEngine 
    RPC_PROTOCOL_BUFFER ((short) 3); // Use ProtobufRpcEngine
    final static short MAX_INDEX = RPC_PROTOCOL_BUFFER.value; // used for array size
    private static final short FIRST_INDEX = RPC_BUILTIN.value;    
    public final short value; //TODO make it private

    RpcKind(short val) {
      this.value = val;
    } 

实际使用的就后面两种。

上面的通信具体使用了 ClientProtocol 协议，它是hadoop ipc 协议，根本就不是YARNRPC 的协议，如AMRMProtocol等。

从执行过程来看并不是因为客户端和服务端使用了不一致的依赖的问题。

所以呢，估计是配置的问题。

可能靠谱点的分析：http://www.marshut.com/mkvtr/hadoop-mapreduce-examples-not-working-when-using-yarn.pdf