hive datanucleus cache 不一致问题

现象：

最近有用户反应hive里面对某张表修改了几个字段名后，在另外的窗口中还是看到原字段名，比较奇怪。

我一开始想到可能是cache的问题，由于我们启用了两个metastore server，而用户客户端是随机选择一个建立链接的，所以有可能是修改表结构在第一个server中，而查询在第二个server，各自都有datanucleus cache所以导致不一致，我简单模拟了场景

模拟场景：

hive client 1:

1. set hive.metastore.local=false;

2. set hive.metastore.uris=thrift://10.x.x.1:9083;

3. create table test(aaa string);

4. desc test;

5. ALTER TABLE test CHANGE aaa bbb STRING;

hive client 2:

1. set hive.metastore.local=false;

2. set hive.metastore.uris=thrift://10.x.x.2:9083;

3. //client 1执行完第3步后执行，test表放入datanucleus cache

    desc test;

4. // client 1执行完第5步后

    desc test; 

报错

FAILED: Error in metadata: at least one column must be specified for the table

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

解决方式：

关闭L2 cache后问题解决

<property>
  <name>datanucleus.cache.level2.type</name>
  <value>none</value>
</property>

JDO Cache：

Caching is an essential mechanism in providing efficient usage of resources in many systems. Data management using JDO is no different and provides a definition of caching at 2 levels. Caching allows objects to be retained and returned rapidly without having to make an extra call to the datastore. The 2 levels of caching available with DataNucleus are

1. Level 1 Cache - mandated by the JDO specification, and represents the caching of instances within a PersistenceManager

2. Level 2 Cache - represents the caching of instances within a PersistenceManagerFactory (across multiple PersistenceManager's)

You can think of a cache as a Map, with values referred to by keys. In the case of JDO, the key is the object identity (identity is unique in JDO).

By default the Level 2 Cache is enabled. The user can configure the Level 2 Cache if they so wish. This is controlled by use of the persistence property datanucleus.cache.level2.type. You set this to "type" of cache required. With the Level 2 Cache you currently have the following options.

1. none - turn OFF Level 2 caching.

2. weak - use the internal (weak reference based) L2 cache. Provides support for the JDO 2 interface of being able to pin objects into the cache, and unpin them when required.This option does not support distributed caching, solely running within the JVM of the client application. Weak references are held to non pinned objects.

3. soft - use the internal (soft reference based) L2 cache. Provides support for the JDO 2 interface of being able to pin objects into the cache, and unpin them when required.This option does not support distributed caching, solely running within the JVM of the client application. Soft references are held to non pinned objects.

本文链接http://blog.csdn.net/lalaguozhe/article/details/9184593，转载请注明