alter table census partition (year=1960) set cached in 'pool_name';
测试数据37460946条
测试结果表明加不加缓存影响不明显,速度差不多。
2.hbase的测试
1.hmaster在HA的情况下,挂了一台另一台可以立刻顶上。
2.hmaster两台全挂的情况下,对读写数据没有任何影响,但是如果此时regionserver也挂了会报错,并且影响读写。
3.regionserver只要有一台存在,就可以读写数据。在其他regionserver挂了的时候,活着的regionserver会保证时刻拥有建表时候的region数量来写入数据。
4.8核,16G内存三台regionserver,最大每秒写入3W条,在开启WAL的情况下。
3.hbase新特性
1.API的变化,比如放弃了客户端几个重要的API:( HTableInterface, HTable,HBaseAdmin),建议程序升级到新的API,老的PAI会在2.X中删除。
2.不重启regionserver的情况下读取配置文件。
3.master节点也运行regionserver。master与regionserver共享RPC端口。
4.global memstore和block cache大小的自动调整。
5.Bucket cache改进(data blocks的可用性和压缩方面)
6.集群中一个新的可插拔式的终端来复制成自定义存储。
(A new pluggable replication endpoint to plug in to HBase's inter-cluster replication to replicate to a custom data store)
7.新增truncate table 命令
8.新增meta table来替代zk来进行更快的region分配(此功能默认是关闭都)
9.广泛的文档改进
10.[HBASE-12511] - namespace permissions - add support from table creation privilege in a namespace 'C'
11.位Thrift Server添加Thrift-over-HTTPS 和 doAs支持(?)
12.bulkload复制表
13.使用前缀设置一个行的开始和结束更加easy了。
14.脚本停止regionserver使用RPC
15.客户端的原子性检查和变化的支持。
16.自动增加heap size的值如果开始没有指定大小的话。
17.regionserver协处理器终端。
18.添加了在一个regionserver上压缩所有region的命令。
19.添加LongComparator过滤器
20.[shell]在线加载配置文件的改变。
21.REST server的一些修复。
22.需要zk3.4.X的支持。
23.需要hadoop2.x的支持
24.jdk1.7,不再支持1.6
25.默认的端口号已经由600xx改成了160XX
26.默认使用hfile version3
27.Slab cache已经被移除,使用bucket cache
28.默认的regionserver堆内存由1G改为jdk默认的数值,一般为物理内存的四分之一。
29.mapred包和类已经被废除。
30.加入两个新的模块:hbase-annotations 和 hbase-rest
4.hive新特性
Bug
[HIVE-5631] - Index creation on a skew table fails
描述:在倾斜表中创建index失败。
[HIVE-5664] - Drop cascade database fails when the db has any tables with indexes
描述:删除数据库时,如果该数据库中的表建了索引,会报出表找不到异常。因为删除表的时候获取的是正常表和index表列表,但是删除正常表的同时也会删除对应的index表,删除index表的时候会因为找不到index表而报错。
[HIVE-6198] - ORC file and struct column names are case sensitive
描述:ORC 文件和结构列大小写敏感。
[HIVE-6468] - HS2 & Metastore using SASL out of memory error when curl sends a get request
[HIVE-7270] - SerDe Properties are not considered by show create table Command
[HIVE-8099] - IN operator for partition column fails when the partition column type is DATE
0.13.0
[HIVE-8295] - Add batch retrieve partition objects for metastore direct sql
[HIVE-8374] - schematool fails on Postgres versions < 9.2
描述:schematool命令在 Postgres 版本<9.2会失败。
[HIVE-8485] - HMS on Oracle incompatibility
描述:参数 hive.metastore.orm.retrieveMapNullsAsEmptyStrings默认值为false。对于oracle等把空转化成null的数据库需要把该值设为true。
[HIVE-8706] - Table statistic collection on counter failed due to table name character case.
描述:表名大小写导致表统计失败。hive会忽略表名的大小写,在hive的metastore中会转化成小写,但是Counters/TezCounters是大小写敏感的,当Hive collection table使用Counters,而hive使用table name 作为counter组名时这可能导致表的统计汇总失败。
影响版本:0.13.1
[HIVE-8715] - Hive 14 upgrade scripts can fail for statistics if database was created using auto-create
描述:当数据库是通过auto-create,hive不能使用schemeTool升级
影响版本:
[HIVE-8739] - handle Derby and Oracle errors with joins and filters in Direct SQL in a invalid-DB-specific path
描述:
[HIVE-8784] - Querying partition does not work with JDO enabled against PostgreSQL
描述:通过PostgreSQL 使用 JDO 查询一个分区会失败。
影响版本: 0.14.1
[HIVE-8805] - CBO skipped due to SemanticException: Line 0:-1 Both left and right aliases encountered in JOIN 'avg_cs_ext_discount_amt'
[HIVE-8807] - Obsolete default values in webhcat-default.xml
[HIVE-8811] - Dynamic partition pruning can result in NPE during query compilation
[HIVE-8827] - Remove SSLv2Hello from list of disabled protocols
描述:
[HIVE-8830] - hcatalog process don't exit because of non daemon thread
描述:HiveClientCache 清理线程而不是进程,会导致hcatalogclient挂起。
影响版本: 0.14.0
[HIVE-8845] - Switch to Tez 0.5.2
[HIVE-8866] - Vectorization on partitioned table throws ArrayIndexOutOfBoundsException when partitions are not of same #of columns
[HIVE-8870] - errors when selecting a struct field within an array from ORC based tables
0.13.0
[HIVE-8873] - Switch to calcite 0.9.2
[HIVE-8876] - incorrect upgrade script for Oracle (13->14)
描述:升级脚本对oracle13-14不适用。
[HIVE-8880] - non-synchronized access to split list in OrcInputFormat
[HIVE-8886] - Some Vectorized String CONCAT expressions result in runtime error Vectorization: Unsuported vector output type: StringGroup
[HIVE-8888] - Mapjoin with LateralViewJoin generates wrong plan in Tez
[HIVE-8947] - HIVE-8876 also affects Postgres < 9.2
描述:升级hive时的脚本对postgre数据库低于9.2的版本是错误的。
影响版本: 0.14.0
[HIVE-8966] - Delta files created by hive hcatalog streaming cannot be compacted
?
[HIVE-9003] - Vectorized IF expr broken for the scalar and scalar case
描述:if表达式被分成scalar和scalar case?
影响版本:0.14.0
[HIVE-9025] - join38.q (without map join) produces incorrect result when testing with multiple reducers
描述:当设置mapred.reduce.tasks大于1时候,做join会产生错误的结果。
影响版本: 0.14.0
[HIVE-9038] - Join tests fail on Tez
描述:
[HIVE-9051] - TezJobMonitor in-place updates logs too often to logfile
描述:0.15版本UI监控日志太频繁,改成与老版本一样。
影响版本:0.15
[HIVE-9053] - select constant in union all followed by group by gives wrong result
0.13.0
[HIVE-9067] - OrcFileMergeOperator may create merge file that does not match properties of input files
0.14.0
[HIVE-9090] - Rename "Tez File Merge Work" to smaller name
描述:这是一个小改动, Tez File Merge Work这个名字过长改成更短。
[HIVE-9108] - Fix for HIVE-8735 is incorrect (stats with long paths)
[HIVE-9111] - Potential NPE in OrcStruct for list and map types
描述:对 OrcStruct中list,map的检查中没有做null判断,可能导致空指针异常。
[HIVE-9112] - Query may generate different results depending on the number of reducers
描述:设置不同的reducer可能导致不同的结果,甚至出错。
[HIVE-9114] - union all query in cbo test has undefined ordering
描述:union all在其他平台上测试结果不一致
[HIVE-9126] - Backport HIVE-8827 (Remove SSLv2Hello from list of disabled protocols) to 0.14 branch
[HIVE-9141] - HiveOnTez: mix of union all, distinct, group by generates error
描述:当设置 hive.execution.engine为tez时,一条sql语句中使用union all,distinct和group by会产生错误。
[HIVE-9155] - HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql
描述: HL_LAST_HEARTBEAT and HL_ACQUIRED_AT使用了bigint来代替int以容纳更大的数据。
[HIVE-9162] - stats19 test is environment-dependant
[HIVE-9166] - Place an upper bound for SARG CNF conversion
[HIVE-9168] - Vectorized Coalesce for strings is broken
[HIVE-9205] - Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist
描述:如果用户使用tez 启动hive shell,会在/usr/下面建目录,权限不足会抛出异常。现在使用/tmp目录,以满足权限问题。
[HIVE-9234] - HiveServer2 leaks FileSystem objects in FileSystem.CACHE
0.12.0
[HIVE-9249] - java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
描述:类型转化错误,HiveVarcharWritable 不能转化成HiveVarchar。
影响版本: 0.14.0
[HIVE-9278] - Cached expression feature broken in one case
描述: hive.cache.expr.evaluation设置为true时,查询结果无输出,并且如果udf函数嵌套其他udf函数会导致错误结果。
[HIVE-9317] - move Microsoft copyright to NOTICE file
版权问题
[HIVE-9359] - Export of a large table causes OOM in Metastore and Client
描述:导出大表导致Metastore 和client OOM问题。
[HIVE-9361] - Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable
[HIVE-9390] - Enhance retry logic wrt DB access in TxnHandler
[HIVE-9401] - Backport: Fastpath for limited fetches from unpartitioned tables
0.14.0
[HIVE-9404] - NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()
描述:org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()方法抛出空指针异常。
影响版本: 0.15.0
[HIVE-9436] - RetryingMetaStoreClient does not retry JDOExceptions
描述:RetryingMetaStoreClient 代码一个正则匹配错误,有些情况会匹配不到导致不会抛出异常。
影响版本:0.14.0, 0.13.1
[HIVE-9473] - sql std auth should disallow built-in udfs that allow any java methods to be called
描述:标准sql授权不应该允许能调用任何java方法的内置udfs运行。
[HIVE-9514] - schematool is broken in hive 1.0.0
描述:使用bin/schematool时产生的错误
[HIVE-9632] - inconsistent results between year(), month(), day(), and the actual values in formulas
描述: 时间的返回不一致问题。例如:2015-01-09这个时间,year(2015-01-09) month(2015-01-09) day(2015-01-09)会返回2015,1,9year(2015-01-09)*10000会返回20150000,month(2015-01-09)*100会返回100,day(2015-01-09)会返回9但是 year(2015-01-09)*10000+month(2015-01-09*100+day(2015-01-09) 会返回 20353515
影响版本:0.14.0
[HIVE-9684] - Incorrect disk range computation in ORC because of optional stream kind
Improvement
[HIVE-3280] - Make HiveMetaStoreClient a public API
描述:HiveMetaStoreClient 变成了一个公共API
[HIVE-8933] - Check release builds for SNAPSHOT dependencies
描述:Hive 0.14.0带有SNAPSHOT依赖,不再需要使用maven enforcer plugin这个插件.
Task
[HIVE-6977] - Delete Hiveserver1
描述:删除了hiveserver1