shark应用cache

转自官网

Unlike Hive, Shark allows users to exploit this temporallocality by caching their working set of data, or in database terms, to createin-memory materialized views. Common data types can be cached in a columnarformat (as Java primitives arrays), which is very efficient for storage andgarbage collection, yet provides maximum performance (orders of magnitudefaster than reading data from disk).

To create a cached table from the rows (or subset ofrows) of an existing table, set the shark.cache table property:

CREATE TABLE ... TBLPROPERTIES ("shark.cache" ="true") AS SELECT ...

We also extend HiveQL to provide a shortcut for thissyntax. Simply append _cached to the table name when using CREATE TABLE AS SELECT, andthat table will be cached in memory. To disable this shortcut, see theconfiguration options section. Below is an example:

CREATE TABLE logs_last_month_cached AS

SELECT * FROM logs WHERE time > date(...);

以上，建表即可，表名加后缀_cached。同时设置这个参数shark.cache.flag.checkTableName为true。

说明：shark.cache.flag.checkTableName# 'true' or 'false', whether to cache tables ending in "_cached"

 

Once this table has been created, we can query it likeany other Hive table.

SELECT count(*) from logs_last_month_cached;

Note that queries which shuffle data require you to setthe number of reducers. This will be automatically determined in the nextversion of Shark:

set mapred.reduce.tasks=[num_tasks];

SELECT page, count(*) c FROM logs_last_month_cached

GROUP BY page ORDER BY c DESC LIMIT 10;

In addition to caching, Shark employs a number ofoptimization techniques such as limit push downs and hash-based shuffle, whichcan provide significant speedups in query processing. You can directly inspectthe execution plan for a given query using the explain statement:

explain SELECT * FROM logs_last_month ORDER BY timestamp LIMIT 10;