Hive中metastore(元数据存储)的三种方式:
内嵌Derby方式
Local方式
Remote方式
[一]、内嵌Derby方式
这个是Hive默认的启动模式,一般用于单元测试,这种存储方式有一个缺点:在同一时间只能有一个进程连接使用数据库。
hive-site.xml 中jdbc URL、驱动、用户名、密码等的配置信息如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
javax.jdo.option.ConnectionURL
jdbc:derby:;databaseName=metastore_db;create=true
JDBC connect string for a JDBC metastore
javax.jdo.option.ConnectionDriverName
org.apache.derby.jdbc.EmbeddedDriver
Driver class name for a JDBC metastore
javax.jdo.option.ConnectionUserName
APP
username to use against metastore database
javax.jdo.option.ConnectionPassword
mine
password to use against metastore database
hive.metastore.warehouse.dir
file:///Users/micmiu/tmp/hive/warehouse
unit test data goes in here on your local filesystem
执行初始化命令:schematool -dbType derby -initSchema
查看初始化后的信息: schematool -dbType derby -info
配置完成后就可在shell中以CLI的方式访问hive 进行操作验证。
[二]、Local方式
以本地Mysql数据库为例:创建好用户:hive;database:hive。
配置文件 hive-site.xml 中jdbc URL、驱动、用户名、密码等属性值配置如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
javax.jdo.option.ConnectionURL
jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true
JDBC connect string for a JDBC metastore
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
Driver class name for a JDBC metastore
javax.jdo.option.ConnectionUserName
hive
username to use against metastore database
javax.jdo.option.ConnectionPassword
micmiu
password to use against metastore database
hive.metastore.warehouse.dir
/user/hive/warehouse
location of default database for the warehouse
ps:需要把mysql的驱动包copy到目录 /lib 中
如果是第一次需要执行初始化命令:schematool -dbType mysql -initSchema
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
micmiu-mbp:mysql micmiu$schematool-dbType mysql-initSchema
14/02/1714:46:08INFO Configuration.deprecation:mapred.input.dir.recursive isdeprecated.Instead,usemapreduce.input.fileinputformat.input.dir.recursive
14/02/1714:46:08INFO Configuration.deprecation:mapred.max.split.size isdeprecated.Instead,usemapreduce.input.fileinputformat.split.maxsize
14/02/1714:46:08INFO Configuration.deprecation:mapred.min.split.size isdeprecated.Instead,usemapreduce.input.fileinputformat.split.minsize
14/02/1714:46:08INFO Configuration.deprecation:mapred.min.split.size.per.rack isdeprecated.Instead,usemapreduce.input.fileinputformat.split.minsize.per.rack
14/02/1714:46:08INFO Configuration.deprecation:mapred.min.split.size.per.node isdeprecated.Instead,usemapreduce.input.fileinputformat.split.minsize.per.node
14/02/1714:46:08INFO Configuration.deprecation:mapred.reduce.tasks isdeprecated.Instead,usemapreduce.job.reduces
14/02/1714:46:08INFO Configuration.deprecation:mapred.reduce.tasks.speculative.execution isdeprecated.Instead,usemapreduce.reduce.speculative
Metastore connection URL:jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true
Metastore Connection Driver:com.mysql.jdbc.Driver
Metastore connection User:hive
Starting metastore schema initialization to0.12.0
Initialization script hive-schema-0.12.0.mysql.sql
Initialization script completed
schemaTool completeted
查看初始化后信息 schematool -dbType mysql -info
初始化后查看mysql中表情况:show tables;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
mysql>show tables;
+---------------------------+
|Tables_in_hive|
+---------------------------+
|BUCKETING_COLS|
|CDS|
|COLUMNS_V2|
|DATABASE_PARAMS|
|DBS|
|DB_PRIVS|
|DELEGATION_TOKENS|
|GLOBAL_PRIVS|
|IDXS|
|INDEX_PARAMS|
|MASTER_KEYS|
|NUCLEUS_TABLES|
|PARTITIONS|
|PARTITION_EVENTS|
|PARTITION_KEYS|
|PARTITION_KEY_VALS|
|PARTITION_PARAMS|
|PART_COL_PRIVS|
|PART_COL_STATS|
|PART_PRIVS|
|ROLES|
|ROLE_MAP|
|SDS|
|SD_PARAMS|
|SEQUENCE_TABLE|
|SERDES|
|SERDE_PARAMS|
|SKEWED_COL_NAMES|
|SKEWED_COL_VALUE_LOC_MAP|
|SKEWED_STRING_LIST|
|SKEWED_STRING_LIST_VALUES|
|SKEWED_VALUES|
|SORT_COLS|
|TABLE_PARAMS|
|TAB_COL_STATS|
|TBLS|
|TBL_COL_PRIVS|
|TBL_PRIVS|
|TYPES|
|TYPE_FIELDS|
|VERSION|
+---------------------------+
41rows inset(0.00sec)
配置完成后就可在shell中以CLI的方式访问hive 进行操作验证。
[三]、Remote方式
以Mysql数据库(192.168.6.77)为例:创建好用户:hive;database:hive_meta。Remote方式需要分别配置服务端和客户端的配置文件:
服务端的 hive-site.xml 中jdbc URL、驱动、用户名、密码等属性值配置如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
javax.jdo.option.ConnectionURL
jdbc:mysql://192.168.6.77/hive_meta?createDatabaseIfNotExist=true
JDBC connect string for a JDBC metastore
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
Driver class name for a JDBC metastore
javax.jdo.option.ConnectionUserName
hive
username to use against metastore database
javax.jdo.option.ConnectionPassword
micmiu
password to use against metastore database
hive.metastore.warehouse.dir
/user/hive/warehouse
base hdfs path :location of default database for the warehouse
ps:需要把mysql的驱动包copy到目录 /lib 中
如果是第一次需要执行初始化命令:schematool -dbType mysql -initSchema
客户端中配置内容修改如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
hive.metastore.uris
thrift://192.168.6.77:9083
Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.
hive.metastore.warehouse.dir
/user/hive/warehouse
location of default database for the warehouse
hive metastore 服务端启动命令:
hive --service metastore -p
如果不加端口默认启动:hive --service metastore,则默认监听端口是:9083 ,注意客户端中的端口配置需要和启动监听的端口一致。服务端启动正常后,客户端就可以执行hive操作了。
参考:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin