最近线上的hive升级到了0.13,遇到不少问题。权限上面,设置了hive.security.authorization.createtable.owner.grants 在hive0.13中,用户自己创建的表也没有权限。通过对源码的分析和debug找到了rc并fix,下面记录下。
1.首先在hive0.11中和hive0.13中分别做建表测试,通过查看数据库中的元数据,发现在hive0.11中如果设置了owner的参数在表创建之后,用户会有这个表的all的权限(具体可以分析db_privs ,tbl_privs表,hive.security.authorization.createtable.owner.grants设置为了all),而在hive0.13中则为空的。
2.通过
hive -hiveconf hive.root.logger=DEBUG,console
对比前后的日志,发现在hive0.11的时候,创建的表的属性中有权限的设置
而在hive0.13中这个是为空的
hive0.11:
14/07/16 17:05:39 DEBUG hive.log: DDL: struct ericni4 { string a}14/07/16 17:05:39 DEBUG
lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
columnNames=[a] columnTypes=[string] separator=[[B@5b224686] nullstring=\N
lastColumnTakesRest=false14/07/16 17:05:39 INFO metastore.HiveMetaStore: 0:
create_table: Table(tableName:ericni4, dbName:temp, owner:ericni2, createTime:1405501539,
lastAccessTime:0, retention:0,sd:StorageDescriptor(cols:[FieldSchema(name:a,
type:string, comment:null)], location:null,
inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
compressed:false, numBuckets:-1,
serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.
LazySimpleSerDe,parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false),partitionKeys:[],
parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE,
privileges:PrincipalPrivilegeSet(userPrivileges:{ericni2=[PrivilegeGrantInfo(privilege:
all, createTime:-1, grantor:ericni2, grantorType:USER, grantOption:true)]},
groupPrivileges:null, rolePrivileges:null)) #hive0.11创建表的时候增加了这个设置
hive0.13
14/07/16 17:10:07 DEBUG hive.log: DDL: struct ericni4 { string a}14/07/16 17:10:07 DEBUG
lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
columnNames=[a] columnTypes=[string] separator=[[B@1667e92a] nullstring=\N
lastColumnTakesRest=false14/07/16 17:10:07 INFO metastore.HiveMetaStore: 0:
create_table: Table(tableName:ericni4, dbName:temp, owner:ericni1, createTime:1405501807,
lastAccessTime:0, retention:0,sd:StorageDescriptor(cols:[FieldSchema(name:a,
type:string, comment:null)], location:null,
inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
compressed:false, numBuckets:-1,
serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe,parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false),partitionKeys:[],
parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
3.会不会是参数没有生效?在hive0.13中注释掉这个设置做对比,发现create_table的属性并没有改变,说明至少在现在的0.13环境下这个参数并没有生效。而在0.11中前后是有变化的。
4.在hive0.11中我们设置一个错误的值来获取堆栈的调用信息:
得到0.11的调用栈:
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.checkPrivilege(CreateTableAutomaticGrant.java:118)
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.getGrantorInfoList(CreateTableAutomaticGrant.java:97)
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.create(CreateTableAutomaticGrant.java:52)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:275)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
再来分析源码
在hive0.11中,在初始化session时,调用SessionState的start方法,会直接根据CreateTableAutomaticGrant类设置createTableGrants属性。
SessionState.start:
try {
startSs. authenticator = HiveUtils.getAuthenticator(
startSs.getConf(),HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
startSs. authorizer = HiveUtils.getAuthorizeProviderManager(
startSs.getConf(), HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER,
startSs. authenticator);
startSs. createTableGrants = CreateTableAutomaticGrant.create(startSs
.getConf()); //设置owner的相关权限
} catch (HiveException e) {
throw new RuntimeException(e);
}
然后在创建表的时候,调用Hive.createTable的方法
(public void createTable(Table tbl, boolean ifNotExists))
public CreateTableAutomaticGrant getCreateTableGrants() {
return createTableGrants;
}
而在hive0.13中,即使设置错误也不会有报错产生,通过debug源码发现这里的调用关系有些变化,SessionState的getCreateTableGrants方法调用的是setupAuth方法。
public CreateTableAutomaticGrant getCreateTableGrants() {
setupAuth(); //调用setupAuth方法设置createTableGrants
return createTableGrants;
}
其中setupAuth方法如下:
private void setupAuth() {
if (authenticator != null) {
// auth has been initialized
return;
}
try {
authenticator = HiveUtils.getAuthenticator( conf,
HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
authenticator.setSessionState( this);
authorizer = HiveUtils.getAuthorizeProviderManager( conf,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER, authenticator, true );
if (authorizer == null) { //在authorizer 的取值为null时,才会生成权限
// if it was null, the new authorization plugin must be specified in
// config
HiveAuthorizerFactory authorizerFactory = HiveUtils.getAuthorizerFactory(conf ,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER);
authorizerV2 = authorizerFactory.createHiveAuthorizer(new HiveMetastoreClientFactoryImpl(),
conf, authenticator);
authorizerV2.applyAuthorizationConfigPolicy(conf );
// create the create table grants with new config
createTableGrants = CreateTableAutomaticGrant.create( conf);
}
} catch (HiveException e) {
throw new RuntimeException(e);
}
if(LOG.isDebugEnabled()){
Object authorizationClass = getAuthorizationMode() == AuthorizationMode. V1 ?
getAuthorizer() : getAuthorizerV2();
LOG.debug("Session is using authorization class " + authorizationClass.getClass());
}
return;
}
可以看到在在authorizer 的取值为null时,才会有生产ower权限的操作(CreateTableAutomaticGrant.create(conf))
authorizer 的值由参数hive.security.authorization.manager获得,在hive0.13里面这个是有默认值的:
HIVE_AUTHORIZATION_MANAGER("hive.security.authorization.manager",
"org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider"),
这样就导致在setupAuth方法中并不会通过CreateTableAutomaticGrant.create(conf)来设置owner的权限,rc找到了。fix也比较简单。
try {
authenticator = HiveUtils.getAuthenticator( conf,
HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
authenticator.setSessionState( this);
authorizer = HiveUtils.getAuthorizeProviderManager( conf,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER, authenticator, true );
createTableGrants = CreateTableAutomaticGrant.create(conf); //不管authorizer 是否为空都设置相关权限
if (authorizer == null) {
.......
bug fix之后,使用错误的参数,堆栈如下,说明生效了:
14/07/18 12:33:38 WARN session.SessionState: authenticator is null ,return,auth has been initialized
14/07/18 12:33:38 WARN session.CreateTableAutomaticGrant: Privilege is null
FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found.
14/07/18 12:33:38 ERROR ql.Driver: FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found.
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found.
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:410)
at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:979)
at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:990)
at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:508)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:461)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359)
at org.apache.hadoop.hive.cli.CliDriver.proce***eader(CliDriver.java:456)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:466)
at org.apache.hadoop.hive.cli.CliDriver.processInitFiles(CliDriver.java:502)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found.
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.validatePrivilege(CreateTableAutomaticGrant.java:129)
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.getGrantorInfoList(CreateTableAutomaticGrant.java:105)
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.create(CreateTableAutomaticGrant.java:57)
at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:392)
... 24 more
转载于:https://blog.51cto.com/caiguangguang/1440128