空中的鱼

原创 impala HA

目的：为impala jdbc提供统一的接口，作用参照http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_proxy.html步骤：安装haproxy选择一台非impalad的机器安装haproxy yum install haproxy编辑/etc/ha...

原创 impala HA

impalad的机器安装haproxy yum install haproxy编辑/etc/haproxy/haproxy.cfg，参考global # To have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by ad

原创 impala集成LDAP

目的：为解决kerberos安全机制下的impala，resin cache kerberos tgt maxrenewlife天失效问题。说明：impala启用LDAP后，会优先选择LDAP用户密码认证，当LDAP认证失败时自动选择kerberos安全认证。步骤：关闭防火墙，设置开机不启动防火墙sudo /etc/init.d/iptables statuss...

原创 impala集成LDAP

失败时自动选择kerberos安全认证。步骤：关闭防火墙，设置开机不启动防火墙sudo /etc/init.d/iptables statussudo /etc/init.d/iptables stop / sudo service iptables stopsudo chkconfig iptables off安装LDAPyum install db4 db4-utils db4-devel cyrus-sasl* krb5-server-ldap -yyum ins

原创 security cdh mapreduce access hbase

执行mapreduce的用户必须是可以访问hdfs相应目录和执行mapreduce的账户，例如hive。指定hive的节点kinit获取执行权限在mapreduce main代码中加入访问hbase的权限，例如：import java.io.IOException;import org.apache.hadoop.hbase.HBaseConfiguration;impor...

原创 security cdh mapreduce access hbase

原创 hive gateway(client) configuration

配置hive gateway机器Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate fa...

原创 hive gateway(client) configuration

tiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297) at org.apache.thrift.transport.TSaslClientT

原创 hcatalog读取hive数据并写入hive

参考http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_table_access_mapreduce.htmlhttps://github.com/cloudera/hcatalog-examples.git命令：for jarfile in `ls /logdata/CDH-5....

原创 hcatalog读取hive数据并写入hive

ata/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib/.`; do       export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:/logdata/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib/$jarfile"done  hadoop jar bigdata-mapreduce-0.0.1-SNAPSHOT.jar com.yeahmobi.bigdata.mapreduce.G

原创 CDH5.0.2升级至CDH5.2.0

升级需求1.为支持spark kerberos安全机制2.为满足impala trunc函数3.为解决impala import时同时query导致impala hang问题升级步骤参考http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation_upgrade...

原创 CDH5.0.2升级至CDH5.2.0

loudera/en/documentation/core/latest/topics/installation_upgrade.html优先升级cloudera manager，再升级cdh1.准备工作： 统一集群root密码，需要运维帮忙操作下 agent自动重启关闭 事先下载好parcals包2.CM升级 登录cmserver安装的主机，执行命令： cat /etc/cloudera-scm-server/db.properties 备份CM数据： pg_