使用kylin踩过的坑

转载：http://www.th7.cn/db/mssql/201609/206145.shtml

我的kylin.properties配置： 

### SERVICE ###
# Kylin server mode, valid value [all, query, job]
kyin.server.mode=all
# Optional information for the owner of kylin platform, it can be your team's email
# Currently it will be attached to each kylin's htable attribute
kylin.owner=whoami@kylin.apache.org
# List of web servers in use, this enables one web server instance to sync up with other servers.
kylin.rest.servers=192.168.64.16:7070
# Display timezone on UI,format like[GMT+N or GMT-N]
kylin.rest.timezone=GMT+8
### SOURCE ###
# Hive client, valid value [cli, beeline]
kylin.hive.client=cli
# Parameters for beeline client, only necessary if hive client is beeline
#kylin.hive.beeline.params=-n root --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*' -u 'jdbc:hive2://localhost:10000'
kylin.hive.keep.flat.table=false
### STORAGE ###
# The metadata store in hbase
kylin.metadata.url=kylin_metadata@hbase
# The storage for final cube file in hbase
kylin.storage.url=hbase
# In seconds (2 days)
kylin.storage.cleanup.time.threshold=172800000
# Working folder in HDFS, make sure user has the right access to the hdfs directory
kylin.hdfs.working.dir=/kylin
# Compression codec for htable, valid value [none, snappy, lzo, gzip, lz4]
kylin.hbase.default.compression.codec=none
# HBase Cluster FileSystem, which serving hbase, format as hdfs://hbase-cluster:8020
# Leave empty if hbase running on same cluster with hive and mapreduce
kylin.hbase.cluster.fs=hdfs://master1:8020
# The cut size for hbase region, in GB.
kylin.hbase.region.cut=5
# The hfile size of GB, smaller hfile leading to the converting hfile MR has more reducers and be faster.
# Set 0 to disable this optimization.
kylin.hbase.hfile.size.gb=2
kylin.hbase.region.count.min=1
kylin.hbase.region.count.max=500
### JOB ###
# max job retry on error, default 0: no retry
kylin.job.retry=0
kylin.job.jar=$KYLIN_HOME/lib/kylin-job-1.5.4.jar
kylin.coprocessor.local.jar=$KYLIN_HOME /lib/kylin-coprocessor-1.5.4.jar
# If true, job engine will not assume that hadoop CLI reside on the same server as it self
# you will have to specify kylin.job.remote.cli.hostname, kylin.job.remote.cli.username and kylin.job.remote.cli.password
# It should not be set to "true" unless you're NOT running Kylin.sh on a hadoop client machine 
# (Thus kylin instance has to ssh to another real hadoop client machine to execute hbase,hive,hadoop commands)
kylin.job.run.as.remote.cmd=false
# Only necessary when kylin.job.run.as.remote.cmd=true
kylin.job.remote.cli.hostname=
kylin.job.remote.cli.port=22
# Only necessary when kylin.job.run.as.remote.cmd=true
kylin.job.remote.cli.username=
# Only necessary when kylin.job.run.as.remote.cmd=true
kylin.job.remote.cli.password=
# Used by test cases to prepare synthetic data for sample cube
kylin.job.remote.cli.working.dir=/tmp/kylin
# Max count of concurrent jobs running
kylin.job.concurrent.max.limit=10
# Time interval to check hadoop job status
kylin.job.yarn.app.rest.check.interval.seconds=10
# Hive database name for putting the intermediate flat tables
kylin.job.hive.database.for.intermediatetable=default
# The percentage of the sampling, default 100%
kylin.job.cubing.inmem.sampling.percent=100
# Whether get job status from resource manager with kerberos authentication
kylin.job.status.with.kerberos=false
kylin.job.mapreduce.default.reduce.input.mb=500
kylin.job.mapreduce.max.reducer.number=500
kylin.job.mapreduce.mapper.input.rows=1000000
kylin.job.step.timeout=7200
### CUBE ###
# 'auto', 'inmem', 'layer' or 'random' for testing
kylin.cube.algorithm=auto
kylin.cube.algorithm.auto.threshold=8
kylin.cube.aggrgroup.max.combination=4096
kylin.dictionary.max.cardinality=5000000
kylin.table.snapshot.max_mb=300
### QUERY ###
kylin.query.scan.threshold=10000000
# 3G
kylin.query.mem.budget=3221225472
kylin.query.coprocessor.mem.gb=3
# Enable/disable ACL check for cube query
kylin.query.security.enabled=true
kylin.query.cache.enabled=true
### SECURITY ###
# Spring security profile, options: testing, ldap, saml
# with "testing" profile, user can use pre-defined name/pwd like KYLIN/ADMIN to login
kylin.security.profile=testing
### SECURITY ###
# Default roles and admin roles in LDAP, for ldap and saml
acl.defaultRole=ROLE_ANALYST,ROLE_MODELER
acl.adminRole=ROLE_ADMIN
# LDAP authentication configuration
ldap.server=ldap://ldap_server:389
ldap.username=
ldap.password=
# LDAP user account directory;
ldap.user.searchBase=
ldap.user.searchPattern=
ldap.user.groupSearchBase=
# LDAP service account directory
ldap.service.searchBase=
ldap.service.searchPattern=
ldap.service.groupSearchBase=
## SAML configurations for SSO
# SAML IDP metadata file location
saml.metadata.file=classpath:sso_metadata.xml
saml.metadata.entityBaseURL=https://hostname/kylin
saml.context.scheme=https
saml.context.serverName=hostname
saml.context.serverPort=443
saml.context.contextPath=/kylin
### MAIL ###
# If true, will send email notification;
mail.enabled=false
mail.host=
mail.username=
mail.password=
mail.sender=
### WEB ###
# Help info, format{name|displayName|link}, optional
kylin.web.help.length=4
kylin.web.help.0=start|Getting Started|
kylin.web.help.1=odbc|ODBC Driver|
kylin.web.help.2=tableau|Tableau Guide|
kylin.web.help.3=onboard|Cube Design Tutorial|
# Guide user how to build streaming cube
kylin.web.streaming.guide=http://kylin.apache.org/
# Hadoop url link, optional
kylin.web.hadoop=
#job diagnostic url link, optional
kylin.web.diagnostic=
#contact mail on web page, optional
kylin.web.contact_mail=
crossdomain.enable=true 
1.运行./bin/find-hive-dependency.sh看Hive环境是否配置正确，提示找不到HCAT_HOME路径。 

解决方法：exportHCAT_HOME=$HIVE_HOME/hcatalog

然后重新运行脚本

2. 在kylin web界面load hive表失败，提示failed to take action。 

解决方法：

vi ./bin/kylin.sh 需要对此脚本做两点修改： 1. export KYLIN_HOME=/home/grid/kylin# 改成绝对路径 2. export HBASE_CLASSPATH_PREFIX=${tomcat_root}/bin/bootstrap.jar:${tomcat_root}/bin/tomcat-juli.jar:${tomcat_root}/lib/*:$hive_dependency:$HBASE_CLASSPATH_PREFIX# 在路径中添加$hive_dependency

3.Kylin如何添加登录用户 

官方doc给出解决思路：Kylin是采用Spring security framework做用户认证的，需要配置${KYLIN_HOME}/tomcat/webapps/kylin/WEB-INF/classes/kylinSecurity.xml 的sandbox,testing部分

...


... 

password需要spring加密：

org.springframework.security
spring-security-core
4.0.0.RELEASE

String password = "123456"; 
org.springframework.security.crypto.password.PasswordEncoder encoder
= new org.springframework.security.crypto.bcrypt.BCryptPasswordEncoder();
String encodedPassword = encoder.encode(password);
System.out.print(encodedPassword); 4. 建立cube时报错FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 

莫名其妙的错误，在kylin.log看不到root cause，需要去hive配置的log查看（log4j中设置，默认目录是/tmp/$user/），找到原因是error message: "Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z"

原来是压缩格式的问题，kylin默认并没有采用hadoop的lzo压缩格式，而是采用了snappy。

有3个解决方案：

1.用kylin apache-kylin-1.5.2.1-HBase1.x-bin.tar.gz 代替apache-kylin-1.5.2.1-bin.tar.gz重新部署，因为我用的是hbase0.98，所以被pass。。

2. 改换成lzo压缩，要麻烦一点，具体查考http://kylin.apache.org/docs15/install/advance_settings.html

3. hive和hbase不采用压缩（cube build时间也许会变长，具体自行评估），在配置文件conf/kylin.properties和conf/*.xml (grep snappy)，然后全部删掉snappy和compress的配置。

5. 建立cube的step3Extract Fact Table Distinct Columns报错java.net.ConnectException: Call From master1/192.168.64.11 to localhost:18032 failed on connection exception: java.net.ConnectException: Connection refused 

解决方案：

这个issue花费了太多时间，网上查到都说是yarn端口配置问题，但是我修改了yarn-site.xml之后还是不行。后来又以为是hive metastore server的问题。但是修改了之后还是同样问题。

没办法，我最后只好换成HBase 1.1.6，同时kylin版本也要找到对应hbase1.x版。问题解决。。。

6. cube创建成功后，查询sql出现error in coprocessor 

解决方法：

这个问题真的困扰了好几天，我kylin.coprocessor.local.jar=/../kylin/lib/kylin-coprocessor-1.5.4.jar已经配好了的，网上的解决方法是find-hbase-dependency.sh脚本里 hbase_dependency=绝对路径/habse-1.1.6/lib，但貌似还是没什么用。