2015年06月_chenfangfang_2015

原创 How-to: Enable yarn ResourceManager HA

Update yarn-site.xml. Add following in yarn-site.xml(master.chff.dc and slave01.chff.dc are resourcemanager nodes): yarn.resourcemanager.ha.enabled true

2015-06-25 14:52:07 1128

原创 How-to: resolve hbase "org.apache.hadoop.hbase.TableExistsException: hbase:namespace"

Error:2015-06-24 13:34:05,251 FATAL [master:60000.activeMasterManager] master.HMaster: Failed to become active masterorg.apache.hadoop.hbase.TableExistsException: hbase:namespace at org.a

2015-06-24 14:36:13 3642

原创 How-to: enable HDFS HA at a new cluster

deploy hadoop cluster with non-HA: make sure hadoop could work normallyConfiguration update:hdfs-site.xml: dfs.nameservices dfscluster

2015-06-24 13:05:52 911

原创 How-to resolve hbase shutdown caused by "KeeperErrorCode = ConnectionLoss for /hbase"

Error log:2015-06-23 17:35:05,995 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=slave.chff.dc:2183,master.chff.dc:2183,slave01.chff.dc:2183, exception=org.apache.

2015-06-23 18:22:14 4408 1

原创 How-to: make spark streaming collect data from Kafka topics and store data into hdfs

Develop steps:Develop class which is used for connect kafka topics and store data into hdfs.In spark project:./examples/src/main/scala/org/apche/spark/examples/streaming/Kafka.scalapackage o

2015-06-18 15:49:43 872

原创 How-to: Enbale HMaster HA(high availability) and High Available Reads

At first, please make sure the backup HMaster node's hostname is configured at all hbase nodes /etc/hosts file. Then add following information in hbase-site.xml of all hbase node(including HMaster, re

2015-06-16 12:38:53 1663

原创 How-to: resolve regionserver died with "No lease on /hbase/oldWALs/..."

Error Log from died regionserver: 2015-06-11 16:23:03,072 ERROR [regionserver/slave04/172.31.34.64:60020] regionserver.HRegionServer: Shutdown / close of WAL failed: org.apache.hadoop.hdfs.server.na

2015-06-16 11:10:08 3487

原创 How-to: install puppet via yum

Master:sudo rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-5.noarch.rpmsudo yum install puppet-serverAdd following in /etc/puppet/puppet.conf [main]: dns_alt_names = master.chff.dcsu

2015-06-15 13:12:55 444

原创 how-to: resolve "Connection refused" during connext hiveserver2 via beeline

Issue:At hiveserver2 node, could connect hiveserver2 via localhost/127.0.0.1, but connection refused when using ip.At other node, could not access hiveserver2 via ipConnect command is like:/op

2015-06-11 11:13:35 1461

原创 how-to: resolve "java.lang.OutOfMemoryError: Java heap space" during using beeline && hiveserver2

Error log in hive.log file:2015-06-10 00:33:18,207 ERROR [HiveServer2-Handler-Pool: Thread-47]: thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error processing OpenSession

2015-06-11 11:12:29 3677

原创 How-to: resolve "Unapproved licenses:" issue during building with mvn+rat

This issue happened during building flume, but this has nothing to do with flume.mvn verbose output will mention about what files is marked "Unapproved license" from target/rat.txt. Like:

2015-06-09 12:29:15 897

原创 How-to: resolve " java.lang.OutOfMemoryError: unable to create new native thread" for hbase thrift

Error:Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at jav

2015-06-08 17:49:15 2076

原创 How-to: enable hbase thrift support

Reference:https://thrift.apache.org/docs/install/centoshttp://yannramin.com/2008/07/19/using-facebook-thrift-with-python-and-hbase/http://blog.cloudera.com/blog/2013/09/how-to-use-the-hbase-thri

2015-06-03 16:53:45 579

转载大数据，TB、PB、EB，你了解多少？

http://www.caf.ac.cn/html/xxh/20130503/20446.html“大数据”作为时下最火热的IT行业的词汇，随之数据仓库、数据安全、数据分析、数据挖掘等等围绕大数量的商业价值的利用逐渐成为行业人士争相追捧的利润焦点。大数据有多大，一直没有清晰的概念，故此简单的科普研究，分享至此：　　　下面先按顺序给出所有单位：Byte、KB、MB、G

2015-06-02 11:13:35 9222

原创大数据平台搭建之components building commands

Spark: http://spark.apache.org/docs/1.3.0/building-spark.html- export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"- ./make-distribution.sh --tgz -Pyarnflume:

2015-06-01 19:18:27 450

原创大数据平台搭建之deploy components to nexus

Add following configuration in mvn setting.xml:bigDataInfraDeploymentRepoadminadmin123bigdatainfra-default-profiletruehttp://hostname:8081/nexus/content/repositories/BigDataInf

2015-06-01 19:15:30 401

原创大数据平台搭建之build environment

Install following tools:Scala 2.10SBT hereJava 1.7.0_72mvn 3.2.5: export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"protobuf 2.5.0yum -y install lzo-devel

2015-06-01 19:10:48 536

原创 Hadoop cluster security 1: How to enable HDFS permission ACl

解决方案：使用linux自带组用户管理+hdfs ACL。可以做到多个groups，多个users对同一目录或者文件的权限设定。假设有三个组（或者根据产品？），技术管理部、业务组、app开发组，权限设定类似于以下表格，如果整个hdfs只是咱们组用的话，用户划分会更简单。每个group中的每个user，对一个文件或目录的权限为由三部分组成：r(read), w(write),x(exe

2015-06-01 18:50:24 800

原创大数据平台搭建之nexus私服

System RequirementsProcessors: 2 Core 2.5Ghz+Available Space: 100G+ depending on number and size of artifactsAvailable RAM: 4Gb+JRE Versions: jdk7u60 Reference： https://suppor

2015-06-01 18:42:30 762

原创大数据平台搭建之components版本选择

确定依据：CDH5内的所有版本都兼容JDK:same one with CDH 5.4.X release: Oracle JDK 1.7.0_75 Open Source components versions:Based on: CDH 5 (storm to be determined), cdh5-*_5.4.1(latest github br

2015-06-01 18:39:24 685

lulynn的博客