Linux. Hadoop Hbase & MapReduce

• Using Map Reduce to read/write table in HBase

---

install Hadoop & Hbase

• hadoop.apache.org
• hbase.apache.org

install hadoop

• install hadoop tutorial -> link
• some questions I met and solutions I found -> link

install hbase

• suppose you already have ‘hbase-1.3.0-src.tar.gz’ -> link

• unzip hbase-1.3.0-bin.tar.gz to /usr/local

sudo tar zxvf hbase-1.3.0-bin.tar.gz -C /usr/local

• rename to ‘hbase’

sudo mv /usr/local/hbase-1.3.0/ /usr/local/hbase/

• modify file permissions (specifically speaking, modify the owner of ‘hbase’, after that you don’s hava to ‘sudo’ or enter password or meet many other problems)

sudo chown -R your_user_name /usr/local/hbase/

• Set the environment variable

sudo vim /etc/profile
//then append the following content
#set hbase path
export PATH=$PATH:/usr/local/hbase/bin

• enable the environment variable (only works in the current shell if you modify /ect/profile and ‘source’ it)

source /etc/profile

• configure hbase

vim /usr/local/hbase/conf/hbase-env.sh
//append content
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_101
export HBASE_MANAGES_ZK=true

vim /usr/local/hbase/conf/hbase-site.xml
//add this <property>s in <configuration>
<property>
        <name>hbase.rootdir</name>
        <value>hdfs://localhost:9000/hbase</value>
</property>
<property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
</property>

• start hbase (make sure hadoop was started)

  • using shell command ‘jps’ to list all the java process running on this PC
  • after successfully starting hadoop, you will see ‘NameNode’ ‘DataNode’ ‘SecondaryNameNode’ ‘NodeManager’ ‘ResourceManager’ via ‘jps’ command
  • then hbase, ‘HMaster’ ‘HRegionServer’

• enter hbase shell

/usr/local/hbase/bin/hbase shell
2017-03-26 23:29:47,401 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.3.0, re359c76e8d9fd0d67396456f92bcbad9ecd7a710, Tue Jan  3 05:31:38 MSK 2017

hbase(main):001:0>  list
TABLE                                                                                 
Course                                                                                
SC                                                                                    
Student                                                                               
person                                                                                
test1                                                                                 
test2                                                                                 
6 row(s) in 0.2590 seconds

=> ["Course", "SC", "Student", "person", "test1", "test2"]
hbase(main):002:0> scan 'test1'
ROW                    COLUMN+CELL                                                    
 s01