Hive&hadoop 安装配置

Hive&hadoop 安装配置

hadoop配置文档

  tar zxvf hadoop-0.20.2.tar.gz
  cd conf
  vim core-config.xml

增加以下内容

 <?xml version="1.0"?>
 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 3 
 4 
 5 
 6 <configuration>
 7     <property>
 8         <name>fs.default.name</name>
 9         <value>hdfs://172.16.18.202:9000</value>
 10     </property>
 11 <property>
 12     <name>mapred.job.tracker</name>
 13     <value>172.16.18.202:9001</value>
 14   </property>
 15   <property>
 16     <name>dfs.replication</name>
 17     <value>1</value>
 18   </property>
 19 </configuration>

怎么做授权

 172.16.60.96 需要授权guojinyong  到 172.16.60.202
 先在202 useradd guojinyong

 然后在96上 
 ssh-keygen -t rsa
 一直回车,会有一个id_rsa.pub的文件

 scp id_rsa.pub guojinyong@172.16.60.202:/tmp/
 cat /tmp/id_rsa.pub >> /home/guojinyong/.ssh/authorized_keys

这样就okey了.

hive配置文档

 配置文档
  tar -xzvf hive-x.y.z.tar.gz
  cd hive-x.y.z
  export HIVE_HOME=`pwd`
  export PATH=$HIVE_HOME/bin:$PATH

  配置hadoop 环境变量 
  export HADOOP_HOME=/work/guojinyong/hadoop-0.20.2  //假设我的hadoop解压在这里
  bin/hive
  hive> show databases;

ok搞定.

hive实战步骤

 vim /tmp/guojinyong/test/mobile.py

 import sys
 import datetime
 
 for line in sys.stdin:
   line = line.strip()
   name,num = line.split('\t')
   str1 = 'ddddd'
   str2 = 'ggggg'
   print '\t'.join([name,num,str1,str2])

 add file /tmp/guojinyong/test/mobile.py
 insert into table test select transform (str) using 'python mobile.py' as (name,num,str1,str2) from test_tmp;