Hive&hadoop 安装配置
hadoop配置文档
tar zxvf hadoop-0.20.2.tar.gz cd conf vim core-config.xml
增加以下内容
<?xml version="1.0"?> 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 3 4 5 6 <configuration> 7 <property> 8 <name>fs.default.name</name> 9 <value>hdfs://172.16.18.202:9000</value> 10 </property> 11 <property> 12 <name>mapred.job.tracker</name> 13 <value>172.16.18.202:9001</value> 14 </property> 15 <property> 16 <name>dfs.replication</name> 17 <value>1</value> 18 </property> 19 </configuration>
怎么做授权
172.16.60.96 需要授权guojinyong 到 172.16.60.202 先在202 useradd guojinyong
然后在96上 ssh-keygen -t rsa 一直回车,会有一个id_rsa.pub的文件 scp id_rsa.pub guojinyong@172.16.60.202:/tmp/ cat /tmp/id_rsa.pub >> /home/guojinyong/.ssh/authorized_keys
这样就okey了.
hive配置文档
配置文档 tar -xzvf hive-x.y.z.tar.gz cd hive-x.y.z export HIVE_HOME=`pwd` export PATH=$HIVE_HOME/bin:$PATH
配置hadoop 环境变量 export HADOOP_HOME=/work/guojinyong/hadoop-0.20.2 //假设我的hadoop解压在这里 bin/hive hive> show databases;
ok搞定.
hive实战步骤
vim /tmp/guojinyong/test/mobile.py
import sys import datetime for line in sys.stdin: line = line.strip() name,num = line.split('\t') str1 = 'ddddd' str2 = 'ggggg' print '\t'.join([name,num,str1,str2])
add file /tmp/guojinyong/test/mobile.py insert into table test select transform (str) using 'python mobile.py' as (name,num,str1,str2) from test_tmp;