Bash 实现MR

继承上一篇: "Python 实现MR", 此文将采用bash实现MR, 测试数据和执行方式完全一样，这里指给出mapper和reducer的bash版本

1. mapper

[root@hadoop ~]# cat mapper.sh
#!/bin/bash

while  read line
do
air=$(echo ${line}|cut -d';' -f1)
airname=$(echo ${air}|cut -d':' -f1)
airnum=$(echo ${air}|cut -d':' -f2)
hotel=$(echo ${line}|cut -d';' -f2)
hotelname=$(echo ${hotel}|cut -d':' -f1)
hotelnum=$(echo ${hotel}|cut -d':' -f2)
echo "${airname} ${airnum}"
echo "${hotelname} ${hotelnum}"
done

2. reducer

[root@hadoop ~]# cat reducer.sh
#!/bin/bash

airnum=0
hotelnum=0

while  read line
do
 name=$(echo ${line}|cut -d' ' -f1)
 num=$(echo ${line}|cut -d' ' -f2)
 if [[ ${name} == "air" ]]; then
  ((airnum=${airnum}+${num}))
 else
  ((hotelnum=${hotelnum}+${num}))
 fi
done

echo "air ${airnum}"
echo "hotel ${hotelnum}"

3.执行

[root@hadoop ~]# hadoop jar /usr/hadoop/hadoop-1.2.1/contrib/streaming/hadoop-streaming-1.2.1.jar -mapper /root/mapper.sh -reducer /root/reducer.sh -input /usr/egencia/tour -output /usr/egencia/tour/out

4.验证：

[root@hadoop ~]# hadoop fs -cat /usr/egencia/tour/out/part-00000
/usr/hadoop/hadoop-1.2.1/libexec/../conf/hadoop-env.sh: line 59: export: `mapred.tasktracker.reduce.tasks.maximum=4': not a valid identifier
Warning: $HADOOP_HOME is deprecated.

air 236
hotel 324

得出的结果和上个文章得出的结果是一样的