mysql hadoop applier_mysql hadoop applier install and configure

#./happlier

a error is

[root@dmining05 mysql2hdfs]# ./happlier

The default data warehouse directory in HDFS will be set to /usr/hive/warehouse

Change the default data warehouse directory? (Y or N) n

Enter either Y or N:N

loadFileSystems error:

(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)

hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:

(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)

Couldnot connect to HDFS file system

the error may be due to the following cause

http://stackoverflow.com/questions/21064140/hadoop-c-hdfs-test-running-exception

#echo $CLASSPATH

/root/hadoop-2.6.0/etc/hadoop:

/root/hadoop-2.6.0/share/hadoop/common/lib/*:

/root/hadoop-2.6.0/share/hadoop/common/*:

/root/hadoop-2.6.0/share/hadoop/hdfs:

/root/hadoop-2.6.0/share/hadoop/hdfs/lib/*:

/root/hadoop-2.6.0/share/hadoop/hdfs/*:

/root/hadoop-2.6.0/share/hadoop/yarn/lib/*:

/root/hadoop-2.6.0/share/hadoop/yarn/*:

/root/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:

/root/hadoop-2.6.0/share/hadoop/mapreduce/*:

/root/hadoop-2.6.0/contrib/capacity-scheduler/*.jar

For hadoop versions 2.0.0 and above, the classpath doesn't support wild characters. If you add the jars explicitly to the CLASSPATH, your app will work.

Can JNI be made to honour wildcard expansion in the classpath?

Ｏne way is to build a file name setclass.sh

#!/bin/bash

hcp=$(hadoop classpath)

#echo $hcp

arr=(${hcp//:/ })

len=${#arr[@]}

let len-=1

echo $len

j=0

export CLASSPATH=/etc/hadoop/conf

for i in ${arr[@]}

do

# echo $i

if [ $j -eq 0 ]; then

export CLASSPATH=$i

elif [ $j -eq $len ]; then

echo $i

else

export CLASSPATH=$CLASSPATH:$i

fi

let j+=1

done

then source it, not to execute it

#source setclass.sh

#echo $CLASSPATH

This work well in current session, if want to make it work gloably,

#$echo $CLASSPATH >> ~/.bashrc

#source  ~/.bashrc

open another consele to check it .

mysql configuration in /etc/my.cnf likes

[mysqld]

basedir = /usr/local/mysql

datadir = /var/lib/mysql

port = 3306

#socket = /var/lib/mysql/mysql.sock

user=mysql

bind_address = ::

#bin log conf

log_bin = masterbin_log

binlog_checksum = NONE

binlog_format = ROW

server-id = 2

#./happlier mysql://root@127.0.0.1:3306 hdfs://localhost:9000

the msg is

the right way is

#./happlier mysql://root:123456@127.0.0.1:3306 hdfs://localhost:9000

(you should provide password to access mysql)

but this warning still exists

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Refrences

http://innovating-technology.blogspot.com/2013/04/mysql-hadoop-applier-part-1.html

http://innovating-technology.blogspot.com/2013/04/mysql-hadoop-applier-part-2.html

http://paddy-w.iteye.com/blog/2023656

http://www.tuicool.com/articles/NfArA3i