启动namenode失败,log中提示错误:
INFO org.apache.hadoop.hdfs.server.common.Storage: Cannot lock storage /opt/data/hadoop/hdfs/name. The directory is already locked.
2013-08-06 09:54:45,052 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
因为在root账户下启动了下hadoop,虽然Ctrl+C终止了,但是master上的namenode在root下启动了,这时候切换到hadoop账户下再启动hadoop会出现目录被root账户下的namenode进程锁定,导致FSNameSystem启动失败。
解决办法:切换至root,用jps查看启动的进程,kill掉就OK。
root账户下启动hadoop(前提是hadoop权限账户不是root账户),终止后切换到hadoop账户,会出现Permission Denied
2013-08-06 09:51:50,882 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
java.io.FileNotFoundException: /opt/data/hadoop/hdfs/name/in_use.lock (Permission denied)
是因为root会把文件权限更改,使用chown -R hadoop:hadoop /opt/data/hadoop/hdfs/*更改用户权限即可
由于slave没有root@master的ssh无密钥登陆,所以不会对slave机进行操作,如果有的话对slave机进行相应的操作即可
如果怕误操作,可以用如下的shell脚本启动
#!/bin/bash
username=`whoami`
echo "current user : $username"
if [ "$username" != "hadoop" ]; then
echo "hadoop need to start by user:hadoop"
#echo "please change your account and try it later"
#exit 1
su - hadoop <<!
echo "changing to hadoop:"
/opt/modules/hadoop/hadoop-1.1.2/bin/start-all.sh
exit
!
echo "back to $username"
exit 0
fi
echo "start hadoop by hadoop"
/opt/modules/hadoop/hadoop-1.1.2/bin/start-all.sh
2.Hive配置Mysql存储metadata出现权限的问题:
在配置mysql的时候,需要手动创建hive账户和配置hive密码
使用mysql -uroot -p输入密码后登陆mysql,如果首次启动mysql需要先设置密码
其设置密码有三种方法:
a. ./mysqladmin -u root -p oldpassword newpasswd(记住这个命令是在/usr/local/mysql/bin中外部命令)
b. SET PASSWORD FOR root=PASSWORD(’new password’);(对登录数据库后这种方式)
c. UPDATE user SET password=PASSWORD(”new password”) WHERE user=’root’; (对登录数据库后这种方式)
创建hive账户:
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hivepasswd';
更新权限
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' WITH GRANT OPTION;//一定要设置为其他主机也可以访问的,不然一直会启动hive后执行任何语句都会出现没有实例化之类的错误,我就在这栽了好久
hive-site.xml的配置文件
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://hive-mysql:3306/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hivepasswd</value>
<description>password to use against metastore database</description>
</property>
下载jdbc的jar包放到hive/lib下
wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.11.tar.gz/from/http://mysql.he.net/
tar -xvzf mysql-connector-java-5.1.11.tar.gz
cp mysql-connector-java-5.1.11/*.jar /data/soft/hive/lib
启动hive即可