hadoop_cluser

原创 2015年11月18日 08:29:01



修改主机名:vi /etc/sysconfig/network
修改主机名和IP的映射关系:vim /etc/hosts
java安装
卸载之前版本的java
1.卸载java:rpm -qa | grep java
  卸载:rpm -e --nodeps  查询出来的包名
1.cd /opt
tar -zxvf jdk-7u75-linux-x64.tar.gz 
2.vi /etc/profile
        export JAVA_HOME=/opt/jdk1.7.0_75
        export PATH=$PATH:$JAVA_HOME/bin
防火墙
        #查看防火墙状态
        service iptables status
        #关闭防火墙
        service iptables stop
        #查看防火墙开机启动状态
        chkconfig iptables --list
        #关闭防火墙开机启动
        chkconfig iptables off


安装zookeeper
1.安装配置zooekeeper集群
1.1解压
tar -zxvf zookeeper-3.4.3.tar.gz 
1.2修改配置
cd /opt/zookeeper-3.4.3/conf/
cp zoo_sample.cfg zoo.cfg
vim /opt/zookeeper-3.4.3/conf/zoo.cfg
修改:
dataDir=/opt/zookeeper-3.4.3/data
dataLogDir=/opt/zookeeper-3.4.3/logs
server.1=s1:2888:3888
server.2=s2:2888:3888
server.3=s3:2888:3888
保存退出
然后创建一个tmp文件夹
mkdir /opt/zookeeper-3.4.3/data
mkdir /opt/zookeeper-3.4.3/logs
再创建一个空文件
touch /opt/zookeeper-3.4.3/data/myid
最后向该文件写入IDaa
echo 1 > /opt/zookeeper-3.4.3/data/myid
1.3将配置好的zookeeper拷贝到其他节点(首先分别在hadoop2hadoop3
scp -r /opt/zookeeper-3.4.3/ hadoop2:/opt/
scp -r /opt/zookeeper-3.4.3/ hadoop3:/opt/
注意:修改hadoop1、hadoop2对应/opt/zookeeper-3.4.3/tmp/myid

hadoop2:
echo 2 > /opt/zookeeper-3.4.3/data/myid
hadoop3:
echo 3 > /opt/zookeeper-3.4.3/data/myid
    1.4启动zookeeper
 /opt/zookeeper-3.4.3/bin/zkServer.sh start
 /opt/zookeeper-3.4.3/bin/zkServer.sh status
 报错
[root@s1 bin]# /opt/zookeeper-3.4.3/bin/zkServer.sh status 
JMX enabled by default 
Using config: /opt/zookeeper-3.4.3/bin/../conf/zoo.cfg 
Error contacting service. It is probably not running.

多查看zookeeper.out得知
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /opt/zookeeper-3.4.3/bin/../conf/zoo.cfg 
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:121) 
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101) 
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) 
Caused by: java.lang.IllegalArgumentException: serverid null is not a number 
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:358) 
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:117) 
        ... 2 more
多一个节点:QuorumPeerMain

解决方法:
1.核对了myid之后,发现确实有问题,修改之,问题 依旧
2.文件夹都在的
3.修改zkServer.sh文件,问题依旧:http://blog.sina.com.cn/s/blog_72827fb101018yn9.html
4.现在查询2181端口:netstat -an |grep 2181已经启动,而且zookeeper.out里面确实没有报错信息了,问题仍没有解决,

tail -f zookeeper.out报错    
2015-02-01 19:22:52,136 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@764] - Notification time out: 60000 
2015-02-01 19:23:52,139 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@368] - Cannot open channel to 2 at election address s2/10.6.0.211:3888 
java.net.ConnectException: 拒绝连接 
at java.net.PlainSocketImpl.socketConnect(Native Method) 
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) 
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) 
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) 
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 
at java.net.Socket.connect(Socket.java:579) 
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:354) 
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:388) 
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:755) 
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:716) 
2015-02-01 19:23:52,144 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@368] - Cannot open channel to 3 at election address s3/10.6.0.86:3888 

[root@hadoopname snapshot]# echo ruok | nc hadoopName 2181

imok



每一个客户端都要启动一次zookeeper



安装hadoop

1.

 hadoop-env.sh

export JAVA_HOME=/opt/jdk1.7.0_75

export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/hadoop-2.4.1/lib/native

export HADOOP_OPTS="-Djava.library.path=/opt/hadoop-2.4.1/lib"

2.core-site.xml做如下配置:

<!-- 指定hdfs的nameservice为ns1 --> 

<property><name>fs.defaultFS</name><value>hdfs://ns1</value></property> 

<!-- 指定zookeeper地址 --> 

<property><name>ha.zookeeper.quorum</name><value>s1:2181,s2:2181,s3:2181</value></property> 

<property> 

  <name>io.compression.codecs</name> 

  <value>org.apache.hadoop.io.compress.GzipCodec, 

         org.apache.hadoop.io.compress.DefaultCodec, 

         com.hadoop.compression.lzo.LzoCodec, 

         com.hadoop.compression.lzo.LzopCodec, 

         org.apache.hadoop.io.compress.BZip2Codec</value> 

</property> 

<property><name>io.compression.codec.lzo.class</name><value>com.hadoop.compression.lzo.LzoCodec</value></property> 

<property><name>hadoop.tmp.dir</name><value>/opt/hadoop-2.4.1/tmp</value></property> 

<property><name>hadoop.proxyuser.hadoop.hosts</name><value>*</value></property> 

<property><name>hadoop.proxyuser.hadoop.groups</name><value>*</value></property> 

<property><name>hadoop.native.lib</name><value>true</value></property> 

<property><name>ha.zookeeper.session-timeout.ms</name><value>60000</value></property> 

<property><name>ha.failover-controller.cli-check.rpc-timeout.ms</name><value>60000</value></property> 

<property><name>ipc.client.connect.timeout</name><value>20000</value></property>



3.hdfs-site.xml

alias vi=vim


<configuration> 
<!--指定hdfs的nameservice为ns1,需要和core-site.xml中的保持一致 --> 
<property><name>dfs.nameservices</name><value>ns1</value></property> 
<!-- ns1下面有两个NameNode,分别是nn1,nn2 --> 
<property><name>dfs.ha.namenodes.ns1</name><value>nn1,nn2</value></property> 
<!-- nn1的RPC通信地址 --> 
<property><name>dfs.namenode.rpc-address.ns1.nn1</name><value>s1:9000</value></property> 
<!-- nn1的http通信地址 --> 
<property><name>dfs.namenode.http-address.ns1.nn1</name><value>s1:50070</value></property> 
<!-- nn2的RPC通信地址 --> 
<property><name>dfs.namenode.rpc-address.ns1.nn2</name><value>s2:9000</value></property> 
<!-- nn2的http通信地址 --> 
<property><name>dfs.namenode.http-address.ns1.nn2</name><value>s2:50070</value></property> 
<!-- 指定NameNode的元数据在JournalNode上的存放位置 --> 
<property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://s1:8485;s2:8485;s3:8485/ns1</value></property> 
<!-- 指定JournalNode在本地磁盘存放数据的位置 --> 
<property><name>dfs.journalnode.edits.dir</name><value>/opt/hadoop-2.4.1/journal</value></property> 
<!-- 开启NameNode失败自动切换 --> 
<property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property> 
<!-- 配置失败自动切换实现方式 --> 
<property><name>dfs.client.failover.proxy.provider.ns1</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property> 
<!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行--> 
<property><name>dfs.ha.fencing.methods</name> 
<value> 
sshfence 
shell(/bin/true) 
</value></property> 
<!-- 使用sshfence隔离机制时需要ssh免登陆 --> 
<property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value></property> 
<!-- 配置sshfence隔离机制超时时间 --> 
<property><name>dfs.ha.fencing.ssh.connect-timeout</name><value>30000</value></property> 
<property><name>dfs.replication</name><value>2</value></property> 
</configuration>



4.cp mapred-site.xml.template mapred-site.xml

mapred-site.xml

<configuration>
<!-- 指定mr框架为yarn方式 -->
<property><name>mapreduce.framework.name</name><value>yarn</value></property>
<property><name>mapred.map.output.compress</name><value>true</value></property>
<property><name>mapred.map.output.compress.codec</name><value>com.hadoop.compression.lzo.LzoCodec</value></property>
</configuration>


5.yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties --> 
<!-- 指定resourcemanager地址 -->
<property><name>yarn.resourcemanager.hostname</name><value>s1</value></property>
<!-- 指定nodemanager启动时加载server的方式为shuffle server -->
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>
<property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property> 
<property><name>yarn.resourcemanager.address</name><value>s1:8032</value></property> 
<property><name>yarn.log-aggregation-enable</name><value>true</value></property> 
<!-- YARN 的 HA 配置 --> 
<property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property> 
<property><name>yarn.resourcemanager.cluster-id</name><value>yarnHA</value></property>
<property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property>
<property><name>yarn.resourcemanager.hostname.rm1</name><value>s1</value></property>
<property><name>yarn.resourcemanager.hostname.rm2</name><value>s2</value></property>
<property><name>yarn.resourcemanager.zk-address</name><value>s1:2181,s2:2181,s3:2181</value></property>
</configuration> 

6.slaves
修改slaves(slaves是指定子节点的位置,因为要在zhangxin01上启动HDFS、在zhangxin03启动yarn,所以zhangxin01上的slaves文件指定的是datanode的位置,zhangxin03上的slaves文件指定的是nodemanager的位置)
s1
s2
s3

2.5启动zookeeper集群
./zkServer.sh status

2.6启动journalnode(在zhangxin01上启动所有journalnode,注意:是调用的hadoop-daemons.sh这个脚本,注意是复数s的那个脚本)
sbin/hadoop-daemons.sh start journalnode(daemons走的是ssh协议,可以同时启用多个进程)
#运行jps命令检验,s2,s3上多了JournalNode进程
2.7格式化HDFS
#在s11上执行命令:
hadoop namenode -format
#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件,这里我配置的是/zhangxin/hadoop-2.2.0/tmp,
#然后将/zhangxin/hadoop-2.2.0/tmp拷贝到zhangxin02的/zhangxin/hadoop-2.2.0/下。
scp -r tmp/ zhangxin:/zhangxin/hadoop-2.2.0/
2.8格式化ZK(在zhangxin01上执行即可)
hdfs zkfc -formatZK
zkCli.sh 
2.9启动HDFS(在zhangxin01上执行)
sbin/start-dfs.sh

2.10启动YARN(#####注意#####:是在zhangxin03上执行start-yarn.sh,把namenode和resourcemanager分开是因为性能问题,因为他们都要占用大量资源,所以把他们分开了,他们分开了就要分别在不同的机器上启动)
sbin/start-yarn.sh



到此,hadoop2.4.1配置完毕,可以统计浏览器访问:
http://10.0.8.2:50070

http://192.168.1.202:50070
验证HDFS HA
首先向hdfs上传一个文件
hadoop fs -put /etc/profile /profile
hadoop fs -ls /
然后再kill掉active的NameNode
kill -9 <pid of NN>

sbin/hadoop-daemon.sh start namenode 
验证YARN:
运行一下hadoop提供的demo中的WordCount程序:
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /profile /out

netstat -ntpl |grep 50070

hdfs namenode –bootstrapStandby

测试:hadoop jar /opt/hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount /core-site.xml /out

报错
解决方法:
安装lzo

一、安装lzop:
yum -y install lzop

二、安装lzo
1、wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz
2、tar -zxvf lzo-2.06.tar.gz
3、mv lzo-2.06 lzo && cd lzo
4、export CFLAGS=-m64
5、./configure -enable-shared

报错:

configure: error: no acceptable C compiler found in $PATH

解决:yum install gcc




6、make && make install【默认安装在了/usr/local/lib下:liblzo2.a  liblzo2.la  liblzo2.so  liblzo2.so.2  liblzo2.so.2.0.0】
7、在/etc/ld.so.conf.d/目录下新建lzo.conf文件,内容:
/usr/local/lib
8、让lzo.conf生效:/sbin/ldconfig -v

三、安装Hadoop-LZO
1、下载源码:https://github.com/twitter/hadoop-lzo
2、解压后是hadoop-lzo-master,进入hadoop-lzo-master目录
3、export CFLAGS=-m64
4、export CXXFLAGS=-m64
5、export C_INCLUDE_PATH=/usr/local/include/lzo
6、export LIBRARY_PATH=/usr/local/lib

maven下载


wget http://maven.apache.org/download.cgi/apache-maven-3.0.5-bin.tar.gz


在dev3上的安装路径:/usr/local/maven
b.解压:tar -zxvf apache-maven-3.0.5-bin.tar.gz 
c.重命名:rm -rf apache-maven-3.0.5-bin.tar.gz 
d.设置环境变量 vi /etc/profile
MAVEN_HOME=/usr/local/maven
export MAVEN_HOME
export PATH=${PATH}:${MAVEN_HOME}/bin
e.输入验证信息,查看是否安装成功:mvn -v
报错:解压
解决:yum install gzip

7、mvn clean package -Dmaven.test.skip=true【可能在hadoop-lzo-master目录下还有一个hadoop-lzo-master目录,就会抛MissingProjectException异常,找不到源码,需要进入hadoop-lzo-master/hadoop-lzo-master目录下编译,或者去掉一层hadoop-lzo-master目录】

8、在当前目录下生成了target,下面有个native/Linux-amd64-64/lib目录,将lib目录下的文件拷贝到hadoop的lib/native目录下:tar -cBf - -C target/native//Linux-amd64-64/lib . | tar -xBvf - -C /usr/local/hadoop/lib/native

cp /opt/hadoop-lzo-master/target/native/Linux-amd64-64/lib/* /opt/hadoop-2.4.1/lib/native/

cp /opt/hadoop-lzo-master/target/native/Linux-amd64-64/* /opt/hadoop-2.4.1/lib/native/




9、将target下的hadoop-lzo-xxx.jar拷贝到hadoop/lib下

cp /opt/hadoop-lzo-master/target/hadoop-lzo-0.4.20-*.jar /opt/hadoop-2.4.1/lib     

cp /opt/hadoop-lzo-master/target/hadoop-lzo-*.jar /opt/hadoop-2.4.1//share/hadoop/mapreduce/lib/

尼玛,还是不行啊

14/03/13 17:25:41 ERROR lzo.LzoCodec: Cannot load native-lzo without native-hadoop 
14/03/13 17:25:43 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file /test2.lzo, size 0.00 GB... 
Exception in thread "main" java.lang.RuntimeException: native-lzo library not available 
at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:91) 
at com.hadoop.compression.lzo.LzoIndex.createIndex(LzoIndex.java:222) 
at com.hadoop.compression.lzo.LzoIndexer.indexSingleFile(LzoIndexer.java:117) 
at com.hadoop.compression.lzo.LzoIndexer.indexInternal(LzoIndexer.java:98) 
at com.hadoop.compression.lzo.LzoIndexer.index(LzoIndexer.java:52) 
at com.hadoop.compression.lzo.LzoIndexer.main(LzoIndexer.java:137) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
十几年前,小学生吃辣条,大学生喝咖啡,十几年后,小学生吃哈根达斯,大学生吃辣条,这不是重点,重点是吃辣条的还是那群人!




Hive只在一个节点上安装即可
 1
1.长传tar包
 
2.解压S
    tar -zxvf hive-0.9.0.tar.gz -C /itcast
3.配置mysql metastore(切换到root用户)
    配置HIVE_HOME环境变量
    擦除:rpm -qa | grep mysql
    解除依赖:rpm -e mysql-libs-5.1.66-2.el6_3.i686 --nodeps
5.安装mysqlyum install -y mysql-server mysql mysql-deve
  查看mysql的版本: rpm -qi mysql-server
  启动mysql:service mysqld start
  设置mysql开机启动:chkconfig mysqld on
    root账号设置密码为 root:mysqladmin -u root password 'root'
    查看端口:
netstat -ntpl |grep 3306
    进入mysql:mysql -uroot -proot
    配置文件所在地:vi /etc/my.cnf
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root' WITH GRANT OPTION;  --修改比较复杂的用户名和密码
FLUSH PRIVILEGES;
查看user表的信息:select * from user;
创建一个新用户:create user 'hive'@'10.6.0.127' identified by 'hive'; 

进入mysql的命令:mysql -h10.6.0.127 -phive -uhive
mysql -h172.31.15.174 -phive -uhive
5.安装hive和mysq完成后,将mysql的连接jar包拷贝到$HIVE_HOME/lib目录下
    如果出现没有权限的问题,在mysql授权(在安装mysql的机器上执行)
 
4.配置hive
    cp hive-default.xml.template hive-site.xml
    修改hive-site.xml(删除所有内容,只留一个<configuration></configuration>)
    添加如下内容:/
<configuration>
    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://10.6.0.127:3306/hive?createDatabaseIfNotExist=true</value>
      <description>JDBC connect string for a JDBC metastore</description>
    </property>
    <property> 
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>Driver class name for a JDBC metastore</description>
    </property>
    <property> 
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>root</value>
      <description>username to use against metastore database</description>
    </property>
    <property> 
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>123456</value>
      <description>password to use against metastore database</description>
    </property>
</configuration>


 


测试hive

hive> SELECT * FROM pokes; 
OK 
Failed with exception java.io.IOException:java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork! 
Time taken: 3.047 seconds 

hive> 
hive
vi /opt/hadoop-2.4.1/etc/hadoop/hadoop-env.sh 添加如下内容
export  HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/lib/hadoop-lzo.jar
vi /opt/hadoop-2.4.1/etc/hadoop/mapred-site.xml
<property><name>mapreduce.framework.name</name><value>yarn</value></property> 
<property><name>mapred.map.output.compress</name><value>true</value></property> 
<property><name>mapred.map.output.compress.codec</name><value>com.hadoop.compression.lzo.LzoCodec</value></property>

测试hadoop-lzo
[root@s1 native]# hadoop fs -text /d_channel.csv.lzo 
15/02/12 18:02:51 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
15/02/12 18:02:51 WARN lzo.LzoCompressor: java.lang.NoSuchFieldError: lzoCompressLevelFunc
15/02/12 18:02:51 ERROR lzo.LzoCodec: Failed to load/initialize native-lzo library
-text: Fatal internal error
java.lang.RuntimeException: native-lzo library not available

        at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:104)
        at com.hadoop.compression.lzo.LzopCodec.createInputStream(LzopCodec.java:89)
        at org.apache.hadoop.fs.shell.Display$Text.getInputStream(Display.java:150)
        at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:98)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
        at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)




启动(必须所有的先ssh通之后可以启动起来,不然不同)

 /opt/zookeeper-3.4.3/bin/zkServer.sh start (s1,s2,s3)
 /opt/zookeeper-3.4.3/bin/zkServer.sh status
/opt/hadoop-2.4.1/sbin/hadoop-daemons.sh start journalnode (s1)
/opt/hadoop-2.4.1/sbin/start-dfs.sh  (s1)
/opt/hadoop-2.4.1/sbin/start-yarn.sh  (s1)
--bwlimit=1500kb

停止
 /opt/zookeeper-3.4.3/bin/zkServer.sh stop(s1,s2,s3)
 /opt/zookeeper-3.4.3/bin/zkServer.sh status
/opt/hadoop-2.4.1/sbin/stop-all.sh
sbin/hadoop-daemons.sh stop journalnode (s1)
/opt/hadoop-2.4.1/sbin/stop-all.sh   (s1)
sbin/stop-yarn.sh  (s1)

查看端口:netstat -ntpl 
访问hdfs的状态信息:10.6.0.127:50070
查看yarn-job的执行状态:http://10.6.0.127:8088/cluster

异常
15/03/19 20:11:04 WARN retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo over s2/10.0.8.3:9000. Not retrying because failovers (15) exceeded maximum allowed (15) 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby 
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) 
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1639) 
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1193) 
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3514) 
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:779)


hive权限问题解决了,然后就是这个异常了

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes  

解决方法:http://blog.csdn.net/wind520/article/details/39890967
mysql -h10.6.0.127 -uroot -p123456 进入hive
SET character_set_client = utf8 ;  
SET character_set_connection = utf8 ;  
SET character_set_database = utf8 ;  
SET character_set_results = utf8 ;  
SET character_set_server = utf8 ;  
SET collation_connection = utf8 ;  
SET collation_database = utf8 ;  
 SET collation_server = utf8 ;  
SET NAMES 'utf8';  

SHOW VARIABLES LIKE 'character%';  

退出mysql


hive --hiveconf hive.root.logger=DEBUG,console 

错误依旧




spark 部署

1.tar -zxvf spark-1.1.0-bin-hadoop2.4.tgz
2.mv spark-1.1.0-bin-hadoop2.4 spark
3.环境变量:export SPARK_HOME=/opt/spark (s1,s2,s3) 
4.cd /opt/spark/conf
   cp spark-env.sh.template spark-env.sh   
   vi spark-env.sh   

export SPARK_MASTER_IP=s1 
export SPARK_MASTER_PORT=7077 
export SPARK_WORKER_CORES=1 
export SPARK_WORKER_MEMORY=1g 
export SPARK_WORKER_INSTANCES=1 
export SPARK_WORKER_DIR=/opt/spark


5.启动spark
1)/opt/spark/bin/spark-shell --master spark://s1:7077

2)./spark-shell master=spark://s1:7077 
3)/opt/spark/bin/spark-shell --master spark://s1:7077,s2:7077,s3:7077


解决方式:
1./etc/hosts  。del 127.0.0.1 xxx 
2.关闭防火墙
root#service iptables stop
root#service ip6tables stop
root#chkconfig iptables off
root#chkconfig ip6tables off
3.无密码登陆 ssh    ssh s1 date  
4.master小写

cd /usr/local/spark-1.1.0-bin-hadoop2.42 bin/run-example SparkPi


   



版权声明:本文为博主原创文章,未经博主允许不得转载。

相关文章推荐

jee webserver cluser web 集群

  • 2012-03-24 16:24
  • 2.32MB
  • 下载

quartz cluser

项目中使用Quartz集群分享 在公司分享了Quartz,发布出来,希望大家讨论补充.  CRM使用Quartz集群分享  一:CRM对定时任务的依赖与问题  二:什么...

hadoop权威指南第四版

  • 2017-10-11 19:07
  • 11.77MB
  • 下载

Hadoop源代码分析

  • 2017-10-12 00:07
  • 5.90MB
  • 下载

Hadoop安装时遇到的一些小问题

这篇是用来记录 Hadoop 安装记录时遇到的问题的,我在 ubuntu16.04、14.02 上都折腾过 2.7 和 3.0,遇到的问题和解决方法大同小异。具体配置流程整理一下后单独写一篇文章。

Hadoop 2.7.2 最小集群搭建

猫哥根据自身经验和踩过的坑,教你一步一步搭建一个原生态hadoop最小化集群(1 master,2 slave)。

hadoop-eclipse-plugin-2.7.4

  • 2017-10-13 16:48
  • 33.59MB
  • 下载

hadoop/bin文件

  • 2017-10-12 11:39
  • 1023KB
  • 下载
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:深度学习:神经网络中的前向传播和反向传播算法推导
举报原因:
原因补充:

(最多只允许输入30个字)