自己架设hadoop集群的配置

自己的配置：

[@tc_12_113 hadoop]# cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
? Licensed under the Apache License, Version 2.0 (the "License");
? you may not use this file except in compliance with the License.
? You may obtain a copy of the License at

??? http://www.apache.org/licenses/LICENSE-2.0

? Unless required by applicable law or agreed to in writing, software
? distributed under the License is distributed on an "AS IS" BASIS,
? WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
? See the License for the specific language governing permissions and
? limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

?? <property>

???? <name>fs.defaultFS</name>

???? <value>hdfs://hadoopmaster:9000</value>

?? </property>

?? <property>

???? <name>io.file.buffer.size</name>

???? <value>131072</value>

?? </property>

? <property>
??? <name>Hadoop.tmp.dir</name>
??? <value>/tmp/hadoop-root</value>
? </property>

?????? <property>

?????????????? <name>hadoop.proxyuser.hduser.hosts</name>

?????????????? <value>*</value>

?????? </property>

?????? <property>

?????????????? <name>hadoop.proxyuser.hduser.groups</name>

?????????????? <value>*</value>

?????? </property>

</configuration>

[@tc_12_113 hadoop]# cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
? Licensed under the Apache License, Version 2.0 (the "License");
? you may not use this file except in compliance with the License.
? You may obtain a copy of the License at

??? http://www.apache.org/licenses/LICENSE-2.0

? Unless required by applicable law or agreed to in writing, software
? distributed under the License is distributed on an "AS IS" BASIS,
? WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
? See the License for the specific language governing permissions and
? limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

?????? <property>

??????????????? <name>dfs.namenode.secondary.http-address</name>

?????????????? <value>hadoopmaster:9001</value>

??????? </property>

???????? <property>

????????????????? <name>dfs.namenode.name.dir</name>

???????????????? <value>file:/tmp/hadoop/dfs/name</value>

??????????? </property>

?????????? <property>

??????????????????? <name>dfs.datanode.data.dir</name>

??????????????????? <value>file:/tmp/hadoop/dfs/data</value>

??????????? </property>

??????????? <property>

???????????????????? <name>dfs.replication</name>

???????????????????? <value>2</value>

???????????? </property>

???????????? <property>

???????????????????? <name>dfs.webhdfs.enabled</name>

???????????????????? <value>true</value>

???????? </property>

<property>

<name>dfs.permission</name>

<value>false</value>

</property>

</configuration>

[@tc_12_113 hadoop]# cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
? Licensed under the Apache License, Version 2.0 (the "License");
? you may not use this file except in compliance with the License.
? You may obtain a copy of the License at

??? http://www.apache.org/licenses/LICENSE-2.0

? Unless required by applicable law or agreed to in writing, software
? distributed under the License is distributed on an "AS IS" BASIS,
? WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
? See the License for the specific language governing permissions and
? limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

??????????????? <property>

??????????????????????????????? <name>mapreduce.framework.name</name>

??????????????????????????????? <value>yarn</value>

??????????????? </property>

??????????????? <property>

??????????????????????????????? <name>mapreduce.jobhistory.address</name>

??????????????????????????????? <value>hadoopmaster:10020</value>

??????????????? </property>

??????????????? <property>

?????????????? <name>mapreduce.jobhistory.webapp.address</name>

?????????????? <value>hadoopmaster:19888</value>

?????? </property>

</configuration>

[@tc_12_113 hadoop]# cat yarn-site.xml
<?xml version="1.0"?>
<!--
? Licensed under the Apache License, Version 2.0 (the "License");
? you may not use this file except in compliance with the License.
? You may obtain a copy of the License at

??? http://www.apache.org/licenses/LICENSE-2.0

? Unless required by applicable law or agreed to in writing, software
? distributed under the License is distributed on an "AS IS" BASIS,
? WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
? See the License for the specific language governing permissions and
? limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->

? <property>

?????????????? <name>yarn.nodemanager.aux-services</name>

?????????????? <value>mapreduce_shuffle</value>

??????? </property>

???????????????? <property>

?????????????? <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

?????????????? <value>org.apache.hadoop.mapred.ShuffleHandler</value>

??????? </property>

??????? <property>

?????????????? <name>yarn.resourcemanager.address</name>

?????????????? <value>hadoopmaster:8032</value>

?????? </property>

??????????????? <property>

?????????????? <name>yarn.resourcemanager.scheduler.address</name>

?????????????? <value>hadoopmaster:8030</value>

?????????????? </property>

?????????????? <property>

?????????????????????? <name>yarn.resourcemanager.resource-tracker.address</name>

??????????????????????? <value>hadoopmaster:8031</value>

?????????????? </property>

?????????????? <property>

?????????????????????? <name>yarn.resourcemanager.admin.address</name>

??????????????????????? <value>hadoopmaster:8033</value>

?????????????? </property>

??????????????? <property>

?????????????? <name>yarn.resourcemanager.webapp.address</name>

?????????????? <value>hadoopmaster:8088</value>

?????? </property>

</configuration>

[@tc_12_113 hadoop]# cat slaves
10.11.12.114
10.10.126.66

[@tc_12_113 hadoop]# cat /etc/hosts

10.11.12.114 hadoopmaster tc_12_113

10.10.126.66 hadoopslave localhost.localdomain

运行中遇到的问题基本都是host 原因……