Hadoop是一个由Apache基金会所开发的分布式系统基础架构,主要用于存储和计算。
Hadoop安装包下载
到Hadoop官网下载对应版本的安装包,本文以hadoop 2.7.5为例。
将下载好的hadoop-2.7.5.tar.gz上传到/opt/ansible/roles/files目录下,本文将所有tar安装包都进行了重命名操作,比如将hadoop-2.7.5.tar.gz解压后重命名为hadoop并打成hadoop.tar.gz包。
编写主机hosts
编辑/opt/ansible/hosts文件添加部署Zookeeper机器IP
[hadoop_servers]
172.18.18.120 namenode_active=true namenode_standby=false datanode=true
172.18.18.121 namenode_active=false namenode_standby=true datanode=true
172.18.18.122 namenode_active=false namenode_standby=false datanode=true
编写Hadoop配置文件
将hadoop-2.7.5.tar.gz解压后的配置文件hadoop-env.sh、core-site.xml、hdfs-site.xml及slaves修改成自己的配置并重命名为hadoop-env.sh.j2、core-site.xml.j2、hdfs-site.xml.j2及slaves.hadoop.j2,然后上传到/opt/ansible/roles/templates/ 目录下。由于配置文件较多,本文不进行内容展示。
编写参数配置main.yml
编辑/opt/ansible/roles/vars/main.yml文件,添加变量
BigdataDir: /opt/hzgc
AnsibleDir: /opt/ansible
编写执行文件hadoop.yml
编辑新建/opt/ansible/yml/hadoop.yml文件,添加内容如下:
- hosts: hadoop_servers remote_user: root roles: - roles tasks: - name: install dependency package yum: name={{ item }} state=present with_items: - openssh - rsync - name: delete {{BigdataDir}}/hadoop file: path={{BigdataDir}}/hadoop/ state=absent - name: copy and unzip hadoop unarchive: src={{AnsibleDir}}/roles/files/hadoop.tar.gz dest={{BigdataDir}} - name: create hadoop logs directory file: dest={{BigdataDir}}/hadoop/logs mode=0775 state=directory - name: install configuration file hadoop-env.sh.j2 for hadoop template: src={{AnsibleDir}}/roles/templates/hadoop-env.sh.j2 dest={{BigdataDir}}/hadoop/etc/hadoop/hadoop-env.sh - name: install configuration file core-site.xml.j2 for hadoop template: src={{AnsibleDir}}/roles/templates/core-site.xml.j2 dest={{BigdataDir}}/hadoop/etc/hadoop/core-site.xml - name: install configuration file hdfs-site.xml.j2 for hadoop template: src={{AnsibleDir}}/roles/templates/hdfs-site.xml.j2 dest={{BigdataDir}}/hadoop/etc/hadoop/hdfs-site.xml - name: install configuration file slaves.j2 for hadoop template: src={{AnsibleDir}}/roles/templates/slaves.hadoop.j2 dest={{BigdataDir}}/hadoop/etc/hadoop/slaves - name: change shell sbin file file: dest={{BigdataDir}}/hadoop/sbin mode=0755 recurse=yes - name: change shell bin file file: dest={{BigdataDir}}/hadoop/bin mode=0755 recurse=yes - name: start journalnode shell: sh {{BigdataDir}}/hadoop/sbin/hadoop-daemon.sh start journalnode become: true become_method: su become_user: root when: datanode == "true" - name: format active namenode hdfs shell: sh {{BigdataDir}}/hadoop/bin/hdfs namenode -format -force become: true become_method: su become_user: root when: namenode_active == "true" - name: start active namenode hdfs shell: sh {{BigdataDir}}/hadoop/sbin/hadoop-daemon.sh start namenode become: true become_method: su become_user: root when: namenode_active == "true" - name: format standby namenode hdfs shell: sh {{BigdataDir}}/hadoop/bin/hdfs namenode -bootstrapStandby -force become: true become_method: su become_user: root when: namenode_standby == "true" - name: stop active namenode hdfs shell: sh {{BigdataDir}}/hadoop/sbin/hadoop-daemon.sh stop namenode become: true become_method: su become_user: root when: namenode_active == "true" - name: format ZKFC shell: sh {{BigdataDir}}/hadoop/bin/hdfs zkfc -formatZK -force become: true become_method: su become_user: root when: namenode_active == "true" - name: stop journalnode shell: sh {{BigdataDir}}/hadoop/sbin/hadoop-daemon.sh stop journalnode become: true become_method: su become_user: root when: datanode == "true" - name: start namenode shell: sh {{BigdataDir}}/hadoop/sbin/start-dfs.sh become: true become_method: su become_user: root when: namenode_active == "true" tags: - start hadoop |
执行Hadoop安装部署 及启动
ansible-playbook -i hosts yml/hadoop.yml
查看是否安装成功:jps
注:因为hadoop依赖zookeeper通信,所以在安装启动hadoop的过程中要保证zookeeper正常启动,zookeeper的自动化部署参考Zookeeper自动化安装
文章目录: