Ansible实现自动decommission Hadoop node

介绍

如何用ansible的playbook实现自动decommission hadoop 的node ?

 

正文

1. 对于如何部署安装ansible ，请看http://haohaozhang.blog.51cto.com/9176600/1610499

2. hadoop部署，请看http://haohaozhang.blog.51cto.com/9176600/1607089

3. 使用hadoop用户创建目录ansible_decom,里面的目录结构如下：

ansible_decom/
ansible_decom/roles
ansible_decom/roles/nn
ansible_decom/roles/nn/tasks
ansible_decom/roles/nn/tasks/main.yml
ansible_decom/roles/decom
ansible_decom/roles/decom/tasks
ansible_decom/roles/decom/tasks/main.yml
ansible_decom/roles/rm
ansible_decom/roles/rm/tasks
ansible_decom/roles/rm/tasks/main.yml
ansible_decom/roles/get-status
ansible_decom/roles/get-status/tasks
ansible_decom/roles/get-status/tasks/main.yml
ansible_decom/group_vars
ansible_decom/group_vars/all
ansible_decom/exclude-iplist
ansible_decom/site.yml
ansible_decom/hosts

以yml结尾的是playbook文件，其中site.yml是主文件，他会调用roles下面的子yml文件。

hosts文件定义的是主机列表。

[rmservers]
10.9.214.151
10.9.214.15
10.9.214.105
10.9.214.113
[nnservers]
10.9.214.151
10.9.214.15
10.9.214.105
10.9.214.113
[client]
10.9.214.149
[decomnodes]
10.9.214.149
10.9.214.160

exclude-iplist定义的是要decommission的主机列表。

10.9.214.160
10.9.214.149

site.yml

---
# this playbook displays decommission nodes 
- name: add node to mapred-exclude and refreshnode
  hosts: rmservers
  remote_user: hadoop
  roles: 
    - rm
- name: add node to hdfs-exclude and refreshnodes
  hosts: nnservers
  remote_user: hadoop
  roles:
    - nn
- name: stop nodemanager and datanode service
  hosts: decomnodes
  remote_user: hadoop
  roles:
    - decom
- name: get status for hosts
  hosts: client
  roles:
    - get-status

roles/decom/tasks/main.yml

---
# stop services
- name: stop nodemanager
  command: /opt/hadoop-2.6.0/sbin/yarn-daemon.sh stop nodemanager
- name: stop datanode
  command: /opt/hadoop-2.6.0/sbin/hadoop-daemon.sh stop datanode

roles/nn/tasks/main.yml

---
# step1. add decom nodes to hdfs-excluse file
# step2. refresh nodes 
- name: add decom nodes to hdfs-exclude file
  copy: src=exclude-iplist dest=/opt/hadoop-2.6.0/etc/hadoop/
- shell: cd /opt/hadoop-2.6.0/etc/hadoop/ ; cat exclude-iplist >> hdfs-exclude ; rm exclude-iplist
- name: refresh nodes
  shell: /opt/hadoop-2.6.0/bin/hdfs dfsadmin -refreshNodes

roles/rm/tasks/main.yml

---
# step1. add decom nodes to mapred-excluse file
# step2. refresh nodes 
- name: add decom nodes to mapred-exclude file
  copy: src=exclude-iplist dest=/opt/hadoop-2.6.0/etc/hadoop/
  
- shell: cd /opt/hadoop-2.6.0/etc/hadoop/ ; cat exclude-iplist >> mapred-exclude ; rm exclude-iplist
- name: refresh nodes
  shell: /opt/hadoop-2.6.0/bin/yarn rmadmin -refreshNodes

 

4. 执行ansible的playbook

asible-playbook -i hosts site.yml

执行若有失败会报错的。

5. 完成之后查看http://10.9.214.151:50070看Live Node和Dead Node的数目和个数是不是变了。