在线扩容datanode脚本（用fabric）

最新推荐文章于 2023-11-20 13:16:00 发布

都在未来

最新推荐文章于 2023-11-20 13:16:00 发布

阅读量399

点赞数

分类专栏： IT/大数据

本文链接：https://blog.csdn.net/weixin_39511658/article/details/78386307

版权

IT/大数据专栏收录该内容

6 篇文章 0 订阅

订阅专栏

环境说明：

在扩容之前，我的集群是三台机器，并且已经开启了的

hadoop版本：hadoop-2.7.3

192.168.40.140 hd1 NameNode

192.168.40.144 hd4 即将被扩容的datanode机器

第一步：将NameNode和datanode各自的root用户的秘钥拷贝过去，以免运行脚本时自己输入

生成公钥和私钥

ssh-keygen -t rsa

将公钥和私钥拷贝到目标主机

ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.40.140

在两台机器上都执行以上命令（需更改目标主机ip）

第二步：脚本,我将脚本命名为amplify_datanode.py

from fabric.api import run, local, env, roles
import os
import sys
import getopt

env.roledefs = {'master': ['192.168.40.140'], 'datanode': ['192.168.40.144']}
env.hosts = '192.168.40.140'
@roles('master')
def master(local_ip, hostname):
    run('echo %s %s >> /etc/hosts' % (local_ip, hostname))
    run('echo %s>>/usr/local/hadoop/hadoop-2.7.3/etc/hadoop/slaves' % hostname)
    run('scp -r /usr/jdk1.8.0_131 %s:/usr' % hostname)
    run('scp -r /usr/local/hadoop %s:/usr/local' % hostname)
def datanode(local_ip, hostname):
    local('useradd hadoop')
    local('echo "hadoop" | passwd --stdin hadoop')
    local('mkdir /home/hadoop/.ssh')
    local('chown -R hadoop:hadoop /home/hadoop')
    local('cp ~/.ssh/authorized_keys /home/hadoop/.ssh')
    local('chown -R hadoop:hadoop /usr/local/hadoop')
    local('echo input /etc/profile')
    local('echo export JAVA_HOME=/usr/jdk1.8.0_131>>/etc/profile')
    local('echo export JAVA_BIN=/usr/jdk1.8.0_131/bin>>/etc/profile')
    local('echo export PATH =$PATH:/usr/jdk1.8.0_131/bin>>/etc/profile')
    local('echo export CLASSPATH=.:/usr/jdk1.8.0_131/lib/dt.jar:/usr/jdk1.8.0_131/lib/tools.jar>>/etc/profile')
    local('echo export JAVA_HOME JAVA_BIN PATH CLASSPATH>>/etc/profile')
    local('source /etc/profile')
    local('echo write into /etc/network')
    local('echo NETWORKING = yes>/etc/sysconfig/network')
    local('echo HOSTNAME = %s>>/etc/sysconfig/network' % hostname)
    local('echo input /etc/hosts')
    local('echo 192.168.40.140 hd1>>/etc/hosts')
    local('echo 192.168.40.141 hd2>>/etc/hosts')
    local('echo 192.168.40.142 hd3>>/etc/hosts')
    local('echo %s %s>>/etc/hosts' % (local_ip, hostname))
    local('echo input .bash_profile')
    local('echo export JAVA_HOME=/usr/jdk1.8.0_131>>/home/hadoop/.bash_profile')
    local('echo export HADOOP_HOME=/usr/local/hadoop/hadoop-2.7.3>>/home/hadoop/.bash_profile')
    local('echo export PATH=$PATH:/usr/jdk1.8.0_131/bin:/usr/local/hadoop/hadoop-2.7.3/bin>>/home/hadoop/.bash_profile')
    local('echo export JAVA_HOME HADOOP_HOME PATH>>/home/hadoop/.bash_profile')
    local('source /home/hadoop/.bash_profile')
    local('/usr/local/hadoop/hadoop-2.7.3/sbin/hadoop-daemon.sh start datanode')


def do_work(local_ip,hostname):
    master(local_ip, hostname)
    datanode(local_ip, hostname)