spark、hadoop集群安装实录

规划

查看图片请看公众号

三节点

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-IEWRJIU3-1624288146884)(C:\Users\wqkant\AppData\Roaming\Typora\typora-user-images\1624100996991.png)]

vagrant 安装

版本

• vagrant:https://www.vagrantup.com/
• virtualbox:https://www.virtualbox.org/

插件

vagrant-hostmanager

Vagrantfile里配置

  config.hostmanager.enabled = true
  config.hostmanager.manage_host = true
  config.hostmanager.manage_guest = true

在定义虚拟机的时候，可以添加 hostname 来设置虚拟机的主机名：

  config.vm.define "node1" do |node1|
    node1.vm.network "private_network", ip: "192.168.10.11"
    node1.vm.hostname="node1"
  end

启动虚拟机以后会自动更新虚拟机的主机名，同时也会更新本地主机上的 hosts 文件里的内容。

或者我们也可以手工的去更新，执行命令：

vagrant hostmanager

集群配置

vagrant init后修改VagrantFile

# 2代表有两个defile
Vagrant.configure("2") do |config|
    (1..2).each do |i|
      config.vm.define "slave#{i}" do |node|
          node.vm.box = "ubuntu/xenial64"
          node.vm.hostname = "hadoop-slave#{i}"
          node.vm.network :private_network, ip: "192.168.10.1#{i}"
          node.vm.network "forwarded_port", guest: 22, host: "222#{i}", id: "ssh", disabled: "true"
          node.vm.network "forwarded_port", guest: 22, host: "222#{i+1}"
          node.vm.provider "virtualbox" do |vb|
          vb.memory = "512"
          # node.vm.provision "shell", path: "slaver_init.sh", privileged: false
        end
      end
    end
    
    config.vm.define :master1, primary: true do |master|
      master.vm.provider "virtualbox" do |v|
        v.customize ["modifyvm", :id, "--name", "hadoop-master1", "--memory", "1024"]
      end
      master.vm.box = "ubuntu/xenial64"
      master.vm.hostname = "hadoop-master1"
      # 注意：在解决多个SSH端口时，需要先禁用默认的ssh转发，再添加自定义转发，才能生效
      master.vm.network :private_network, ip: "192.168.10.10"
      master.vm.network "forwarded_port", guest: 22, host: 2219, id: "ssh", disabled: "true"
      master.vm.network "forwarded_port", guest: 22, host: 2220
      # 初始化脚本， reload会重新执行哟！！！
      master.vm.provision "shell", path: "master_init.sh", privileged: false
    end
  #manage hosts file 
  config.hostmanager.enabled = true
  config.hostmanager.manage_host = true
  config.hostmanager.manage_guest = true

end

vagrant支持以下三种网络配置：

Forwarded port(端口映射)

是指将宿主计算机的端口映射到虚拟机上的某个端口上，访问宿主计算机的该端口时，请求实际会被转发到虚拟机上指定的端口，配置文件设置语法为：