Nomad 多job/group/task调度测试

最新推荐文章于 2024-06-16 09:44:10 发布

之江小林

最新推荐文章于 2024-06-16 09:44:10 发布

阅读量955

点赞数 3

分类专栏： Nomad 文章标签： docker

本文链接：https://blog.csdn.net/weixin_43739110/article/details/119995289

版权

Nomad 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

Nomad 多job/group/task调度测试

关系：job 包含 group 包含 task

1 搭建nomad集群

本测试使用三台ubuntu18.04虚拟机，IP地址分别为：

虚拟机1：192.168.60.10
虚拟机2：192.168.60.11
虚拟机3：192.168.60.12

具体搭建方法见Nomad集群自身高可用测试。

2 测试driver=docker

使用的docker镜像为nginx的本地拷贝镜像nginx:v1，见Nomad Nginx 暴露IP端口和重启/重调度验证。

2.1 多job测试

将以下job同时执行四次nomad job run nginx[1-4].nomad，注意每次都要将job、group、task名字修改：

job "nginx1" {
  datacenters = ["dc1"]
  type = "service"
  group "nginxg1" {
    count = 1
    network {
      port "nginxport" {
        static = 8765
        to = 80
      }
    }
    task "nginxt1" {
      driver = "docker"
      config {
        image = "nginx:v1"
        ports = ["nginxport"]
      }
    }
  }
}

static = 8765这个端口号需要不同，否则肯定会分配到不同的虚拟机上，避免端口被占用。

再执行nomad job status如下：

ubuntu1@ubuntu1$ nomad job status          
ID      Type     Priority  Status   Submit Date
nginx1  service  50        running  2021-08-30T13:35:02+08:00
nginx2  service  50        running  2021-08-30T13:35:16+08:00
nginx3  service  50        running  2021-08-30T13:36:17+08:00
nginx4  service  50        running  2021-08-30T13:36:34+08:00

在每个虚拟机中docker ps查看结果，虚拟机2运行了4个nginx。

如果我在第一个job里规定只能在虚拟机1运行，后面三个也都会在虚拟机1运行。

这个机制我也不太明白，可能是这些job还不够占资源。

停止job：

nomad job stop nginx1
nomad job stop nginx2
nomad job stop nginx3
nomad job stop nginx4

2.2 多group测试

执行以下job文件nomad job run nginx.nomad，1个job包含了4个group，每个group有1个task：

job "nginx" {
  datacenters = ["dc1"]

  group "nginxg1" {
    count = 1
    network {
      port "nginxport" {
        static = 8765
        to = 80
      }
    }
    task "nginxt1" {
      driver = "docker"
      config {
        image = "nginx:v2"
        ports = ["nginxport"]
      }
    }
  }

  group "nginxg2" {
    count = 1
    network {
      port "nginxport" {
        static = 8764
        to = 80
      }
    }
    task "nginxt2" {
      driver = "docker"
      config {
        image = "nginx:v2"
        ports = ["nginxport"]
      }
    }
  }

  group "nginxg3" {
    count = 1
    network {
      port "nginxport" {
        static = 8763
        to = 80
      }
    }
    task "nginxt3" {
      driver = "docker"
      config {
        image = "nginx:v2"
        ports = ["nginxport"]
      }
    }
  }

  group "nginxg4" {
    count = 1
    network {
      port "nginxport" {
        static = 8762
        to = 80
      }
    }
    task "nginxt4" {
      driver = "docker"
      config {
        image = "nginx:v2"
        ports = ["nginxport"]
      }
    }
  }
}

结果与2.1 多job测试相同。

停止job：nomad job stop nginx

2.3 多task测试

执行以下job文件nomad job run nginx.nomad，1个job包含了1个group，每个group有4个task：

job "nginx" {
  datacenters = ["home"]

  group "nginxg" {
    count = 1
    network {
      port "nginxport1" {
        static = 8765
        to = 80
      }

      port "nginxport2" {
        static = 8764
        to = 80
      }

      port "nginxport3" {
        static = 8763
        to = 80
      }

      port "nginxport4" {
        static = 8762
        to = 80
      }
    }

    task "nginxt1" {
      driver = "docker"
      config {
        image = "nginx:v2"
        ports = ["nginxport1"]
      }
    }

    task "nginxt2" {
      driver = "docker"
      config {
        image = "nginx:v2"
        ports = ["nginxport2"]
      }
    }

    task "nginxt3" {
      driver = "docker"
      config {
        image = "nginx:v2"
        ports = ["nginxport3"]
      }
    }

    task "nginxt4" {
      driver = "docker"
      config {
        image = "nginx:v2"
        ports = ["nginxport4"]
      }
    }

  }
}

运行后发现，虚拟机2上运行了4个nginx：

ubuntu2@ubuntu2:~$ sudo docker ps 
CONTAINER ID   IMAGE      COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
4be8151eaf2a   nginx:v1   "/docker-entrypoint.…"   20 seconds ago   Up 18 seconds   192.168.60.11:8762->80/tcp, 192.168.60.11:8762->80/udp   nginxt4-bd4038a2-a662-eb24-e8e5-f83906a2ece6
b090014faf2e   nginx:v1   "/docker-entrypoint.…"   20 seconds ago   Up 18 seconds   192.168.60.11:8764->80/tcp, 192.168.60.11:8764->80/udp   nginxt2-bd4038a2-a662-eb24-e8e5-f83906a2ece6
fc3be23df0ea   nginx:v1   "/docker-entrypoint.…"   20 seconds ago   Up 18 seconds   192.168.60.11:8765->80/tcp, 192.168.60.11:8765->80/udp   nginxt1-bd4038a2-a662-eb24-e8e5-f83906a2ece6
b2670ffe0f10   nginx:v1   "/docker-entrypoint.…"   20 seconds ago   Up 18 seconds   192.168.60.11:8763->80/tcp, 192.168.60.11:8763->80/udp   nginxt3-bd4038a2-a662-eb24-e8e5-f83906a2ece6

说明在driver=docker（type=service）时，1个group内的所有task都会被分配到同一个client上。

3 测试driver=raw_exec

编写简单程序main.c：

#include <stdio.h>
int main()
{
	while(1){sleep(10);}
	return 0;
}

gcc main.c -o main.out编译。

sudo cp main.out /usr/将main.out拷贝到/usr下面。

在/etc/nomad.d/nomad.hcl文件中添加以下内容：

plugin "raw_exec" {
  config {
    enabled = true
  }
}

三个虚拟机都要完成以上操作。

3.1 多job测试

将以下job同时执行4次nomad job run exec[1-4].nomad，注意每次都要将job、group、task名字修改：

job "exec1" {
  datacenters = ["dc1"]
  type = "batch"
  
  group "execg1" {
    count = 1
    task "exect1" {
       driver = "raw_exec"
       config {
          command = "/usr/main.out"
       }
    }
  }
}

再执行nomad job status如下：

ubuntu1@ubuntu1$ nomad job status          
ID      Type     Priority  Status          Submit Date
exec1   batch    50        running         2021-08-31T11:11:09+08:00
exec2   batch    50        running         2021-08-31T11:11:13+08:00
exec3   batch    50        running         2021-08-31T11:11:16+08:00
exec4   batch    50        running         2021-08-31T11:11:19+08:00

发现在ubuntu2和ubuntu3下分别都运行了2个main.out进程，ubuntu1没有。

说明在driver=raw_exec（type=batch）时，1个client可以运行多个job，且并不会优先分配给没有job的client。

停止job：

nomad job stop exec1
nomad job stop exec2
nomad job stop exec3
nomad job stop exec4

3.2 多group测试

执行以下job文件nomad job run exec.nomad，1个job包含了4个group，每个group有1个task：

job "exec" {
    datacenters = ["dc1"]
    type = "batch"

    group "execg1" {
        count = 1
        task "exect1" {
            driver = "raw_exec"
            config {
                command = "/usr/main.out"
            }
        }
    }

    group "execg2" {
        count = 1
        task "exect2" {
            driver = "raw_exec"
            config {
                command = "/usr/main.out"
            }
        }
    }

    group "execg3" {
        count = 1
        task "exect3" {
            driver = "raw_exec"
            config {
                command = "/usr/main.out"
            }
        }
    }

    group "execg4" {
        count = 1
        task "exect4" {
            driver = "raw_exec"
            config {
                command = "/usr/main.out"
            }
        }
    }
}

也可以只有1个group，但是count=4，效果相同

结果ubuntu1运行了1个main.out进程，ubuntu3运行了3个，ubuntu2没有。

说明在driver=raw_exec（type=batch）时，1个client可以运行多个group，且并不会优先分配给没有group的client。

停止job：nomad job stop exec

3.3 多task测试

执行以下job文件nomad job run exec.nomad，1个job包含了1个group，每个group有4个task：

job "exec" {
    datacenters = ["dc1"]
    type = "batch"

    group "execg" {
        count = 1

        task "exect1" {
            driver = "raw_exec"
            config {
                command = "/usr/main.out"
            }
        }

        task "exect2" {
            driver = "raw_exec"
            config {
                command = "/usr/main.out"
            }
        }

        task "exect3" {
            driver = "raw_exec"
            config {
                command = "/usr/main.out"
            }
        }

        task "exect4" {
            driver = "raw_exec"
            config {
                command = "/usr/main.out"
            }
        }
    }
}