这一篇由Naveen K T撰写的客座博客,并带有原始博客资源。
我最近为一个客户做了关于评估Nomad的PoC工作。由于存在一些约束限制了客户环境潜在的能力,因此我在个人工作站上整合了一些“便捷”的东西,来弄清楚Alluxio与Nomad完美运行所需条件。
启动和运行Nomad非常简单快速:下载压缩二进制文件,解压并在开发模式启动Nomada代理。完成!事实证明在Nomad上运行Alluxio比我想像有点更复杂。我在实践中很早就遇到的一个主要问题,是Momad本身还不支持持久化存储(预期在下个版本推出)。
自从我使用Docker部署Alluxio以来,用Docker卷插件来集成存储系统似乎是个可行选择,并且经过一些网上调研后,我选择了GlusterFS。我使用VMWare Fusion,最终创建了7个Linux虚拟机,设置了静态IP,但这些设置在Virtualbox中也应该可行。
GlusterFS集群
我在Fedora 30 VM上架设了一个3节点GlusterFS集群,由于这是一个沙盒环境,我给操作系统和存储使用了单个磁盘。我的3节点集群如下:
Storage Node 1:
- Hostname: http://gfs1.ddl.io
- IP address: 172.16.4.11
- Mount point: /gfsvolume/gv0
Storage Node 2:
- Hostname: http://gfs2.ddl.io
- IP address: 172.16.4.12
- Mount point: /gfsvolume/gv0
Storage Node 3:
- Hostname: http://gfs1.ddl.io
- IP address: 172.16.4.13
- Mount point: /gfsvolume/gv0
安装和配置GlusterFS
安装时,在每个节点上打开终端,在命令提示符处键入以下内容:
$ sudo dnf update
$ sudo dnf install glusterfs-server
$ sudo systemctl enable glusterd
$ sudo systemctl start glusterd
现在已经安装了GlusterFS,我们将配置存储卷。在节点1,终端提示符处键入以下内容来创建可信存储池。
$ sudo gluster peer probe 172.16.4.12
$ sudo gluster peer probe 172.16.4.13
接下来,创建分布式GlusterFS卷并启动。
$ sudo gluster volume create nomadvolume transport \
tcp 172.16.4.11:/gfsvolume/gv0 172.16.4.12:/gfsvolume/gv0 \
172.16.4.13:/gfsvolume/gv0
$ sudo gluster volume start nomadvolume
有关GlusterFS的其他信息,可查看GlusterFS文档。
Nomad集群
Nomad集群由1个服务器和3个客户端节点组成。每个客户端节点还运行一个Consul代理和Docker,并且Consul服务器运行在Nomad服务器节点。我们将使用Consul作为Docker覆盖网络的KV存储。
Nomad Server:
- Hostname: http://ns1.ddl.io
- IP address: 172.16.4.21
Nomad Client Node 1:
-
Hostname: http://nc1.ddl.io
-
IP address: 172.16.4.22
Nomad Client Node 2:
-
Hostname: http://nc2.ddl.io
-
IP address: 172.16.4.23
Nomad Client Node 3:
-
Hostname: http://nc3.ddl.io
-
IP address: 172.16.4.24
安装Consul
为安装Consul服务器(172.16.4.21),解压二进制文件到/usr/local/consul,并将路径添加到环境变量。Consul启动可通过在终端提示符下运行以下命令:
$ consul agent -ui -dev -bind="172.16.4.21" -client="0.0.0.0"
接下来,我们将安装Consul代理。我们将在每个Nomad客户端安装代理。为安装和启动代理,在每个Nomad客户端节点执行以下操作。
-
解压Consul二进制文件到/usr/local/consul,并将路径加入环境变量。
-
在终端提示符,键入以下内容:
$ sudo mkdir /etc/consul.d
$ sudo tee /etc/consul.d/consul.hcl > /dev/null <<EOF
datacenter = "dc1"
data_dir = "/opt/consul"
performance {raft_multiplier = 1}
retry_join = ["172.16.4.21"]
client_addr = "172.16.4.22"
bind_addr = "{{ GetInterfaceIP \"ens33\" }}"
log_level = "TRACE"
enable_syslog = true
EOF
将client_addr更改为运行中客户端节点的地址。
为启动Consul客户端,执行以下命令
$ sudo env PATH=$PATH consul agent -config-dir=/etc/consul.d
其他信息,请参考Consul文档。
安装Nomad
安装Nomad服务器(172.16.4.21),解压二进制文件到/usr/local/nomad,并将路径添加到环境变量。
配置服务器,在终端提示符执行以下命令:
$ sudo mkdir /etc/nomad.d
$ sudo tee /etc/nomad.d/server.hcl > /dev/null <<EOF
enable_syslog = true
log_level = "DEBUG"
bind_addr = "0.0.0.0"
datacenter = "dc1"
# Setup data dir
data_dir = "/home/pocuser/nomad/server"
advertise {
# This should be the IP of THIS MACHINE and must be routable by every node
# in your cluster
rpc = "172.16.4.21:4647"
}
# Enable the server
server {
enabled = true
# Self-elect, should be 3 or 5 for production
bootstrap_expect = 1
}
EOF
执行以下命令,启动Nomad服务器:
$ sudo env PATH=$PATH nomad agent -config=/etc/nomad.d
安装和启动Nomad客户端,在每个Nomad客户端节点执行以下操作:
- 解压Nomad二进制文件到/usr/local/nomad,将路径加入环境变量。
- 配置客户端,在终端提示符执行以下命令:
$ sudo mkdir /etc/nomad.d
$ sudo tee /etc/nomad.d/client.hcl > /dev/null <<EOF
data_dir = "/home/ntogar/nomad/data"
bind_addr = "0.0.0.0"
ports {
http = 4646
rpc = 4647
serf = 4648
}
client {
enabled = true
network_speed = 10
servers = ["172.16.4.21:4647"]
options {
"docker.privileged.enabled" = "true"
"driver.raw_exec.enable" = "1"
}
}
consul {
address = "172.16.4.22:8500"
}
EOF
Nomad客户端应始终连接到Consul客户端,不可直接连到服务器。将IP改为在客户端节点上运行的Consul代理。
启动Nomad客户端
$ sudo env PATH=$PATH nomad agent -config=/etc/nomad.d
有关其他信息,请参阅Nomad文档。
Alluxio
在Nomad上运行Alluxio需要一些准备工作。
1.在容器内创建Docker卷来挂载GlusterGS分布式存储。
2.配置Alluxio,创建Docker镜像
3.设置本地Docker注册表
4.配置Docker以使用Consul作为KV存储
5.创建Docker覆盖网络Docker卷
由于Nomad本身不支持持久性存储,我们将为GlusterFS安装Docker卷插件。安装和创建卷,运行以下命令:
$ sudo docker plugin install --alias glusterfs mikebarkmin/glusterfs
$ sudo docker volume create -d glusterfs -o \
servers=172.16.4.11,172.16.4.12,172.16.4.13 \
-o volname=nomadvolume nomadvolume
配置Alluxio
然后我们将配置Alluxio以使用分布式存储。下载Alluxio,解压至~/alluxio。配置Alluxio属性。
$ cd ~/alluxio/integration/docker
$ cp conf/alluxio-site.properties.template \
conf/alluxio-site.properties
编辑conf/alluxio-site.properties,添加以下行:
alluxio.master.mount.table.root.ufs=/mnt/gfsvol.
接下来,我们将创建Docker镜像
$ sudo docker build -t alluxio-docker .
Nomad在运行时从注册表下载镜像,因此我们将设置本地Docker注册表并将Alluxio镜像推送到注册表。
更改Docker
Docker需配置为使用Consul作为存储以使覆盖网络生效。首先在Nomad客户端节点创建/etc/docker/daemon.json,并粘贴以下内容。
{
“insecure-registries”: [“172.16.4.21:5000”],
“debug”: false,
“cluster-store”: “consul://172.16.4.21:8500”,
“cluster-advertise”: “172.16.4.22:0”
}
重启Docker守护程序
$ sudo systemctl restart docker
创建Docker覆盖网络
为创建覆盖网络,运行以下命令
$ sudo docker network create --driver overlay --subnet \
192.168.0.0/24 nomad_cluster
f761da29406153b07986b8459150bffa3b06f9872d018b929158e352b25ce725
现在你可以在(http://72.16.4.21:8500)打开Consul UI,并在“键/值”下查看网络。
Running a Nomad Job
Nomad Job
创建一个Nomad作业说明,Alluxio.nomad来运行1个Alluxio主节点和2个工作节点。我们将创建两个作业组,1个为主节点创建,1个为工作节点创建。
job "alluxio-docker" {
datacenters = ["dc1"]
type = "service"
update {
stagger = "30s"
max_parallel = 1
}
group "alluxio-masters" {
count = 1
#### Alluxio Master Node ####
task "master" {
driver = "docker"
config {
image = "172.16.4.21:5000/alluxio-docker"
args = ["master"]
network_aliases = ["alluxiomaster"]
network_mode = "nomad_cluster"
volumes = ["nomadvolume:/mnt/gfsvol"]
volume_driver = "glusterfs"
dns_servers = ["169.254.1.1"]
port_map = {
master_web_port = 19999
}
}
service {
name = "alluxio"
tags = ["master"]
port = "master_web_port"
check {
type = "tcp"
port = "master_web_port"
interval = "10s"
timeout = "2s"
}
}
resources {
network {
mbits = 5
port "master_web_port" {
static = 19999
}
}
}
}
}
group "alluxio-workers" {
count = 2
#### Alluxio Worker Node ####
task "worker" {
driver = "docker"
config {
image = "172.16.4.21:5000/alluxio-docker"
args = ["worker"]
network_aliases = ["alluxioworker"]
network_mode = "nomad_cluster"
shm_size = 1073741824
volumes = ["nomadvolume:/mnt/gfsvol"]
volume_driver = "glusterfs"
dns_servers = ["169.254.1.1"]
port_map = {
worker_web_port = 30000
}
}
env {
ALLUXIO_JAVA_OPTS = "-Dalluxio.worker.memory.size=1G -Dalluxio.master.hostname=alluxiomaster"
}
service {
name = "alluxio"
tags = ["worker"]
port = "worker_web_port"
check {
type = "tcp"
port = "worker_web_port"
interval = "10s"
timeout = "2s"
}
}
resources {
cpu = 1000
memory = 1024
network {
mbits = 5
port "worker_web_port" {
static = 30000
}
}
}
}
}
}
现在,运行作业
$ nomad job run alluxio.nomad
==> Monitoring evaluation "4470746e"
Evaluation triggered by job "alluxio-docker"
Evaluation within deployment: "b471e355"
Allocation "62224435" created: node "d6313535", group "alluxio-workers"
Allocation "bbaa6675" created: node "66b502e3", group "alluxio-workers"
Allocation "c8ea32ff" created: node "d6313535", group "alluxio-masters"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "4470746e" finished with status "complete"
检查运行中作业状态
$ nomad job status alluxio-docker
ID = alluxio-docker
Name = alluxio-docker
Submit Date = 2019-08-06T18:16:03+05:30
Type = service
Priority = 50
Datacenters = dc1
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
alluxio-masters 0 0 1 0 0 0
alluxio-workers 0 0 2 0 0 0
Latest Deployment
ID = b471e355
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
alluxio-masters 1 1 1 0 2019-08-06T18:26:38+05:30
alluxio-workers 2 2 2 0 2019-08-06T18:27:03+05:30
Allocations
ID Node ID Task Group Version Desired Status Created Modified
62224435 d6313535 alluxio-workers 0 run running 1m13s ago 15s ago
bbaa6675 66b502e3 alluxio-workers 0 run running 1m13s ago 13s ago
c8ea32ff d6313535 alluxio-masters 0 run running 1m13s ago 37s ago
如果查看作业状态输出,则在分配(Allocations)标签页下,我们注意到两个allocation运行在ID为d6313535的客户端节点,一个allocation运行在ID为66b502e3的客户端节点。我们可以调出Nomad UI (http://172.16.4.21:4646),在客户端下看到集群中不同节点。
检查运行容器的节点
Nomad Client Node 1
Nomad Client Node 3
测试集群
在运行于节点1的Alluxio工作节点容器上打开终端,运行Alluxio测试。
$ sudo docker exec -it 8fc5526d467f /bin/bash
运行测试,在/mnt/gfsvol(参考Nomad作业说明卷)查看输出。
bash-4.4# cd /opt/alluxio
bash-4.4# ./bin/alluxio runTests
2019-08-06 13:25:17,290 INFO NettyUtils - EPOLL_MODE is available
2019-08-06 13:25:17,472 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=8fc5526d467f, rack=null)
2019-08-06 13:25:18,297 INFO ConfigurationUtils - Alluxio client has loaded configuration from meta master alluxiomaster/192.168.0.4:19998
2019-08-06 13:25:18,305 INFO ConfigurationUtils - Alluxio client (version 2.0.0) is trying to load cluster level configurations
2019-08-06 13:25:18,318 INFO ConfigurationUtils - Alluxio client has loaded cluster level configurations
2019-08-06 13:25:18,318 INFO ConfigurationUtils - Alluxio client (version 2.0.0) is trying to load path level configurations
2019-08-06 13:25:18,380 INFO ConfigurationUtils - Alluxio client has loaded path level configurations
runTest BASIC CACHE_PROMOTE MUST_CACHE
2019-08-06 13:25:19,563 INFO BasicOperations - writeFile to file /default_tests_files/BASIC_CACHE_PROMOTE_MUST_CACHE took 635 ms.
2019-08-06 13:25:19,659 INFO BasicOperations - readFile file /default_tests_files/BASIC_CACHE_PROMOTE_MUST_CACHE took 96 ms.
Passed the test!
runTest BASIC_NON_BYTE_BUFFER CACHE_PROMOTE MUST_CACHE
2019-08-06 13:25:19,751 INFO BasicNonByteBufferOperations - writeFile to file /default_tests_files/BASIC_NON_BYTE_BUFFER_CACHE_PROMOTE_MUST_CACHE took 41 ms.
2019-08-06 13:25:19,775 INFO BasicNonByteBufferOperations - readFile file /default_tests_files/BASIC_NON_BYTE_BUFFER_CACHE_PROMOTE_MUST_CACHE took 24 ms.
Passed the test!
...................
bash-4.4# ls -l /mnt/gfsvol/default_tests_files/
total 9
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_ASYNC_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_CACHE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_PROMOTE_ASYNC_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_PROMOTE_CACHE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_PROMOTE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_ASYNC_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_PROMOTE_ASYNC_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_PROMOTE_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_PROMOTE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_NO_CACHE_ASYNC_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_NO_CACHE_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_NO_CACHE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_NO_CACHE_ASYNC_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_NO_CACHE_CACHE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_NO_CACHE_THROUGH
接下来,我们将在运行于节点3的Alluxio工作节点容器上打开终端,查看输出的结果。
$ sudo docker exec -it acdfccaf3b81 /bin/bash
#### Container #####
bash-4.4# ls -l /mnt/gfsvol/default_tests_files/
total 9
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_ASYNC_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_CACHE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_PROMOTE_ASYNC_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_PROMOTE_CACHE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_PROMOTE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_ASYNC_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_PROMOTE_ASYNC_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_PROMOTE_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_PROMOTE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_NO_CACHE_ASYNC_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_NO_CACHE_CACHE_THROUGH
-rw-r--r-- 1 root root 84 Aug 6 13:25 BASIC_NON_BYTE_BUFFER_NO_CACHE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_NO_CACHE_ASYNC_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_NO_CACHE_CACHE_THROUGH
-rw-r--r-- 1 root root 80 Aug 6 13:25 BASIC_NO_CACHE_THROUGH
总结
这是我第一次使用Nomad,虽然有些事情我花了一些时间才能运行起来,但安装相对简单。Nomad的简单性和有限范围是设计使然,但在一些公司,Nomad支持自带持久化存储或网络命名空间可能是个问题。这将在预期9月发布的0.10版本中解决,会使Nomad成为更具吸引力的替代Kubernets的方案之一。
希望您觉得这有用。如有任何疑问,请随时与我联系,感谢任何反馈。谢谢!