OpenStack（Stein）版配置Zun组件

配置Zun还需要一些其他的组件和软件支持，比如kuryr-libnetwork，docker之类的。

基础环境

Docker

新版

安装一点必备的依赖

yum install -y yum-utils device-mapper-persistent-data lvm2

配置仓库

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装

yum install -y docker-ce docker-ce-cli containerd.io

启动服务，自启动

systemctl start docker containerd.service 
systemctl enable docker containerd.service

19.03

好像最近的docker版本升级了，变成了20版本，然后kuryr就用不成了。之前用的19.03没问题，就换成这个，安装这个版本的。
通过本地包安装

wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-19.03.14-3.el7.x86_64.rpm
wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-cli-19.03.14-3.el7.x86_64.rpm
wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.3.9-3.1.el7.x86_64.rpm

yum localinstall  -y  docker-ce-cli-19.03.14-3.el7.x86_64.rpm
yum localinstall  -y  containerd.io-1.3.9-3.1.el7.x86_64.rpm
yum localinstall  -y  docker-ce-19.03.14-3.el7.x86_64.rpm

Etcd

Etcd for RHEL and CentOS
直接安装

yum install -y etcd

编辑配置文件/etc/etcd/etcd.conf

#[Member]
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.1.104:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.104:2379"
ETCD_NAME="controller"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.104:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.104:2379"
ETCD_INITIAL_CLUSTER="controller=http://192.168.1.104:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER_STATE="new"

启动服务，自启动

systemctl enable etcd
# systemctl start etcd

Kuryr-libnetwork

控制节点

Install and configure controller node
创建用户

[root@controller OpenStack (keystone_admin)]#openstack user create --domain default --password-prompt kuryr
User Password:
Repeat User Password:
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| enabled             | True                             |
| id                  | df33445232c345978e520490244e6769 |
| name                | kuryr                            |
| options             | {
   }                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

添加角色

openstack role add --project service --user kuryr admin

计算节点

Install and configure a compute node for Ubuntu
创建用户

groupadd --system kuryr
useradd --home-dir "/var/lib/kuryr" --create-home --system --shell /bin/false -g kuryr kuryr

创建文件夹，然后授权

mkdir -p /etc/kuryr
chown kuryr:kuryr /etc/kuryr

安装epel，再换个源

yum install -y epel-release
wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum clean all && yum makecache
yum install -y python-pip

下载kuryr-libnetwork，复制到/var/lib/kuryr，因为我下不下来，所以提前下好了。

#直接下载
cd /var/lib/kuryr
git clone -b stable/stein https://git.openstack.org/openstack/kuryr-libnetwork.git
#使用本地继续
tar -zxvf kuryr-libnetwork-stable_stein.tar.gz 
cp -r kuryr-libnetwork /var/lib/kuryr/kuryr-libnetwork
cd /var/lib/kuryr/
chown -R kuryr:kuryr kuryr-libnetwork
cd kuryr-libnetwork/
git init

这时候再换一个源，换pip源，创建~/.pip/pip.conf

[global]
index-url=http://mirrors.aliyun.com/pypi/simple/

[install]
trusted-host=mirrors.aliyun.com

安装kuryr-libnetwork的依赖

pip install -r requirements.txt
python setup.py install

这时候报错了

Marker evaluation failed, see the following error.  For more information see: http://docs.openstack.org/pbr/latest/user/using.html#environment-markers
ERROR:root:Error parsing
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pbr/core.py", line 96, in pbr
    attrs = util.cfg_to_args(path, dist.script_args)
  File "/usr/lib/python2.7/site-packages/pbr/util.py", line 258, in cfg_to_args
    kwargs = setup_cfg_to_setup_kwargs(config, script_args)
  File "/usr/lib/python2.7/site-packages/pbr/util.py", line 456, in setup_cfg_to_setup_kwargs
    if pkg_resources.evaluate_marker('(%s)' % env_marker):
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1364, in evaluate_marker
    return interpret(parser.expr(text).totuple(1)[1])
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1342, in interpret
    return op(nodelist)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1307, in atom
    return interpret(nodelist[2])
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1342, in interpret
    return op(nodelist)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1324, in comparison
    raise SyntaxError(repr(cop)+" operator not allowed in environment markers")
SyntaxError: '<' operator not allowed in environment markers
error in setup command: Error parsing /var/lib/kuryr/kuryr-libnetwork/setup.cfg: SyntaxError: '<' operator not allowed in environment markers

好像是setuptools版本不够，升级一下

pip install --upgrade pip
pip install --upgrade setuptools

然后重新执行python setup.py install。
复制一下配置文件

su -s /bin/sh -c "./tools/generate_config_file_samples.sh" kuryr
su -s /bin/sh -c "cp etc/kuryr.conf.sample /etc/kuryr/kuryr.conf" kuryr

编辑配置文件/etc/kuryr/kuryr.conf，第一项，官网写的是/usr/local/linexec文件夹，但是系统里/usr/libexec里有内容，我就先用系统的路径试试，正好默认也是这个。

[DEFAULT]
bindir = /usr/libexec/kuryr

[neutron]
www_authenticate_uri = http://controller:5000
auth_url = http://controller:35357
username = kuryr
user_domain_name = Default
password = kuryr
project_name = service
project_domain_name = Default
auth_type = password

创建服务文件/etc/systemd/system/kuryr-libnetwork.service，这里官网是/usr/local/bin/kuryr-server但是我的这里没有东西，/usr/bin/kuryr-server这里有，我就用后者了先。

[Unit]
Description = Kuryr-libnetwork - Docker network plugin for Neutron

[Service]
ExecStart = /usr/bin/kuryr-server --config-file /etc/kuryr/kuryr.conf
CapabilityBoundingSet = CAP_NET_ADMIN

[Install]
WantedBy = multi-user.target

启动服务，自启动

systemctl enable kuryr-libnetwork
systemctl start kuryr-libnetwork

systemctl restart docker

验证

这里其实已经可以使用kuryr-libnetwork创建网络了，驱动选择kuryr就好了，会有一点坑。

报错

Error response from daemon: legacy plugin: Post http://127.0.0.1:23750/Plugin.Activate: dial tcp 127.0.0.1:23750: connect: connection refused

docker network create with kuryr
执行

docker network create --driver kuryr --ipam-driver kuryr --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net

报错

Error response from daemon: legacy plugin: Post http://127.0.0.1:23750/Plugin.Activate: dial tcp 127.0.0.1:23750: connect: connection refused

查看一下docker.service的status，systemctl status docker.service

Sep 17 21:31:16 controller dockerd[30029]: time="2020-09-17T21:31:16.642992083+08:00" level=warning msg="Unable to connect to plugin: 127.0.0.1:23750/Plugin.Activate: Post http://127.0.0.1:23750/Plugin.Activate: dial tcp 127.0.0.1:23750: connect: connection refused, retrying in 1s"
Sep 17 21:31:17 controller dockerd[30029]: time="2020-09-17T21:31:17.643707810+08:00" level=warning msg="Unable to connect to plugin: 127.0.0.1:23750/Plugin.Activate: Post http://127.0.0.1:23750/Plugin.Activate: dial tcp 127.0.0.1:23750: connect: connection refused, retrying in 2s"
Sep 17 21:31:19 controller dockerd[30029]: time="2020-09-17T21:31:19.644635277+08:00" level=warning msg="Unable to connect to plugin: 127.0.0.1:23750/Plugin.Activate: Post http://127.0.0.1:23750/Plugin.Activate: dial tcp 127.0.0.1:23750: connect: connection refused, retrying in 4s"
Sep 17 21:31:23 controller dockerd[30029]: time="2020-09-17T21:31:23.645450218+08:00" level=warning msg="Unable to connect to plugin: 127.0.0.1:23750/Plugin.Activate: Post http://127.0.0.1:23750/Plugin.Activate: dial tcp 127.0.0.1:23750: connect: connection refused, retrying in 8s"
Sep 17 21:31:31 controller dockerd[30029]: time="2020-09-17T21:31:31.646711235+08:00" level=error msg="Handler for POST /v1.40/networks/create returned error: legacy plugin: Post http://127.0.0.1:23750/Plugin.Activate: dial tcp 127.0.0.1:23750: connect: connection refused"

是端口没有开放，查看一下kuryr发现是failed，好奇怪，failed为什么不报错

systemctl status kuryr-libnetwork.service 

● kuryr-libnetwork.service - Kuryr-libnetwork - Docker network plugin for Neutron
   Loaded: loaded (/etc/systemd/system/kuryr-libnetwork.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2020-09-17 21:50:35 CST; 1min 55s ago
  Process: 34251 ExecStart=/usr/bin/kuryr-server --config-file /etc/kuryr/kuryr.conf (code=exited, status=1/FAILURE)
 Main PID: 34251 (code=exited, status=1/FAILURE)

Sep 17 21:50:35 controller kuryr-server[34251]: 2020-09-17 21:50:35.015 34251 ERROR kuryr     self.auth_ref = self.get_auth_ref(session)
Sep 17 21:50:35 controller kuryr-server[34251]: 2020-09-17 21:50:35.015 34251 ERROR kuryr   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line