kolla-ansible 部署OpenStack queens版本笔记
一. 实验环境:
- 6台主机安装CentOS7 minimal系统32G内存,1T+500G双硬盘(其中一个为后期ceph部署做准备),三张千兆网卡(一张用于后期ceph(enp3s0f0),另两张网卡分别作为控制网络(enp0s31f6)以及neutron桥接(enp3s0f1)网络)
- 网络规划:
host | IP address | remark |
---|---|---|
controller01 | 10.132.226.51 | 1 |
controller02 | 10.132.226.52 | 2 |
controller03 | 10.132.226.53 | 3 |
compute01 | 10.132.226.54 | 4 |
compute02 | 10.132.226.55 | 5 |
kolla | 10.132.226.200 | 6 |
virtulal IP | 10.132.226.70 | |
虚拟地址池 | 10.132.226.71-99 |
二. 控制以及计算节点初始化操作:
- 使用以下脚本对每个计算机进行初始化配置(kolla除外)执行 sh initnode.sh n(n代表第几台主机)
# /usr/bin/bash
ls -l /etc/sysconfig/network-scripts|awk '/ifcfg-enp[0-9]*/ {print $9}' > default_name.txt
i=5
c=0
cat default_name.txt | while read line
do
cd /etc/sysconfig/network-scripts
cp $line ${line}.bak
sed -i 's/BOOTPROTO=dhcp/BOOTPROTO=static/g' $line
sed -i 's/ONBOOT=no/ONBOOT=yes/g' $line
c=$(expr $c + 1)
if [ "${c}" == "1" ]; then
continue;
elif [ "${c}" == "2" ]; then
i=$(expr $i + 2)
echo -e "\nIPADDR=10.132.226.${i}${1}" >> $line
echo "NETMASK=255.255.255.0" >> $line
echo "GATEWAY=10.132.226.254" >> $line
echo "DNS1=192.168.0.1" >> $line
echo "DNS2=114.114.114.114" >> $line
else
sed -i '1,11d' $line
sed -i '1i\OVS_BRIDGE=br-ex' $line
sed -i '1i\DEVICETYPE=ovs' $line
sed -i '1i\BOOTPROTO=none' $line
sed -i '1i\TYPE=OVSPort' $line
fi
done
systemctl stop firewalld && systemctl disable firewalld && systemctl status firewalld
yum update -y
yum install -y wget vim net-tools
wget -P /etc/yum.repos.d/ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce
mkdir -pv /etc/docker
systemctl restart docker && systemctl status docker
echo -e "\n10.132.226.200\tkolla\n10.132.226.51\tcontroller01\n10.132.226.52\tcontroller02\n10.132.226.53\tcontroller03\n10.132.226.54\tcompute01\n10.132.226.55\tcompute02" >> /etc/hosts
if [ $1 -lt 4 ]; then
echo "controller0${1}" > /etc/hostname
else
num=$(expr $1 - 3)
echo "compute0${num}" > /etc/hostname
fi
reboot
-
各节点主机初始化内容:
- 配置网卡信息
- 关闭防火墙
- 安装docker
- 修改hostname以及添加hosts信息
-
设置各节点主机之间免密登录
- 在各主机依次执行
ssh-keygen ssh-copy-id -i ~/.ssh/id_rsa.pub root@controller01
- 将authorized_key文件发放到各主机的~/.ssh/目录
scp authorized_keys kolla:~/.ssh/ scp authorized_keys controller01:~/.ssh/ scp authorized_keys controller02:~/.ssh/ scp authorized_keys controller03:~/.ssh/ scp authorized_keys compute01:~/.ssh/ scp authorized_keys compute02:~/.ssh/
三. kolla主机配置
-
安装docker:
- 从阿里云下载docker的repo文件:
[root@kolla ~]# wget -P /etc/yum.repos.d/ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
- 安装docker-ce:
[root@kolla ~]# yum install -y docker-ce
- 配置国内镜像:
[root@kolla ~]# mkdir -p /etc/docker [root@kolla ~]# vim /etc/docker/daemon.json { "registry-mirrors": ["https://7g5a4z30.mirror.aliyuncs.com"] }
- 启动docker
[root@kolla ~]# systemctl daemon-reload && systemctl enable docker && systemctl restart docker
- 检查镜像站点配置是否正确
[root@kolla ~]# docker pull hello-world
-
安装依赖软件
- 安装pip并更新
[root@kolla ~]# yum insatll epel-release -y [root@kolla ~]# yum insatll python-pip -y [root@kolla ~]# pip install -U pip
- 修改pip源
[root@kolla ~]# mkdir ~/.pip [root@kolla ~]# vim ~/.pip/pip.conf [global] trusted-host = pypi.douban.com index-url = http://pypi.douban.com/simple
- 安装其他依赖包
[root@kolla ~]# yum install python-devel libffi-devel gcc openssl-devel libselinux-python -y
-
安装配置ansible:
- 先使用pip安装再使用yum安装,可以防止某些py包版本太低
[root@kolla ~]# pip install ansible [root@kolla ~]# yum install ansible -y
- 在/etc/ansible/ansible.cfg配置文件中添加以下内容:
[defaults] host_key_checking=False pipelining=True forks=100
-
安装配置kolla-ansible:
- 使用pip安装kolla-ansible:
pip install kolla-ansible
- 复制global.yml和password.yml文件到/etc/kolla目录:
cp -r /usr/share/kolla-ansible/etc_examples/kolla /etc/kolla/
- 复制all-in-one 和multinode 文件到当前操作目录:
cp /usr/share/kolla-ansible/ansible/inventory/* .
- 修改global.yml文件
global.yml - 拉取镜像
kolla-ansible pull -vvv
- 再次修改global.yml文件(因为上一个文件拉取的镜像缺少nova-compute等镜像)
global.yml - 拉取镜像
kolla-ansible pull -vvv
-
上传镜像到本地registry仓库:
- 配置Docker共享挂载:
[root@kolla ~]# mkdir -p /etc/systemd/system/docker.service.d [root@kolla ~]# vim /etc/systemd/system/docker.service.d/kolla.conf [Service] MountFlags=shared [root@kolla ~]# systemctl daemon-reload && systemctl restart docker && systemctl status docker
- 启动registry容器,并将端口映射到4000端口
[root@kolla /]# docker run -d --name registry --restart=always -p 4000:5000 -v /opt/registry:/var/lib/registry registry:2.6.2
- 修改Docker服务配置,信任本地Registry服务
[root@kolla /]# vim /usr/lib/systemd/system/docker.service ExecStart=/usr/bin/dockerd --insecure-registry kolla:4000
- 重新启动docker服务
systemctl daemon-reload && systemctl restart docker
- 测试registry服务是否正常:
[root@kolla ~]# curl -X GET http://kolla:4000/v2/_catalog {"repositories":[]}
- 修改镜像tag:
for i in `docker images|grep -v registry|grep -v R|awk '{print $1}'`;do docker image tag $i:queens kolla:4000/$i:queens;done
- push到本地库
for i in `docker images|grep kolla:4000|awk '{print $1}'`;do docker push $i:queens;done
- 查看镜像是否上传成功:
curl -XGET http://kolla:4000/v2/_catalog { "repositories": [ "kolla/centos-source-aodh-api", "kolla/centos-source-aodh-evaluator", "kolla/centos-source-aodh-listener", "kolla/centos-source-aodh-notifier", "kolla/centos-source-barbican-api", "kolla/centos-source-barbican-keystone-listener", "kolla/centos-source-barbican-worker", "kolla/centos-source-blazar-api", "kolla/centos-source-blazar-manager", "kolla/centos-source-ceilometer-central", "kolla/centos-source-ceilometer-compute", "kolla/centos-source-ceilometer-notification", "kolla/centos-source-ceph-mds", "kolla/centos-source-ceph-mgr", "kolla/centos-source-ceph-mon", "kolla/centos-source-ceph-nfs", "kolla/centos-source-ceph-osd", "kolla/centos-source-ceph-rgw", "kolla/centos-source-chrony", "kolla/centos-source-cinder-api", "kolla/centos-source-cinder-backup", "kolla/centos-source-cinder-scheduler", "kolla/centos-source-cinder-volume", "kolla/centos-source-cloudkitty-api", "kolla/centos-source-cloudkitty-processor", "kolla/centos-source-collectd", "kolla/centos-source-congress-api", "kolla/centos-source-congress-datasource", "kolla/centos-source-congress-policy-engine", "kolla/centos-source-cron", "kolla/centos-source-designate-api", "kolla/centos-source-designate-backend-bind9", "kolla/centos-source-designate-central", "kolla/centos-source-designate-mdns", "kolla/centos-source-designate-producer", "kolla/centos-source-designate-sink", "kolla/centos-source-designate-worker", "kolla/centos-source-dnsmasq", "kolla/centos-source-elasticsearch", "kolla/centos-source-etcd", "kolla/centos-source-fluentd", "kolla/centos-source-freezer-api", "kolla/centos-source-glance-api", "kolla/centos-source-gnocchi-api", "kolla/centos-source-gnocchi-metricd", "kolla/centos-source-gnocchi-statsd", "kolla/centos-source-grafana", "kolla/centos-source-haproxy", "kolla/centos-source-heat-api", "kolla/centos-source-heat-api-cfn", "kolla/centos-source-heat-engine", "kolla/centos-source-horizon", "kolla/centos-source-influxdb", "kolla/centos-source-ironic-api", "kolla/centos-source-ironic-conductor", "kolla/centos-source-ironic-inspector", "kolla/centos-source-ironic-pxe", "kolla/centos-source-iscsid", "kolla/centos-source-karbor-api", "kolla/centos-source-karbor-operationengine", "kolla/centos-source-karbor-protection", "kolla/centos-source-keepalived", "kolla/centos-source-keystone", "kolla/centos-source-kibana", "kolla/centos-source-kolla-toolbox", "kolla/centos-source-kuryr-libnetwork", "kolla/centos-source-magnum-api", "kolla/centos-source-magnum-conductor", "kolla/centos-source-manila-api", "kolla/centos-source-manila-data", "kolla/centos-source-manila-scheduler", "kolla/centos-source-manila-share", "kolla/centos-source-mariadb", "kolla/centos-source-memcached", "kolla/centos-source-mistral-api", "kolla/centos-source-mistral-engine", "kolla/centos-source-mistral-executor", "kolla/centos-source-mongodb", "kolla/centos-source-multipathd", "kolla/centos-source-murano-api", "kolla/centos-source-murano-engine", "kolla/centos-source-neutron-bgp-dragent", "kolla/centos-source-neutron-dhcp-agent", "kolla/centos-source-neutron-l3-agent", "kolla/centos-source-neutron-lbaas-agent", "kolla/centos-source-neutron-metadata-agent", "kolla/centos-source-neutron-openvswitch-agent", "kolla/centos-source-neutron-server", "kolla/centos-source-neutron-server-opendaylight", "kolla/centos-source-neutron-sriov-agent", "kolla/centos-source-neutron-vpnaas-agent", "kolla/centos-source-nova-api", "kolla/centos-source-nova-compute", "kolla/centos-source-nova-compute-ironic", "kolla/centos-source-nova-conductor", "kolla/centos-source-nova-consoleauth", "kolla/centos-source-nova-libvirt", "kolla/centos-source-nova-novncproxy", "kolla/centos-source-nova-placement-api", "kolla/centos-source-nova-scheduler"] }
-
修改部署配置文件
- 修改当前目录下的multinode文件:mutinode
- 修改/etc/kolla/global.yml文件:global.yml
-
部署:
- 生产随机密码文件:
kolla-genpwd
- 修改horizon登录界面admin密码:
[root@kolla ~]# vim /etc/kolla/passwords.yml keepalived_password: mFbTVxF6XyrrT8NqaN5UpFB098GEXuZ9oQyfQI14 keystone_admin_password: admin # 更改此处 keystone_database_password: C4EzIx0zhoFjsG9dA9TBRaZfbFIdT3f9sCe7jGyg
- 引导配置各节点依赖软件:
kolla-ansible -i ./multinode bootstrap-servers PLAY RECAP ************************************************************************************************************************************************************* compute01 : ok=38 changed=7 unreachable=0 failed=0 compute02 : ok=38 changed=7 unreachable=0 failed=0 controller01 : ok=38 changed=7 unreachable=0 failed=0 controller02 : ok=39 changed=17 unreachable=0 failed=0 controller03 : ok=38 changed=7 unreachable=0 failed=0 localhost : ok=1 changed=0 unreachable=0 failed=0
- 进行预部署检查:
kolla-ansible -i ./multinode prechecks PLAY RECAP ************************************************************************************************************************************************************ compute01 : ok=26 changed=1 unreachable=0 failed=0 compute02 : ok=26 changed=1 unreachable=0 failed=0 controller01 : ok=91 changed=1 unreachable=0 failed=0 controller02 : ok=87 changed=1 unreachable=0 failed=0 controller03 : ok=87 changed=1 unreachable=0 failed=0 localhost : ok=6 changed=1 unreachable=0 failed=0
- Cinder出现错误
TASK [cinder : Checking LVM volume group exists for Cinder] *********************************************************************************************************** skipping: [controller01] skipping: [controller02] skipping: [controller03] [DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|failed` use `result is failed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. fatal: [compute01]: FAILED! => {"changed": false, "cmd": ["vgs", "cinder-volumes"], "delta": "0:00:00.009794", "end": "2018-10-13 18:33:13.868282", "failed_when_result": true, "msg": "non-zero return code", "rc": 5, "start": "2018-10-13 18:33:13.858488", "stderr": " Volume group \"cinder-volumes\" not found\n Cannot process volume group cinder-volumes", "stderr_lines": [" Volume group \"cinder-volumes\" not found", " Cannot process volume group cinder-volumes"], "stdout": "", "stdout_lines": []} [DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|failed` use `result is failed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. fatal: [compute02]: FAILED! => {"changed": false, "cmd": ["vgs", "cinder-volumes"], "delta": "0:00:00.010114", "end": "2018-10-13 18:33:13.860281", "failed_when_result": true, "msg": "non-zero return code", "rc": 5, "start": "2018-10-13 18:33:13.850167", "stderr": " Volume group \"cinder-volumes\" not found\n Cannot process volume group cinder-volumes", "stderr_lines": [" Volume group \"cinder-volumes\" not found", " Cannot process volume group cinder-volumes"], "stdout": "", "stdout_lines": []}
* 解决方案: [root@compute02 .ssh]# vgdisplay --- Volume group --- VG Name centos System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 4 VG Access read/write VG Status resizable MAX LV 0 Cur LV 3 Open LV 3 Max PV 0 Cur PV 1 Act PV 1 VG Size <1.82 TiB PE Size 4.00 MiB Total PE 476806 Alloc PE / Size 476806 / <1.82 TiB Free PE / Size 0 / 0 VG UUID FEgDXH-SBlh-x29N-qU0f-Wajd-2sJ6-rbUre5 [root@compute02 .ssh]# dd if=/dev/zero of=./disk.img count=200 bs=512MB 200+0 records in 200+0 records out 102400000000 bytes (102 GB) copied, 509.072 s, 201 MB/s [root@compute02 .ssh]# losetup -f /dev/loop0 [root@compute02 .ssh]# losetup /dev/loop0 disk.img [root@compute02 .ssh]# pvcreate /dev/loop0 Physical volume "/dev/loop0" successfully created. [root@compute02 .ssh]# vgcreate cinder-volumes /dev/loop0 Volume group "cinder-volumes" successfully created
- 进行实际部署:
kolla-ansible -i ./multinode deploy
-
初始化OpenStack
- 删除ipadress的py包并重新安装(版本过低下一步客户端安装会出错,原先安装其他包的时候作为依赖包安装的ipaddress无法通过pip删除并升级,只能手动删除再安装最新版本):
[root@kolla ~]# cd /usr/lib/python2.7/site-packages/ [root@kolla site-packages]# rm -rf ipaddress* [root@kolla site-packages]# pip install ipaddress
- 安装OpenStack CLI客户端:
[root@kolla site-packages]# pip install python-openstackclient python-glanceclient python-neutronclient
- 设置环境变量:
[root@kolla site-packages]# . /etc/kolla/admin-openrc.sh
- 编辑初始化脚本中的网络配置:
[root@kolla ~]# vim /usr/share/kolla-ansible/init-runonce EXT_NET_CIDR='10.132.226.0/24' EXT_NET_RANGE='start=10.132.226.130,end=10.132.226.169' EXT_NET_GATEWAY='10.132.226.254'
- 执行初始化脚本:
[root@kolla ~]# . /usr/share/kolla-ansible/init-runonce Checking for locally available cirros image. None found, downloading cirros image. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 12.1M 100 12.1M 0 0 2040k 0 0:00:06 0:00:06 --:--:-- 2716k Creating glance image. ······ Done. To deploy a demo instance, run: openstack server create \ --image cirros \ --flavor m1.tiny \ --key-name mykey \ --nic net-id=89a1f674-e89f-4e6d-b96d-2875446adc1e \ demo1