openshift节点扩容

本文是公司内部集群扩容笔记，只做参考

软硬件环境

• 阿里云ecs，1个deploy,3个master,3个infra,其余为计算节点
• centos7.6
• docker-1.13.1
• openshfit 3.11

1. 扩容节点配置

1.1 ssh到新的节点

 hostnamectl set-hostname computeXX.youdomain.local

1.2 禁用swap

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

1.3 启用selinux

setenforce 0
sed -i 's/^SELINUX=disabled$/SELINUX=permissive/' /etc/selinux/config

1.4 禁用防火墙

systemctl stop firewalld
systemctl disable firewalld

1.5 安装、更新包

yum install -y docker-1.13.1  wget git net-tools bind-utils yum-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct   PyYAML python-ipaddress yum-utils telnet curl lrzsz jq perf strace vim iotop httpd-tools python-passlib patch java-1.8.0-openjdk-headless
yum update -y
systemctl enable  NetworkManager
systemctl start NetworkManager

1.6 重启

注: 阿里云ecs有bug，修改selinux后直接重启无法进入系统

 fixfiles onboot  && reboot

2. deploy节点操作

XX为节点序号

2.1. 配置host，/etc/hosts

xxxx.xxxx.xxxx.xxxx computeXX.yourdomain.local computeXX

2.2. 修改/etc/ansible/hosts，新增

[new_nodes]
computeXX.yourdomain.local openshift_node_group_name='node-config-compute'

2.3. 配置免密

ssh-copy-id -i ~/.ssh/id_rsa.pub computeXX.yourdomain.local

2.4. 拷贝host到新的节点 （外部读者请忽略）

ansible all -m copy -a "src=/etc/hosts dest=/etc/hosts owner=root group=root mode=0755"

这里有三台master和3台infra节点，对外暴露使用了阿里的负载均衡器(有bug)，目前如果负载到内部自身那个节点流量会进不来，需要master和infra的host需要单独再修改，内部通信不走负载均衡使用自身节点的apiserver

ssh master1
192.168.1.77 openshift-internal.yourdomain.local
ssh master2
192.168.1.81 openshift-internal.yourdomain.local
ssh master3
192.168.1.83 openshift-internal.yourdomain.local

分别ssh infra1 ssh infra2 ssh infra3，注释掉registry内部ip，走公网流量

#192.168.1.97 registry.yourdomain.com

2.5. 重启域名服务

ansible all -m  shell -a 'systemctl daemon-reload && systemctl restart dnsmasq'

2.6.安装lxcfs (opt）

ansible new_nodes -m shell -a 'mkdir -p /root/lxcfs-rpm'

ansible new_nodes -m copy -a "src=/root/lxcfs-rpm/lxcfs-2.0.5-3.el7.centos.x86_64.rpm dest=/root/lxcfs-rpm/lxcfs-2.0.5-3.el7.centos.x86_64.rpm owner=root group=root mode=0755"

ansible new_nodes -m  shell -a 'yum install -y /root/lxcfs-rpm/lxcfs-2.0.5-3.el7.centos.x86_64.rpm && systemctl enable lxcfs && systemctl start lxcfs'

2.7. 执行安装

## 扩容计算节点
ansible-playbook -vv -i /etc/ansible/hosts /root/openshift-ansible/playbooks/openshift-node/scaleup.yml
## 如果扩容master节点
ansible-playbook -vv -i /etc/ansible/hosts /root/openshift-ansible/playbooks/openshift-master/scaleup.yml

2.8. 待成功后，注释/etc/ansible/hosts

[new_nodes]
#computeXX.youdomain.local openshift_node_group_name='node-config-compute'

2.9. 给新的节点打污点 key=value

oc label node computeXX.youdomain.local key=value

kubectl taint nodes computeXX.youdomain.local key=value:NoExecute