Copy of 联城科技paas平台离线安装文档
一.环境说明
| Ip列表 | Host | 备注 |
|---|---|---|
| 192.168.12.76 | master1.qingyuanos.com | master/node 角色 |
| 192.168.12.77 | master2.qingyuanos.com | master/node 角色 |
| 192.168.12.78 | master3.qingyuanos.com | master/node 角色 |
| 192.168.12.5 | node1.qingyuanos.com | node角色 |
| 192.168.12.6 | node2.qingyuanos.com | node角色 |
| 192.168.12.7 | node3.qingyuanos.com | node角色 |
二.基础环境的配置
1.配置主机名及hosts
hostnamectl set-hostname master1.qingyuanos.com
hostnamectl set-hostname master2.qingyuanos.com
hostnamectl set-hostname master3.qingyuanos.com
hostnamectl set-hostname node1.qingyuanos.com
hostnamectl set-hostname node2.qingyuanos.com
hostnamectl set-hostname node3.qingyuanos.com
hosts 文件配置如下 注：所有主机都修改hosts文件
[root@master3 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.12.78 master1.qingyuanos.com master.qingyuanos.com
192.168.12.77 master2.qingyuanos.com
192.168.12.76 master3.qingyuanos.com
192.168.12.5 node1.qingyuanos.com
192.168.12.6 node1.qingyuanos.com
192.168.12.7 node1.qingyuanos.com
2.配置面密钥登陆（在master3上配置）
[root@master3 ~]# ssh-keygen #一路回车
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
36:90:90:0f:24:86:29:9d:aa:dd:24:1c:f3:c9:44:46 root@master3.qingyuanos.com
The key's randomart image is:
+--[ RSA 2048]----+
| +o=E. |
|+.=o+. . |
|.o * +o |
|. o = .. |
|.. + S |
|. . . . . |
| |
| |
| |
[root@master3 ~]# ssh-copy-id 192.168.12.76 # yes ，然后输入密码
[root@master3 ~]# ssh-copy-id 192.168.12.77
[root@master3 ~]# ssh-copy-id 192.168.12.78
[root@master3 ~]# ssh-copy-id 192.168.12.5
[root@master3 ~]# ssh-copy-id 192.168.12.6
[root@master3 ~]# ssh-copy-id 192.168.12.7
3.安装httpd 做本地yum源（以下为搭建paas平台的所有包）
进入http_rpm 目录
[root@master3 http_rpm]# rpm -ivh *.rpm
修改http端口（修改42行，改为81端口）
[root@master3 oc]# grep 81 -n /etc/httpd/conf/httpd.conf
42:Listen 81
[root@master3 oc]# httpd
将本地源的tar包openshift-reop-3.9.tar.gz 拷贝到节点，并解压到http根目录下
[root@master3 oc]# tar zxf openshift-v3.6.tar.gz -C /var/www/html/
备份原repo文件并指定本地源
[root@master3 yum.repos.d]# mkdir /etc/yum.repos.d/repo.bak
[root@master3 yum.repos.d]# mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/repo.bak/
[root@master3 yum.repos.d]# cat openshift.repo
[centos-openshift-origin]
name=CentOS OpenShift Origin
baseurl=http://192.168.12.78:81/openshift-v3.6
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS
此时需要为防火墙添加一个81端口，防止跑脚本的时候防火墙启动以后无法访问本地源
[root@master3 yum.repos.d]# iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 81 -j ACCEPT
4.安装基础依赖包
所有节点执行
执行前先检查有无NetworkManager,有的情况下
yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion -y
否则
yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion NetworkManager -y
启动服务（所有节点执行）
#systemctl enable NetworkManager; systemctl start NetworkManager
三.安装oc
1.安装docker （所有节点执行）
[root@master3 yum.repos.d]# yum -y install docker
2.load docekr镜像（所有节点执行）
[root@node1 ~]# tar zxf origin-images-3.6.tar.gz
[root@node1 ~]# cd origin-images-3.6
[root@node1 origin-images-3.6]# ls
origin-deployer.tar origin-docker-registry.tar origin-pod.tar
origin-docker-builder.tar origin-haproxy-router.tar origin-sti-builder.tar
[root@node1 origin-images-3.6]# for i in `ls ` ;do docker load -i $i ;done
[root@node3 ~]# docker load -i kubernetes.tar.gz
3.安装配置ansible（在做无密钥登陆的节点上安装）
[root@master3 oc]# yum -y install ansible
[root@master3 oc]# tar zxf openshift-ansible-3.6wanda.tar.gz
[root@master3 oc]# cd openshift-ansible-openshift-ansible-3.6.173.0.63-1/
修改 ./roles/openshift_repos/templates/CentOS-OpenShift-Origin36.repo.j2
配置文件
将内容替换为以下，注意修改IP地址为http节点的IP地址
[root@master3 openshift-ansible-openshift-ansible-3.6.173.0.63-1]# cat ./roles/openshift_repos/templates/CentOS-OpenShift-Origin36.repo.j2
[centos-openshift-origin]
name=CentOS OpenShift Origin
baseurl=http://192.168.12.78:81/openshift-v3.6
enabled=1
gpgcheck=0
4.配置hosts文件
# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes
etcd
# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=root
# If ansible_ssh_user is not root, ansible_become must be set to true
#ansible_become=true
deployment_type=origin
# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_cluster_method = native
openshift_rolling_restart_mode = services
os_sdn_network_plugin_name=redhat/openshift-ovs-multitenant
openshift_master_portal_net=172.30.0.0/16
openshift_node_proxy_mode=iptables
osm_cluster_network_cidr=10.128.0.0/14
osm_host_subnet_length=9
openshift_disable_check=memory_availability,disk_availability,docker_storage,docker_storage_driver,docker_image_availability,package_version,package_availability,package_update
#openshift_router_selector='region=infra'
#openshift_registry_selector='region=infra'
openshift_use_openshift_sdn=true
openshift_master_default_subdomain=qingyuanos.com
openshift_master_cluster_method=native
openshift_master_cluster_hostname=master.qingyuanos.com
openshift_master_cluster_public_hostname=master.qingyuanos.com
openshift_clock_enabled=true
openshift_public_ip=192.168.12.76
#openshift_master_ca_certificate={'certfile': '/root/openshift-ansible/custom_ca/ca.crt', 'keyfile': '/root/openshift-ansible/custom_ca/ca.key'}
# host group for masters
[masters]
master1.qingyuanos.com ansible_host=192.168.12.76
master2.qingyuanos.com ansible_host=192.168.12.77
master3.qingyuanos.com ansible_host=192.168.12.78
[etcd]
master1.qingyuanos.com
master2.qingyuanos.com
master3.qingyuanos.com
# Specify load balancer host
# host group for nodes, includes region info
[nodes]
master1.qingyuanos.com ansible_host=192.168.12.76 openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
master2.qingyuanos.com ansible_host=192.168.12.77 openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
master3.qingyuanos.com ansible_host=192.168.12.78 openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
node1.qingyuanos.com ansible_host=192.168.12.5 openshift_node_labels="{'region': 'primary', 'zone': 'default'}"
node2.qingyuanos.com ansible_host=192.168.12.6 openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
四.配置dns
如果route 和镜像仓库起启动的情况下部署，配置dnsmasq服务在所有节点
[root@master origin-images-3.9]# cat /etc/dnsmasq.d/origin-dns.conf
no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
dns-forward-max=5000
cache-size=5000
bind-dynamic
except-interface=lo
listen-address=172.28.90.84 #修改自定义域名/本地ip地址
address=/.qyos.com/192.168.1.121 #修改自定义域名/本地外网ip地址
address=/docker-registry.default.svc/172.30.136.106 #私有仓库的IP地址 通过oc get svc 进行查看
# End of config
启动dns并设置为开机自启
[root@master3 ~]# systemctl enable dnsmasq
[root@master3 ~]# systemctl start dnsmasq
五. 安装Hawkular
1.安装相关依赖包
yum install httpd-tools java-1.8.0-openjdk-headless java-1.8.0-openjdk-headless -y
2.load 镜像（所有节点都执行一下）
[root@master2 ~]# ls
anaconda-ks.cfg hawkular.3.6.tar.gz
[root@master2 ~]# tar zxf hawkular.3.6.tar.gz
[root@master2 ~]# cd hawkular/
[root@master2 hawkular]# ls
origin-metrics-cassandra.tar origin-metrics-deployer.tar origin-metrics-hawkular-metrics.tar.gz origin-metrics-heapster.tar
[root@master2 hawkular]# for i in `ls`;do docker load -i $i;done
3.ansible 安装hawkular 注意修改域名（openshift_metrics_hawkular_hostname)这一项
ansible-playbook -i hosts playbooks/byo/openshift-cluster/openshift-metrics.yml \
-e openshift_metrics_install_metrics=True \
-e openshift_metrics_hawkular_hostname=hawkular-metrics-openshift-infra.qingyuanos.com \
-e openshift_metrics_image_version=v3.6.0 \
-e openshift_metrics_image_prefix=openshift/origin- \
-e openshift_metrics_resolution=60s \
-e openshift_metrics_duration=1 \
4.如果想重新安装，先运行下列命令清除上次安装的。
oc delete all --selector="metrics-infra" -n openshift-infra
oc delete sa --selector="metrics-infra" -n openshift-infra
oc delete templates --selector="metrics-infra" -n openshift-infra
oc delete secrets --selector="metrics-infra" -n openshift-infra
oc delete pvc --selector="metrics-infra" -n openshift-infra
六. 仓库添加持久化存储
1.搭建nfs
[root@master3 ~]# yum -y install nfs-utils
2.修改配置文件
[root@master3 ~]# sed -n 5p /etc/idmapd.conf
[root@master3 ~]# cat /etc/exports
/opt/ *(rw,sync,no_subtree_check,no_root_squash,fsid=8)
3.创建相关文件夹
mkdir /opt/gogs
mkdir /opt/influxdb
mkdir /opt/logging
mkdir /opt/mongo
mkdir /opt/monito
mkdir /opt/mysql
mkdir /opt/prometheus
mkdir /opt/registry
4.添加防火墙规则
iptables -I INPUT -p tcp -m tcp --dport 2049 -j ACCEPT
iptables -I INPUT -p udp -m udp --dport 2049 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 53 -j ACCEPT
iptables -I INPUT -p udp -m udp --dport 53 -j ACCEPT
当前环境直接生效，为避免重启之后被刷新，这里需要
[root@master3 ~]# service iptables save
5.启动服务，并添加到开机自启中
[root@master3 ~]# service nfs start
[root@master3 ~]# systemctl enable nfs
6.给私有仓库创建pv，以及pvc 进行持久化
首先切换项目到default
[root@master3 ~]# oc project default
[root@master3 oc]# cat pv.yaml #修改IP地址和路径以及存储大小
apiVersion: v1
kind: PersistentVolume
metadata:
name: registry
spec:
capacity:
storage: 2000Gi
accessModes:
- ReadWriteOnce
nfs:
# FIXME: use the right IP
server: 192.168.12.78
path: /opt/registry
[root@master3 oc]# cat pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: registry-pvc
labels:
app: registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2000Gi
volumeName: registry
[root@master3 oc]# oc create -f pv.yaml
[root@master3 oc]# oc create -f pvc.yaml
修改挂载pv
oc volume deploymentconfigs/docker-registry \
--add --name=registry-storage -t pvc --claim-name=registry-pvc --overwrite
查看是否挂载成功
[root@master3 oc]# oc get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
registry 2000Gi RWO Retain Bound default/registry-pvc 14h
七.安装cicd
1.把镜像传到所有节点
[root@master3 cicd]# ll /oc/cicd/images/
total 1414384
-rw-------. 1 qy qy 151784448 May 18 2017 gogs.tar
-rw-------. 1 qy qy 679963136 May 18 2017 jenkins1.tar
-rw-------. 1 qy qy 616568320 May 18 2017 slave.tar
[root@master3 cicd]# cd /oc/cicd/
[root@master3 cicd]# scp -r images/ master1.qingyuanos.com:/root
2.在所有节点load镜像
[root@master3 images]# for i in `ls`;do docker load -i $i ;done
3.修改pv.yaml文件中的nfs地址以及挂载路径
[root@master3 cicd]# cat pv.yaml | grep 192 -n -A 1
15: server: 192.168.12.78
16- path: /opt/gogs
4.修改nfs挂载目录的权限
[root@master3 opt]# chmod -R 777 gogs/ registry/
5.修改脚本后执行脚本（把$3修改为$2）
[root@master3 cicd]# cat start-cicd.sh | grep "\$2" -n
7:export HUB=`oc get svc -n default|grep docker-registry|awk '{print $2}'`:5000
[root@master3 cicd]# bash -x start-cicd.sh
6.gog初始化
a.首先查看gogs的域名
[root@master3 cicd]# oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
gogs gogs-cicd.qingyuanos.com gogs 3000 None
jenkins jenkins-cicd.qingyuanos.com jenkins <all> None
b.登陆gogs进行初始化
c.登陆录http://gogs-cicd.qingyuanos.com/ 账号gogs，密码password
点击+号，创建新的仓库
只需要填写仓库名称：
openshift-tasks
d.登陆到安装cicd的节点（注意get remote 路径）
[root@master3 cicd]# cd /oc/cicd/openshift-tasks/
[root@master3 cicd]# rm -rf .git ##注意git前面一个点
[root@master3 cicd]# git init
[root@master3 cicd]# git config --global user.email you@example.com
[root@master3 cicd]# git add .
[root@master3 cicd]# git commit -a -m "init"
[root@master3 cicd]# git remote add origin http://gogs-cicd.qingyuanos.com/gogs/openshift-tasks.git
[root@master3 cicd]# git push -u origin master (账号：gogs 密码password)
八.部署普罗米修斯
1.创建一个project，用来部署prometheus.建议名字定为“monitor”
[root@master3 prometheus]# oc adm new-project monitor --node-selector=''
2.解压文件并执行脚本，需要一个参数——在步骤1里创建的project的name：
[root@master3 oc]# tar zxf prometheus.tar.gz
[root@master3 oc]# cd prometheus/
[root@master3 prometheus]# ./sa-scc.sh monitor
3.执行脚本，需要两个参数——镜像仓库地址、project name
[root@master3 prometheus]# REGISTRY=`oc get svc -n default| grep docker-registry| awk '{print $2":5000"}'`
[root@master3 prometheus]# ./tar-to-image.sh $REGISTRY monitor
[root@master3 prometheus]# bash -x prom.sh $REGISTRY monitor
九.安装ux
1.添加权限
oadm policy add-scc-to-user anyuid system:serviceaccount:qybe:default
oadm policy add-cluster-role-to-user cluster-admin admin
oadm policy add-role-to-user cluster-admin admin
oadm policy add-scc-to-group anyuid system:authenticated -n qybe
oadm policy add-scc-to-user anyuid -z qybe
2.执行以下命令
cd images
for i in `ls *.tar`; do docker load -i $i ;done
oc login -u system:admin
oadm policy add-cluster-role-to-user cluster-admin admin
oadm policy add-role-to-user cluster-admin admin
oc new-project qybe --display-name="QingYuan OS"
export HUB=`oc get svc -n default|grep docker-registry|awk '{print $2}'`:5000
docker tag qybe/auth-server $HUB/qybe/auth-server
docker tag qybe/paas-service $HUB/qybe/paas-service
docker tag qybe/ng2-qyweb $HUB/qybe/ng2-qyweb
docker tag 172.30.187.6:5000/dcsp/api-gateway $HUB/qybe/api-gateway
docker tag qybe/eureka-service $HUB/qybe/eureka
docker tag qybe/config-server $HUB/qybe/config-server
docker tag qybe/caas-service:latest $HUB/qybe/alert
docker tag 172.30.117.248:5000/qybe/mysql $HUB/qybe/mysql
docker tag mongo $HUB/qybe/mongo
oc login -u admin -p cow
export TOKEN=`oc whoami -t`
docker login -u admin -p $TOKEN -e ftwang@qingyuanos.com $HUB
docker push $HUB/qybe/auth-server
docker push $HUB/qybe/paas-service
docker push $HUB/qybe/ng2-qyweb
docker push $HUB/qybe/api-gateway
docker push $HUB/qybe/eureka
docker push $HUB/qybe/config-server
docker push $HUB/qybe/mysql
docker push $HUB/qybe/mongo
docker push $HUB/qybe/alert
3.将ux文件夹的qybe全部文件以及隐藏文件拷到nfs的qybe目录
[root@master3 ocux]# cp -r qybe/ /opt/
4.修改yaml文件
vi api-gateway/api-gateway.yaml
- name: openshift_url
value: https://10.0.1.10:8443 #修改成三台master 其中之一的，也可以事域名
- name: prometheus_route
value: http://prometheus-kube-system.eu.qingyuanos.com $修改成prometheus 的route 通过oc get route -n monitor 进行查看
vi config-server/config-server-pv.yaml
nfs:
# FIXME: use the right IP
server: 10.0.1.11 # nfs的IP地址
path: /opt/qybe #根据实际情况进行修改
3. vi mongo/mongo.yaml
nfs:
# FIXME: use the right IP
server: 10.0.1.11 # nfs的IP地址
path: /opt/mongo #根据实际情况进行修改
4. vi mysql/mysql-pv.yaml
nfs:
# FIXME: use the right IP
server: 10.0.1.11 # nfs的IP地址
path: /mnt/mysql #根据实际情况进行修改
vi ng2-qyweb/ng2-qyweb.yaml
- name: QY_OAUTH2_ADDR
value: http://api-gateway.eu.qingyuanos.com/uaa # 注意域名ansible 中hosts 文件指定什么域名这里就改为什么
- name: QY_OS_WS_ADDR
value: wss://10.0.1.10:8443 #修改成三台master 其中之一的，也可以事域名
- name: QY_NODE_ADDR
value: http://ng2-qybe.eu.qingyuanos.com # 注意域名ansible 中hosts 文件指定什么域名这里就改为什么
- name: QY_WS_GW_ADDR
value: ws://ng2-ws-qybe.eu.qingyuanos.com # 注意域名ansible 中hosts 文件指定什么域名这里就改为什么
paas-service/paas-service.yaml
value:jdbc:mysql://mysql-svc:3306/qyweb_paas?useUnicode=true&characterEncoding=utf8&useSSL=false
需创建Mysql库，需修改mysql svc地址。
5.使用oc命令创建pod （注：一定按照以下顺序 创建时候 running一个后 再去创建下一个）
[root@master3 ~]# oc project qybe
[root@master3 ~]# cd /oc/ocux/
[root@master3 ocux]# oc create -f mongo/
[root@master3 ocux]# oc create -f mysql/
插入mysql权限
[root@master3 ocux]# oc exec -it `oc get po | grep mysql | awk '{print $1}'` bash
root@mysql-1-b9wzv:/# mysql -u root -p1q2w3e4r
use qy_oauth2;
insert into user_role values(5,1,1);
create database qyweb_paas;
exit
exit
[root@master3 ocux]# oc create -f config-server/
[root@master3 ocux]# oc create -f auth-server/
[root@master3 ocux]# oc create -f eureka/
[root@master3 ocux]# oc create -f api-gateway/
[root@master3 ocux]# oc create -f paas-service/
[root@master3 ocux]# oc create -f alert/
[root@master3 ocux]# oc create -f ng2-qyweb
6.打一些 label
oc label user admin org.test-org1=admin
oc describe user admin
oc label namespace cicd organization=test-org1
oc label namespace default organization=test-org1
oc label namespace dev organization=test-org1
oc label namespace kube-syste morganization=test-org1
oc label namespace logging organization=test-org1
oc label namespace management-infra organization=test-org1
oc label namespace openshift organization=test-org1
oc label namespace openshift-infra organization=test-org1
oc label namespace qyosiaas organization=test-org1
oc label namespace stage organization=test-org1
oc label namespace monitor organization=test-org1
扫一扫
9542
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
