什么是fabric8
fabric8是个开源的ci工具,通过这个工具,开发者可以在上面进行代码编写,版本管理,测试以及发布等一系列流程。fabric8依赖于k8s,而k8s则依赖于docker,我们的最终目的是搭建一个比较大型的ci开发流程工具网站。
前提:
1.docker安装
2.k8s 安装
3.ceph集群
在k8s集群上安装fabric8
1.master节点操作:
curl -sS https://get.fabric8.io/download.txt | bash
$ export PATH=$PATH:$HOME/.fabric8/bin
如果下载贼慢,可以从官网上直接下载:
https://github.com/fabric8io/fabric8/releases
坑点:
一开始下载最新的版本0.4.176,在执行的过程中各种问题,换成0.4.167后各种pod安装成功。
2.部署:
gofabric8 deploy -d ss.ss.com
kubectl get pods -n default
查看default空间pod发现有一些已经起来了,但是有几个还是创建的状态。这时候查看pvc状态,都是pengding,需要创建STORAGECLASS 动态卷来支持pvc的创建。
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fabric8-docker-registry-storage pending pvc-d72b8dca-7672-11ea-a7bd-ac162d7b06e0 5Gi 1d
gogs-data pending pvc-d7479f36-7672-11ea-a7bd-ac162d7b06e0 100Mi 1d
jenkins-jobs pending pvc-d7663ae6-7672-11ea-a7bd-ac162d7b06e0 1Gi 1d
jenkins-mvn-local-repo pending pvc-9dadcb49-772c-11ea-a7bd-ac162d7b06e0 1Gi 21h
jenkins-workspace pending pvc-d7a32baa-7672-11ea-a7bd-ac162d7b06e0 1Gi 1d
nexus-storage pending pvc-d7c1c4b7-7672-11ea-a7bd-ac162d7b06e0 100Mi 1d
3.配置ceph的存储类
我这里使用的是现成的ceph集群。首先在k8s上创建ceph集群的secret凭证。
$ grep key /etc/ceph/ceph.client.admin.keyring |awk '{printf "%s", $NF}'|base64
QVFCWXB0RmIzK2dqTEJBQUtsYm4vaHU2NWZ2eHlaaGRnM2hwc1E9PQ==
$ vim ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
namespace: default
type: "kubernetes.io/rbd"
data:
key: QVFCWXB0RmIzK2dqTEJBQUtsYm4vaHU2NWZ2eHlaaGRnM2hwc1E9PQ==
$ kubectl apply -f ceph-secret.yaml
secret/ceph-secret created
$ kubectl get secret
ceph的storageclass
$ vim ceph-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: jax-ceph
provisioner: kubernetes.io/rbd
parameters:
monitors: 10.10.3.150:6789,10.10.3.151:6789,10.10.3.152:6789
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: default
pool: rbd
userId: admin
userSecretName: ceph-secret
$ kubectl apply -f ceph-storageclass.yaml
storageclass.storage.k8s.io/jax-ceph created
$ kubectl get storageclass
这个时候再次查看pvc状态
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fabric8-docker-registry-storage Bound pvc-d72b8dca-7672-11ea-a7bd-ac162d7b06e0 5Gi RWO standard 1d
gogs-data Bound pvc-d7479f36-7672-11ea-a7bd-ac162d7b06e0 100Mi RWO standard 1d
jenkins-jobs Bound pvc-d7663ae6-7672-11ea-a7bd-ac162d7b06e0 1Gi RWO standard 1d
jenkins-mvn-local-repo pengding standard
jenkins-workspace Bound pvc-d7a32baa-7672-11ea-a7bd-ac162d7b06e0 1Gi RWO standard 1d
nexus-storage Bound pvc-d7c1c4b7-7672-11ea-a7bd-ac162d7b06e0 100Mi RWO standard 1d
查看jenkins-mvn-local-repo 详情,发现这个pvc的访问方式是RWM,而ceph rdb只支持ROM和RWO,不支持RWM。
cephfs是支持RWM的,但是k8s1.9不支持cephfs的strageclass。……😦
我这里将 jenkins-mvn-local-repo ,手动改成RWO
$ kubectl describe pvc jenkins-mvn-local-repo
$ kubectl get pvc jenkins-mvn-local-repo -o yaml > /tmp/pvc-jenkins-mvn-local-repo.yaml
$ kubectl delete pvc jenkins-mvn-local-repo
$ vi /tmp/pvc-jenkins-mvn-local-repo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
volume.beta.kubernetes.io/storage-class: standard
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/rbd
creationTimestamp: 2020-04-04T12:50:20Z
labels:
group: io.fabric8.devops.apps
project: jenkins
provider: fabric8
version: 2.2.311
name: jenkins-mvn-local-repo
namespace: default
resourceVersion: "85826000"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/jenkins-mvn-local-repo
uid: d784bbf7-7672-11ea-a7bd-ac162d7b06e0
spec:
accessModes:
- ReadWriteOnce ##改这里
resources:
requests:
storage: 1Gi
$ kubectl create -f /tmp/pvc-jenkins-mvn-local-repo.yaml
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fabric8-docker-registry-storage Bound pvc-d72b8dca-7672-11ea-a7bd-ac162d7b06e0 5Gi RWO standard 1d
gogs-data Bound pvc-d7479f36-7672-11ea-a7bd-ac162d7b06e0 100Mi RWO standard 1d
jenkins-jobs Bound pvc-d7663ae6-7672-11ea-a7bd-ac162d7b06e0 1Gi RWO standard 1d
jenkins-mvn-local-repo Bound pvc-9dadcb49-772c-11ea-a7bd-ac162d7b06e0 1Gi RWO standard 21h
jenkins-workspace Bound pvc-d7a32baa-7672-11ea-a7bd-ac162d7b06e0 1Gi RWO standard 1d
nexus-storage Bound pvc-d7c1c4b7-7672-11ea-a7bd-ac162d7b06e0 100Mi RWO standard 1d
$ gofabric8 validate ##检查fabric8安装是否完成。
可以看到所有的pvc都已经bound上了。
其中有个pod gogs没启动起来,看日志显示目录权限的问题,可以在containers的上面一行加上下面配置
initContainers:
- name: init
image: busybox
command:
- chmod
- '777'
- /app/gogs/data
resources: {}
volumeMounts:
- name: gogs-data
mountPath: /app/gogs/data
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
截图:
现在看看fabric8依赖的pod已经running,并且gofabric8 validate
检查fabric8安装成功。
4.为infress创建外部访问,暴露在k8s集群外(loadblancer)
$ kubectl expose deployment gofabric8 --type=LoadBalancer --name=myfabric
查看端口
$ kubectl get service
myfab LoadBalancer 10.98.231.216 <pending> 9090:32451/TCP,9191:31460/TCP 18h
这时我们访问master所在机器的32451端口就可以了。
5.为ingress授权
访问fabric8 web的时候各种接口403,报错如下:
secrets is forbidden: User "system:serviceaccount:default:default" cannot list secrets
授权,复制cluster-admin的权限
kubectl create clusterrolebinding nginx-ingress-clusterrolebinding --clusterrole=cluster-admin --serviceaccount=default:default --namespace=default
这时再次访问web就正常了。