单点jumpserver结合ceph，在k8s下运行

总结：

实现的效果，数据在ceph中持久化，单点jumpserver可以在k8s集群中任意node迁移。

前提说明：

1，有ceph集群环境

2，有k8s集群环境

创建pool

ceph osd pool create jumpserver 36 36
注： 36 这个值根据实际情况，填写适当的大小。

初始化pool

rbd pool init jumpserver

创建用户

ceph auth get-or-create client.jumpserver mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=jumpserver' -o ceph.client.jumpserver.keyring

注： client.jumpserver 中 jumpserver 为用户名
pool=jumpserver 中 jumpserver 为 pool 名

创建块设备

rbd create  jumpserver/data --size 10G --image-feature layering
rbd create  jumpserver/mysql --size 10G --image-feature layering

创建k8s名称空间

kubectl create namespace jumpserver

创建secret

# 获取到key值
# ceph auth get-key client.jumpserver
AQCvGldfbQ7cKRAALqD+0u7RcISQ4R6CcJVjNQ==

#创建secret
kubectl create secret generic jumpserver-rbd --from-literal=key=AQCvGldfbQ7cKRAALqD+0u7RcISQ4R6CcJVjNQ== -n jumpserver

创建pv

# cat jumpserver-pv-mysql.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jumpserver-mysql
  namespace: jumpserver
  labels: 
    app: jumpserver
    type: mysql
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  rbd:
    secretRef:
      name: jumpserver-rbd
    user: jumpserver 
    monitors:
    - 192.168.8.138:6789
    - 192.168.8.139:6789
    - 192.168.8.140:6789
    pool: jumpserver
    image: mysql

# cat jumpserver-pv-data.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jumpserver-data
  namespace: jumpserver
  labels:
    app: jumpserver
    type: data
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  rbd:
    secretRef:
      name: jumpserver-rbd
    user: jumpserver
    monitors:
    - 192.168.8.138:6789
    - 192.168.8.139:6789
    - 192.168.8.140:6789
    pool: jumpserver
    image: data

创建pvc

# cat jumpserver-pvc-mysql.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jumpserver-mysql
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  selector:
    matchLabels:
      app: jumpserver
      type: mysql
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

# cat jumpserver-pvc-data.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: jumpserver
  name: jumpserver-data
  labels:
    app: jumpserver
    type: data
spec:
  selector:
    matchLabels:
      app: jumpserver
      type: data
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

生成jumpserver的key

# cat key.sh 
if [ ! "$SECRET_KEY" ]; then
  SECRET_KEY=`cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 50`;
  echo "SECRET_KEY=$SECRET_KEY" >> ~/.bashrc;
  echo $SECRET_KEY;
else
  echo $SECRET_KEY;
fi  
if [ ! "$BOOTSTRAP_TOKEN" ]; then
  BOOTSTRAP_TOKEN=`cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 16`;
  echo "BOOTSTRAP_TOKEN=$BOOTSTRAP_TOKEN" >> ~/.bashrc;
  echo $BOOTSTRAP_TOKEN;
else
  echo $BOOTSTRAP_TOKEN;
fi


# sh key.sh

deployment 编排

# cat jumpserver-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jumpserver
  namespace: jumpserver
  labels:
    app: jumpserver
spec:
  strategy:
    rollingUpdate:
      maxSurge: 100%
      maxUnavailable: 100%
    type: RollingUpdate
  replicas: 1
  selector:
    matchLabels:
      app: jumpserver
  template:
    metadata:
      labels:
        app: jumpserver
    spec:
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: jumpserver-data
      - name: mysql
        persistentVolumeClaim:
          claimName: jumpserver-mysql
      containers:
      - name: jumpserver
        image: jumpserver/jms_all:v2.2.2
        volumeMounts:
        - name: data
          mountPath: /opt/jumpserver/data
        - name: mysql
          mountPath: /var/lib/mysql
        env:
        - name: SECRET_KEY
          value: XUEtKwz8ealO9atO8mZZNqBcW4rWY3Kz49P1cr2TquYlxDtadv
        - name: BOOTSTRAP_TOKEN
          value: IbI6XTB8MjDRa6YE
        ports:
        - name: http1
          containerPort: 80
        - name: http2
          containerPort: 2222

创建service，命令行登陆使用NodePort，对外暴露

apiVersion: v1
kind: Service
metadata:
  name: jumpserver-service-web
  namespace: jumpserver
spec:
  selector:
    app: jumpserver 
  ports:
  - name: http1
    port: 80
    targetPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: jumpserver-service-sshd
  namespace: jumpserver
spec:
  selector:
    app: jumpserver
  type: NodePort
  ports:
  - name: http2
    port: 2222
    targetPort: 2222
    nodePort: 32323

web端使用ingress对外暴露

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jumpserver
  namespace: jumpserver
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 10G
spec:
  rules:
  - host: "jumpserver.xqy.com"
    http:
      paths:
      - path: /
        backend: 
          serviceName: jumpserver-service-web
          servicePort: 80 

---

#apiVersion: extensions/v1beta1
#kind: Ingress
#metadata:
#  name: jumpserver-ssh
#  namespace: jumpserver
#  annotations:           
#    #ingress使用那种软件 
#    kubernetes.io/ingress.class: nginx
#    #配置websocket 需要的配置   
#    nginx.ingress.kubernetes.io/configuration-snippet: |
#      proxy_set_header Upgrade "websocket";
#      proxy_set_header Connection "Upgrade";
#spec:
#  rules:
#  - host: "jumpserver-ssh.xqy.com"
#    http:
#      paths:
#      - path: /
#        backend:
#          serviceName: jumpserver-service-sshd
#          servicePort: 2222