在kubernetes中部署springcloud dataflow

https://dataflow.spring.io/docs/2.5.0.SNAPSHOT/installation/kubernetes/kubectl/

Version: 2.5.0 SHAPSHOT

------------------------------------------------
docker 18.06
kubernetes 1.17.3
------------------------

Deploying with kubectl

下载源码
git clone https://github.com/spring-cloud/spring-cloud-dataflow

cd spring-cloud-dataflow

git checkout master
-------------------------------------------

Deploy Services, Skipper, and Data Flow
You must deploy a number of services and the Data Flow server. The following subsections describe how to do so:

Deploy MySQL
Deploy Prometheus and Grafana
Create Data Flow Role Bindings and Service Account
Deploy Skipper
Deploy the Data Flow Server
-------------------------------------------

Deploy MySQL

------
[root@master spring-cloud-dataflow]# kubectl create -f src/kubernetes/mysql/
---
deployment.apps/mysql created
persistentvolumeclaim/mysql created
secret/mysql created
service/mysql created

------
查看
[root@master spring-cloud-dataflow]# kubectl get all -l app=mysql
------
清理
kubectl delete all,pvc,secrets -l app=mysql

-------------------------------------------
Deploy Prometheus and Grafana

------
Run the following commands to create the cluster role, binding, and service account:
---
kubectl create -f src/kubernetes/prometheus/prometheus-clusterroles.yaml
kubectl create -f src/kubernetes/prometheus/prometheus-clusterrolebinding.yaml
kubectl create -f src/kubernetes/prometheus/prometheus-serviceaccount.yaml
输出信息
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
serviceaccount/prometheus created

------
Run the following commands to deploy Prometheus RSocket Proxy:
---
kubectl create -f src/kubernetes/prometheus-proxy/
输出信息
clusterrolebinding.rbac.authorization.k8s.io/prometheus-proxy created
deployment.apps/prometheus-proxy created
service/prometheus-proxy created
serviceaccount/prometheus-proxy created

---
查看
kubectl get all -l app=prometheus-proxy
---
清理
kubectl delete all,cm,svc -l app=prometheus-proxy

------
Run the following commands to deploy Prometheus:
---
kubectl create -f src/kubernetes/prometheus/prometheus-configmap.yaml
kubectl create -f src/kubernetes/prometheus/prometheus-deployment.yaml
kubectl create -f src/kubernetes/prometheus/prometheus-service.yaml
输出信息
configmap/prometheus created
deployment.apps/prometheus created
service/prometheus created

---
查看
kubectl get all -l app=prometheus

---
清理
kubectl delete clusterrole,clusterrolebinding,sa -l app=prometheus

------
Run the following command to deploy Grafana:
---
kubectl create -f src/kubernetes/grafana/
输出信息
configmap/grafana created
deployment.apps/grafana created
secret/grafana created
service/grafana created

---
查看
kubectl get all -l app=grafana
---
清理
kubectl delete all,cm,svc,secrets -l app=grafana

-------------
Grafana dashboard 默认用户 admin/password

如果不部署Prometheus 和 Grafana, 修改文件src/kubernetes/server/server-config.yaml

-------------------------------------------------
Create Data Flow Role Bindings and Service Account

------
To create Role Bindings and Service account, run the following commands:
---
kubectl create -f src/kubernetes/server/server-roles.yaml
kubectl create -f src/kubernetes/server/server-rolebinding.yaml
kubectl create -f src/kubernetes/server/service-account.yaml
输出信息
role.rbac.authorization.k8s.io/scdf-role created
rolebinding.rbac.authorization.k8s.io/scdf-rb created
serviceaccount/scdf-sa created

---
kubectl get roles and kubectl get sa
---
kubectl delete role scdf-role
kubectl delete rolebinding scdf-rb
kubectl delete serviceaccount scdf-sa

-------------------------------------------------

部署rabbitmq
$ kubectl create -f src/kubernetes/rabbitmq/

部署skipper
$ kubectl create -f src/kubernetes/skipper/skipper-config-rabbit.yaml

Run the following commands to start Skipper as the companion server for Spring Cloud Data Flow:

$ kubectl create -f src/kubernetes/skipper/skipper-deployment.yaml
$ kubectl create -f src/kubernetes/skipper/skipper-svc.yaml

重启pod的方式
kubectl get pod {podname} -n {namespace} -o yaml | kubectl replace --force -f -

$ kubectl get pod scdf-server-646bbfb7-rsgbj -o yaml | kubectl replace --force -f -

-------------------------------------------------

Deploy Data Flow Server

定义文件
src/kubernetes/server/server-deployment.yaml

数据流服务器使用Fabric8 Java客户端库连接到Kubernetes集群。 
有多种方法可以配置客户端以连接群集。 
我们使用环境变量来设置将数据流服务器部署到Kubernetes时所需的值。 
我们还使用Spring Cloud Kubernetes库访问Kubernetes ConfigMap和Secrets设置。

MySQL机密位于src/kubernetes/mysql/mysql-secrets.yaml文件中。
如果您修改了MySQL的密码，则应在src/kubernetes/mysql/mysql-secrets.yaml文件中进行更改。
任何秘密都必须以base64编码提供。

To create the configuration map with the default settings, run the following command:
---
默认设置
kubectl create -f src/kubernetes/server/server-config.yaml

Now you need to create the server deployment, by running the following commands:
---
kubectl create -f src/kubernetes/server/server-svc.yaml
kubectl create -f src/kubernetes/server/server-deployment.yaml
输出信息
service/scdf-server created
deployment.apps/scdf-server created

---
查看
kubectl get all -l app=scdf-server
---
清理
kubectl delete all,cm -l app=scdf-server

查看外部访问地址
[root@master spring-cloud-dataflow]# kubectl get svc scdf-server
NAME          TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
scdf-server   LoadBalancer   10.102.20.49   <pending>     80:31402/TCP   95s

-----------------------------------------------------
验证部署状态

[root@master spring-cloud-dataflow]# kubectl get pod
NAME                                READY   STATUS     RESTARTS   AGE
grafana-cd64cf7dd-p2wss             1/1     Running    0          39m
mysql-b94654bd4-vfv62               0/1     Pending    0          65m
nginx-5b6fb6dd96-jb66v              1/1     Running    0          4h39m
nginx-5b6fb6dd96-vgfb5              1/1     Running    0          4h38m
prometheus-85c4bb756-drmk6          1/1     Running    0          41m
prometheus-proxy-658c445796-p8l9n   1/1     Running    0          56m
scdf-server-646bbfb7-zjgnl          0/1     Init:0/1   0          22m

---------
发现scdf-server状态异常
kubectl describe pod scdf-server-646bbfb7-zjgnl

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  22m   default-scheduler  Successfully assigned default/scdf-server-646bbfb7-zjgnl to master
  Normal  Pulling    22m   kubelet, master    Pulling image "busybox"
  Normal  Pulled     22m   kubelet, master    Successfully pulled image "busybox"
  Normal  Created    22m   kubelet, master    Created container init-mysql-wait
  Normal  Started    22m   kubelet, master    Started container init-mysql-wait

发现mysql状态异常
kubectl describe pod mysql-b94654bd4-vfv62

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  2m58s (x51 over 68m)  default-scheduler  error while running "VolumeBinding" filter plugin for pod "mysql-b94654bd4-vfv62": pod has unbound immediate PersistentVolumeClaims

---------------------------------------------
kubectl get pvc mysql
kubectl edit pvc mysql

kubectl create -f src/kubernetes/mysql/

先删除
kubectl delete -f src/kubernetes/mysql/

kubectl describe pod mysql-b94654bd4-wsgqz

kubectl get pv,pvc

kubectl describe pv,pvc
---
no persistent volumes available for this claim and no storage class is set

一番查找和尝试,得到上面的错误
用这个提示搜到一篇博客文章   --通过搭建MySQL掌握k8s（Kubernetes）重要概念（上）：网络与持久卷
看来是基础问题, 先学习一下

1. docker 部署 mysql

docker run --name test-mysql -p 3306:33060 -e MYSQL_ROOT_PASSWORD=root -d mysql:5.7
docker logs test-mysql
docker ps
docker inspect test-mysql
docker exec -it test-mysql mysql -uroot -proot

2. 在k8s上安装MySQL分成三个部分，创建部署文件，创建服务文件和安装测试。

mysql-deployment.yaml
mysql-service.yaml
mysql-pv.yaml
mysql-pvc.yaml

storageclass, pv, pvc  本地模式  名称,路径,容量大小 一一对应才能正常启动
容器上还支持nfs等存储

参考
https://segmentfault.com/a/1190000019612162
https://www.jianshu.com/p/be8d7ca39f38
https://stackoverflow.com/questions/55112404/local-persistent-volume-1-nodes-didnt-find-available-persistent-volumes-to-bi
----------------------------------------------------------------------------

kubectl describe pod scdf-server-646bbfb7-zjgnl

------
如下错误
Events:
  Type     Reason   Age                   From             Message
  ----     ------   ----                  ----             -------
  Normal   Pulling  13h (x5 over 13h)     kubelet, master  Pulling image "springcloud/spring-cloud-dataflow-server:2.5.0.BUILD-SNAPSHOT"
  Warning  Failed   15m (x175 over 13h)   kubelet, master  Error: ImagePullBackOff
  Warning  Failed   2m12s (x11 over 13h)  kubelet, master  Error: ErrImagePull
  Warning  Failed   2m12s (x3 over 13h)   kubelet, master  Failed to pull image "springcloud/spring-cloud-dataflow-server:2.5.0.BUILD-SNAPSHOT": rpc error: code = Unknown desc = context canceled

镜像拉取失败
-------------

先在docker中拉取
docker pull springcloud/spring-cloud-dataflow-server:2.5.0.BUILD-SNAPSHOT
-----------------
可能需要将registry-1.docker.io对应的ip配置到/etc/hosts中
yum install bind-utils
dig @114.114.114.114 registry-1.docker.io
dig @223.5.5.5 registry-1.docker.io
当前docker环境所使用的dns查询
vi /etc/host
34.197.189.129 registry-1.docker.io
systemctl restart docker
重新docker pull 
-----------------

docker images
springcloud/spring-cloud-dataflow-server                          2.5.0.BUILD-SNAPSHOT   e137e5c2dc27        9 hours ago         342MB

------------------------
kubectl get deployments
kubectl get pods -l app=scdf-server -o wide
kubectl get services scdf-server
---
[root@master ~]# kubectl get services scdf-server
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
scdf-server   LoadBalancer   10.96.195.218   <pending>     80:31348/TCP   17m

------------------------------------------------------
kubernetes：发布的 loadbanlence service 一直 pending
问题原因：
查看了官方文档，自己搭建的k8s集群 （公有云环境除外），是没有LB能力的
解决办法：
GitHub上面有一个项目https://github.com/danderson/metallb，metallb可以完美解决问题
--------
参考https://blog.csdn.net/textdemo123/article/details/99400237?utm_source=distribute.pc_relevant.none-task

搭建好了metallb之后
[root@master yaml-resource]# kubectl get services scdf-server
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
scdf-server   LoadBalancer   10.96.195.218   192.168.1.152   80:31348/TCP   4h12m

http://192.168.1.152/ 无法访问

删除重新建立,应该是忘记执行server-config.yaml导致
-----------------
[root@master spring-cloud-dataflow]# kubectl delete all,cm -l app=scdf-server

[root@master spring-cloud-dataflow]# kubectl create -f src/kubernetes/server/server-config.yaml
configmap/scdf-server created
[root@master spring-cloud-dataflow]# kubectl create -f src/kubernetes/server/server-svc.yaml
service/scdf-server created
[root@master spring-cloud-dataflow]# kubectl create -f src/kubernetes/server/server-deployment.yaml
deployment.apps/scdf-server created

------------------------------
等待几分钟后
访问http://192.168.1.152/,返回了json数据

仪表盘
http://192.168.1.152/dashboard

左侧不展示 /about 500
查看日志
kubectl logs -f scdf-server-646bbfb7-rsgbj
---
错误, 之前没有部署skipper
Not enough variable values available to expand 'SKIPPER_SERVICE_HOST'
重新部署后,解决