kubeflow--安装使用pipeline

安装使用pipeline

安装pipeline组件

export PIPELINE_VERSION=0.2.2
kubectl apply -f https://storage.googleapis.com/ml-pipeline/pipeline-lite/$PIPELINE_VERSION/crd.yaml
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -f https://storage.googleapis.com/ml-pipeline/pipeline-lite/$PIPELINE_VERSION/namespaced-install.yaml

查看安装结果

kubectl get all -nj kubeflow

[root@master pipelines]# kubectl get pods -n kubeflow
NAME                                                  READY   STATUS             RESTARTS   AGE
pod/ambassador-cd476cb56-jk79h                        2/2     Running            0          26h
pod/ambassador-cd476cb56-md5x5                        2/2     Running            0          26h
pod/ambassador-cd476cb56-qr5l4                        2/2     Running            0          26h
pod/centraldashboard-7d45f8cbc8-vdksx                 1/1     Running            0          26h
pod/conditional-execution-pipeline-k6sm2-1061071197   0/2     Completed          0          156m
pod/conditional-execution-pipeline-k6sm2-1124728528   0/2     Completed          0          159m
pod/conditional-execution-pipeline-k6sm2-2948537851   0/2     Completed          0          156m
pod/controller-manager-5cbdfbc5bd-lnwfc               1/1     Running            0          3h45m
pod/jupyter-mockdone                                  1/1     Running            0          24h
pod/jupyter-yanggang                                  1/1     Running            0          24h
pod/metadata-deployment-88c674dbb-s4t8w               0/1     ImagePullBackOff   0          3h45m
pod/metadata-envoy-7566f8975b-qswk7                   1/1     Running            0          3h45m
pod/metadata-writer-69677d58cd-cw6fq                  1/1     Running            0          3h45m
pod/minio-b67997596-pc87g                             1/1     Running            0          3h45m
pod/ml-pipeline-5569cb7bdb-cqmls                      1/1     Running            0          3h45m
pod/ml-pipeline-persistenceagent-f5db76589-6x8g8      1/1     Running            0          3h45m
pod/ml-pipeline-scheduledworkflow-67567b8cc5-mdkzl    1/1     Running            0          3h45m
pod/ml-pipeline-ui-ccf5995c8-2xn9b                    1/1     Running            0          3h45m
pod/ml-pipeline-viewer-crd-5fc8d465c8-gj4bd           0/1     ImagePullBackOff   0          3h45m
pod/ml-pipeline-visualizationserver-f94b84d58-qpqtf   1/1     Running            0          3h45m
pod/mysql-5f8b56b877-4dg8k                            1/1     Running            0          3h45m
pod/tf-hub-0                                          1/1     Running            0          26h
pod/tf-job-dashboard-9fd7d588-z9nc8                   1/1     Running            0          26h
pod/tf-job-operator-8d98cd89b-vbv97                   1/1     Running            0          26h
pod/workflow-controller-864d8bd5f9-vdmhs              1/1     Running            0          3h45m
pod/xgboost-trainer-6rcpz-1335953530                  0/2     Error              0          155m
pod/xgboost-trainer-6rcpz-1524690980                  1/2     ImagePullBackOff   0          145m

查看service

[root@master 2020]# kubectl get svc -n kubeflow
NAME                              TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
ml-pipeline                       ClusterIP      10.97.54.58      <none>        8888/TCP,8887/TCP   3h48m
ml-pipeline-ui                    ClusterIP      10.111.88.29     <none>     80:32591/TCP        3h48m
ml-pipeline-visualizationserver   ClusterIP      10.99.7.16       <none>        8888/TCP            3h48m

编辑service ml-pipeline-ui ，修改其类型为LoadBalancer，暴露给外部访问

kubectl -n kubeflow edit svc ml-pipeline-ui
修改:
spec:
 clusterIP: 10.111.88.29
 externalTrafficPolicy: Cluster
 ports:
 - nodePort: 32591
   port: 80
   protocol: TCP
   targetPort: 3000
 selector:
   app: ml-pipeline-ui
   application-crd-id: kubeflow-pipelines
 sessionAffinity: None
 type: LoadBalancer  ###修改为LoadBalancer

再次查看该服务，可获得其loadbalancer地址

[root@master pipelines]# kubectl -n kubeflow get svc |grep pipe
ml-pipeline                       ClusterIP      10.97.54.58      <none>        8888/TCP,8887/TCP   3h52m
ml-pipeline-ui                    LoadBalancer   10.111.88.29     10.18.5.34    80:32591/TCP        3h52m
ml-pipeline-visualizationserver   ClusterIP      10.99.7.16       <none>        8888/TCP            3h52m

使用kubeflow运行自定义pipeline

浏览器访问 http://10.18.5.34 ，进入管理页面

上传编译好的pipeline压缩文件

创建后如下所示

创建运行实例(create run)

执行start,在k8s集群内查看pod

pod/conditional-execution-pipeline-9hss8-2599113536   0/2     Completed          0          112s
pod/conditional-execution-pipeline-9hss8-2619729557   0/2     Completed          0          108s
pod/conditional-execution-pipeline-9hss8-4071122163   0/2     Completed          0          104s

查看页面结果，pipeline成功运行

本文使用的测试pipeline为官方示例，使用步骤如下

##获取代码
git clone https://github.com/kubeflow/pipelines.git
cd pipelines/samples/core
##安装sdk
pip install https://storage.googleapis.com/ml-pipeline/release/0.1.20/kfp.tar.gz --upgrade   
##编译打包
dsl-compile --py .\condition\condition.py --output test.tgz
##得到的test.tgz即为上传时所需的包