Argo 101 - 3 : Download & Upload with 「script」 template

上一篇中成功连通Argo与Storage - Minio，那么现在就可以使用 script 这种 template 来 upload 和 download 数据；对于Data Engineering或者Machine Learning来说，这一步都是必须的；

download

既然是download，那首先需要了解download的 from 和 to 都是哪儿

• from，肯定是Minio了，但是连接Minio需要mc(Minio client)的帮助，所以需要使用minio/mc这个image
• to，肯定是Argo的pod了，但是pod并没有持久化存储数据的能力；更重要的，如果多个pod需要共享数据，那么这里就必须使用 pv/pvc，

最终的脚本如下：

script:
  image: minio/mc:RELEASE.2020-04-17T08-55-48Z
  volumeMounts:
  - name: "rex-pvc"
    mountPath: /assets
  command: [sh]
  source: |
    mc config host add minio <your host> <accesskey> <secretkey>
    mc cp minio/minio-seldon/data/input-data.txt /assets/input-data.txt

upload

script:
  image: minio/mc:RELEASE.2020-04-17T08-55-48Z
  volumeMounts:
  - name: "rex-pvc"
    mountPath: /assets
  command: [sh]
  source: |
    mc config host add minio <your host> <accesskey> <secretkey>
    mc cp /assets/input-data.txt minio/minio-seldon/data/output-data.txt

最终的Yaml

逻辑很简单，并没有涉及数据的处理，仅仅是将input-data.txt从Minio上download下来，换个名字再upload上去;

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: rex-download-upload
  namespace: argo
spec:
  entrypoint: download-upload-process
  volumeClaimTemplates:
  - metadata:
      name: "rex-pvc"
      ownerReferences:
      - apiVersion: argoproj.io/v1alpha1
        blockOwnerDeletion: true
        kind: Workflow
        name: "{{`{{workflow.name}}`}}"
        uid: "{{`{{workflow.uid}}`}}"
    spec:
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: "1Mi"
  templates:
  - name: download-upload-process
    steps:
    - - name: download-object-store
        template: download-object-store-template
    - - name: upload-object-store
        template: upload-object-store-template

  - name: download-object-store-template
    script:
      image: minio/mc:RELEASE.2020-04-17T08-55-48Z
      volumeMounts:
      - name: "rex-pvc"
        mountPath: /assets
      command: [sh]
      source: |
        mc config host add minio <your host> <accesskey> <secretkey>
        mc cp minio/minio-seldon/data/input-data.txt /assets/input-data.txt

  - name: upload-object-store-template
    script:
      image: minio/mc:RELEASE.2020-04-17T08-55-48Z
      volumeMounts:
      - name: "rex-pvc"
        mountPath: /assets
      command: [sh]
      source: |
        mc config host add minio <your host> <accesskey> <secretkey>
        mc cp /assets/input-data.txt minio/minio-seldon/data/output-data.txt

在download数据之前，先准备好input-data.txt

之后就可以argo submit了

Argoworkflow成功运行完毕，去Minio看一下，有没有output-data.txt

Nice，没有问题！