KubeEdge-Ianvs及其增量学习终身学习案例

Leventcoco

已于 2023-06-04 00:15:01 修改

阅读量301

点赞数

文章标签：人工智能

于 2023-04-14 12:13:04 首次发布

本文链接：https://blog.csdn.net/Leventcoco/article/details/130141567

版权

1 安装ianvs

1.1 创建虚拟环境

(1). 如果想运行增量学习案例：

python版本选择3.6，过程将比较丝滑。

conda create --name python36 python=3.6

查看已有虚拟环境

conda env list

(2). 如果想运行终身学习案例：

使用python=3.8.5以及torch=1.8

conda create --name py3.8.5_torch1.8_lifelong python=3.8.5

进入该虚拟环境

conda activate py3.8.5_torch1.8_lifelong

1.2 下载源码

因为服务器不能访问github，因此我的方式是，直接下载源码解压至/home/QXY/ianvs

也可以

git clone https://github.com/kubeedge/ianvs.git

在这里插入图片描述

1.3 安装第三方依赖

cd ianvs
python -m pip install ./examples/resources/third_party/*
python -m pip install -r requirements.txt

conda install -c conda-forge pycocotools

1.4 安装ianvs并测试

python setup.py install

ianvs -v

输出正确版本号即为安装成功

在这里插入图片描述

2 运行增量学习demo

2.1 准备数据及模型

2.1.1 数据

先下载到本地，再传至服务器此文件夹中，并解压

/home/QXY/ianvs/dataset

https://kubeedge.obs.cn-north-1.myhuaweicloud.com:443/ianvs/pcb-aoi/dataset.zip

在这里插入图片描述

2.1.2 模型

先下载到本地，再传至服务器此文件夹中，无需解压

/home/QXY/ianvs/initial_model/model.zip

https://kubeedge.obs.cn-north-1.myhuaweicloud.com:443/ianvs/pcb-aoi/model.zip

2.2 配置

2.2.1 testenv.yaml

/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/testenv/testenv.yaml

改一些url，和自己的路径对上，带QXY的是需要修改的地方

testenv:
  # dataset configuration
  dataset:
    # the url address of train dataset index; string type;
    train_url: "/home/QXY/ianvs/dataset/train_data/index.txt"
    # the url address of test dataset index; string type;
    test_url: "/home/QXY/ianvs/dataset/test_data/index.txt"

  # model eval configuration of incremental learning;
  model_eval:
    # metric used for model evaluation
    model_metric:
      # metric name; string type;
      name: "f1_score"
      # the url address of python file
      url: "/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/testenv/f1_score.py"

    # condition of triggering inference model to update
    # threshold of the condition; types are float/int
    threshold: 0.01
    # operator of the condition; string type;
    # values are ">=", ">", "<=", "<" and "=";
    operator: ">="

  # metrics configuration for test case's evaluation; list type;
  metrics:
      # metric name; string type;
    - name: "f1_score"
      # the url address of python file
      url: "/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/testenv/f1_score.py"
    - name: "samples_transfer_ratio"

  # incremental rounds setting of incremental learning; int type; default value is 2;
  incremental_rounds: 2

2.2.2 f1_score.py 和 basemodel.py

不用动

2.2.3 fpn_algorithm.yaml

类似2.2.1

/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/testalgorithms/fpn/fpn_algorithm.yaml

algorithm:
  # paradigm type; string type;
  # currently the options of value are as follows:
  #   1> "singletasklearning"
  #   2> "incrementallearning"
  paradigm_type: "incrementallearning"
  incremental_learning_data_setting:
    # ratio of training dataset; float type;
    # the default value is 0.8.
    train_ratio: 0.8
    # the method of splitting dataset; string type; optional;
    # currently the options of value are as follows:
    #   1> "default": the dataset is evenly divided based train_ratio;
    splitting_method: "default"
  # the url address of initial model for model pre-training; string url;
  initial_model_url: "/home/QXY/ianvs/initial_model/model.zip"

  # algorithm module configuration in the paradigm; list type;
  modules:
    # type of algorithm module; string type;
    # currently the options of value are as follows:
    #   1> "basemodel": contains important interfaces such as train、 eval、 predict and more; required module;
    - type: "basemodel"
      # name of python module; string type;
      # example: basemodel.py has BaseModel module that the alias is "FPN" for this benchmarking;
      name: "FPN"
      # the url address of python module; string type;
      url: "/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/testalgorithms/fpn/basemodel.py"

      # hyperparameters configuration for the python module; list type;
      hyperparameters:
        # name of the hyperparameter; string type;
        - momentum:
            # values of the hyperparameter; list type;
            # types of the value are string/int/float/boolean/list/dictionary
            values:
              - 0.95
              - 0.5
        - learning_rate:
            values:
              - 0.1
      #  2> "hard_example_mining": check hard example when predict ; optional module;
    - type: "hard_example_mining"
      # name of python module; string type;
      name: "IBT"
      # the url address of python module; string type;
      url: "/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/testalgorithms/fpn/hard_example_mining.py"
      # hyperparameters configuration for the python module; list type;
      hyperparameters:
        # name of the hyperparameter; string type;
        # threshold of image; value is [0, 1]
        - threshold_img:
            values:
              - 0.9
        # predict box of image; value is [0, 1]
        - threshold_box:
            values:
              - 0.9

2.2.4 安装FPN_TensorFlow

cd /home/QXY/ianvs/examples/resources/algorithms
python -m pip installFPN_TensorFlow-0.1-py3-none-any.whl

在这里插入图片描述

2.2.5 benchmarkingjob.yaml

类似2.2.1

/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/benchmarkingjob.yaml

benchmarkingjob:
  # job name of bechmarking; string type;
  name: "benchmarkingjob"
  # the url address of job workspace that will reserve the output of tests; string type;
  workspace: "/home/QXY/ianvs/incremental_learning_bench/workspace"

  # the url address of test environment configuration file; string type;
  # the file format supports yaml/yml;
  testenv: "/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/testenv/testenv.yaml"

  # the configuration of test object
  test_object:
    # test type; string type;
    # currently the option of value is "algorithms",the others will be added in succession.
    type: "algorithms"
    # test algorithm configuration files; list type;
    algorithms:
      # algorithm name; string type;
      - name: "fpn_incremental_learning"
        # the url address of test algorithm configuration file; string type;
        # the file format supports yaml/yml
        url: "/home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/testalgorithms/fpn/fpn_algorithm.yaml"

  # the configuration of ranking leaderboard
  rank:
    # rank leaderboard with metric of test case's evaluation and order ; list type;
    # the sorting priority is based on the sequence of metrics in the list from front to back;
    sort_by: [ { "f1_score": "descend" }, { "samples_transfer_ratio": "ascend" } ]

    # visualization configuration
    visualization:
      # mode of visualization in the leaderboard; string type;
      # There are quite a few possible dataitems in the leaderboard. Not all of them can be shown simultaneously on the screen.
      # In the leaderboard, we provide the "selected_only" mode for the user to configure what is shown or is not shown.
      mode: "selected_only"
      # method of visualization for selected dataitems; string type;
      # currently the options of value are as follows:
      #  1> "print_table": print selected dataitems;
      method: "print_table"

    # selected dataitem configuration
    # The user can add his/her interested dataitems in terms of "paradigms", "modules", "hyperparameters" and "metrics",
    # so that the selected columns will be shown.
    selected_dataitem:
      # currently the options of value are as follows:
      #   1> "all": select all paradigms in the leaderboard;
      #   2> paradigms in the leaderboard, e.g., "singletasklearning"
      paradigms: [ "all" ]
      # currently the options of value are as follows:
      #   1> "all": select all modules in the leaderboard;
      #   2> modules in the leaderboard, e.g., "basemodel"
      modules: [ "all" ]
      # currently the options of value are as follows:
      #   1> "all": select all hyperparameters in the leaderboard;
      #   2> hyperparameters in the leaderboard, e.g., "momentum"
      hyperparameters: [ "all" ]
      # currently the options of value are as follows:
      #   1> "all": select all metrics in the leaderboard;
      #   2> metrics in the leaderboard, e.g., "F1_SCORE"
      metrics: [ "f1_score", "samples_transfer_ratio" ]

    # model of save selected and all dataitems in workspace `./rank` ; string type;
    # currently the options of value are as follows:
    #  1> "selected_and_all": save selected and all dataitems;
    #  2> "selected_only": save selected dataitems;
    save_mode: "selected_and_all"

2.3 运行并查看

2.3.1 运行

ianvs -f /home/QXY/ianvs/examples/pcb-aoi/incremental_learning_bench/benchmarkingjob.yaml

2.3.2 查看测试结果

在这里插入图片描述
测试结果在：

/home/QXY/ianvs/incremental_learning_bench/workspace

3 运行终身学习demo

3 Reference

(1). 安装ianvs

https://ianvs.readthedocs.io/en/latest/guides/how-to-install-ianvs.html

(2). 官方文档

https://ianvs.readthedocs.io/

(3). singletask_learning_bench

https://github.com/kubeedge/ianvs/blob/main/docs/guides/quick-start.md

https://blog.csdn.net/VectorYang/article/details/126025276

4 Debug

(1). ERROR: Cannot uninstall ‘certifi’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.