Tensorflow multi-models and multi-version

1. Multi-models

预备备：docker使用的是17.05.0-ce及以上的版本，之前用的13.XX的版本没有--mount这样的命令

0）下载安装tensorflow

#官网上抄的，啊哈哈

docker pull tensorflow/serving

git clone https://github.com/tensorflow/serving

1）创建一个多模型配置文件

#从tensorflow-serving git文档上抄下来的，啊哈哈

sudo vim {你的tf-serving配置文件根目录}/models.config

贴入以下内容

model_config_list: {

  config: {

    name: "half_plus_two",

    base_path: "/models/half_plus_two",

    model_platform: "tensorflow"

  },

  config: {

    name: "half_plus_three",

    base_path: "/models/half_plus_three",

    model_platform: "tensorflow"

  }

}

2）启动一个docker container

sudo docker run -p 8500:8500 -p 8501:8501 \

--mount type=bind,source={你的tf-serving根目录}/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_cpu,target=/models/half_plus_two\

--mount type=bind,source={你的tf-serving根目录}/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_three,target=/models/half_plus_three\

--mount type=bind,source={你的tf-serving配置文件根目录}/models.config,target=/models/models.config \

-t tensorflow/serving --model_config_file=/models/models.config&

3）测试多个模型调用

curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://127.0.0.1:8501/v1/models/half_plus_two:predict

curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://127.0.0.1:8501/v1/models/half_plus_three:predict

 

2. How to serve multiple versions of model via standard tensorflow serving docker image?

This require a ModelServerConfig, which will be supported by the next docker image tensorflow/serving release 1.11.0 (available since 5. Okt 2018). Until then, you can create your own docker image, or use tensorflow/serving:nightly or tensorflow/serving:1.11.0-rc0 as stated here. See that thread for how to implement multiple models.

If you on the other hand want to enable multiple versions of a single model, you can use the following config file called "models.config":

model_config_list: {
    config: {
        name: "my_model",
        base_path: "/models/my_model",
        model_platform: "tensorflow",
        model_version_policy: {
            all: {}
        }
    }
}

here "model_version_policy: {all:{ } }" make every versions of the model available. Then run the docker:

docker run -p 8500:8500 8501:8501 \
    --mount type=bind,source=/path/to/my_model/,target=/models/my_model \
    --mount type=bind,source=/path/to/my/models.config,target=/models/models.config \
    -t tensorflow/serving:nightly --model_config_file=/models/models.config

Edit:
Now that version 1.11.0 is available, you can start by pulling the new image:

docker pull tensorflow/serving

Then run the docker image as above, using tensorflow/serving instead of tensorflow/serving:nightly.