Xinference如何注册自定义模型

环境：

Xinference

问题描述：

Xinference如何注册自定义模型

解决方案：

1.写个model_config.json，内容如下

{
  "version": 1,
  "context_length": 2048,
  "model_name": "custom-llama-3",
  "model_lang": [
    "en",
    "ch"
  ],
  "model_ability": [
    "generate",
    "chat"
  ],
  "model_family": "other", 
  "model_specs": [
    {
      "model_format": "ggufv2",
      "model_size_in_billions": 8,
      "quantizations": [
        "4-bit",
        "8-bit",
        "none"
      ],
      "model_id": "Llama3-8B-Chinese-Chat.Q6_K",
      "model_uri": "/mnt/e/7B/koboldcpp1.63/koboldcpp1.63",
      "model_file_name_template": "llama-3-8b-ggmlv3.{quantization}.bin"
    }
  ]
}

2.运行注册命令

 xinference register -f model_config.json

3.查看自定义模型，出现了就成功
4.最后运行模型