安装
fastapi安装,需要安装自身包和一个ASGI的服务器
pip install fastapi
pip install "uvicorn[standard]"
与WSGI类似,ASGI( Asynchronous Server Gateway Interface)描述了Python Web应用程序和Web服务器之间的通用接口。与WSGI不同的是,ASGI允许每个应用程序有多个异步事件。
常见ASGI如uvicorn,gunicorn,hypercorn
简单使用
参考
浏览器访问:http://127.0.0.1:8000
LLM部署
可以命令行运行:python -m uvicorn main:app --reload
可以放在python文件里:uvicorn.run(app, host='127.0.0.1', port=8000, workers=1)
curl测试:curl -X POST "http://127.0.0.1:8000/ask_post" -H "Content-Type:application/json" -d '{"prompt":"test"}'
curl -X POST "http://127.0.0.1:8000/ask_post" \
-H 'Content-Type: application/json' \
-d '{"query": "早上好,你早饭吃的什么?", "params":{"temperature":0.7, "num_return_sequences":5}}'
完整代码
import os # 注意要放前面
os.environ["WANDB_DISABLED"]="true" # 把wandb关了
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from transformers import AutoTokenizer, AutoModelForCausalLM
from fastapi import FastAPI, Request
import argparse
import uvicorn
import json
MODEL_NAME="Qwen2-7B-Instruct"
app = FastAPI()
def load_model_and_tokenizer(model_name):
path = f'/data1/shares/{model_name}'
model = AutoModelForCausalLM.from_pretrained(path)
tokenizer = AutoTokenizer.from_pretrained(path)
# model = model.to(f'cuda:{device}')
model = model.cuda()
print(f'Load model to cuda successfully... ')
return model, tokenizer
def generate(model, tokenizer, query_text, params):
default_params = {'temperature': 0,
'num_return_sequences': 2,
'max_new_tokens': 256}
if params is None:
params = default_params
else:
for k, v in default_params.items():
if k not in params:
params[k] = v
inputs = query_text
inputs_num = len(inputs)
inputs = tokenizer(inputs, return_tensors="pt")
inputs = {k:v.cuda() for k, v in inputs.items()}
outputs = model.generate(**inputs, **params,
eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id)
generate_strs = tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)
generate_strs = [str[inputs_num:] for str in generate_strs]
return generate_strs, generate_strs[0]
@app.get("/model_name")
def read_root():
return {"MODEL_NAME": MODEL_NAME}
#http://127.0.0.1:8000/ask?query=query
@app.get("/ask")
def ask(query):
global model, tokenizer
return {"query": query, "embed_query": generate(model, tokenizer, query, None)}
@app.post("/ask_post")
async def ask_post(request: Request):
global model, tokenizer # 声明全局变量以便在函数内部使用模型和分词器
json_post_raw = await request.json() # 获取POST请求的JSON数据
json_post = json.dumps(json_post_raw) # 将JSON数据转换为字符串
final_json = json.loads(json_post) # 将字符串转换为Python对象
print(f'>>> final_json:\n{final_json}')
params = None
if "params" in final_json:
params = final_json['params']
query = final_json['query']
data = generate(model, tokenizer, query, params)
return {"data": data, "query": query}
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("--model", type=str, default="Qwen2-7B-Instruct",
choices=['bge-large-zh-v1.5', 'Qwen2-7B-Instruct'])
args = parser.parse_args()
MODEL_NAME = args.model
model, tokenizer = load_model_and_tokenizer(MODEL_NAME)
model.eval()
uvicorn.run(app, host='127.0.0.1', port=8000, workers=1)
request访问
def fun(prompt, params):
data = {"query": prompt, "params": params}
data["params"]["num_return_sequences"] = 1
json_str_data = json.dumps(data, ensure_ascii=False)
response = requests.post(url='http://127.0.0.1:8000/ask_post', headers=self.headers, data=json_str_data)
return response.json()["data"][0]
常见问题
- 运行的时候报错unicorn没装上,但是已经装了
'uvicorn' is a package and cannot be directly executed
:因为在环境变量中找到的python版本和pip对应的python版本不一致。如export PATH=$PATH:/usr/local/python3/bin
。或者使用python -m
命令。【最重要的是,安装的是uvicorn不是unicorn,我装错了。。。】用python -m
命令更安全。避免用其他python的bin目录下的uvicorn。