最近流行的ChatGPT,好奇他的流文字是怎么传输,所以去研究了,并复现了一下。
后端用的是langchain+fastapi,用到了starlette的一个插件包,sse_starlette返回
先定义langchain的CallbackHandler:
import queue
import sys
from typing import Any, Dict, List, Union
from langchain.callbacks.base import BaseCallbackHandler
from langchain.schema import LLMResult
class StreamingCallbackHandler(BaseCallbackHandler):
def __init__(self):
self.tokens = queue.Queue()
self.stream_end_flag = False
super(BaseCallbackHandler, self).__init__()
def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
self.tokens.put(token)
sys.stdout.write(token)
sys.stdout.flush()
def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> None:
self.tokens.put(StopIteration)
import asyncio
from fastapi import FastAPI
from typing import Annotated
from langchain.llms import OpenAI
from sse_starlette.sse import EventSourceResponse
from stream_callback import StreamingCallbackHandler
app = FastAPI()
@app.post('/simpleChat', response_class=EventSourceResponse)
async def simple_chat(data: Annotated[dict, Body()]):
app_input = data.get('appInput')
callback_handler = StreamingCallbackHandler()
chat_prompt = PromptTemplate(
input_variables=['human_input'],
template='''{human_input}'''
)
chain = LLMChain(
llm=OpenAI(
temperature=0.8,
request_timeout=setting.REQUEST_TIMEOUT,
max_retries=1,
max_tokens=2048,
streaming=True,
),
prompt=chat_prompt
)
task = chain.aapplly([{'human_input': app_input}], callbacks=[callback_handler])
loop = asyncio.get_event_loop()
asyncio.run_coroutine_threadsafe(task, loop)
def resp():
while True:
try:
tk = callback_handler.tokens.get()
if tk is StopIteration: raise tk
yield tk
except StopIteration:
raise StopIteration
return EventSourceResponse(resp())
前端用的是vue, 由于源生sse并不支持post的方式请求,因此使用fetch-event-source包进行post的请求。
npm install @microsoft/fetch-event-source # 使用npm工具安装
<template>
<div>
<span>{{ content }}</span>
</div>
<div>
<el-form :model="form">
<el-form-item>
<el-input v-model="form.appInput" />
<el-button type="primary" @click="submitChat"/>
</el-form-item>
</el-form>
</div>
</template>
<script setup lang='ts'>
import {fetchEventSource} from "@microsoft/fetch-event-source"
const form = reactive({
appInput: ''
});
const content = ref<string>('')
const submitChat = () => {
if (form.appInput !== ''){
content.value = ''
fetchEventSource('/api/v1/simpleChat', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body:JSON.stringify({
chatInput: form.appInput,
}),
onmessage(ev) {
content.value+=ev.data
}
})
}
}
</script>