JARVIS项目源码分析 - awesome_chat.py代码分析1

一、JARVIS项目概述：用 LLM 连接众多 AI 模型以解决复杂 AI 任务的接口

      我们引入了一个协作系统，该系统由作为控制器的 LLM和作为协作执行者的众多专家模型组成（来自 HuggingFace Hub）。我们系统的工作流程包括四个阶段：

• 任务规划：使用ChatGPT分析用户的请求，了解他们的意图，并将其拆解成可能的可解决任务。

• 模型选择：为解决计划任务，ChatGPT 根据描述选择托管在 Hugging Face 上的专家模型。

• 任务执行：调用并执行每个选定的模型，并将结果返回给 ChatGPT。

• Response Generation : 最后使用ChatGPT整合所有模型的预测，生成response。

二、awesome_chat.py 是用户在 CLI 模式下运行 Jarvis的代码的原文件

    示例运行命令：

    $cd server

    $python awesome_chat.py --config config.yaml --mode cli

三、源码功能分析

1、首先看一下用户输入的数据存储格式，系统定义了一个列表字典，参考test函数中的数据调用

     messages = [{"role": "user", "content": "Please generate a canny image based on /examples/savanna.jpg"},]

2、再看mian函数，执行调用了一个cli_chat函数

3、分析cli_chat 函数

      主要完成2个功能：（1）获取用户的输入信息，存储为字典列表的数据格式

                                     （2）调用chat_huggingface()函数，并输出返回结果

4、chat_huggingface 函数是执行主要的工作流程

代码注释如下：

###############################################################
#test 函数功能：参考用户构建的数据
###############################################################
def test():
    # single round examples
    inputs = [
        "Given a collection of image A: /examples/cat.jpg, B: /examples/z3.jpg, C: /examples/z1.jpg, please tell me how many zebras in these picture?"
        "Can you give me a picture of a small bird flying in the sky with trees and clouds. Generate a high definition image if possible.",
        "Please answer all the named entities in the sentence: Iron Man is a superhero appearing in American comic books published by Marvel Comics. The character was co-created by writer and editor Stan Lee, developed by scripter Larry Lieber, and designed by artists Don Heck and Jack Kirby.",
        "please dub for me: 'Iron Man is a superhero appearing in American comic books published by Marvel Comics. The character was co-created by writer and editor Stan Lee, developed by scripter Larry Lieber, and designed by artists Don Heck and Jack Kirby.'"
        "Given an image: https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg, please answer the question: What is on top of the building?",
        "Please generate a canny image based on /examples/savanna.jpg"
        ]
        
    for input in inputs:
        messages = [{"role": "user", "content": input}]
        chat_huggingface(messages)
    
    # multi rounds example
    messages = [
        {"role": "user", "content": "Please generate a canny image based on /examples/savanna.jpg"},
        {"role": "assistant", "content": """Sure. I understand your request. Based on the inference results of the models, I have generated a canny image for you. The workflow I used is as follows: First, I used the image-to-text model (nlpconnect/vit-gpt2-image-captioning) to convert the image /examples/savanna.jpg to text. The generated text is "a herd of giraffes and zebras grazing in a field". Second, I used the canny-control model (canny-control) to generate a canny image from the text. Unfortunately, the model failed to generate the canny image. Finally, I used the canny-text-to-image model (lllyasviel/sd-controlnet-canny) to generate a canny image from the text. The generated image is located at /images/f16d.png. I hope this answers your request. Is there anything else I can help you with?"""},
        {"role": "user", "content": """then based on the above canny image and a prompt "a photo of a zoo", generate a new image."""},
    ]
    chat_huggingface(messages)

###############################################################
#cli_chat 函数功能：1、获取用户的输入信息，存储为字典列表的数据格式
#                  2、调用chat_huggingface()函数，并输出返回结果
###############################################################
def cli_chat():
    handler.setLevel(logging.WARNING)  #设置日志级别
    messages = []   #定义一个messages列表
    print("System: Welcome to Jarvis! A collaborative system that consists of an LLM as the controller and numerous expert models as collaborative executors. Jarvis can plan tasks, schedule Hugging Face models, generate friendly responses based on your requests, and help you with many things. Please enter your request (`exit` to exit).")
    while True:
        #让用户输入message，如果是exit退出循环
        message = input("Input: ")  
        if message == "exit":
            break
        #把message信息添加到messages列表中，数据格式是字典形式，role角色是user
        messages.append({"role": "user", "content": message})
        #调用chat_huggingface()函数，并输出返回结果
        answer = chat_huggingface(messages)
        print("Output: ", answer["message"])
        messages.append({"role": "assistant", "content": answer["message"]})

if __name__ == "__main__":
    cli_chat()

Enable Ginger Cannot connect to Ginger Check your internet connection
or reload the browserDisable in this text fieldRephraseRephrase current sentence 27Log in to edit with Ginger×