Story-Adapter：长序列故事帧生成模型

1.论文介绍

Story-Adapter: A Training-free Iterative Framework For Long Story Visualization
StoryAdapter
文本提出了一个迭代式生成长序列故事帧的模型。
文本设计了一个即插即用的全局参考交叉注意力模块（GRCA）来增强迭代生成方式的有效性。
本文设计了一个线性加权策略来融合视觉与文本两类模态的信息。

模型图
采用类似ipadapter的方法，将生成的所有帧数据，最为下一次迭代生成的参考图像。

2.模型的部署

模型的权重下载

需要将权重下载到如下的文件夹里

需要修改的参数

因为我下载的text_encoderder权重是fp16，所以我要进这个config文件中把roech_dtype改为float16

3.模型生成的结果

运行模型的配置文件的参数

--base_model_path
/story-adapter-main/RealVisXL_V4.0
--image_encoder_path
/story-adapter-main/IP-Adapter/sdxl_models/image_encoder
--ip_ckpt
/story-adapter-main/IP-Adapter/sdxl_models/ip-adapter_sdxl.bin

run.py文件中的参数需要修改为以下
增加了一下两个，应为瞎子啊的unet的权重的fp16，safetensor的
use_safetensors=True,
variant=“fp16”

pipe = StableDiffusionXLPipeline.from_pretrained(
    base_model_path,
    torch_dtype=torch.float16,
    scheduler=noise_scheduler,
    feature_extractor=None,
    safety_checker=None,
    use_safetensors=True,
    variant="fp16"
)

用GPT写的文本

story2 = [
    "a little lion running in the savanna.",
    "a little lion sleeping under a tree in the savanna.",
    "a little lion chasing a butterfly in the savanna.",
    "a little lion playing with a ball in the savanna.",
    "a little lion jumping over a log in the savanna.",
    "a little lion walking beside a river in the savanna.",
    "a little lion running with a herd of zebras in the savanna.",
    "a little lion watching a sunset in the savanna.",
    "a little lion hiding behind a bush in the savanna.",
    "a little lion playing with its reflection in a waterhole.",
    "a little lion running under the stars in the savanna night.",
    "a little lion chasing its tail in the savanna.",
    "a little lion roaring proudly in the savanna.",
    "a little lion playing with a lion cub in the savanna.",
    "a little lion walking beside its mother in the savanna.",
    "a little lion hunting with its pride in the savanna.",
    "a little lion and a giraffe standing together under the sun.",
    "a little lion running past a herd of antelopes.",
    "a little lion resting under the shade of a tree in the savanna.",
    "a little lion playing in the grass with a lion cub.",
    "a little lion watching a bird flying overhead in the savanna.",
    "a little lion climbing a small hill in the savanna.",
    "a little lion running beside a group of elephants in the savanna.",
    "a little lion chasing a bird in the sky.",
    "a little lion sitting on a rock, looking at the horizon.",
    "a little lion playing in the tall grass with a zebra.",
    "a little lion sneaking up on its prey in the savanna.",
    "a little lion resting by a tree, listening to the sounds of nature.",
    "a little lion walking through the dry grass in the savanna.",
    "a little lion running across the plain at sunrise.",
    "a little lion standing on a hill, looking down at the savanna below.",
    "a little lion practicing its roar in front of a tree.",
    "a little lion taking a nap under the stars in the savanna night.",
    "a little lion running beside a riverbank, chasing its shadow.",
    "a little lion sprinting through the savanna with the wind in its mane.",
    "a little lion running alongside a cheetah in the savanna.",
    "a little lion running through a rainstorm in the savanna.",
    "a little lion running through the mist of a savanna morning.",
    "a little lion watching a sunset from a cliff in the savanna.",
    "a little lion running after a butterfly in a field of flowers.",
    "a little lion climbing a tree in the savanna.",
    "a little lion walking through a field of tall grass at dusk.",
    "a little lion watching a herd of buffalo passing by in the savanna.",
    "a little lion chasing a bird across the savanna.",
    "a little lion playing with a beetle on the savanna floor.",
    "a little lion running beside a lioness in the savanna.",
    "a little lion playing in the mud after the rain in the savanna.",
    "a little lion running on a dirt path in the savanna at dawn.",
    "a little lion sitting in the grass, looking up at the stars in the savanna night.",
    "a little lion chasing a small rabbit in the savanna.",
    "a little lion running past a group of wildebeest in the savanna.",
    "a little lion rolling in the grass, playing with its siblings.",
    "a little lion gazing at the moon in the savanna night.",
    "a little lion standing proudly on a rock, overlooking the savanna.",
    "a little lion resting on a sun-warmed rock in the savanna.",
    "a little lion running in circles with a butterfly in the savanna.",
    "a little lion and a young giraffe drinking water together at a waterhole.",
    "a little lion running in the rain, enjoying the droplets in the savanna.",
    "a little lion playing hide and seek with its siblings in the savanna.",
    "a little lion resting with its family under a large tree in the savanna.",
    "a little lion practicing hunting skills in the tall grass.",
    "a little lion chasing a meerkat in the savanna.",
    "a little lion watching a rainbow appear after a rainstorm.",
    "a little lion sniffing the air, enjoying the fresh breeze in the savanna.",
    "a little lion running alongside a herd of zebras in the evening.",
    "a little lion playing with a fallen leaf in the savanna.",
    "a little lion watching a cheetah run across the savanna.",
    "a little lion standing proudly in the tall grass, with its mane blowing in the wind.",
    "a little lion chasing its shadow in the savanna.",
    "a little lion watching the stars twinkle in the savanna night.",
    "a little lion running through the fields with its pride.",
    "a little lion playing with a stick it found on the savanna ground.",
    "a little lion observing a group of birds flying across the savanna.",
    "a little lion and a lioness resting together under a tree.",
    "a little lion looking at its reflection in the waterhole.",
    "a little lion running past a herd of elephants in the savanna.",
    "a little lion resting in the shade of a tree after a long run in the savanna.",
    "a little lion running along the banks of a river in the savanna.",
    "a little lion exploring the savanna at dusk.",
    "a little lion running through the open plains of the savanna at sunset.",
]

生成的结果如下