阿里InContextLoRA：更强ID一致性！基于黑森林F1身份一致性连贯视频分镜图集，10组风格无限创意

最新推荐文章于 2025-03-08 18:14:51 发布

小敢摘葡萄

最新推荐文章于 2025-03-08 18:14:51 发布

阅读量1.1k

点赞数 13

文章标签：音视频人工智能大数据 ai AIGC AI

本文链接：https://blog.csdn.net/xiaoganbuaiuk/article/details/143701298

版权

In-Context LoRA简介

近日有阿里团队发布了一款名为In-Context LoRA 的微调文生图框架（基于ai-toolkit），该框架能够生成具有可定制内在关系的图像集，可以选择性地基于一组图像进行条件生成，从而实现对广泛任务的适应。

在使用扩散变换器（DiTs）框是进行任务无关图像生成的方法，是通过简单地在图像间连接注意力令牌。然而，尽管投入了大量的计算资源，生成图像的保真度仍然不尽如人意。阿里研发团队采用了通过假设文本到图像的DiTs天生就具有上下文生成能力，只需要最小的调整就能激活它们，重新评估并简化了框架。并通过多样化的任务实验，定性地证明现有的文生图的DiTs架构是可以在没有任何调整的情况下有效地执行上下文生成。基于这一洞见，研发团队提出了一个非常简单的流程来利用DiTs的上下文能力：

1. 连接图像而不是令牌，
1. 对多幅图像进行联合描述
3. 使用小数据集（例如，20到100个样本）而不是全参数调整和大数据集进行任务特定的LoRA调整。

In-Context LoRA（IC-LoRA）方法不需要对原始DiT模型进行任何修改，只需要改变训练数据。IC-LoRA流程生成了高保真度的图像集，具有更好地遵循提示。虽然在调整数据方面是任务特定的，但是IC-LoRA框架在架构和流程上仍然是任务无关的。

• Github：https://github.com/ali-vilab/In-Context-LoRA
• huggface：https://huggingface.co/ali-vilab/In-Context-LoRA/tree/main
• 项目主页：https://ali-vilab.github.io/In-Context-LoRA-Page/

演示案例

01

This set of four images illustrates a young artist's creative process in a bright and inspiring studio; [IMAGE1] she stands before a large canvas, brush in hand, adding vibrant colors to a partially completed painting, [IMAGE2] she sits at a cluttered wooden table, sketching ideas in a notebook with various art supplies scattered around, [IMAGE3] she takes a moment to step back and observe her work, and [IMAGE4] she experiments with different textures by mixing paints directly on the palette, her focused expression showcasing her dedication to her craft.

02

In a tender exploration of first love, [IMAGE1] we see Jamie nervously arranging flowers in a park, glancing around as if waiting for someone special, [IMAGE2] transitioning to the moment arrives, their eyes locking in a shy smile that speaks volumes, [IMAGE3] finally showing them seated on a bench, sharing stories and laughter, surrounded by blooming blossoms, embodying the magic of young romance.

03

This set of four images showcases a teenage girl with curly black hair wearing a stylish denim jacket, each image highlighting her dynamic personality in urban settings; [IMAGE1] she is skateboarding down a graffiti-covered alley, a confident smile on her face as she maneuvers around obstacles; [IMAGE2] she is seated at a trendy café, typing on her laptop with focused determination, the bustling city life visible through the large windows behind her; [IMAGE3] she stands on a rooftop at sunset, her hair blowing in the breeze as she gazes thoughtfully over the city skyline; and [IMAGE4] she is laughing with friends at a vibrant street market, colorful lights and stalls creating a lively atmosphere around her.

04

This set of four images captures a lively day spent at a beach between a mother and her son, highlighting their playful connection and shared joy; [IMAGE1] the boy runs towards the water, his arms wide open, with the mother following behind, smiling as she watches him; [IMAGE2] they are knee-deep in the ocean, laughing as they splash each other, the sunlight reflecting off the water; [IMAGE3] they sit on the sand, the boy intently building a sandcastle while the mother assists, both focused and relaxed; [IMAGE4] the final image shows the two walking along the shore at sunset, the mother’s arm draped protectively around her son’s shoulders, their footprints trailing behind them in the sand.

05

The set of four images features a minimalist handwriting font for casual use. [IMAGE1] shows "Everyday" on a coffee cup; [IMAGE2] displays "Notes" on a small journal; [IMAGE3] has "Live Simply" on a white pillow; [IMAGE4] shows "Good Vibes" on a cozy blanket, perfect for lifestyle and home decor branding.

06

This set of four images captures a colorful, nature-inspired living space with touches of green and earthy textures; [IMAGE1] features a cozy nook with a woven chair draped in green blankets, surrounded by potted plants and botanical prints on the wall; [IMAGE2] highlights a rustic wooden shelf adorned with small planters, candles, and woven baskets; [IMAGE3] displays a serene bedroom with a bed made up in white linens, a natural wood nightstand, and a forest-themed mural; [IMAGE4] shows a close-up of a large plant pot with unique textures beside a patterned area rug.

07

This set of four images showcases a rustic-themed PowerPoint template for a culinary workshop; [IMAGE1] introduces "Farm to Table Cooking" in warm, earthy tones; [IMAGE2] organizes workshop sections like "Ingredients," "Preparation," and "Serving"; [IMAGE3] displays ingredient lists for seasonal produce; [IMAGE4] includes chef profiles with short bios.

08

This image pair showcases the transformation of a cyclist through a sandstorm visual effect; [IMAGE1] features a cyclist in vibrant gear pedaling steadily on a clear, open road with a serene sky in the background, highlighting focus and determination, [IMAGE2] transforms the scene as the cyclist becomes enveloped in a fierce sandstorm, with sand particles swirling intensely around the bike and rider against a stormy, darkened backdrop, emphasizing chaos and power.

更多演示案例：https://ali-vilab.github.io/In-Context-LoRA-Page/

In-Context LoRA ComfyUI体验

同时研发团队还提供Flux体验工作流和10 款多样化风格的Flux LORA模型。如果已经安装过Flux模型体验，仅需导入工作流，并下载对应风格LORA模型即可。风格包括：情侣头像设计（Couple Profile Design）、电影故事板（Film Storyboard）、字体设计（Font Design）、家居装饰（Home Decoration）、肖像画（Portrait Illustration）、肖像摄影（Portrait Photography）、PPT模板（PPT Template）、沙尘暴视觉效果（Sandstorm Visual Effect）、烟花视觉效果（Sparklers Visual Effect）、视觉身份设计（Visual Identity Design。另外，最新BizyAir也已经支持，低显存用户可考虑。

Flux文生图工作流

本文涉及ComfyUI工作流和模型均可在LIBLIBAI上下载或在线运行体验：

• STOIQO NewReality_FLUX：

https://www.liblib.art/modelinfo/abdbdadbaba64560aa55bf3e5167c85d?versionUuid=b903fe96b9ec4ee1887e8ae37c06f3e1

• F.1-绮梦流光-水湄凝香：

https://www.liblib.art/modelinfo/134c6dd95aef48e98a22b24e003e026b

• 工作流-Flux文|图生图+LORA+提示反推一键切换工作流：

https://www.liblib.art/modelinfo/782aacd70f604da39e83368c696a02a8

另外LIBLIBAI已支持本地客户端使用可首页下载体验。

本地体验演示:

In-Context LoRA工作流

In-Context LoRA工作流已上传LIBLIB平台：

https://www.liblib.art/modelinfo/fd5f7b35250d47a68beb27701096b505?versionUuid=d6526844494c427eac11d68546742a8d

01.电影故事版

[MOVIE-SHOTS] In a tender exploration of first love, [IMAGE1] we see Jamie nervously arranging flowers in a park, glancing around as if waiting for someone special, [IMAGE2] transitioning to the moment arrives, their eyes locking in a shy smile that speaks volumes, [IMAGE3] finally showing them seated on a bench, sharing stories and laughter, surrounded by blooming blossoms, embodying the magic of young romance.

02.肖像画

This set of four images captures the serene moments of an elderly woman tending to her garden. [IMAGE1] She kneels beside a bed of blooming flowers, her hands gently pruning a rose bush, the soft morning light illuminating her silver hair; [IMAGE2] she stands with a watering can, her face calm and peaceful as she nurtures her plants; [IMAGE3] a close-up reveals her content smile as she examines a budding flower in her hand, a sense of pride and joy evident; [IMAGE4] she sits on a small bench, sipping tea with her garden behind her, surrounded by the vibrant colors of her hard work.

03.字体设计

The set of four images features a minimalist handwriting font for casual use. [IMAGE1] shows "Everyday" on a coffee cup; [IMAGE2] displays "Notes" on a small journal; [IMAGE3] has "Live Simply" on a white pillow; [IMAGE4] shows "Good Vibes" on a cozy blanket, perfect for lifestyle and home decor branding.

04.家具装饰

This vibrant set of four image captures a lively home decor scene filled with color and eclectic charm; [IMAGE1] the first image showcases a cozy living area with pastel-colored walls, a soft blue sofa, wooden storage units displaying colorful accents, and a unique layered pendant light, [IMAGE2] the second image features a kitchen setup with open shelves holding assorted kitchenware, a wire grid for organizing mugs above a white sink, and warm sunlight streaming onto the countertop, [IMAGE3] the third image highlights a bold art wall with an array of colorful, abstract paintings above a sage green sofa adorned with bright cushions, and [IMAGE4] the fourth image shows a cheerful dining nook with a blue table, vividly striped cushions, framed artwork on the sunny yellow wall, and a distinctive green pendant lamp casting a soft glow over the space.

05.PPT模板设计

The set of four images presents a PowerPoint template designed for a charity fundraiser; [IMAGE1] introduces "Help Make a Difference" in large, bold text over a background of hands reaching out; [IMAGE2] lists causes like “Education,” “Healthcare,” and “Water Access” with heart icons; [IMAGE3] displays donation statistics; [IMAGE4] includes a call-to-action slide with links to donate and volunteer.

06.情侣头像设计

The pair of images depicts a couple in a cartoon-style grocery shopping scene; [IMAGE1] one character reaches for a snack on a high shelf with a playful grin, while [IMAGE2] the other character with wide eyes and a towering cart of food holds a grocery list, all set in a colorful grocery aisle.

07.视觉身份设计

In this set of two images, a modern mystical brand identity comes to life. [IMAGE1] Against a deep navy background, intricate star and moon motifs in metallic silver and soft blush pink shimmer in various sizes, creating a cosmic, dreamlike atmosphere. The brand name “Celestial Glow” is displayed in a sleek, geometric font that radiates a mystical yet minimalist vibe. [IMAGE2] The design is adapted onto a glowing glass misting bottle and a crystal-infused body lotion bottle, arranged on a soft, cloud-like velvet fabric with crystals and candles, showing the brand’s ethereal charm in self-care products.

08.肖像插画

The image pair contrasts a photographic portrait with its illustrated counterpart, showcasing an artistic reinterpretation; [IMAGE1] the initial photo shows a woman with a high bun, dressed in a classic black trench coat, holding a bright yellow umbrella, standing on a rainy street, while [IMAGE2] the illustration accentuates her pose with exaggerated features, making the umbrella the focal point with vivid yellows and reds, transforming the rain into playful, curving lines.

09.沙尘暴视觉效果

This image pair showcases the transformation of a cyclist through a sandstorm visual effect; [IMAGE1] features a cyclist in vibrant gear pedaling steadily on a clear, open road with a serene sky in the background, highlighting focus and determination, [IMAGE2] transforms the scene as the cyclist becomes enveloped in a fierce sandstorm, with sand particles swirling intensely around the bike and rider against a stormy, darkened backdrop, emphasizing chaos and power.

写在最后

感兴趣的小伙伴，赠送全套AIGC学习资料，包含AI绘画、AI人工智能等前沿科技教程和软件工具，具体看这里。
在这里插入图片描述

AIGC技术的未来发展前景广阔，随着人工智能技术的不断发展，AIGC技术也将不断提高。未来，AIGC技术将在游戏和计算领域得到更广泛的应用，使游戏和计算系统具有更高效、更智能、更灵活的特性。同时，AIGC技术也将与人工智能技术紧密结合，在更多的领域得到广泛应用，对程序员来说影响至关重要。未来，AIGC技术将继续得到提高，同时也将与人工智能技术紧密结合，在更多的领域得到广泛应用。

在这里插入图片描述