1.Datawhale AI夏令营 AIGC TASK1

神迹小卒

已于 2024-08-12 12:02:47 修改

阅读量762

点赞数 17

分类专栏： AIGC 文章标签：人工智能 AIGC

于 2024-08-08 20:58:01 首次发布

本文链接：https://blog.csdn.net/aiqq136/article/details/141036091

版权

AIGC 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

0808

1.AIGC TASK1 Mission

效果图
在这里插入图片描述

Task1将于8月11日晚上23:59分截止，
缺少任意一次打卡都将无法获得结营证书哦～
JUST REMEMBER TO FINISH ALL TASK

可图Kolors-LoRA风格故事挑战赛

Kolors:large-scale text-to-image generation model
LoRA:Low-Rank Adaptation:maintaining the original model's capabilities while adapting it to new or specific tasks based on latent diffusion techniques, developed by the Kuaishou Kolors team

https://github.com/Kwai-Kolors/Kolors

赛事链接
https://tianchi.aliyun.com/s/ce4dc8bf800db1e58d51263ff357d28f

latent diffusion

Compression into Latent Space
Diffusion Process
Controlled Generation
Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs)

参赛者需在可图Kolors 模型的基础上训练LoRA 模型，生成无限风格，如水墨画风格、水彩风格、赛博朋克风格、日漫风格…
基于LoRA模型生成 8 张图片组成连贯故事，故事内容可自定义；基于8图故事，评估LoRA风格的美感度及连贯性

样例：偶像少女养成日记

赛选手需提交训练的LoRA 模型文件、

LORA 模型的介绍、

以及使用该模型生成的至少8张图片和对应 prompt，

以便我们能够复现生成效果，

对于生成效果明显无法复现的，取消获奖资格

将训练好的LoRA 模型上传到魔搭模型库

LoRA模型命名为：队伍名称-可图Kolors训练-xxxxxx

freshman2233-可图Kolors训练-freshman2233-46119

LoRA 上传地址：https://modelscope.cn/models/create?template=text-to-image-lora

作品发布在比赛品牌馆讨论区，https://modelscope.cn/brand/view/Kolors?branch=0&tree=11

发布标题格式为：天池平台报名队伍名称+LoRA模型链接地址+作品图（8图故事）

2.Text-to-Image Generation

1.规则和模板匹配

The rule matches the template

2.统计模型和机器学习

Statistical models and machine learning

概率图模型和统计语言模型

Probability graph model and statistical language model

3.深度学习，尤其是卷积神经网络（CNN）和生成对抗网络（GAN）的发展

Deep learning, especially the development of convolutional neural networks (CNNS) and generative adversarial networks (Gans)

unsupervised machine learning

Discriminator:determine whether a given data instance is real (from the actual dataset) or fake (created by the generator).

If the data instance is indistinguishable from real data in the eyes of the discriminator, it’s considered a success.

4.大规模预训练模型如OpenAI的CLIP、DALL-E以及Stable Diffusion

CLIP通过大规模的文本和图像配对数据训练，能够理解和生成高度一致的文本和图像；DALL-E和Stable Diffusion进一步提升了生成图像的创意和细节表现能力，使得通过简单的文本描述生成高质量、复杂图像成为可能。

3.文生图基础知识介绍

提示词

主体描述，细节描述，修饰词，艺术风格，艺术家

【promts】Beautiful and cute girl, smiling, 16 years old, denim jacket, gradient background, soft colors, soft lighting, cinematic edge lighting, light and dark contrast, anime, super detail, 8k

【负向prompts】(lowres, low quality, worst quality:1.2), (text:1.2), deformed, black and white,disfigured, low contrast, cropped, missing fingers

主体描述

【提示】美丽可爱的女孩，微笑，16岁，牛仔夹克，渐变背景，柔和的色彩，柔和的灯光，电影边缘照明，明暗对比，动漫，超级细节，8k

【负向提示】(低分辨率，低质量，最差质量:1.2)，(文本:1.2)，变形，黑白，毁损，低对比度，裁剪，缺手指

Lora

“Low-Rank Adaptation”，即低秩适应

ComfyUI

工作流工具，主要用于简化和优化 AI 模型的配置和训练过程。通过直观的界面和集成的功能，用户可以轻松地进行模型微调、数据预处理、图像生成等任务，从而提高工作效率和生成效果

参考图控制ControlNet

预训练的扩散模型（如Stable Diffusion模型）上的可训练神经网络模块

从随机噪声逐渐生成图像的过程

姿势关键点、分割图、深度图、颜色

OpenPose姿势控制

Canny精准绘制

Hed绘制:获取渐变线条的线稿图控制方式

深度图Midas

颜色color控制

4.小白零基础 30 分钟速通指南

Step0：开通阿里云PAI-DSW试用

Platform for AI
Data Science Workshop

在魔搭社区进行授权

Step1：报名赛事！(点击即可跳转)

Step2：在魔搭社区创建PAI实例！（点击即可跳转）

Step3：30 分钟体验一站式 baseline！

git lfs install

git clone https://www.modelscope.cn/datasets/maochase/kolors.git

Git Large File Storage

Data-Juicer：数据处理和转换工具，旨在简化数据的提取、转换和加载过程

DiffSynth-Studio：高效微调训练大模型工具

!pip install simple-aesthetics-predictor

!pip install -v -e data-juicer

!pip uninstall pytorch-lightning -y

!pip install peft lightning pandas torchvision

!pip install -e DiffSynth-Studio

from modelscope.msdatasets import MsDataset

ds = MsDataset.load(

    'AI-ModelScope/lowres_anime',

    subset_name='default',

    split='train',

    cache_dir="/mnt/workspace/kolors/data"

)

import json, os

from data_juicer.utils.mm_utils import SpecialTokens

from tqdm import tqdm

os.makedirs("./data/lora_dataset/train", exist_ok=True)

os.makedirs("./data/data-juicer/input", exist_ok=True)

with open("./data/data-juicer/input/metadata.jsonl", "w") as f:

    for data_id, data in enumerate(tqdm(ds)):

        image = data["image"].convert("RGB")

        image.save(f"/mnt/workspace/kolors/data/lora_dataset/train/{data_id}.jpg")

        metadata = {"text": "二次元", "image": [f"/mnt/workspace/kolors/data/lora_dataset/train/{data_id}.jpg"]}

        f.write(json.dumps(metadata))

        f.write("\n")

data_juicer_config = """
# global parameters
project_name: 'data-process'
dataset_path: './data/data-juicer/input/metadata.jsonl'  # path to your dataset directory or file
np: 4  # number of subprocess to process your dataset

text_keys: 'text'
image_key: 'image'
image_special_token: '<__dj__image>'

export_path: './data/data-juicer/output/result.jsonl'

# process schedule
# a list of several process operators with their arguments
process:
    - image_shape_filter:
        min_width: 1024
        min_height: 1024
        any_or_all: any
    - image_aspect_ratio_filter:
        min_ratio: 0.5
        max_ratio: 2.0
        any_or_all: any
"""
with open("data/data-juicer/data_juicer_config.yaml", "w") as file:
    file.write(data_juicer_config.strip())

!dj-process --config data/data-juicer/data_juicer_config.yaml

import pandas as pd

import os, json

from PIL import Image

from tqdm import tqdm

texts, file_names = [], []

os.makedirs("./data/lora_dataset_processed/train", exist_ok=True)

with open("./data/data-juicer/output/result.jsonl", "r") as file:

    for data_id, data in enumerate(tqdm(file.readlines())):

        data = json.loads(data)

        text = data["text"]

        texts.append(text)

        image = Image.open(data["image"][0])

        image_path = f"./data/lora_dataset_processed/train/{data_id}.jpg"

        image.save(image_path)

        file_names.append(f"{data_id}.jpg")

data_frame = pd.DataFrame()

data_frame["file_name"] = file_names

data_frame["text"] = texts

data_frame.to_csv("./data/lora_dataset_processed/train/metadata.csv", index=False, encoding="utf-8-sig")

data_frame

from diffsynth import download_models

download_models(["Kolors", "SDXL-vae-fp16-fix"])

!python DiffSynth-Studio/examples/train/kolors/train_kolors_lora.py -h

import os

cmd = """

python DiffSynth-Studio/examples/train/kolors/train_kolors_lora.py \

  --pretrained_unet_path models/kolors/Kolors/unet/diffusion_pytorch_model.safetensors \

  --pretrained_text_encoder_path models/kolors/Kolors/text_encoder \

  --pretrained_fp16_vae_path models/sdxl-vae-fp16-fix/diffusion_pytorch_model.safetensors \

  --lora_rank 16 \

  --lora_alpha 4.0 \

  --dataset_path data/lora_dataset_processed \

  --output_path ./models \

  --max_epochs 1 \

  --center_crop \

  --use_gradient_checkpointing \

  --precision "16-mixed"

""".strip()

os.system(cmd)

from diffsynth import ModelManager, SDXLImagePipeline

from peft import LoraConfig, inject_adapter_in_model

import torch

def load_lora(model, lora_rank, lora_alpha, lora_path):

    lora_config = LoraConfig(

        r=lora_rank,

        lora_alpha=lora_alpha,

        init_lora_weights="gaussian",

        target_modules=["to_q", "to_k", "to_v", "to_out"],

    )

    model = inject_adapter_in_model(lora_config, model)

    state_dict = torch.load(lora_path, map_location="cpu")

    model.load_state_dict(state_dict, strict=False)

    return model

# Load models

model_manager = ModelManager(torch_dtype=torch.float16, device="cuda",

                             file_path_list=[

                                 "models/kolors/Kolors/text_encoder",

                                 "models/kolors/Kolors/unet/diffusion_pytorch_model.safetensors",

                                 "models/kolors/Kolors/vae/diffusion_pytorch_model.safetensors"

                             ])

pipe = SDXLImagePipeline.from_model_manager(model_manager)

# Load LoRA

pipe.unet = load_lora(

    pipe.unet,

    lora_rank=16, # This parameter should be consistent with that in your training script.

    lora_alpha=2.0, # lora_alpha can control the weight of LoRA.

    lora_path="models/lightning_logs/version_0/checkpoints/epoch=0-step=500.ckpt"

)

torch.manual_seed(0)

image = pipe(

    prompt="二次元，一个紫色短发小女孩，在家中沙发上坐着，双手托着腮，很无聊，全身，粉色连衣裙",

    negative_prompt="丑陋、变形、嘈杂、模糊、低对比度",

    cfg_scale=4,

    num_inference_steps=50, height=1024, width=1024,

)

image.save("1.jpg")

please generating 6 sequence prompt using Chinese about 二次元

描述一个女性勇者打败恶龙救出公主的故事：

要求按照顺序，生成8段剧情连续的提示词

参考案例：

prompt=“二次元，一个紫色短发小女孩，在家中沙发上坐着，双手托着腮，很无聊，全身，粉色连衣裙”,