FLUX.1 团队已将从 Pro 提取的 SFT 转成 Diffusion 架构

Flux.1 是由 Black Forest Labs 黑森林实验室推出的文生图模型套件，说到 Black Forest Labs 大家可能觉得陌生，它是 Stability AI 前核心成员 Robin Rombach 创立的新团队，成员也多来自 Stable Diffusion 的原始开发团队，因此具备强大的生成式模型开发能力。

之前，我也第一时间分享了关于FLUX.1的相关文章。而此次，FLUX.1 团队已将从 Dev 提取的 SFT 转成 Diffusion 架构，下载和原来的路径一致，如果还在用SFT的，可能需要等第三方提供了（由于一些工具早期适配SFT）。不过官方已经表示ComfyUI和Diffusers都已经支持，相信不久就会全面转向标准架构。

目录

black-forest-labs/FLUX.1-dev/
├── ae.safetensors
├── dev_grid.jpg
├── flux1-dev.safetensors
├── LICENSE.md
├── model_index.json
├── README.md
├── scheduler
│   └── scheduler_config.json
├── text_encoder
│   ├── config.json
│   └── model.safetensors
├── text_encoder_2
│   ├── config.json
│   ├── model-00001-of-00002.safetensors
│   ├── model-00002-of-00002.safetensors
│   └── model.safetensors.index.json
├── tokenizer
│   ├── merges.txt
│   ├── special_tokens_map.json
│   ├── tokenizer_config.json
│   └── vocab.json
├── tokenizer_2
│   ├── special_tokens_map.json
│   ├── spiece.model
│   ├── tokenizer_config.json
│   └── tokenizer.json
├── transformer
│   ├── config.json
│   ├── diffusion_pytorch_model-00001-of-00003.safetensors
│   ├── diffusion_pytorch_model-00002-of-00003.safetensors
│   ├── diffusion_pytorch_model-00003-of-00003.safetensors
│   └── diffusion_pytorch_model.safetensors.index.json
└── vae
    ├── config.json
    └── diffusion_pytorch_model.safetensors

代码演示

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
# pipe.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Remove this if you have enough GPU power

# pipe.vae.enable_tiling()
# pipe.vae.enable_slicing()
pipe.enable_sequential_cpu_offload() #save some VRAM by offloading the model to CPU. Remove this if you have enough GPU power
# pipe.enable_xformers_memory_efficient_attention()

prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=50,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
# image.save("flux-dev.png")
image

Sat Aug 17 15:12:47 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
|  0%   37C    P8             32W /  515W |    1092MiB /  24564MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1137      G   /usr/lib/Xorg                                 167MiB |
|    0   N/A  N/A      1611      G   /usr/bin/sddm-greeter-qt6                     146MiB |
|    0   N/A  N/A      5759      C   ...conda3/envs/ai-train/bin/python3.10        746MiB |
+-----------------------------------------------------------------------------------------+

感觉也不是很吃资源，可以支持更多ControlNet和Lora了。