Colossal-AI训练diffusion经验记录

已于 2023-02-28 21:40:29 修改
阅读量1.9k 收藏 5
点赞数 1
分类专栏： 深度学习 文章标签： python 深度学习 人工智能
于 2023-02-22 15:08:01 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_37236149/article/details/129146869
版权

Colossal-AI训练diffusion经验记录

基本使用

参考链接
调用colossalai包和 pytorch-lightning包

  1. train
#case1: HybridAdam
from lightning.pytorch import trainer, LightningModule
from colossalai.nn.optimizer import HybridAdam
class MyDiffuser(LightningModule):    
	...    
	def configure_sharded_model(self) -> None:        
	# create your model here        
	self.model = construct_diffuser_model(...)        
	...    
	def configure_optimizers(self):        
	# use the specified optimizer        
	optimizer = HybridAdam(self.model.parameters(), self.lr)        
	...
model = MyDiffuser()
trainer = Trainer(accelerator="gpu", devices=1, precision=16, strategy="colossalai")
trainer.fit(model)

#case2: ColossalAIStrategy显存优化
from lightning.pytorch import trainer, LightningModule
from lightning.pytorch.strategies import ColossalAIStrategy
Mystrategy = ColossalAIStrategy(use_chunk=True, enable_distributed_storage=True, placement_policy=auto)
trainer = Trainer(accelerator="gpu", devices=4, precision=16, strategy=Mystrategy)
trainer.fit(model)
  1. finetune：低成本，通过较少资源短时间训练出可以生成自己风格的模型
    基于 HuggingFace 上开源的 Stable Diffusion 模型权重进行微调
    步骤：
    • 修改 Dataloader 载入微调数据集
    • 读取预训练权重
    • 简单修改参数配置 yaml 文件并运行训练脚本
      python main.py --logdir /your_log_dir -t -b config/train_colossalai.yaml
model:  
	target: ldm.models.diffusion.ddpm.LatentDiffusion  
	params:    
		your_sub_module_config:
			target: your.model.import.path     
		 	params:        
		 		from_pretrained: 'your_file_path/unet/diffusion_pytorch_model.bin'        
		 		...
lightning:  
	trainer:    
		strategy:      
			target: pytorch_lightning.strategies.ColossalAIStrategy      
			params:        
				...
  1. inference：支持原生 Stable Diffusion 推理管道
    完成训练或精调后，只需直接调用 diffuser 库+加载自己保存的模型参数
    ->模型推理对数值精度不敏感
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained("your_ColoDiffusion_checkpoint_path").to("cuda")
image = pipe('your prompt', num_inference_steps=50)["sample"][0]
image.save('file path')

Colossal-AI+diffusion

We provide the script train_colossalai.sh to run the training task with colossalai, and can also use train_ddp.sh to run the training task with ddp to compare.

可以通过 train_colossalai.sh 来执行训练任务：

python main.py --logdir /tmp/ --train --base configs/train_colossalai.yaml --ckpt 512-base-ema.ckpt

You can change the --logdir to decide where to save the log information and the last checkpoint.
You will find your ckpt in logdir/checkpoints or logdir/diff_tb/version_0/checkpoints
You will find your train config yaml in logdir/configs

You can add the --ckpt if you want to load the pretrained model, for example 512-base-ema.ckpt

You can change the --base to specify the path of config yaml
Training config

train_colossalai.yaml中可以修改的一些参数

devices: device number used for training, default 8
max_epochs: max training epochs, default 2
precision: the precision type used in training, default 16 (fp16), you must use fp16 if you want to apply colossalai

环境配置

  • 使用Colossal-AI需下载pytorch-lightning>=1.8.1

    Colossal-AI 已集成作为 PyTorch Lightning 的官方大模型解决方案
    Step 2: install lightning
    Install Lightning version later than 2022.01.04. We suggest you install lightning from source.

    注意太早的 pytorch-lightning 版本没有strategy更没有ColossalAIStrategy，
    经测试1.4.1和1.6.1版本都报错或者warning了，最后换到1.8.1成功跑通
    pip install pytorch-lightning==1.8.1

  • 3090需下载 CUDA >= 11.6版本
    否则报如下提示

    NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
    The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.

    参考如下文章
    解决：NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with …

  • xformer需要pytorch>=1.13.1

  1. pytorch 1.13.1
    torchvision 0.14.1
    cuda 11.7
    按environment.yaml配的初始环境会报如下错误：
/home/user/anaconda3/envs/colossalai/lib/python3.7/site-packages/torchvision/io/image.py:11: UserWarning: Failed to load image Python extension: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
  warn(f"Failed to load image Python extension: {e}")

应该时torchvision有问题，重新安装pytorch，经测试如下版本可成功跑通
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

  1. 师兄采用
    pytorch 1.12.1
    torchvision 1.13.1
    cuda 11.6
    conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
    会报如下提示
...
=========================================================================================
No pre-built kernel is found, build and load the fused_optim kernel during runtime now
=========================================================================================
Detected CUDA files, patching ldflags
Emitting ninja build file /home/fangfei/.cache/colossalai/torch_extensions/torch1.12_cu11.6/build.ninja...
Building extension module fused_optim...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
...
[6/7] /home/fangfei/.conda/envs/tmm/bin/nvcc  -DTORCH_EXTENSION_NAME=fused_optim -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/kernels/include -I/home/fangfei/.conda/envs/tmm/include -isystem /home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include -isystem /home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/TH -isystem /home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/THC -isystem /home/fangfei/.conda/envs/tmm/include -isystem /home/fangfei/.conda/envs/tmm/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 --use_fast_math -lineinfo -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -std=c++14 -c /home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu -o multi_tensor_l2norm_kernel.cuda.o
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu: In function ‘std::tuple<at::Tensor, at::Tensor> multi_tensor_l2norm_cuda(int, at::Tensor, std::vector<std::vector<at::Tensor> >, c10::optional<bool>)’:
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:289:89: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
       tensor_lists[0][0].scalar_type(), 0, "multi_tensor_l2norm_cuda",
                                                                                         ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:290:9: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
       multi_tensor_apply<1>(
         ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:291:120: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
           BLOCK_SIZE, chunk_size, noop_flag, tensor_lists,
                                                                                                                        ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:292:40: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
           L2NormFunctor<scalar_t_0>(), output.DATA_PTR<float>(),
                                        ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:305:115: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
   cleanup<<<per_tensor ? ntensors : 1, 512, 0, stream>>>(
                                                                                                                   ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:305:163: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
   cleanup<<<per_tensor ? ntensors : 1, 512, 0, stream>>>(
                                                                                                                                                                   ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:305:196: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
   cleanup<<<per_tensor ? ntensors : 1, 512, 0, stream>>>(
                                                                                                                                                                                                    ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:305:241: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
   cleanup<<<per_tensor ? ntensors : 1, 512, 0, stream>>>(
                                                                                                                                                                                                                                                 ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu: In function ‘void multi_tensor_norm_out_cuda(int, at::Tensor, std::vector<std::vector<at::Tensor> >, at::Tensor, float, float, int)’:
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:349:90: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
         tensor_lists[0][0].scalar_type(), 0, "multi_tensor_maxnorm_cuda",
                                                                                          ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:349:125: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
         tensor_lists[0][0].scalar_type(), 0, "multi_tensor_maxnorm_cuda",
                                                                                                                             ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:351:91: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
             BLOCK_SIZE, chunk_size, noop_flag, tensor_lists,
                                                                                           ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:351:126: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
             BLOCK_SIZE, chunk_size, noop_flag, tensor_lists,
                                                                                                                              ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:356:89: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
         tensor_lists[0][0].scalar_type(), 0, "multi_tensor_l2norm_cuda",
                                                                                         ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:356:124: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
         tensor_lists[0][0].scalar_type(), 0, "multi_tensor_l2norm_cuda",
                                                                                                                            ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:358:89: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
             BLOCK_SIZE, chunk_size, noop_flag, tensor_lists,
                                                                                         ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:358:124: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
             BLOCK_SIZE, chunk_size, noop_flag, tensor_lists,
                                                                                                                            ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:376:101: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
   cleanup_v2<<<ntensors, 512, 0, stream>>>(
                                                                                                     ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:376:136: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
   cleanup_v2<<<ntensors, 512, 0, stream>>>(
                                                                                                                                        ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:376:157: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
   cleanup_v2<<<ntensors, 512, 0, stream>>>(
                                                                                                                                                             ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:376:178: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
   cleanup_v2<<<ntensors, 512, 0, stream>>>(
                                                                                                                                                                                  ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh: In instantiation of ‘void multi_tensor_apply(int, int, const at::Tensor&, const std::vector<std::vector<at::Tensor> >&, T, ArgTypes ...) [with int depth = 1; T = L2NormFunctor<float>; ArgTypes = {float*, float*, bool, int}]’:
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:290:57:   required from here
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh:104:150: warning: ‘T* at::Tensor::data() const [with T = int]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
                 multi_tensor_apply_kernel<<<loc_block_info, block_size, 0, stream>>>(
                                                                                                                                                      ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh: In instantiation of ‘void multi_tensor_apply(int, int, const at::Tensor&, const std::vector<std::vector<at::Tensor> >&, T, ArgTypes ...) [with int depth = 1; T = L2NormFunctor<c10::Half>; ArgTypes = {float*, float*, bool, int}]’:
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:292:88:   required from here
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh:104:150: warning: ‘T* at::Tensor::data() const [with T = int]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
                 multi_tensor_apply_kernel<<<loc_block_info, block_size, 0, stream>>>(
                                                                                                                                                      ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh: In instantiation of ‘void multi_tensor_apply(int, int, const at::Tensor&, const std::vector<std::vector<at::Tensor> >&, T, ArgTypes ...) [with int depth = 1; T = MaxNormFunctor<float>; ArgTypes = {float*, float*, bool, int}]’:
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:350:27:   required from here
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh:104:150: warning: ‘T* at::Tensor::data() const [with T = int]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
                 multi_tensor_apply_kernel<<<loc_block_info, block_size, 0, stream>>>(
                                                                                                                                                      ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh: In instantiation of ‘void multi_tensor_apply(int, int, const at::Tensor&, const std::vector<std::vector<at::Tensor> >&, T, ArgTypes ...) [with int depth = 1; T = MaxNormFunctor<c10::Half>; ArgTypes = {float*, float*, bool, int}]’:
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu:352:28:   required from here
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/colossalai/kernel/cuda_native/csrc/multi_tensor_apply.cuh:104:150: warning: ‘T* at::Tensor::data() const [with T = int]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
                 multi_tensor_apply_kernel<<<loc_block_info, block_size, 0, stream>>>(
                                                                                                                                                      ^
/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:235:1: note: declared here
   T * data() const {
 ^ ~~
[7/7] c++ colossal_C_frontend.o multi_tensor_sgd_kernel.cuda.o multi_tensor_scale_kernel.cuda.o multi_tensor_adam.cuda.o multi_tensor_l2norm_kernel.cuda.o multi_tensor_lamb.cuda.o -shared -L/home/fangfei/.conda/envs/tmm/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/home/fangfei/.conda/envs/tmm/lib64 -lcudart -o fused_optim.so
Loading extension module fused_optim...
Time to load fused_optim op: 67.05005216598511 seconds
Segmentation fault (core dumped)

triton

A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
...

虽然不影响程序运行 不过去查了一下好像是xformer的原因
参考文章：解决stable diffusion wenUI安装xformers出错问题

Segmentation fault (core dumped)

...
Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads.
DiffusionWrapper has 865.91 M params.
=========================================================================================
No pre-built kernel is found, build and load the cpu_adam kernel during runtime now
=========================================================================================
Emitting ninja build file /home/fangfei/.cache/colossalai/torch_extensions/torch1.13_cu11.7/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.413104295730591 seconds
=========================================================================================
No pre-built kernel is found, build and load the fused_optim kernel during runtime now
=========================================================================================
Detected CUDA files, patching ldflags
Emitting ninja build file /home/fangfei/.cache/colossalai/torch_extensions/torch1.13_cu11.7/build.ninja...
Building extension module fused_optim...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_optim...
Time to load fused_optim op: 1.8385069370269775 seconds
Segmentation fault (core dumped)

因为之前跑stable diffusion不会出现这个问题，分析要么是colossalai加的代码有问题，要么是包冲突。
已向colossalai的github提交issue等待回复中…

  1. 尝试输出报错信息进行定位

在我们运行程序的时候，经常会遇到Segmentation fault (core dumped)的问题，这种问题多见于内存操作不当，比如空指针、野指针的读写操作，数组越界访问，常量被破坏等。对于一个比较大的程序，比较难定位错误具体的位置，因此我们需要利用core文件进行错误的查找。

[解决问题]查找产生Segmentation fault (core dumped)错误的真正原因
如何生成和调试 Linux 程序崩溃产生的 core 文件
查资料得知core dumped问题可以通过core文件查看报错
1.ulimit -c unlimited #不限制core文件大小
此时，程序产生core dump时会自动生成core文件在默认生成路径中
2.如果没有看到core文件可以修改生成路径到当前文件夹
例如：
sudo sysctl -w kernel.core_pattern=core.%p
3.core文件查看方式
gdb sysbench core.37795
或者进入gdb后
core-file core.7416

...
[New LWP 39080]
Core was generated by `python main.py --logdir tmp --train --base configs/train_colossalai_cifar10.yaml'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fba24575540 in ?? ()
[Current thread is 1 (LWP 37795)]

报错的是这一句Program terminated with signal SIGSEGV
SIGSEGV意思是程序企图向分配的内存以外的区域读写，或者向只读区域进行写操作（访问空指针，内存越界访问，访问已经释放的内存…）
非常难受的是，范围太广了难以定位。

尝试gdb调试python
GDB调试指南(入门，看这篇够了)
使用gdb调试Python进程
使用gdb调试Python程序

待解决…

天线狗子
关注
  • 1
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
专栏目录
巨大的约翰逊「Colossal Johnson」-crx插件
03-13
将'鲍里斯约翰逊'的文本替换为'庞克洛克逊鲍里斯约翰逊'。 继鲍里斯·约翰逊（Boris Johnson）最近的滑稽动作之后，他改用了更合适的名字。 单词交换代码和灵感的全部功劳归功于Eric Bailey https://github.com/ericwbailey/millennials-to-snake-people-我只是复制了他的工作并更改了一些单词。 支持语言:English (UK)
实战：手把手教你colossal-AI复现Chatgpt的流程
02-23 1082
相信很多人都看了使用colossal-AI复现Chatgpt的流程的文章，但实际上看过了，不免有人发出“说得贼明白，就是自己做不出来”的感叹吧。本文手把手教你如何实现这个流程。
colossal cave adventure
12-12
使用Fortran开发的经典游戏 colossal cave adventure巨洞文字冒险游戏，是电脑游戏史上的里程碑，珍贵稀有的精品源代码及可执行文件。
Colossal Johnson-crx插件
04-06
语言:English (UK) 用'巨大的plonker borisjohnson'替换文本'borisjohnson'。 用一些似乎更真实的形式替换'boris johnson'。 单词交换代码和灵感的所有信用都转到eric bailey https://github.com/ercwbailey/millennials-to-snake-people - 我只是重复他的工作并改变了几个字。 </ div>
克隆ChatGPT，开源模型完整复现
03-29
以 ChatGPT、GPT4 为代表的 AI 应用和大模型火爆全球，被视为开启了新的科技工业革命和 AGI （通用人工智能）的新起点。不仅科技巨头间你追我赶，争相推出新品，许多学术界、工业界的 AI 大佬也纷纷投入投身相关创业浪潮。生成式 AI 正以“天”为单位，快速迭代，持续狂飙！ 然而，OpenAI 并未将其开源，它们背后的技术细节有哪些？如何快速跟进、追赶并参与到此轮技术浪潮中？如何降低 AI 大模型构建和应用的高昂成本？如何保护核心数据与知识产权不会因使用第三方大模型 API 外泄？ 作为当下最受欢迎的开源 AI 大模型解决方案，Colossal-AI 率先建立了包含监督数据集收集->监督微调->奖励模型训练->强化学习微调的完整 RLHF 流程，以 LLaMA 为基础预训练模型，推出 ColossalChat，是目前最接近 ChatGPT 原始技术方案的实用开源项目! 包含以下内容 1、Demo：可直接在线体验模型效果，无需注册或 waitinglist2、训练代码：开源完整 RLHF 训练代码，已开源至含 7B 和 13B 两种模型3、数据集：开源 104K 中、英双语数据
分析解决【No module named ‘triton‘】的问题
热门推荐
Shion's bla bla blog
04-13 3万+
A matching Triton is not available, some optimizations will not be enabled ？？？
Error caught was: No module named ‘triton‘
WGS.
04-07 3894
Error caught was: No module named 'triton'
碰到一个用pip死活装不上模块的问题（ModuleNotFoundError: No module named ‘triton‘）
dalaomanzou的博客
10-18 1368
也是被原项目坑了一手，上面没提醒是在linux平台上用的，下意识以为在windows上可用。折腾这个浪费我不少时间。而一个模块如果只在linux上可用，而在windows上不可用的话，就会这样报错并始终下载不了。折腾了半天才发现问题的根源在于平台不同。命令安装该模块的时候，总会提示。当我在windows上尝试用。
ColossalAI-Chat训练手册（RLHF）
chtw的博客
04-15 2456
ColossalAI-Chat是一款基于人工智能技术的智能聊天机器人，是由Colossal AI开发的一款聊天机器人。该机器人使用了最先进的自然语言处理技术和深度学习算法，可以回答各种问题、提供建议、提供娱乐和与用户进行轻松对话。ColossalAI-Chat可以在多种平台上使用，例如Facebook Messenger、Slack、WeChat等。ColossalAI-Chat通过使用自然语言处理技术和深度学习算法，机器人可以理解人类语言的含义，从而生成更加自然和准确的回答。
Colossal.txt
最新发布
04-23
Colossal.txt
colossal_buck
09-02
colossal_buck
La1-xCexMnO3电子掺杂薄膜中Ce的掺杂比对输运性质和机理的影响
03-13
我们对La1-xCexMnO3进行了实验研究，以研究电子掺杂钙钛矿薄膜的传输特性和场致传输机理。 La1-xCexMnO3膜表现出显着的金属-绝缘体转变，这可以通过Ce的掺杂比来调节。 电阻-温度曲线表明，磁畴和电子-散射主要是在...
Strain effect on colossal oxygen ionic conductivity in nanoscale zirconia electrolytes: A first-principles-based study
02-21
钇稳定氧化锆二维纳米薄膜室温庞氧离子电导特性研究，李峰，陆瑞锋，采用第一性原理计算，研究了三明治结构中晶格失配应变与二维纳米薄膜室温庞氧离子电导现象之间的关系。对KTaO3/YSZ/KTaO3三明治结构而
关于收录现实中文版ChatGPT的各种技术路线，数据及其他资料
04-29
InstructionWild : Colossal AI 收集的中英文双语数据集(104K) GPT-4-LLM : GPT-4标注的中英文双语指令微调数据，提示来自Stanford Alpaca。 ShareGPT : ChatGPT用户分享的聊天数据，大部分分为英文数据，插件维护者...
大模型训练之加速篇 -attention优化【稀疏attention-＞线性化attention-＞分块计算-＞简化attention-＞Transformer-VQ】
weixin_36378508的博客
09-21 2529
加速
背景宽高随文本变化_今日论文|零资源口语建模&多尺度合并蒸馏语义分割持续学习&迷彩动物识别&数据到文本语义准确新指标&自动深度GNN设计&MORE...
weixin_39611072的博客
12-06 4725
11月24日人工智能领域新增论文456篇，AI日读精选其中88篇推荐给大家。这些论文主要来自AAAI，AAMAS，ACCV，CIKM，Artificial Intelligence，CVPR，ECCV，EMNLP，ICASSP，ICLR，ICML，ICONIP，ICPR，ICRA，TKDE，INTERSPEECH，Information Systems，Machine Learning，...
万字长文解读Stable Diffusion的核心插件—ControlNet
xian0710830114的专栏
04-14 2万+
ControlNet在大型预训练扩散模型（Stable Diffusion）的基础上实现了更多的输入条件，如边缘映射、分割映射和关键点等图片加上文字作为Prompt生成新的图片，同时也是stable-diffusion-webui的重要插件。ControlNet因为使用了冻结参数的Stable Diffusion和零卷积，使得即使使用个人电脑在小的数据集上fine-tuning效果也不会下降，从而实现了以端到端方式学习特定任务的条件目的。
基于Pytorch2对比 FlashAttention、Memory-Efficient Attention、CausalSelfAttention
写bug的程旭源
03-19 5573
本文主要是Pytorch2.0 的小实验，在MacBookPro 上体验一下等优化改进后的Transformer Self Attention的性能，具体的有 FlashAttention、Memory-Efficient Attention、CausalSelfAttention 等。主要是torch.compile(model) 和 scaled_dot_product_attention的使用。Pytorch2.0版本来了，带来了很多的新技术。
Colossal-AI
03-27
Colossal-AI是一个由CSDN自主开发的AI助手，它基于强大的自然语言处理技术，可以回答各种问题并提供相关的知识和信息。Colossal-AI可以帮助用户解决技术问题、提供学习资料、进行智能对话等。它具有广泛的应用领域，包括但不限于编程、数据分析、人工智能等。通过与Colossal-AI的交互，用户可以获得准确、及时的帮助和指导。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值