- [rank0]: ValueError: Attempting to unscale FP16 gradients.
将amp=True 改成 amp=False
接着出现:
2. RuntimeError: Input type (float) and bias type (c10::Half) should be the same
我直接暴力
x = x.half()
- [rank0]: ValueError: trying to initialize the default process group twice!
报错的文件前面加上
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '5678'
dist.init_process_group(backend='nccl',rank=0, world_size = 1)
- [rank0]: RuntimeError: probability tensor contains either
inf
,nan
or element < 0
搜索了半天大家的思路是把fp16改成bp16
然后我在config中修改
参考链接
# vit encoder
image_size: 224
drop_path_rate: 0
use_grad_checkpoint: False
vit_precision: "bf16"
freeze_vit: False
然后会出现2的错误,把刚刚暴力改成半精度的删除,就可以run了
终于解决跑起来blip2模型
挂起来的时候又出现新问题
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network…
大概是端口号占用了
因此突发奇想
os.environ[‘MASTER_PORT’] = ‘5678’
把这个5678 换成其他数字 成功!
跑起来了终于。。。