ImageNet 数据集准备（用于分类）

HLBayes

已于 2022-07-17 18:18:42 修改

阅读量5.1k

点赞数 1

分类专栏： Linux 文章标签：分类数据挖掘人工智能

于 2022-05-31 12:12:51 首次发布

本文为博主（HLBayes）原创文章，请经博主允许后转载。

本文链接：https://blog.csdn.net/HLBoy_happy/article/details/125060319

版权

Linux 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

ImageNet 数据集准备（用于分类）

主要内容
常用命令
常见BUG
- 文件传输出现`No space left on device`问题（如下）
- torch导入数据时出现 `RuntimeError: Trying to resize storage that is not resizable`

主要内容

ImageNet数据集较大，解压前有136G(train)+6.3G(val)；解压后有140G(train)+6.4G(val)，并且较常用于图像分割，所以要快速导入ImageNet并用与分类，则需要做些预处理：

将数据存储到/dev/shm/文件夹下
将所有图片resize为相同的 $256\times 256$ 大小

这其中会涉及到“内存大小是否足够、文件夹的删除、重命名”等等操作。

改存储

修改/dev/shm/为160G：

sudo mount -o size=163840M  -o remount /dev/shm

解压

如果下载的数据集是压缩的，可以在下载的文件夹下使用以下命令进行解压和数据移动：

# 在/dev/shm/下创建文件夹；解压数据集到指定文件夹；
mkdir -p /dev/shm/ILSVRC12_256/{train,val} && tar -xvf ILSVRC2012_img_train.tar -C /dev/shm/ILSVRC12_256/train && tar -xvf ILSVRC2012_img_val.tar -C /dev/shm/ILSVRC12_256/val

# 进入/dev/shm/ILSVRC12_256/train 文件夹
cd /dev/shm/ILSVRC12_256/train
# 解压1000个类别压缩包并创建对应的子文件
find . -name "*.tar" | while read NAME ; do mkdir -p "${NAME%.tar}"; tar -xvf "${NAME}" -C "${NAME%.tar}"; rm -f "${NAME}"; done

# 进入/dev/shm/ILSVRC12_256/val文件夹
cd /dev/shm/ILSVRC12_256/val
# 利用线上的sh命令对验证集分类，得到和训练集一样的文件结构
wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash

图片预处理

[可选择操作，使用后在代码中就不用resize了] 对所有图片进行做resize的预处理，加快图片从磁盘加载到内存（处理后68G(train)+2.9G(val)；linux默认没有安装convert，使用sudo apt-get install imagemagick进行安装）：

cd /dev/shm/ILSVRC12_256/val
find . -name "*.JPEG" | xargs -I {} convert {} -resize "256^>" {}

cd /dev/shm/ILSVRC12_256/train
find . -name "*.JPEG" | xargs -I {} convert {} -resize "256^>" {}

常用命令

# 从服务上下载文件
scp -r username@servername:/var/www/remote_dir/（远程目录） /var/www/local_dir（本地目录）
# 查看文件大小
df -h  # 查看根目录下各个主文件夹的大小
du -hs *  # 查看当前文件夹下各个子文件夹和文件的大小
# 文件移动、重命名
mv dir1 dir2  # 如果目录dir2存在，将dir1移动到dir2中；否则，为文件夹重命名，将目录dir1改名为dir2
find . -mindepth 2 -type f -exec mv -i -- {} . \;  # 将子所有文件夹下的文件移动到父目录 ()
find . -depth -mindepth 1 -type d -empty -exec rmdir {} \;  # 删除空目录
# 复制文件夹内容、复制文件夹
cp -a A/. B  # 复制A文件夹下所有的子内容到B内
cp -a A B  # 复制A文件夹本身到B内; 当B不存在时，复制A文件夹重命名为B
# 软链接\快捷方式
ln -s ../A A  # 当前路径下创建文件夹A引向../A文件夹；注意在软链接文件夹内需用使用绝对路径增加新文件软链接
rm -rf ./A  # 删除软连接; 千万别错写成 rm -rf ./A/

常见BUG

文件传输出现`No space left on device`问题（如下）

(base) HLBayes@ubuntu:~/HLBayes/a_Data$ scp -r user@xxx.xxx.xxx.xxx:~/HLBayses/ILSVRC2012_img_train.tar.gz ./
ILSVRC2012_img_train.tar.gz                                                                      100%  136GB   5.1MB/s 7:34:31    
.//ILSVRC2012_img_train.tar.gz: No space left on device

先通过df -h看看是不是磁盘满了（结果显示并没有）

(base) HLBayes@ubuntu:~/HLBayes/a_Data$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             32G     0   32G   0% /dev
tmpfs           6.3G  2.2M  6.3G   1% /run
/dev/sda5       469G  445G   16M 100% /
tmpfs            32G     0   32G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/loop1       56M   56M     0 100% /snap/core18/2128
/dev/loop0      219M  219M     0 100% /snap/gnome-3-34-1804/72
/dev/loop4       51M   51M     0 100% /snap/snap-store/547
/dev/loop3       33M   33M     0 100% /snap/snapd/12704
/dev/loop2       66M   66M     0 100% /snap/gtk-common-themes/1515
/dev/sdb1       1.8T  507G  1.3T  30% /home/HLBayes/sdb1
/dev/sda1       512M  6.6M  506M   2% /boot/efi
/dev/sda4       1.9T  1.0T  800G  57% /home/HLBayes/sda4
/dev/sda2       1.2T  902G  250G  79% /home/HLBayes/sda2
tmpfs           6.3G  8.0K  6.3G   1% /run/user/1000

再用lsof | grep deleted确认是不是某些大文件被删除，但并没有释放空间而只是被标记为(deleted) （结果显示确实是，并可以看到它们的描述符354916、354967…，将它们杀掉就好了。注意：在kill 354916之前先用htop查看是否是自己正在跑的程序，以防把自己在跑的程序弄断了。）

(base) HLBayes@ubuntu:/~/HLBayes/a_Data$ lsof | grep deleted
pulseaudi   16981                                 ws    6u      REG                0,1 67108864       6144 /memfd:pulseaudio (deleted)
pulseaudi   16981   17058 null-sink               ws    6u      REG                0,1 67108864       6144 /memfd:pulseaudio (deleted)
pulseaudi   16981   17084 snapd-gli               ws    6u      REG                0,1 67108864       6144 /memfd:pulseaudio (deleted)
sh         354906                                 ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916                                 ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354917 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354918 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354919 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354920 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354921 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354922 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354943 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354944 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354945 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354916  354946 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967                                 ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354968 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354969 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354970 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354971 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354972 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354976 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354983 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354984 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354985 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       354967  354986 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004                                 ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355005 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355006 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355007 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355008 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355009 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355010 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355014 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355015 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355016 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355017 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355004  355040 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018                                 ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355019 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355020 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355021 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355022 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355023 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355024 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355025 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355026 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355027 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355028 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355029 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355018  355030 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355066                                 ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355066  355067 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355066  355068 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355066  355069 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355066  355070 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355066  355071 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       355066  355072 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       381610                                 ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       381610  381611 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       381610  381612 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       381610  381613 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       381610  381614 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       381610  381615 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       381610  381616 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)
node       381610  381617 node                    ws   99w      REG                8,5        0    2097153 /home/ws/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/vscode-remote-lock.ws.c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5 (deleted)

torch导入数据时出现 `RuntimeError: Trying to resize storage that is not resizable`

数据的dataloader构造时没有使用resize或者crop对样本进行transform。

参考：
IMAGENET
ImageNet(ISLVRC2012)数据集 - 小傻鱼的文章 - 知乎

HLBayes

关注

1
点赞
踩
9

收藏

觉得还不错? 一键收藏
1
评论
ImageNet 数据集准备（用于分类）

ImageNet数据集较大，解压前有136G(train)+6.3G(val)；解压后有140G(train)+6.4G(val)，并且较常用于图像分割，所以要用快速导入ImageNet并用与分类，则需要做以下几点预处理：将数据存储到/dev/shm/文件夹下将所有图片resize为相同的 256×256256\times 256256×256 大小这其中会涉及到“内存大小是否足够、文件夹的删除、重命名”等等操作。查看文件夹大小df -h # 查看根目录下各个主文件夹的大小du -hs
复制链接

扫一扫