Real-word grasp_multiObject_multiGrasp 代码复现demo + 训练 + 测试

Real-word grasp_multiObject_multiGrasp 代码复现demo + 训练 + 测试
论文代码：
链接: [link].接:https://github.com/ivalab/grasp_multiObject_multiGrasp.
复现所用环境：Ubuntu16.04 + cuda8 + cudnn 5.1 + tensorflow-gpu1.2.0 + python2.7;

demo复现

1、首先根据github上提示，下载程序源码，编译运行环境，下载cocoAPI；
2、安装程序运行多需库文件：
shapely scikit-image easydict opencv-python matplotlib Pillow 应该就这些，要是又缺少在运行demo时会提示，根据提示在补充即可；
具体安装方式：。

// An highlighted block
pip install easydict -i https://pypi.tuna.tsinghua.edu.cn/simple;
pip install cython -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install opencv-python -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install matplotlib -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install Pillow -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install scikit-image --upgrade --user -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install shapely -i https://pypi.tuna.tsinghua.edu.cn/simple``下面展示一些 `内联代码片`。

3、运行demo时图片路径的修改：将data/demo下的图片存放到data/demo/results_all_cls文件夹下；同时找到./tools/demo_graspRGD.py 修改109行为

// An highlighted block
im_file = os.path.join(cfg.DATA_DIR, 'demo/results_all_cls', image_name)

4、设置到这一步运行：

// An highlighted block
./tools/demo_graspRGD.py --net res50 --dataset grasp

5、在次遇到一个报错：TypeError: bottleneck() argument after ** must be a mapping, not tuple；解决方法：在python2.7/site-packages/tensorflow/contrib/slim/python/slim/nets/resnet_utils.py文件第203行后加：

// An highlighted block
unit={'depth':unit[0],'depth_bottleneck':unit[1],'stride':unit[2]}

6、到此demo能够顺利跑通；

模型训练

1、数据集下载：cornell datasets 链接: [link]https://github.com/ivalab/grasp_multiObject
2、数据预处理，在本工程中有两种方法：matlab方式根据作者提供的dataPreprocessingTest_fasterrcnn_split.m文件进行数据预处理，这个脚本似乎有问题，我根据自己的理解对脚本做了修改，源码如下：

// An highlighted block
%% created by Fu-Jen Chu on 09/15/2016

close all
clear

%parpool(4)
addpath('/home/dongmingshuai/cornell_datasets/cornell_grasp_data/grasp_datasets')


% generate list for splits
list = [100:949 1000:1034];
list_idx = randperm(length(list));
train_list_idx = list_idx(length(list)/5+1:end);
test_list_idx = list_idx(1:length(list)/5);
train_list = list(train_list_idx);
test_list = list(test_list_idx);


for folder = 1:10
display(['processing folder ' int2str(folder)])

imgDataDir = ['/home/dongmingshuai/cornell_datasets/cornell_grasp_data/grasp_datasets/' sprintf('%02d',folder)];
txtDataDir = ['/home/dongmingshuai/cornell_datasets/cornell_grasp_data/grasp_datasets/' sprintf('%02d',folder)];

%imgDataOutDir = ['/home/dongmingshuai/cornell_datasets/cornell_grasp_data/grasp_datasets/' sprintf('%02d',folder) '_Cropped320_rgd'];
imgDataOutDir = '/home/dongmingshuai/grasp_model/grasp_multiObject_multiGrasp/data/Images';
annotationDataOutDir = '/home/dongmingshuai/grasp_model/grasp_multiObject_multiGrasp/data/Annotations';
imgSetTrain = '/home/dongmingshuai/grasp_model/grasp_multiObject_multiGrasp/data/ImageSets/train.txt';
imgSetTest = '/home/dongmingshuai/grasp_model/grasp_multiObject_multiGrasp/data/ImageSets/test.txt';

imgFiles = dir([imgDataDir '/*.png']);
txtFiles = dir([txtDataDir '/*pos.txt']);

logfileID = fopen('log.txt','a');
%mainfileID = fopen(['/home/fujenchu/projects/deepLearning/deepGraspExtensiveOffline/data/grasps/scripts/trainttt' sprintf('%02d',folder) '.txt'],'a');
for idx = 1:length(imgFiles)
    %% display progress
    %tic
    display(['processing folder: ' sprintf('%02d',folder) ', imgFiles: ' int2str(idx)])

    %% reading data
    imgName = imgFiles(idx).name;
    [pathstr,imgname] = fileparts(imgName);

    filenum = str2num(imgname(4:7));
    if(any(test_list == filenum))
        file_writeID = fopen(imgSetTest,'a');
        %fprintf(file_writeID, '%s\n', [imgDataDir '_Cropped320_rgd/' imgname '_preprocessed_1.png' ] );
        fprintf(file_writeID, '%s\n', [imgname '_preprocessed_1' ] );
        fclose(file_writeID);
        %continue;%2020_7_16
    end

    txtName = txtFiles(idx).name;
    [pathstr,txtname] = fileparts(txtName);

    img = imread([imgDataDir '/' imgname '.png']);
    fileID = fopen([txtDataDir '/' txtname '.txt'],'r');
    sizeA = [2 100];
    bbsIn_all = fscanf(fileID, '%f %f', sizeA);
    fclose(fileID);

    %% data pre-processing
    [imagesOut bbsOut] = dataPreprocessing_fasterrcnn(img, bbsIn_all, 227, 5, 5);
    %display(['imagesOut:=' imagesOut])
    % for each augmented image
    for i = 1:1:size(imagesOut,2)

        % for each bbs
        file_writeID = fopen([annotationDataOutDir '/' imgname '_preprocessed_' int2str(i) '.txt'],'w');
        printCount = 0;
        for ibbs = 1:1:size(bbsOut{i},2)
          A = bbsOut{i}{ibbs};
          xy_ctr = sum(A,2)/4; x_ctr = xy_ctr(1); y_ctr = xy_ctr(2);
          width = sqrt(sum((A(:,1) - A(:,2)).^2)); height = sqrt(sum((A(:,2) - A(:,3)).^2));
          if(A(1,1) > A(1,2))
              theta = atan((A(2,2)-A(2,1))/(A(1,1)-A(1,2)));
          else
              theta = atan((A(2,1)-A(2,2))/(A(1,2)-A(1,1))); % note y is facing down
          end

          % process to fasterrcnn
          x_min = x_ctr - width/2; x_max = x_ctr + width/2;
          y_min = y_ctr - height/2; y_max = y_ctr + height/2;
          %if(x_min < 0 || y_min < 0 || x_max > 227 || y_max > 227) display('yoooooooo'); end
          if((x_min < 0 && x_max < 0) || (y_min > 227 && y_max > 227) || (x_min > 227 && x_max > 227) || (y_min < 0 && y_max < 0)) display('xxxxxxxxx'); break; end
          cls = round((theta/pi*180+90)/10) + 1;

          % write as lefttop rightdown, Xmin Ymin Xmax Ymax, ex: 261 109 511 705  (x水平 y垂直)
          fprintf(file_writeID, '%d %f %f %f %f\n', cls, x_min, y_min, x_max, y_max );
          printCount = printCount+1;
        end
        if(printCount == 0) fprintf(logfileID, '%s\n', [imgname '_preprocessed_' int2str(i) ]);end

        fclose(file_writeID);
        imwrite(imagesOut{i}, [imgDataOutDir '/' imgname '_preprocessed_' int2str(i) '.png']);
        %if(int2str(i) == 1) imwrite(imagesOut{i}, [imgDataDir '_Cropped320_rgd/' imgname '_preprocessed_' int2str(i) '.png']);end
        %display(['int2str(i)=: ' int2str(i) ])
        if(any(test_list == filenum))
            continue;
        end

        % write filename to imageSet 
        file_writeID = fopen(imgSetTrain,'a');
        fprintf(file_writeID, '%s\n', [imgname '_preprocessed_' int2str(i) ] );
        fclose(file_writeID);

    end


    %toc
end
%fclose(mainfileID);

end

第二种数据预处理的方法为用python方式，参考链接: [link]https://blog.csdn.net/sxl1399504891/article/details/106733754；
通过上述两种方式生成的数据存储格式都为：

// An highlighted block
INRIA
|-- data
    |-- Annotations
         |-- *.txt (Annotation files)
    |-- Images
         |-- *.png (Image files)
    |-- ImageSets
         |-- train.txt

3、数据预处理完成后，需要将lib/datasets/factory.py 43 下数据地址改为自己的：

// An highlighted block
graspRGB_devkit_path=‘/home/dongmingshuai/grasp_model/grasp_multiObject_multiGrasp’

4、将output/res50/train/default/ 下demo运行时的模型移走，不然训练iter与模型的一致，则程序会默认训练完成，直接进入测试状态；所以要想重新训练程序，就把上述目录清空；这个目录本质是存放训练过程中训练所得的模型用的；
5、不要忘记将预训练模型res50.ckpt 等 放入data/imagenet_weights文件夹下；
6、通过一下指令训练模型:

// An highlighted block
./experiments/scripts/train_faster_rcnn.sh 0 graspRGB res50

7、2020.8.7跟新：用我上面修改过后的matlab程序在训练时可能会直接跳转到测试阶段，因为matlab数据预处理可能出现空的.txt文件，在作者的程序中有对文件内容的检测，当检测到空时会直接停止训练，程序在：/lib/datasets/graspRGB.py

已经注释掉的部分，要是有小伙伴遇到没有完成训练直接进入测试阶段的，可以检查一下这部分。

模型测试

待更新。。。