尝试GraphVQA的优化器及初探VinVL

代码地址：https://github.com/codexxxl/GraphVQA

https://github.com/pzzhang/VinVL

https://github.com/microsoft/scene_graph_benchmark

1.VinVL 最新目标属性提取器，测试结果

python tools/demo/demo_image.py --config_file sgg_configs/vgattr/vinvl_x152c4.yaml --img_file demo/woman_fish.jpg --save_file output/woman_fish_x152c4.obj.jpg MODEL.WEIGHT pretrained_model/vinvl_vg_x152c4.pth MODEL.ROI_HEADS.NMS_FILTER 1 MODEL.ROI_HEADS.SCORE_THRESH 0.2 TEST.IGNORE_BOX_REGRESSION False

woman_fish_x152c4.obj.jpg 

 运行上面指令，检测后的标注信息

运行检测属性的代码 

python tools/demo/demo_image.py --config_file sgg_configs/vgattr/vinvl_x152c4.yaml --img_file demo/woman_fish.jpg --save_file output/woman_fish_x152c4.attr.jpg --visualize_attr MODEL.WEIGHT pretrained_model/vinvl_vg_x152c4.pth MODEL.ROI_HEADS.NMS_FILTER 1 MODEL.ROI_HEADS.SCORE_THRESH 0.2 TEST.IGNORE_BOX_REGRESSION False

检测结果输出：

[{'rect': [991.9342651367188, 1908.3780517578125, 3000.91748046875, 2725.4501953125], 'class': 504, 'conf': 0.9993057250976562, 'attr': [191, 229, 34], 'attr_conf': [0.3128730356693268, 0.18410372734069824, 0.15922875702381134]}, {'rect': [1977.78759765625, 898.5238037109375, 3707.15380859375, 2725.4501953125], 'class': 1166, 'conf': 0.8882755637168884, 'attr': [322, 70], 'attr_conf': [0.6487608551979065, 0.17789442837238312]}, {'rect': [2338.898193359375, 285.08843994140625, 2973.72802734375, 952.0045166015625], 'class': 625, 'conf': 0.8609470129013062, 'attr': [39], 'attr_conf': [0.7681410312652588]}, {'rect': [2317.74951171875, 285.910400390625, 3056.99609375, 663.0642700195312], 'class': 623, 'conf': 0.7241727113723755, 'attr': [39, 34, 113], 'attr_conf': [0.8176420331001282, 0.06621109694242477, 0.052659668028354645]}, {'rect': [3133.188232421875, 1512.47265625, 4036.172119140625, 1752.4656982421875], 'class': 616, 'conf': 0.6738954782485962, 'attr': [516, 48], 'attr_conf': [0.462296724319458, 0.36743414402008057]}, {'rect': [0.0, 1344.9588623046875, 4027.20849609375, 2725.4501953125], 'class': 1535, 'conf': 0.6564119458198547, 'attr': [59, 39], 'attr_conf': [0.27179425954818726, 0.2638888955116272]}, {'rect': [1863.373046875, 343.1026306152344, 3880.974853515625, 2725.4501953125], 'class': 965, 'conf': 0.5709189176559448, 'attr': [524, 413, 322, 436], 'attr_conf': [0.16003631055355072, 0.12642444670200348, 0.10024773329496384, 0.05118722841143608]}, {'rect': [2449.614990234375, 327.15380859375, 2924.2216796875, 622.8408203125], 'class': 576, 'conf': 0.5566259026527405, 'attr': [39, 34], 'attr_conf': [0.5047386884689331, 0.3504347801208496]}, {'rect': [16.437084197998047, 0.0, 4013.478515625, 1397.109130859375], 'class': 1218, 'conf': 0.5476083159446716, 'attr': [504, 86, 39, 191], 'attr_conf': [0.2666150629520416, 0.2562120258808136, 0.2138122022151947, 0.08350048214197159]}, {'rect': [1945.4022216796875, 230.500244140625, 3718.212890625, 2712.58203125], 'class': 1573, 'conf': 0.48703911900520325, 'attr': [524, 322, 413], 'attr_conf': [0.1432984620332718, 0.13907624781131744, 0.11325392127037048]}, {'rect': [2521.505126953125, 578.2726440429688, 2907.731201171875, 963.7191162109375], 'class': 476, 'conf': 0.4716569781303406, 'attr': [413, 48, 34, 112, 504], 'attr_conf': [0.10887283086776733, 0.07077351957559586, 0.06388427317142487, 0.0602513886988163, 0.05938141047954559]}, {'rect': [3699.579833984375, 322.67523193359375, 4026.988037109375, 1481.635009765625], 'class': 1473, 'conf': 0.391387939453125, 'attr': [193, 459, 229], 'attr_conf': [0.5401939153671265, 0.1762886792421341, 0.06615839153528214]}, {'rect': [2440.798095703125, 348.1719970703125, 2967.064208984375, 628.6888427734375], 'class': 1336, 'conf': 0.3430824875831604, 'attr': [39, 34], 'attr_conf': [0.5874763131141663, 0.2740896940231323]}, {'rect': [3169.352783203125, 780.6634521484375, 3553.637939453125, 1478.3839111328125], 'class': 1473, 'conf': 0.3334616720676422, 'attr': [193, 459, 229], 'attr_conf': [0.6520551443099976, 0.11651214212179184, 0.051287371665239334]}, {'rect': [2891.11279296875, 0.0, 4091.448974609375, 793.7659301757812], 'class': 309, 'conf': 0.3211399018764496, 'attr': [504, 191, 467], 'attr_conf': [0.6393128633499146, 0.08907239884138107, 0.06135357543826103]}, {'rect': [3482.58642578125, 382.056640625, 3749.58642578125, 1415.3408203125], 'class': 1473, 'conf': 0.31957191228866577, 'attr': [193, 459, 320, 229], 'attr_conf': [0.5199567079544067, 0.1837625503540039, 0.05761323869228363, 0.05211116001009941]}, {'rect': [2321.515869140625, 605.0642700195312, 2526.9013671875, 977.3557739257812], 'class': 605, 'conf': 0.29987606406211853, 'attr': [37, 247], 'attr_conf': [0.7735818028450012, 0.09806638211011887]}, {'rect': [2127.5751953125, 2134.81591796875, 2262.715576171875, 2415.89990234375], 'class': 496, 'conf': 0.297390341758728, 'attr': [108, 412, 393, 246], 'attr_conf': [0.1142352744936943, 0.10552464425563812, 0.05669544264674187, 0.05169088393449783]}, {'rect': [2677.3310546875, 134.84388732910156, 4091.448974609375, 1580.820556640625], 'class': 1473, 'conf': 0.29660564661026, 'attr': [193, 459, 229], 'attr_conf': [0.5245699286460876, 0.15207165479660034, 0.05997393652796745]}, {'rect': [2024.552001953125, 2193.329833984375, 2158.512451171875, 2396.92578125], 'class': 496, 'conf': 0.28710007667541504, 'attr': [108, 412, 504], 'attr_conf': [0.12590305507183075, 0.08486489951610565, 0.05081488564610481]}, {'rect': [624.5617065429688, 1420.5489501953125, 759.9111328125, 1486.137451171875], 'class': 157, 'conf': 0.27564460039138794, 'attr': [412, 504], 'attr_conf': [0.5688188076019287, 0.23694151639938354]}, {'rect': [1958.045654296875, 219.3265838623047, 3907.9912109375, 2725.4501953125], 'class': 786, 'conf': 0.2753722369670868, 'attr': [524, 413, 322, 436], 'attr_conf': [0.18565915524959564, 0.1339663416147232, 0.07900091260671616, 0.053055983036756516]}, {'rect': [2745.610595703125, 608.8289184570312, 2854.701171875, 720.16162109375], 'class': 873, 'conf': 0.2748320400714874, 'attr': [229, 34, 32, 412, 112, 247], 'attr_conf': [0.17484165728092194, 0.15115873515605927, 0.13260619342327118, 0.08716263622045517, 0.08315394818782806, 0.051453426480293274]}, {'rect': [1991.775146484375, 1731.2242431640625, 2384.193359375, 2199.8984375], 'class': 613, 'conf': 0.24275298416614532, 'attr': [504, 21], 'attr_conf': [0.20696546137332916, 0.053015097975730896]}, {'rect': [0.0, 932.7134399414062, 3893.390625, 1714.900146484375], 'class': 763, 'conf': 0.23321756720542908, 'attr': [34, 247], 'attr_conf': [0.3523329198360443, 0.2780471444129944]}, {'rect': [2690.24169921875, 752.7881469726562, 2864.73681640625, 844.5185546875], 'class': 846, 'conf': 0.22465139627456665, 'attr': [504, 292, 32, 229], 'attr_conf': [0.38891884684562683, 0.1857149600982666, 0.17027851939201355, 0.05245932564139366]}, {'rect': [3004.89990234375, 819.2578735351562, 3244.568115234375, 1269.077392578125], 'class': 1473, 'conf': 0.21888506412506104, 'attr': [193, 459, 320], 'attr_conf': [0.5380640625953674, 0.18382669985294342, 0.08441270887851715]}, {'rect': [0.0, 915.714599609375, 1703.8875732421875, 1393.787841796875], 'class': 837, 'conf': 0.21269068121910095, 'attr': [34, 193, 48, 129, 191, 39], 'attr_conf': [0.20238496363162994, 0.12581747770309448, 0.06017753854393959, 0.05757256969809532, 0.05753318965435028, 0.05716494470834732]}, {'rect': [3646.268310546875, 1735.0911865234375, 3981.6572265625, 2032.996826171875], 'class': 1413, 'conf': 0.20098179578781128, 'attr': [370, 523, 79, 34], 'attr_conf': [0.3161413371562958, 0.26935118436813354, 0.13629189133644104, 0.12578412890434265]}]

 faster rcnn 的来比较一波

是不是推荐框有点少？（up_2_down_10_100）

 但是强制36个框还是挺强的

2.VinVL Feature extraction 

python tools/test_sg_net.py --config-file sgg_configs/vgattr/vinvl_x152c4.yaml TEST.IMS_PER_BATCH 2 MODEL.WEIGHT models/vinvl/vinvl_vg_x152c4.pth MODEL.ROI_HEADS.NMS_FILTER 1 MODEL.ROI_HEADS.SCORE_THRESH 0.2 DATA_DIR "../maskrcnn-benchmark-1/datasets1" TEST.IGNORE_BOX_REGRESSION True MODEL.ATTRIBUTE_ON True 

To extract relation features (union bounding box's feature), in yaml file, set TEST.OUTPUT_RELATION_FEATURE to True, add relation_feature in TEST.TSV_SAVE_SUBSET.

To extract bounding box features, in yaml file, set TEST.OUTPUT_FEATURE to True, add feature in TEST.TSV_SAVE_SUBSET.