FCN训练自己的数据遇到的问题总结

配置个FCN遇到了无数的坑，真的是醉了，特意记录下来，希望对大家有些帮助，少走弯路。

1.运行solve.py时,出现一下错误:

Traceback (most recent call last):
  File "solve.py", line 26, in <module>
    solver = caffe.SGDSolver('solver.prototxt')
  File "/home/b2403/caffe-master/fcn.berkeleyvision.org/voc-fcn32s/voc_layers.py", line 183, in reshape
    self.label = self.load_label(self.indices[self.idx])
  File "/home/b2403/caffe-master/fcn.berkeleyvision.org/voc-fcn32s/voc_layers.py", line 228, in load_label
    im = Image.open('{}/cls/{}.png'.format(self.voc_dir, idx))

AttributeError: 'SBDDSegDataLayer' object has no attribute 'voc_dir'

解决办法:voc_layers的SBDDSegDataLayer函数中的im = Image.open('{}/cls/{}.png'.format(self.voc_dir, idx))改为

im = Image.open('{}/cls/{}.png'.format(self.sbdd_dir, idx))

2.训练时出现下面的错误,将
I0714 09:49:08.538975  6419 solver.cpp:228] Iteration 1980, loss = 4.3501e+06
I0714 09:49:08.539055  6419 solver.cpp:244]     Train net output #0: loss = 5.72268e+06 (* 1 = 5.72268e+06 loss)
I0714 09:49:08.539077  6419 sgd_solver.cpp:106] Iteration 1980, lr = 1e-11
>>> 2018-07-14 09:49:12.091805 Begin seg tests
Iteration 140, loss = -nan
Iteration 140, loss = -nan
>>> 2018-07-14 09:50:26.481155 Iteration 2000 loss 4225485.24608
>>> 2018-07-14 09:50:26.481276 Iteration 2000 overall accuracy nan
>>> 2018-07-14 09:50:26.481357 Iteration 2000 mean accuracy nan
>>> 2018-07-14 09:50:26.481606 Iteration 2000 mean IU nan

>>> 2018-07-14 09:50:26.481780 Iteration 2000 fwavacc 0.0

3.训练开始时出现Check failed: error == cudaSuccess (2 vs. 0)  out of memory错误

I0714 09:58:01.326063 11372 sgd_solver.cpp:106] Iteration 0, lr = 1e-11
F0714 09:58:01.468791 11372 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
已放弃 (核心已转储)

原因：查到很多博客说是因为内存不足，而我的原因是因为图像没有统一大小，所以就把训练图像，测试图像，以及label全部变成256*256大小的。

4.将图像统一大小后，进行训练时，又遇到了这个错误：