./build/tools/caffe train --solver=examples/mnist/lenet_solver.prototxt
进行调试,目的是跟踪程序的变化,了解caffe的具体执行过程。
发现在
shared_ptr<caffe::Solver<float> >
solver(caffe::SolverRegistry<float>::CreateSolver(solver_param));
之后,除了原来的主线程,多了4个其他线程,进入其他线程,发现是DataReader相关的逻辑。
那为什么是4个?为什么每一个datareader需要一个单独的线程?
调用逻辑上,DataReader::Body::InternalThreadEntry
什么时候开始的新线程?
开始新的线程函数是StartInternalThread
两个地方用到这个函数
设置断点 void BasePrefetchingDataLayer<Dtype>::LayerSetUp 和DataReader::Body::Body
调用栈如下:
* frame #0: 0x000000010017d4df libcaffe.so.1.0.0-rc3`caffe::DataReader::Body::Body(this=0x000000010162d7b0, param=0x0000000101621460) + 95 at data_reader.cpp:66
frame #1: 0x000000010017cfbd libcaffe.so.1.0.0-rc3`caffe::DataReader::Body::Body(this=0x000000010162d7b0, param=0x0000000101621460) + 29 at data_reader.cpp:65
frame #2: 0x000000010017cd79 libcaffe.so.1.0.0-rc3`caffe::DataReader::DataReader(this=0x00000001018155e8, param=0x0000000101621460) + 505 at data_reader.cpp:27
frame #3: 0x000000010017cfed libcaffe.so.1.0.0-rc3`caffe::DataReader::DataReader(this=0x00000001018155e8, param=0x0000000101621460) + 29 at data_reader.cpp:20
frame #4: 0x00000001001f7d72 libcaffe.so.1.0.0-rc3`caffe::DataLayer<float>::DataLayer(this=0x0000000101815000, param=0x0000000101621460) + 82 at data_layer.cpp:17
frame #5: 0x00000001001f7dcd libcaffe.so.1.0.0-rc3`caffe::DataLayer<float>::DataLayer(this=0x0000000101815000, param=0x0000000101621460) + 29 at data_layer.cpp:17
frame #6: 0x00000001001fa4a3 libcaffe.so.1.0.0-rc3`boost::shared_ptr<caffe::Layer<float> > caffe::Creator_DataLayer<float>(param=0x0000000101621460) + 67 at data_layer.cpp:108
frame #7: 0x00000001002990a5 libcaffe.so.1.0.0-rc3`caffe::LayerRegistry<float>::CreateLayer(param=0x0000000101621460) + 709 at layer_factory.hpp:83
frame #8: 0x00000001002e06a2 libcaffe.so.1.0.0-rc3`caffe::Net<float>::Init(this=0x0000000101614170, in_param=0x0000000101610d90) + 3282 at net.cpp:97
frame #9: 0x00000001002df6a7 libcaffe.so.1.0.0-rc3`caffe::Net<float>::Net(this=0x0000000101614170, param=0x0000000101610d90, root_net=0x0000000000000000) + 13495 at net.cpp:27
frame #10: 0x00000001002e5a85 libcaffe.so.1.0.0-rc3`caffe::Net<float>::Net(this=0x0000000101614170, param=0x0000000101610d90, root_net=0x0000000000000000) + 37 at net.cpp:26
frame #11: 0x000000010033400a libcaffe.so.1.0.0-rc3`caffe::Solver<float>::InitTestNets(this=0x000000010171dab0) + 4202 at solver.cpp:184
frame #12: 0x0000000100330ef3 libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Init(this=0x000000010171dab0, param=0x00007fff5fbff2c0) + 1139 at solver.cpp:59
frame #13: 0x000000010033096f libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Solver(this=0x000000010171dab0, param=0x00007fff5fbff2c0, root_solver=0x0000000000000000) + 1663 at solver.cpp:32
frame #14: 0x0000000100356192 libcaffe.so.1.0.0-rc3`caffe::SGDSolver<float>::SGDSolver(this=0x000000010171dab0, param=0x00007fff5fbff2c0) + 66 at sgd_solvers.hpp:19
frame #15: 0x000000010035725d libcaffe.so.1.0.0-rc3`caffe::SGDSolver<float>::SGDSolver(this=0x000000010171dab0, param=0x00007fff5fbff2c0) + 29 at sgd_solvers.hpp:19
frame #16: 0x00000001003621b2 libcaffe.so.1.0.0-rc3`caffe::Solver<float>* caffe::Creator_SGDSolver<float>(param=0x00007fff5fbff2c0) + 50 at sgd_solver.cpp:350
frame #17: 0x000000010000be0b caffe.bin`caffe::SolverRegistry<float>::CreateSolver(param=0x00007fff5fbff2c0) + 475 at solver_factory.hpp:78
frame #18: 0x00000001000047d5 caffe.bin`train() + 5269 at caffe.cpp:236
* frame #0: 0x00000001001d802b libcaffe.so.1.0.0-rc3`caffe::BasePrefetchingDataLayer<float>::LayerSetUp(this=0x0000000102806200, bottom=0x000000010170e370, top=0x000000010170e450) + 27 at base_data_layer.cpp:48
frame #1: 0x000000010019e8cf libcaffe.so.1.0.0-rc3`caffe::Layer<float>::SetUp(this=0x0000000102806200, bottom=0x000000010170e370, top=0x000000010170e450) + 111 at layer.hpp:71
frame #2: 0x00000001002e175d libcaffe.so.1.0.0-rc3`caffe::Net<float>::Init(this=0x000000010170a9f0, in_param=0x00007fff5fbfe200) + 7565 at net.cpp:148
frame #3: 0x00000001002df6a7 libcaffe.so.1.0.0-rc3`caffe::Net<float>::Net(this=0x000000010170a9f0, param=0x00007fff5fbfe200, root_net=0x0000000000000000) + 13495 at net.cpp:27
frame #4: 0x00000001002e5a85 libcaffe.so.1.0.0-rc3`caffe::Net<float>::Net(this=0x000000010170a9f0, param=0x00007fff5fbfe200, root_net=0x0000000000000000) + 37 at net.cpp:26
frame #5: 0x0000000100332dfc libcaffe.so.1.0.0-rc3`caffe::Solver<float>::InitTrainNet(this=0x000000010171b4f0) + 2988 at solver.cpp:105
frame #6: 0x0000000100330ed5 libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Init(this=0x000000010171b4f0, param=0x00007fff5fbff2c0) + 1109 at solver.cpp:57
frame #7: 0x000000010033096f libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Solver(this=0x000000010171b4f0, param=0x00007fff5fbff2c0, root_solver=0x0000000000000000) + 1663 at solver.cpp:32
frame #8: 0x0000000100356192 libcaffe.so.1.0.0-rc3`caffe::SGDSolver<float>::SGDSolver(this=0x000000010171b4f0, param=0x00007fff5fbff2c0) + 66 at sgd_solvers.hpp:19
frame #9: 0x000000010035725d libcaffe.so.1.0.0-rc3`caffe::SGDSolver<float>::SGDSolver(this=0x000000010171b4f0, param=0x00007fff5fbff2c0) + 29 at sgd_solvers.hpp:19
frame #10: 0x00000001003621b2 libcaffe.so.1.0.0-rc3`caffe::Solver<float>* caffe::Creator_SGDSolver<float>(param=0x00007fff5fbff2c0) + 50 at sgd_solver.cpp:350
frame #11: 0x000000010000be0b caffe.bin`caffe::SolverRegistry<float>::CreateSolver(param=0x00007fff5fbff2c0) + 475 at solver_factory.hpp:78
frame #12: 0x00000001000047d5 caffe.bin`train() + 5269 at caffe.cpp:236
frame #13: 0x0000000100009038 caffe.bin`main(argc=2, argv=0x00007fff5fbffb50) + 600 at caffe.cpp:443
net 的下一层是layer,layer之间进行互联,layer本身分为很多种,用type标示出来,有全联接层、accuracy层等,那这些层内部如果实现?
在lmdb中数据是以key value的形势存储下来的,估计就是key len value len的形势。
那是如何读入到内存中,又怎样在在内存中使用的?如何确定label和train数据的对应关系,在训练的过程是是一个数据点一次计算吗?
考虑到读取数据必然调用lmdb的接口,设置断点
db_lmdb.cpp:12
得到如下的程序栈:
* thread #2: tid = 0xa26a38, 0x0000000100387310 libcaffe.so.1.0.0-rc3`caffe::db::LMDB::Open(this=0x0000000102303690, source="examples/mnist/mnist_train_lmdb", mode=READ) + 64 at db_lmdb.cpp:12, stop reason = breakpoint 1.1
* frame #0: 0x0000000100387310 libcaffe.so.1.0.0-rc3`caffe::db::LMDB::Open(this=0x0000000102303690, source="examples/mnist/mnist_train_lmdb", mode=READ) + 64 at db_lmdb.cpp:12
frame #1: 0x000000010017d79b libcaffe.so.1.0.0-rc3`caffe::DataReader::Body::InternalThreadEntry(this=0x000000010170a350) + 203 at data_reader.cpp:75
frame #2: 0x000000010019a3ee libcaffe.so.1.0.0-rc3`caffe::InternalThread::entry(this=0x000000010170a350, device=0, mode=CPU, rand_seed=447230961, solver_count=1, root_solver=true) + 94 at internal_thread.cpp:51
frame #3: 0x000000010019c233 libcaffe.so.1.0.0-rc3`boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>::operator(this=0x000000010170ab60, p=0x000000010170a350, a1=0, a2=CPU, a3=447230961, a4=1, a5=true)(caffe::InternalThread*, int, caffe::Caffe::Brew, int, int, bool) const + 163 at mem_fn_template.hpp:619
frame #4: 0x000000010019c161 libcaffe.so.1.0.0-rc3`void boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> >::operator(this=0x000000010170ab70, (null)=type<void> @ 0x0000700000080e48, f=0x000000010170ab60, a=0x0000700000080e70, (null)=0)<boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>&, boost::_bi::list0&, int) + 241 at bind.hpp:602
frame #5: 0x000000010019c06a libcaffe.so.1.0.0-rc3`boost::_bi::bind_t<void, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> > >::operator(this=0x000000010170ab60)() + 58 at bind_template.hpp:20
frame #6: 0x000000010019b6ac libcaffe.so.1.0.0-rc3`boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> > > >::run(this=0x000000010170a960) + 28 at thread.hpp:116
frame #7: 0x000000010109e804 libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 164
frame #8: 0x00007fff922e29b1 libsystem_pthread.dylib`_pthread_body + 131
frame #9: 0x00007fff922e292e libsystem_pthread.dylib`_pthread_start + 168
frame #10: 0x00007fff922e0385 libsystem_pthread.dylib`thread_start + 13
* thread #3: tid = 0xa29a6a, 0x0000000100387310 libcaffe.so.1.0.0-rc3`caffe::db::LMDB::Open(this=0x0000000102009eb0, source="examples/mnist/mnist_test_lmdb", mode=READ) + 64 at db_lmdb.cpp:12, stop reason = breakpoint 1.1
* frame #0: 0x0000000100387310 libcaffe.so.1.0.0-rc3`caffe::db::LMDB::Open(this=0x0000000102009eb0, source="examples/mnist/mnist_test_lmdb", mode=READ) + 64 at db_lmdb.cpp:12
frame #1: 0x000000010017d79b libcaffe.so.1.0.0-rc3`caffe::DataReader::Body::InternalThreadEntry(this=0x0000000101660900) + 203 at data_reader.cpp:75
frame #2: 0x000000010019a3ee libcaffe.so.1.0.0-rc3`caffe::InternalThread::entry(this=0x0000000101660900, device=0, mode=CPU, rand_seed=310595927, solver_count=1, root_solver=true) + 94 at internal_thread.cpp:51
frame #3: 0x000000010019c233 libcaffe.so.1.0.0-rc3`boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>::operator(this=0x0000000101661060, p=0x0000000101660900, a1=0, a2=CPU, a3=310595927, a4=1, a5=true)(caffe::InternalThread*, int, caffe::Caffe::Brew, int, int, bool) const + 163 at mem_fn_template.hpp:619
frame #4: 0x000000010019c161 libcaffe.so.1.0.0-rc3`void boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> >::operator(this=0x0000000101661070, (null)=type<void> @ 0x0000700000186e48, f=0x0000000101661060, a=0x0000700000186e70, (null)=0)<boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>&, boost::_bi::list0&, int) + 241 at bind.hpp:602
frame #5: 0x000000010019c06a libcaffe.so.1.0.0-rc3`boost::_bi::bind_t<void, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> > >::operator(this=0x0000000101661060)() + 58 at bind_template.hpp:20
frame #6: 0x000000010019b6ac libcaffe.so.1.0.0-rc3`boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> > > >::run(this=0x0000000101660e60) + 28 at thread.hpp:116
frame #7: 0x000000010109e804 libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 164
frame #8: 0x00007fff922e29b1 libsystem_pthread.dylib`_pthread_body + 131
frame #9: 0x00007fff922e292e libsystem_pthread.dylib`_pthread_start + 168
frame #10: 0x00007fff922e0385 libsystem_pthread.dylib`thread_start + 13
这两个线程的source不同,也就是每一个source会有一个单独的读线程,读取的数据放入到data_reader的new_queue_pairs_中,这是一个缓冲队列,多线程安全的。
那什么时候读这个队列呢?
读队列会在data_layer中进行,使用另外的一个线程,
设置断点,data_layer.cpp:58
则* thread #5: tid = 0xa3aae2, 0x00000001001f85a1 libcaffe.so.1.0.0-rc3`caffe::DataLayer<float>::load_batch(this=0x000000010300bc00, batch=0x000000010300c078) + 49 at data_layer.cpp:58, stop reason = breakpoint 2.1
* frame #0: 0x00000001001f85a1 libcaffe.so.1.0.0-rc3`caffe::DataLayer<float>::load_batch(this=0x000000010300bc00, batch=0x000000010300c078) + 49 at data_layer.cpp:58
frame #1: 0x00000001001d8715 libcaffe.so.1.0.0-rc3`caffe::BasePrefetchingDataLayer<float>::InternalThreadEntry(this=0x000000010300bc00) + 325 at base_data_layer.cpp:87
frame #2: 0x000000010019a3ee libcaffe.so.1.0.0-rc3`caffe::InternalThread::entry(this=0x000000010300bc00, device=0, mode=CPU, rand_seed=-1096880597, solver_count=1, root_solver=true) + 94 at internal_thread.cpp:51
frame #3: 0x000000010019c233 libcaffe.so.1.0.0-rc3`boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>::operator(this=0x0000000102015670, p=0x000000010300bc00, a1=0, a2=CPU, a3=-1096880597, a4=1, a5=true)(caffe::InternalThread*, int, caffe::Caffe::Brew, int, int, bool) const + 163 at mem_fn_template.hpp:619
frame #4: 0x000000010019c161 libcaffe.so.1.0.0-rc3`void boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> >::operator(this=0x0000000102015680, (null)=type<void> @ 0x0000700000209e48, f=0x0000000102015670, a=0x0000700000209e70, (null)=0)<boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>&, boost::_bi::list0&, int) + 241 at bind.hpp:602
frame #5: 0x000000010019c06a libcaffe.so.1.0.0-rc3`boost::_bi::bind_t<void, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> > >::operator(this=0x0000000102015670)() + 58 at bind_template.hpp:20
frame #6: 0x000000010019b6ac libcaffe.so.1.0.0-rc3`boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf5<void, caffe::InternalThread, int, caffe::Caffe::Brew, int, int, bool>, boost::_bi::list6<boost::_bi::value<caffe::InternalThread*>, boost::_bi::value<int>, boost::_bi::value<caffe::Caffe::Brew>, boost::_bi::value<int>, boost::_bi::value<int>, boost::_bi::value<bool> > > >::run(this=0x0000000102015470) + 28 at thread.hpp:116
frame #7: 0x000000010109e804 libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 164
frame #8: 0x00007fff922e29b1 libsystem_pthread.dylib`_pthread_body + 131
frame #9: 0x00007fff922e292e libsystem_pthread.dylib`_pthread_start + 168
frame #10: 0x00007fff922e0385 libsystem_pthread.dylib`thread_start + 13
Forward_cpu会将的数据从prefetch_full_中copy到top中,则data layer的数据往上进行了传递
调用堆栈如下:
* thread #1: tid = 0xa2a208, 0x00000001001d8237 libcaffe.so.1.0.0-rc3`caffe::BasePrefetchingDataLayer<float>::Forward_cpu(this=0x000000010300bc00, bottom=0x000000010200b520, top=0x000000010200ab60) + 39 at base_data_layer.cpp:109, queue = 'com.apple.main-thread', stop reason = breakpoint 3.1
* frame #0: 0x00000001001d8237 libcaffe.so.1.0.0-rc3`caffe::BasePrefetchingDataLayer<float>::Forward_cpu(this=0x000000010300bc00, bottom=0x000000010200b520, top=0x000000010200ab60) + 39 at base_data_layer.cpp:109
frame #1: 0x000000010019ee77 libcaffe.so.1.0.0-rc3`caffe::Layer<float>::Forward(this=0x000000010300bc00, bottom=0x000000010200b520, top=0x000000010200ab60) + 199 at layer.hpp:459
frame #2: 0x00000001002f1d9c libcaffe.so.1.0.0-rc3`caffe::Net<float>::ForwardFromTo(this=0x00000001020080d0, start=0, end=5) + 636 at net.cpp:550
frame #3: 0x00000001002f1ac2 libcaffe.so.1.0.0-rc3`caffe::Net<float>::Forward(this=0x00000001020080d0, loss=0x00007fff5fbfdf78) + 82 at net.cpp:570
frame #4: 0x00000001003381be libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Test(this=0x0000000102002060, test_net_id=0) + 2286 at solver.cpp:363
frame #5: 0x0000000100336bec libcaffe.so.1.0.0-rc3`caffe::Solver<float>::TestAll(this=0x0000000102002060) + 124 at solver.cpp:330
frame #6: 0x00000001003354f2 libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Step(this=0x0000000102002060, iters=10000) + 674 at solver.cpp:207
frame #7: 0x0000000100334a31 libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Solve(this=0x0000000102002060, resume_file=0x0000000000000000) + 881 at solver.cpp:293
frame #8: 0x0000000100004df8 caffe.bin`train() + 6840 at caffe.cpp:252
frame #9: 0x0000000100009038 caffe.bin`main(argc=2, argv=0x00007fff5fbffb08) + 600 at caffe.cpp:443
frame #10: 0x00007fff8d0585ad libdyld.dylib`start + 1
在最简单的模型中,模型描述如下:
name: "LeNet" layer { name: "mnist" type: "Data" top: "data" top: "label" include { phase: TRAIN } transform_param { scale: 0.00390625 } data_param { source: "examples/mnist/mnist_train_lmdb" batch_size: 64 backend: LMDB } } layer { name: "mnist" type: "Data" top: "data" top: "label" include { phase: TEST } transform_param { scale: 0.00390625 } data_param { source: "examples/mnist/mnist_test_lmdb" batch_size: 100 backend: LMDB } } layer { name: "ip" type: "InnerProduct" bottom: "data" top: "ip" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 10 weight_filler { type: "xavier" } bias_filler { type: "constant" } } } layer { name: "accuracy" type: "Accuracy" bottom: "ip" bottom: "label" top: "accuracy" include { phase: TEST } } layer { name: "loss" type: "SoftmaxWithLoss" bottom: "ip" bottom: "label" top: "loss" } |
其中只有data ip和softmaxwithloss三个层次,不同层之间通过名字bottom top的方式关联到一起,bottom和top可以有多个,也就形成了网状结构。
data中,batch_size指定了一次计算的数据的大小。
在ip中,num_output标示模型输出的个数,这个地方为10,表示的是输出维度为10,这是和mnist中label的数量为10对应的。
在网络完成初始化之后,开始进行训练,训练的第一步是前向计算,会从网络的第0层开始调用forword_cpu,逐层将数据从底层向上层传递,直到计算出loss。
在上面的例子中,就是void SoftmaxWithLossLayer<Dtype>::Forward_cpu
函数栈如下:
* thread #1: tid = 0xdba216, 0x00000001002abc43 libcaffe.so.1.0.0-rc3`caffe::SoftmaxLayer<float>::Forward_cpu(this=0x0000000102304080, bottom=0x0000000101717fd8, top=0x0000000101717ff0) + 1043 at softmax_layer.cpp:55, queue = 'com.apple.main-thread', stop reason = breakpoint 14.1
* frame #0: 0x00000001002abc43 libcaffe.so.1.0.0-rc3`caffe::SoftmaxLayer<float>::Forward_cpu(this=0x0000000102304080, bottom=0x0000000101717fd8, top=0x0000000101717ff0) + 1043 at softmax_layer.cpp:55
frame #1: 0x000000010019ee77 libcaffe.so.1.0.0-rc3`caffe::Layer<float>::Forward(this=0x0000000102304080, bottom=0x0000000101717fd8, top=0x0000000101717ff0) + 199 at layer.hpp:459
frame #2: 0x00000001002ae1c5 libcaffe.so.1.0.0-rc3`caffe::SoftmaxWithLossLayer<float>::Forward_cpu(this=0x0000000101717cb0, bottom=0x00000001017009d0, top=0x0000000101615070) + 117 at softmax_loss_layer.cpp:92
frame #3: 0x000000010019ee77 libcaffe.so.1.0.0-rc3`caffe::Layer<float>::Forward(this=0x0000000101717cb0, bottom=0x00000001017009d0, top=0x0000000101615070) + 199 at layer.hpp:459
frame #4: 0x00000001002f1d9c libcaffe.so.1.0.0-rc3`caffe::Net<float>::ForwardFromTo(this=0x0000000101625a50, start=0, end=2) + 636 at net.cpp:550
frame #5: 0x00000001002f1ac2 libcaffe.so.1.0.0-rc3`caffe::Net<float>::Forward(this=0x0000000101625a50, loss=0x00007fff5fbfe584) + 82 at net.cpp:570
frame #6: 0x00000001002f3d3d libcaffe.so.1.0.0-rc3`caffe::Net<float>::ForwardBackward(this=0x0000000101625a50) + 29 at net.hpp:88
frame #7: 0x00000001003356eb libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Step(this=0x00000001017003f0, iters=10000) + 1179 at solver.cpp:222
frame #8: 0x0000000100334a31 libcaffe.so.1.0.0-rc3`caffe::Solver<float>::Solve(this=0x00000001017003f0, resume_file=0x0000000000000000) + 881 at solver.cpp:293
frame #9: 0x0000000100004df8 caffe.bin`train() + 6840 at caffe.cpp:252
frame #10: 0x0000000100009038 caffe.bin`main(argc=2, argv=0x00007fff5fbffb08) + 600 at caffe.cpp:443
frame #11: 0x00007fff8d0585ad libdyld.dylib`start + 1