Caffe Python MemoryDataLayer Segmentation Fault
阿新 • • 發佈:2018-12-30
因為利用Pyhon來做資料的預處理比較方便,因此在data_layer選擇上,採用了MemoryDataLayer,可以比較方便的直接用Python 根據set_input_array進行feed資料,然後再呼叫solver進行step就可以了。說一下我碰到的問題,當時檢查了一下感覺沒有哪裡出錯,但是報
Segmentation Fault(Core Abort)
感覺好囧,最怕這個了。一般說段錯誤都是記憶體錯誤,比如陣列越界,無效的指標,引用被釋放的資源等等。經過一步步debug之後發現問題出現在
solver.net.set_input_arrays
solver在將資料傳送到網路低端的時候報錯。那麼接下來找到python
caffe\python\caffe\_caffe.cpp
檔案,這個檔案是基於boost python的,用來將C++的介面匯出,供python呼叫。進一步我們找到相關函式
void Net_SetInputArrays(Net<Dtype>* net, bp::object data_obj, bp::object labels_obj) { // check that this network has an input MemoryDataLayer shared_ptr<MemoryDataLayer<Dtype> > md_layer = boost::dynamic_pointer_cast<MemoryDataLayer<Dtype> >(net->layers()[0]); if (!md_layer) { throw std::runtime_error("set_input_arrays may only be called if the" " first layer is a MemoryDataLayer"); } // check that we were passed appropriately-sized contiguous memory PyArrayObject* data_arr = reinterpret_cast<PyArrayObject*>(data_obj.ptr()); PyArrayObject* labels_arr = reinterpret_cast<PyArrayObject*>(labels_obj.ptr()); CheckContiguousArray(data_arr, "data array", md_layer->channels(), md_layer->height(), md_layer->width()); CheckContiguousArray(labels_arr, "labels array", 1, 1, 1); if (PyArray_DIMS(data_arr)[0] != PyArray_DIMS(labels_arr)[0]) { throw std::runtime_error("data and labels must have the same first" " dimension"); } if (PyArray_DIMS(data_arr)[0] % md_layer->batch_size() != 0) { throw std::runtime_error("first dimensions of input arrays must be a" " multiple of batch size"); } md_layer->Reset(static_cast<Dtype*>(PyArray_DATA(data_arr)), static_cast<Dtype*>(PyArray_DATA(labels_arr)), PyArray_DIMS(data_arr)[0]); }
問題就出在了最後的一個語句
md_layer->Reset(static_cast<Dtype*>(PyArray_DATA(data_arr)),
static_cast<Dtype*>(PyArray_DATA(labels_arr)),
PyArray_DIMS(data_arr)[0]);
當執行reset MemoryDataLayer的Reset函式時出錯。於此同時在github上也發現了同樣的問題,https://github.com/BVLC/caffe/issues/2334也是因為Python MemoryDataLayer引發的段錯誤。他說到,在裡面把傳入的data和label做要給深拷貝就可以解決,估計是執行時資料已經被釋放了,只傳了指標引發了段錯誤。
解決方案:
找到caffe\src\layers\memory_data_layer.cpp
開啟,給Reset函式新增相應的深拷貝程式碼
template <typename Dtype>
void MemoryDataLayer<Dtype>::Reset(Dtype* data, Dtype* labels, int n) {
CHECK(data);
CHECK(labels);
CHECK_EQ(n % batch_size_, 0) << "n must be a multiple of batch size";
// Warn with transformation parameters since a memory array is meant to
// be generic and no transformations are done with Reset().
if (this->layer_param_.has_transform_param()) {
LOG(WARNING) << this->type() << " does not transform array data on Reset()";
}
// data_ = data; 將這裡註釋掉,
// labels_ = labels;
//以下部分是進行深拷貝
if(data_)
delete []data_;
if(labels_)
delete [] labels_;
data_ = new Dtype[n*size_];
labels_ = new Dtype[n * num_tasks_];
memcpy(data_, data, sizeof(Dtype)*n*size_);
memcpy(labels_, labels, sizeof(Dtype) * n * num_tasks_);
n_ = n;
pos_ = 0;
}
Ok進行修改之後,回到Caffe的根目錄,執行make all
,make test
,`make runtest
,make pycaffe
。重新編譯完成之後,重新執行就好了,繼續開始訓練。