1. 程式人生 > >caffe安裝教程2

caffe安裝教程2


0.安裝一些基本依賴項:
$ sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler  
$ sudo apt-get install --no-install-recommends libboost-all-dev
$ sudo apt-get install libopenblas-dev liblapack-dev libatlas-base-dev
$ sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev


一、安裝必要的依賴庫
(1)安裝 matio:
安裝方法1:
sudo apt-get install libmatio-dev

正在讀取軟體包列表... 完成
正在分析軟體包的依賴關係樹       
正在讀取狀態資訊... 完成       
將會安裝下列額外的軟體包:
  libmatio2
下列【新】軟體包將被安裝:
  libmatio-dev libmatio2
升級了 0 個軟體包,新安裝了 2 個軟體包,要解除安裝 0 個軟體包,有 2 個軟體包未被升級。
需要下載 151 kB 的軟體包。
解壓縮後會消耗掉 691 kB 的額外空間。
您希望繼續執行嗎? [Y/n] y
獲取:1 http://cn.archive.ubuntu.com/ubuntu/ trusty/universe libmatio2 amd64 1.5.2-1ubuntu1 [76.8 kB]
獲取:2 http://cn.archive.ubuntu.com/ubuntu/ trusty/universe libmatio-dev amd64 1.5.2-1ubuntu1 [73.7 kB]
下載 151 kB,耗時 1秒 (107 kB/s)       
正在選中未選擇的軟體包 libmatio2:amd64。
(正在讀取資料庫 ... 系統當前共安裝有 258518 個檔案和目錄。)
正準備解包 .../libmatio2_1.5.2-1ubuntu1_amd64.deb  ...
正在解包 libmatio2:amd64 (1.5.2-1ubuntu1) ...
正在選中未選擇的軟體包 libmatio-dev:amd64。
正準備解包 .../libmatio-dev_1.5.2-1ubuntu1_amd64.deb  ...
正在解包 libmatio-dev:amd64 (1.5.2-1ubuntu1) ...
正在設定 libmatio2:amd64 (1.5.2-1ubuntu1) ...
正在設定 libmatio-dev:amd64 (1.5.2-1ubuntu1) ...
正在處理用於 libc-bin (2.19-0ubuntu6.9) 的觸發器 ...
/sbin/ldconfig.real: /usr/lib/nvidia-375/libEGL.so.1 is not a symbolic link

/sbin/ldconfig.real: /usr/lib32/nvidia-375/libEGL.so.1 is not a symbolic link

##########################################################################

安裝方法2:
下載matio(https://sourceforge.net/projects/matio/files/matio/1.5.2/)
tar zxf matio-1.5.2.tar.gz
cd matio-1.5.2
./configure
make
make check
make install
sudo ldconfig

(2)安裝 wget
sudo pip install wget 


[email protected]:~$ sudo pip install wget 
Collecting wget
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading wget-3.2.zip
Building wheels for collected packages: wget
  Running setup.py bdist_wheel for wget ... done
  Stored in directory: /home/gjh/.cache/pip/wheels/6d/98/29/61ccc41148f871009126c2e844e26f73eeb25e12cca92228a5
Successfully built wget
Installing collected packages: wget
Successfully installed wget-3.2

若出錯了則按照下面的命令成功:
pip install --upgrade pip --user
pip install --upgrade setuptools --user
sudo pip install wget


二、下載Deeplabv2並編譯
1、下載程式碼:
git clone https://github.com/xmojiao/deeplab_v2.git
(試過許多Deeplab程式碼,這個最容易編譯成功,所以我用的是這個程式碼編譯的)。
2、對 caffe 進行編譯:
修改deeplab_v2/deeplab-public-ver2/路徑下的 Makefile.config.example檔案,重新命名為Makefile.config,接著修改這個檔案中的內容,將第四行的 “# USE_CUDNN := 1”的 # 去掉。如果需要,因為我用的pycaffe編譯,所以不需要修改python的路徑,儲存退出。
下面為編譯 caffe的命令:
cd ~/Desktop/deeplab_v2/deeplab-public-ver2
make clean 
make all -j8
make pycaffe
make pytest 

錯誤問題1:執行make pycaffe 時出現 ImportError: No module named google.protobuf.internal
解決方案: sudo  pip install protobuf

[email protected]:~/cell_images/deeplab-v2-master$ pip install protobuf
The directory '/home/gjh/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/gjh/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting protobuf
  Downloading protobuf-3.5.0.post1-cp27-cp27mu-manylinux1_x86_64.whl (6.4MB)
    100% |████████████████████████████████| 6.4MB 242kB/s 
Requirement already satisfied: six>=1.9 in /home/gjh/miniconda2/lib/python2.7/site-packages (from protobuf)
Requirement already satisfied: setuptools in /home/gjh/miniconda2/lib/python2.7/site-packages (from protobuf)
Installing collected packages: protobuf
Successfully installed protobuf-3.5.0.post1

錯誤問題2:執行make pytest 時出現:
......
I1222 21:38:29.254962 26335 net.cpp:100] Creating Layer conv
I1222 21:38:29.254967 26335 net.cpp:434] conv <- data
I1222 21:38:29.254973 26335 net.cpp:408] conv -> conv
F1222 21:38:31.349524 26335 cudnn.hpp:113] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0)  CUDNN_STATUS_BAD_PARAM
*** Check failure stack trace: ***
Aborted (core dumped)
make: *** [pytest] 錯誤 134
解決方案:選擇“# USE_CUDNN := 1”去編譯caffe (網上說是因為記憶體不足等原因,所以我就直接不用CUDNN了,)
cd ~/Desktop/deeplab_v2/deeplab-public-ver2
make clean 
make all -j8
make pycaffe
make pytest 
執行無誤,結束!


三:依次建立存放設定資料夾,預測結果資料夾,資料集txt資料夾,log資料夾,model資料夾,evaluation資料夾
mkdir -p ~/home/gjh/cell_images/exper/voc12/config/deeplab_largeFOV
mkdir  -p ~/home/gjh/cell_images/exper/voc12/features/labels
mkdir -p ~/home/gjh/cell_images/exper/voc12/features2/labels
mkdir -p ~/home/gjh/cell_images/exper/voc12/list
mkdir -p ~/home/gjh/cell_images/exper/voc12/log 
mkdir -p ~/home/gjh/cell_images/exper/voc12/model/deeplab_largeFOV
mkdir -p ~/home/gjh/cell_images/exper/voc12/res
(2)資料轉換
因為pascal voc2012增強資料集的label是mat格式的檔案,要把mat格式的label轉為png格式的圖片. 
cd ~/cell_images/exper/voc2012
mkdir cls_png 

[email protected]:~/cell_images/exper/voc2012$ ./mat2png.py /home/gjh/cell_images/cls /home/gjh/cell_images/cls_png
因為pascal voc2012原始資料集的label為三通道RGB影象,但是caffe最後一層softmax loss
層只能識別一通道的label,所以此處我們需要對原始資料集的label進行降維 
cd ~~/cell_images/exper/voc2012
mkdir SegmentationClass_1D
./convert_labels.py /home/gjh/cell_images/SegmentationClass/   /home/gjh/cell_images/trainval.txt   /home/gjh/cell_images/SegmentationClass_1D/

(3)資料融合
此時我們已經處理好好pascal voc2012 增強資料集和pascal voc2012的原始資料集,為了便於train.txt等檔案的呼叫,我們需要將兩個資料夾資料合併到同一個檔案中.目前已有資料檔案如下:

   1. ~/DL_dataset/VOC2012_orig 為原始pascal voc2012資料夾
        images資料集的檔名為:JPEGImages
        labels資料集檔名為:SegmentationClass_1D

    2. ~/DL_dataset/VOC_aug/dataset為pascal voc2012增強資料集資料夾
        images資料集的檔名為:img
        ,jpg圖片數為5073
        labels資料集檔名為:cls_png,png圖片數11355

    現分別pascal voc2012增強資料集裡的images和labels複製到增強資料集中,若重複則覆蓋,合將並資料集的操作如下:

cp /media/gjh/文件/VOC2007-2012資料集/VOC2012/VOCdevkit/VOC2012/SegmentationClass_1D/* /media/gjh/文件/VOC2007-2012資料集/PASCAL_VOC2012/PASCAL_VOC2012/cls_png

cp /media/gjh/文件/VOC2007-2012資料集/VOC2012/VOCdevkit/VOC2012/JPEGImages/* /media/gjh/文件/VOC2007-2012資料集/PASCAL_VOC2012/PASCAL_VOC2012/img/

(4)檔名修改
對應train.txt檔案的資料集檔名,修改檔名。

cd ~/DL_dataset/VOC_aug/dataset
mv ./img ./JPEGImages
mv ./cls_png ./SegmentationClassAug

到此處, ~/DL_dataset/VOC_aug/dataset資料夾中
- images資料集的檔名為:JPEGImages ,jpg圖片數由5073變為17125
- labels資料集檔名為:cls_png ,png圖片數由11355變為12031

(5)修改並執行 run_pascal.sh
進入/Desktop/deeplab_v2/voc2012,修改 run_pascal.sh 檔案,主要是修改路徑,我的修改後的檔案如下:

  #!/bin/sh

## MODIFY PATH for YOUR SETTING
ROOT_DIR=/home/mmt/Desktop/my_dataset

CAFFE_DIR=/home/mmt/Desktop/deeplab_v2/deeplab-public-ver2
CAFFE_BIN=${CAFFE_DIR}/build/tools/caffe.bin
........

## Run

RUN_TRAIN=1   #1時train
RUN_TEST=0    #1時test
RUN_TRAIN2=0
RUN_TEST2=0

........


接下來執行程式碼:
Train:
~/Desktop/deeplab_v2/voc2012
sh ./run_pascal.sh 


Test:
將相應變數改為1:
~/Desktop/deeplab_v2/voc2012
sh ./run_pascal_test.sh
執行結果如下:

因為結果儲存的是mat檔案,如果想轉換成png的形式,執行:
cd ~/Desktop/deeplab_v2/voc2012
修改create_labels_21.py的路徑,然後此目錄執行:
python create_labels_21.py
因為訓練一會,我就暫停了,所以test的結果不好,而且影象的分割後的尺寸變了,不知道什麼原因,不過經過densecrf後會變回來。


(6)修改並執行 run_densecrf.sh
1)首先對densecrf進行編譯。
cd ~/Desktop/deeplab_v2/deeplab-public-ver2/densecrf/
make
有許多warning,但是沒出錯,不用管。
2)資料整理
因為densecrf只識別ppm格式的影象,所以要轉換格式。進入/Desktop/deeplab_v2/deeplab-public-ver2/densecrf/my_script,裡面有自帶的修改ppm 的MATLAB程式,修改路徑,直接執行即可。 
程式碼如下:
############################################################################################# 
% save jpg images as bin file for cpp
%
is_server = 1;

dataset = 'voc2012';  %'coco', 'voc2012'

if is_server
  if strcmp(dataset, 'voc2012')
    img_folder  = '/media/gjh/文件/VOC2007-2012datasaet/dataset/JPEGImages';
    save_folder = '/media/gjh/文件/VOC2007-2012datasaet/dataset/PPMImages';
  elseif strcmp(dataset, 'coco')
    img_folder  = '/rmt/data/coco/JPEGImages';
    save_folder = '/rmt/data/coco/PPMImages';
  end
else
  img_folder = '../img';
  save_folder = '../img_ppm';
end

if ~exist(save_folder, 'dir')
    mkdir(save_folder);
end

img_dir = dir(fullfile(img_folder, '*.jpg'));

for i = 1 : numel(img_dir)
%     fprintf(1, 'processing %d (%d)...\n', i, numel(img_dir));
    img = imread(fullfile(img_folder, img_dir(i).name));
    
    img_fn = img_dir(i).name(1:end-4);
    save_fn = fullfile(save_folder, [img_fn, '.ppm']);
    
    imwrite(img, save_fn);   
end
############################################################################################# 


2)接下來,修改 run_densecrf.sh, 注意把 MODEL_NAME=deeplab_largeFOV修改了,原檔案少了一個 p。
#############################################################################################

#!/bin/bash 

###########################################
# You can either use this script to generate the DenseCRF post-processed results
# or use the densecrf_layer (wrapper) in Caffe
###########################################
DATASET=voc2012
LOAD_MAT_FILE=1

MODEL_NAME=deeplab_largeFOV

TEST_SET=val           #val, test

# the features  folder save the features computed via the model trained with the train set
# the features2 folder save the features computed via the model trained with the trainval set
FEATURE_NAME=features #features, features2
FEATURE_TYPE=fc8

# specify the parameters
MAX_ITER=10

Bi_W=4
Bi_X_STD=49
Bi_Y_STD=49
Bi_R_STD=5
Bi_G_STD=5 
Bi_B_STD=5

POS_W=3
POS_X_STD=3
POS_Y_STD=3


#######################################
# MODIFY THE PATY FOR YOUR SETTING
#######################################
SAVE_DIR=/media/gjh/文件/VOC2007-2012datasaet/dataset/${DATASET}/res/${FEATURE_NAME}/${MODEL_NAME}/${TEST_SET}/${FEATURE_TYPE}/post_densecrf_W${Bi_W}_XStd${Bi_X_STD}_RStd${Bi_R_STD}_PosW${POS_W}_PosXStd${POS_X_STD}

echo "SAVE TO ${SAVE_DIR}"

CRF_DIR=/home/gjh/cell_images/deeplab-v2-master/densecrf
IMG_DIR_NAME=dataset

# if [ ${DATASET} == "voc12" ]
# then
    # IMG_DIR_NAME=pascal/VOCdevkit/VOC2012
# elif [ ${DATASET} == "coco" ]
# then
    # IMG_DIR_NAME=coco
# elif [ ${DATASET} == "voc10_part" ]
# then
   # IMG_DIR_NAME=pascal/VOCdevkit/VOC2012
# fi

# NOTE THAT the densecrf code only loads ppm images
IMG_DIR=/media/gjh/文件/VOC2007-2012datasaet/${IMG_DIR_NAME}/PPMImages

# if [ ${LOAD_MAT_FILE} == 1 ]
# then
    # the features are saved in .mat format
    CRF_BIN=${CRF_DIR}/prog_refine_pascal_v4
    FEATURE_DIR=/media/gjh/文件/VOC2007-2012datasaet/dataset/${DATASET}/${FEATURE_NAME}/${MODEL_NAME}/${TEST_SET}/${FEATURE_TYPE}
# else
    # the features are saved in .bin format (has called SaveMatAsBin.m in the densecrf/my_script)
    # CRF_BIN=${CRF_DIR}/prog_refine_pascal
    # FEATURE_DIR=/rmt/work/deeplab/exper/${DATASET}/${FEATURE_NAME}/${MODEL_NAME}/${TEST_SET}/${FEATURE_TYPE}/bin
# fi

mkdir -p ${SAVE_DIR}

# run the program
${CRF_BIN} -id ${IMG_DIR} -fd ${FEATURE_DIR} -sd ${SAVE_DIR} -i ${MAX_ITER} -px ${POS_X_STD} -py ${POS_Y_STD} -pw ${POS_W} -bx ${Bi_X_STD} -by ${Bi_Y_STD} -br ${Bi_R_STD} -bg ${Bi_G_STD} -bb ${Bi_B_STD} -bw ${Bi_W}

##################################################################################################################################
執行結果:
[email protected]:~/cell_images/deeplab-v2-master/voc2012$ sh ./run_densecrf.sh
SAVE TO /media/gjh/文件/VOC2007-2012datasaet/dataset/voc2012/res/features/deeplab_largeFOV/val/fc8/post_densecrf_W4_XStd49_RStd5_PosW3_PosXStd3
Input Parameters: 
ImgDir:           /media/gjh/文件/VOC2007-2012datasaet/dataset/PPMImages
FeatureDir:       /media/gjh/文件/VOC2007-2012datasaet/dataset/voc2012/features/deeplab_largeFOV/val/fc8
SaveDir:          /media/gjh/文件/VOC2007-2012datasaet/dataset/voc2012/res/features/deeplab_largeFOV/val/fc8/post_densecrf_W4_XStd49_RStd5_PosW3_PosXStd3
MaxIterations:    10
PosW:      3
PosXStd:   3
PosYStd:   3
Bi_W:      4
Bi_X_Std:  49
Bi_Y_Std:  49
Bi_R_Std:  5
Bi_G_Std:  5
Bi_B_Std:  5


3)然後執行 /home/mmt/crf/deeplab-public-ver2/densecrf/my_script/GetDenseCRFResult.m把bin生成圖片格式
注意修改檔案路徑(GetDenseCRFResult.m,SetupEnv在/deeplab_v2/deeplab-public-ver2/matlab/my_script中),
兩個程式的程式碼如下:
##################################################################################################################################
% compute the densecrf result (.bin) to png
%

addpath('/home/gjh/cell_images/deeplab-v2-master/matlab/my_script');
% SetupEnv在/home/gjh/cell_images/deeplab-v2-master/matlab/my_script中
SetupEnv;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% You do not need to change values below
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
if is_server
  if learn_crf
    post_folder = sprintf('post_densecrf_W%d_XStd%d_RStd%d_PosW%d_PosXStd%d_ModelType%d_Epoch%d', bi_w, bi_x_std, bi_r_std, pos_w, pos_x_std, model_type, epoch);

    map_folder = fullfile('/media/gjh/文件/VOC2007-2012datasaet/dataset', dataset, 'densecrf', 'res', feature_name, model_name, testset, feature_type, post_folder); 

    save_root_folder = fullfile('/media/gjh/文件/VOC2007-2012datasaet/dataset', dataset, 'res', feature_name, model_name, testset, feature_type, post_folder); ;
  else
    post_folder = sprintf('post_densecrf_W%d_XStd%d_RStd%d_PosW%d_PosXStd%d', bi_w, bi_x_std, bi_r_std, pos_w, pos_x_std);

    map_folder = fullfile('/media/gjh/文件/VOC2007-2012datasaet/dataset', dataset, 'res', feature_name, model_name, testset, feature_type, post_folder); 

    save_root_folder = map_folder;
  end
else 
  map_folder = '../result';
end

map_dir = dir(fullfile(map_folder, '*.bin'));


fprintf(1,' saving to %s\n', save_root_folder);

if strcmp(dataset, 'voc2012')
  seg_res_dir = [save_root_folder '/results/VOC2012/'];
elseif strcmp(dataset, 'coco')
  seg_res_dir = [save_root_folder, '/results/COCO2014/'];
else
  error('Wrong dataset!');
end

save_result_folder = fullfile(seg_res_dir, 'Segmentation', [id '_' testset '_cls']);

if ~exist(save_result_folder, 'dir')
    mkdir(save_result_folder);
end

for i = 1 : numel(map_dir)
    fprintf(1, 'processing %d (%d)...\n', i, numel(map_dir));
    map = LoadBinFile(fullfile(map_folder, map_dir(i).name), 'int16');

    img_fn = map_dir(i).name(1:end-4);
    imwrite(uint8(map), colormap, fullfile(save_result_folder, [img_fn, '.png']));
end
##################################################################################################################################
##################################################################################################################################
SetupEnv.m
##################################################################################################################################
% set up the environment variables
%

clear all; close all;
load('./pascal_seg_colormap.mat');

is_server       = 1;

crf_load_mat    = 1;   % the densecrf code load MAT files directly (no call SaveMatAsBin.m)
                       % used ONLY by DownSampleFeature.m
learn_crf       = 0;   % NOT USED. Set to 0

is_mat          = 1;   % the results to be evaluated are saved as mat (1) or png (0)
has_postprocess = 0;   % has done densecrf post processing (1) or not (0)
is_argmax       = 0;   % the output has been taken argmax already (e.g., coco dataset). 
                       % assume the argmax takes C-convention (i.e., start from 0)

debug           = 0;   % if debug, show some results

% vgg128_noup (not optimized well), aka DeepLab
% bi_w = 5, bi_x_std = 50, bi_r_std = 10

% vgg128_ms_pool3, aka DeepLab-MSc
% bi_w = 3, bi_x_std = 95, bi_r_std = 3

% vgg128_noup_pool3_cocomix, aka DeepLab-COCO
% bi_w = 5, bi_x_std = 67, bi_r_std = 3

%% these are used for the bounding box weak annotation experiments (i.e., to generate the Bbox-Seg)
% erode_gt (bbox)
% bi_w = 41, bi_x_std = 33, bi_r_std = 4

% erode_gt/bboxErode20
% bi_w = 45, bi_x_std = 37, bi_r_std = 3, pos_w = 15, pos_x_std = 3
 

%
% initial or default values for crf
% these parameter should be the same with the document 'run_densecrf.sh'
bi_w           = 4; 
bi_x_std       = 49;
bi_r_std       = 5;

pos_w          = 3;
pos_x_std      = 3;


%
dataset    = 'voc2012';  %'voc2012', 'coco' % 修改
trainset   = 'train_aug';      % not used
testset    = 'val';            %'val', 'test'

model_name = 'deeplab_largeFOV'; % 修改

feature_name = 'features';
feature_type = 'fc8'; % fc8 / crf

id           = 'comp6';

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% used for cross-validation
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
rng(10)

% downsampling files for cross-validation
down_sample_method = 2;      % 1: equally sample with "down_sample_rate", 2: randomly pick "num_sample" samples
down_sample_rate   = 8;
num_sample         = 100;    % number of samples used for cross-validation

% ranges for cross-validation
range_pos_w = [3];
range_pos_x_std = [3];

range_bi_w = [5];
range_bi_x_std = [49];
range_bi_r_std = [4 5];
##################################################################################################################################
4)至此,deeplabv2 程式已除錯完。