ubuntu16.04 + cuda10.1 + opencv-3.3.0 + caffe

95 阅读 0 评论 63 点赞

我是靠谱客的博主沉默胡萝卜，这篇文章主要介绍ubuntu16.04 + cuda10.1 + opencv-3.3.0 + caffe，现在分享给大家，希望可以做个参考。

1.系统环境

复制代码

1
2
3
4
5
6
系统：ubuntu16.04
GPU驱动：nvidia-driver-418-server
CUDA版本：cuda10.1
CUDNN版本：cudnn7.6.4
Anaconda版本：Anaconda5.2 （python3.6）
opencv版本：opencv-3.3.0

2.安装caffe

opencv-3.3.0的编译可以参考我的：

https://blog.csdn.net/lu_linux/article/details/117187172

环境依赖

复制代码

1
2
3
4
5
6
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libopenblas-dev liblapack-dev libatlas-base-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install git cmake build-essential

clone caffe源码

复制代码

1
2
3
git clone https://github.com/BVLC/caffe.git
cd caffe
cp Makefile.config.example Makefile.config

修改 Makefile.config 文件内容：

1.应用 cudnn

复制代码

1
2
3
4
将
#USE_CUDNN := 1
修改成： 
USE_CUDNN := 1

2.应用 opencv 版本

复制代码

1
2
3
4
5
6
7
8
将
#USE_OPENCV := 0
修改为： 
USE_OPENCV := 1
将
#OPENCV_VERSION := 3 
修改为： 
OPENCV_VERSION := 3

3.使用 python 接口

复制代码

1
2
3
4
将
#WITH_PYTHON_LAYER := 1 
修改为 
WITH_PYTHON_LAYER := 1

4.修改 python 路径

复制代码

1
2
3
4
5
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib 
修改为： 
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

5.修改Python版本

复制代码

1
2
3
4
5
# 推荐使用python3.6的版本
# 把python2.7代码加注释，python3.5前的注释去掉改成
PYTHON_LIBRARIES := boost_python3 python3.6m
PYTHON_INCLUDE := /usr/local/include/python3.6m                  
                  /usr/local/lib/python3.6/site-packages/numpy/core/include

6.修改cuda版本不兼容问题

复制代码

1
2
3
4
5
6
7
8
9
10
11
# 如果cuda版本大于等于9.0，需要注释掉前两行，如下
# -gencode arch=compute_20,code=sm_20 
# -gencode arch=compute_20,code=sm_21 
CUDA_ARCH := -gencode arch=compute_30,code=sm_30             
-gencode arch=compute_35,code=sm_35               
-gencode arch=compute_50,code=sm_50               
-gencode arch=compute_52,code=sm_52               
-gencode arch=compute_60,code=sm_60               
-gencode arch=compute_61,code=sm_61               
-gencode arch=compute_61,code=compute_61

然后修改 caffe 目录下的 Makefile 文件：

复制代码

1
2
3
4
5
将：
NVCCFLAGS +=-ccbin=$(CXX) -Xcompiler-fPIC $(COMMON_FLAGS)
替换为：
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

复制代码

1
2
3
4
5
将：
LIBRARIES += glog gflags protobuf boost_system boost_filesystem m
改为：
LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial

然后修改 host_config.h 文件 :

复制代码

1
2
3
4
5
6
7
8
9
10
root@computer:/home/share/caffe# grep -rn "unsupported GNU" /usr/local/cuda/include/
/usr/local/cuda/include/crt/host_config.h:129:#error -- unsupported GNU version! gcc versions later than 8 are not supported!
root@computer:/home/share/caffe#
从上面日志可以看出我的cuda10.1已经支持到gnu 8了，我本地的gnu版本是7，故不需要修改
如果cuda版本支持的gnu版本小于当前系统gnu版本则需要注释掉，如果日志如下（cuda支持的为4.9）就需要注释掉：
将
#error-- unsupported GNU version! gcc versions later than 4.9 are not supported!
改为
//#error-- unsupported GNU version! gcc versions later than 4.9 are not supported!

开始编译

复制代码

1
make all -j8

报错：

/usr/bin/ld: cannot find -lboost_python3

找不到boost_python3造成的，需要自己编译一个

可以参考：https://blog.csdn.net/u012505617/article/details/88556621

复制代码

1
2
3
4
5
6
7
8
9
10
# 下载boost_1_67_0.tar.gz
wget http://sourceforge.net/projects/boost/files/boost/1.67.0/boost_1_67_0.tar.gz
# 解压文件包
tar -zxvf boost_1_67_0.tar.gz
# 进入文件夹
cd boost_1_67_0/
# 生成 .so 文件
./bootstrap.sh --with-libraries=python --with-toolset=gcc
./b2 --with-python include="/usr/local/include/python3.6m"

创建软连接

复制代码

root@computer:/home/share/boost_1_67_0# cd stage/lib/
root@computer:/home/share/boost_1_67_0/stage/lib# ll
total 1052
drwxr-xr-x 2 root root
4096 May 24 11:00 ./
drwxr-xr-x 3 root root
4096 May 24 10:57 ../
-rw-r--r-- 1 root root
85744 May 24 11:00 libboost_numpy36.a
lrwxrwxrwx 1 root root
26 May 24 10:58 libboost_numpy36.so -> libboost_numpy36.so.1.67.0*
-rwxr-xr-x 1 root root
65192 May 24 10:58 libboost_numpy36.so.1.67.0*
-rw-r--r-- 1 root root 571448 May 24 10:59 libboost_python36.a
lrwxrwxrwx 1 root root
27 May 24 10:58 libboost_python36.so -> libboost_python36.so.1.67.0*
-rwxr-xr-x 1 root root 342584 May 24 10:58 libboost_python36.so.1.67.0*
root@computer:/home/share/boost_1_67_0/stage/lib# cp -rf libboost_python36.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_python-py36.so.1.67.0
root@computer:/home/share/boost_1_67_0/stage/lib# cp -rf libboost_python36.a /usr/lib/x86_64-linux-gnu/libboost_python36.a
root@computer:/home/share/boost_1_67_0/stage/lib# ln -s /usr/lib/x86_64-linux-gnu/libboost_python-py36.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_python3.so
root@computer:/home/share/boost_1_67_0/stage/lib#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@computer:/home/share/boost_1_67_0# cd stage/lib/
root@computer:/home/share/boost_1_67_0/stage/lib# ll
total 1052
drwxr-xr-x 2 root root
4096 May 24 11:00 ./
drwxr-xr-x 3 root root
4096 May 24 10:57 ../
-rw-r--r-- 1 root root
85744 May 24 11:00 libboost_numpy36.a
lrwxrwxrwx 1 root root
26 May 24 10:58 libboost_numpy36.so -> libboost_numpy36.so.1.67.0*
-rwxr-xr-x 1 root root
65192 May 24 10:58 libboost_numpy36.so.1.67.0*
-rw-r--r-- 1 root root 571448 May 24 10:59 libboost_python36.a
lrwxrwxrwx 1 root root
27 May 24 10:58 libboost_python36.so -> libboost_python36.so.1.67.0*
-rwxr-xr-x 1 root root 342584 May 24 10:58 libboost_python36.so.1.67.0*
root@computer:/home/share/boost_1_67_0/stage/lib# cp -rf libboost_python36.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_python-py36.so.1.67.0
root@computer:/home/share/boost_1_67_0/stage/lib# cp -rf libboost_python36.a /usr/lib/x86_64-linux-gnu/libboost_python36.a
root@computer:/home/share/boost_1_67_0/stage/lib# ln -s /usr/lib/x86_64-linux-gnu/libboost_python-py36.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_python3.so
root@computer:/home/share/boost_1_67_0/stage/lib#

make test测试

复制代码

1
sudo make test

复制代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
LD .build_release/src/caffe/test/test_stochastic_pooling.o
LD .build_release/src/caffe/test/test_common.o
LD .build_release/src/caffe/test/test_sigmoid_cross_entropy_loss_layer.o
LD .build_release/src/caffe/test/test_hinge_loss_layer.o
LD .build_release/src/caffe/test/test_reduction_layer.o
LD .build_release/src/caffe/test/test_tile_layer.o
LD .build_release/src/caffe/test/test_hdf5data_layer.o
LD .build_release/src/caffe/test/test_infogain_loss_layer.o
LD .build_release/src/caffe/test/test_scale_layer.o
LD .build_release/src/caffe/test/test_syncedmem.o
LD .build_release/src/caffe/test/test_inner_product_layer.o
LD .build_release/src/caffe/test/test_slice_layer.o
LD .build_release/src/caffe/test/test_net.o
LD .build_release/src/caffe/test/test_batch_norm_layer.o
LD .build_release/src/caffe/test/test_lrn_layer.o
LD .build_release/cuda/src/caffe/test/test_im2col_kernel.o
CXX/LD -o .build_release/test/test_all.testbin src/caffe/test/test_caffe_main.cpp
root@computer:/home/share/caffe#

make runtest测试

复制代码

1
sudo make runtest

失败报错如下:

.build_release/tools/caffe: error while loading shared libraries: libboost_python36.so.1.67.0: cannot open shared object file: No such file or directory

解决方法：创建软连接

复制代码

1
2
root@computer:/home/share/caffe# ln -s /usr/lib/x86_64-linux-gnu/libboost_python-py36.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_python36.so.1.67.0
root@computer:/home/share/caffe#

再次测试

复制代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[----------] 2 tests from EuclideanLossLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN
] EuclideanLossLayerTest/3.TestForward
[
OK ] EuclideanLossLayerTest/3.TestForward (1 ms)
[ RUN
] EuclideanLossLayerTest/3.TestGradient
[
OK ] EuclideanLossLayerTest/3.TestGradient (33 ms)
[----------] 2 tests from EuclideanLossLayerTest/3 (34 ms total)
[----------] 1 test from SolverTypeUpgradeTest
[ RUN
] SolverTypeUpgradeTest.TestSimple
[
OK ] SolverTypeUpgradeTest.TestSimple (1 ms)
[----------] 1 test from SolverTypeUpgradeTest (1 ms total)
[----------] Global test environment tear-down
[==========] 2207 tests from 285 test cases ran. (537753 ms total)
[
PASSED
] 2207 tests.
root@computer:/home/share/caffe#

mnist数据集测试

复制代码

1
2
3
4
5
6
7
8
9
# 下载mnist数据集
cd caffe/data/mnist
sudo sh ./get_mnist.sh
# 在该目录下将有相应图片和标签文件mnist数据格式转换
cd caffe
sudo sh ./examples/mnist/create_mnist.sh
# 将在mnist文件夹（上个步骤的路径）生成LMDB格式数据集
# 训练mnist cd caffe
sudo sh ./examples/mnist/train_lenet.sh

编译pycaffe

在caffe根目录的python文件夹下，有一个requirements.txt的清单文件，上面列出了需要的依赖库，按照这个清单安装就可以了。

在安装scipy库的时候，需要fortran编译器（gfortran)，如果没有这个编译器就会报错，因此，我们可以先安装一下。

首先回到caffe的根目录，然后执行安装代码：

复制代码

1
2
3
4
cd caffe
sudo apt-get install gfortran
cd ./python
for req in $(cat requirements.txt); do pip install $req; done

安装完成以后，再次回到caffe根目录我们可以执行：

复制代码

1
2
cd ..
sudo pip install -r python/requirements.txt

就会看到，安装成功的，都会显示Requirement already satisfied, 没有安装成功的，会继续安装。

编译python接口:

复制代码

1
2
3
make pycaffe
-j8
make distribute

如果没有任何错误，这个时候你会在你的caffe主目录下面看到一个distribute的文件夹。这儿就是我们需要的pycaffe了。接着我们需要将python配置到环境变量里面：

复制代码

1
2
3
4
5
6
7
8
9
sudo gedit ~/.bashrc
将export PYTHONPATH=/home/share/caffe/python:$PYTHONPATH添加到文件中
source ~/.bashrc
vim ~/.bashrc
export PYTHONPATH="/home/share/caffe/python:$PYTHONPATH"
export LD_LIBRARY_PATH="/home/share/caffe/distribute/lib:$LD_LIBRARY_PATH"
#退出vim
source ~/.bashrc

后面你在命令行当中输入python并"import caffe"，如果么有发现错误提示，即代表安装成功。