mesos下載、準備安裝環境、編譯以及測試
參考網址:http://mesos.apache.org/gettingstarted/
實驗環境:mesos版本:0.21.0、作業系統:Red Hat 6
為了瞭解Spark的三種部署standalone、mesos和YARN。所以想嘗試部署下mesos叢集。我根據自己的需求,參考了官方文件的如下步驟:
Downloading Mesos
There are different ways you can get Mesos:
-
Download the latest stable release from Apache (Recommended)
$ wget http://www.apache.org/dist/mesos/0.22.1/mesos-0.22.1.tar.gz $ tar -zxf mesos-0.22.1.tar.gz
CentOS 6.5
-
Following are the instructions for stock CentOS 6.5. If you are using a different OS, please install the packages accordingly.
Mesos 0.21.0+ requires subversion 1.8+ devel package which is not available by default by yum. Add one of the repo that has subversion-devel 1.8 available, i.e: Add new repo /etc/yum.repos.d/wandisco-svn.repo, with: [WandiscoSVN] name=Wandisco SVN Repo baseurl=http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/$basearch/ enabled=1 gpgcheck=0 $ sudo yum groupinstall -y "Development Tools" $ sudo yum install -y python-devel java-1.7.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel # Install maven. $ wget http://mirror.nexcess.net/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz $ sudo tar -zxf apache-maven-3.0.5-bin.tar.gz -C /opt/ $ sudo ln -s /opt/apache-maven-3.0.5/bin/mvn /usr/bin/mvn
Building Mesos
# Change working directory.
$ cd mesos
# Bootstrap (***Skip this if you are not building from git repo***).
$ ./bootstrap
# Configure and build.
$ mkdir build
$ cd build
$ ../configure
$ make
# Run test suite.
$ make check
# Install (***Optional***).
$ make install
Examples
-
Mesos comes bundled with example frameworks written in
C++
,Java
andPython
.# Change into build directory. $ cd build # Start mesos master (***Ensure work directory exists and has proper permissions***). $ ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos # Start mesos slave. $ ./bin/mesos-slave.sh --master=127.0.0.1:5050 # Visit the mesos web page. $ http://127.0.0.1:5050 # Run C++ framework (***Exits after successfully running some tasks.***). $ ./src/test-framework --master=127.0.0.1:5050 # Run Java framework (***Exits after successfully running some tasks.***). $ ./src/examples/java/test-framework 127.0.0.1:5050 # Run Python framework (***Exits after successfully running some tasks.***). $ ./src/examples/python/test-framework 127.0.0.1:5050
NOTE: To build the example frameworks, make sure you build the test suite by doingmake
check
.
但是mesos的構建遠比我想象的要難一些。它花了我大概一天半的時間對它進行下載、準備安裝環境以及簡單的測試。其中幾乎每一個步驟都會遇到問題,所以特意記錄下來,方便將來參考:
下載Mesos:
Apache Mesos官方文件上有兩種方式來獲取Mesos: 1、從 Apache官網上下載(推薦的方式)。 $ wget http://www.apache.org/dist/mesos/0.21.0/mesos-0.21.0.tar.gz
$ tar -zxf mesos-0.21.0.tar.gz
2、從 Mesos
git repository上獲取 (只適用於大神們的方式)
$ git clone https://git-wip-us.apache.org/repos/asf/mesos.git
因為Spark1.3.1支援的是mesos 0.21.0,所以我沒有使用最新的mesos版本,所以使用了0.21.0版本。我採用的是第一種方式下載Mesos。下載下來的Mesos只有是原始碼,而沒有二進位制包。所以需要我們按照自己的作業系統來編譯。在編譯之前,首先確認編譯mesos所需的環境。
對系統的要求:
Mesos執行在Linux (64 Bit) 和 Mac OSX (64 Bit)上:
CentOS 6.51、確保/etc/yum.repos.d下的源是CentOS的,並且版本也和Linux一致。 我的Red Hat 6採用的是CentOS 6的源。如果Ubuntu,可以參考官方文件來做。 我之前Red Hat 6採用了CentOS5的源,導致後面環境更新時出現了問題,所以把CentOS6的源也記錄下: 在/etc/yum.repos.d這個目錄下,檔名可以自己取,我用的是CentOS6.repo 這個檔名,其內容如下: [base]
name=CentOS-6 - Base - 163.com
baseurl=http://mirrors.163.com/centos/6/os/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#gpgcheck=1
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
##released updates
#[updates]
#name=CentOS-6 - Updates - 163.com
#baseurl=http://mirrors.163.com/centos/6/updates/$basearch/
##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#gpgcheck=1
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
##additional packages that may be useful
#[extras]
#name=CentOS-6 - Extras - 163.com
#baseurl=http://mirrors.163.com/centos/6/extras/$basearch/
##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
#gpgcheck=1
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
##additional packages that extend functionality of existing packages
#[centosplus]
#name=CentOS-6 .2- Plus - 163.com
#baseurl=http://mirrors.163.com/centos/6/centosplus/$basearch/
##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#gpgcheck=1
#enabled=0
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
##contrib - packages by Centos Users
#[contrib]
#name=CentOS-6 - Contrib - 163.com
#baseurl=http://mirrors.163.com/centos/6/contrib/$basearch/
##mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
#gpgcheck=1
#enabled=0
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
然後執行:
yum clean all && yum clean metadata && yum clean dbcache && yum makecache && yum update
2、Mesos 0.21.0 + 需要subversion 1.8+ deve這個包,它在預設的yum是獲取不到的。可以加入一個新的repo,/etc/yum.repos.d/wandisco-svn.repo它的內容是:
[WandiscoSVN]
name=Wandisco SVN Repo
baseurl=http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/$basearch/
enabled=1
gpgcheck=0
我不確定/etc/yum.repos.d/wandisco-svn.repo這個檔案是否起到了作用,將wandisco-svn.repo這個檔案新增到/etc/yum.repos.d。
3、執行sudo yum groupinstall -y "Development Tools",遇到如下問題:
public key for systemtap-client-2.5-5.el6.x86_64.rpm is not installed
原因是缺少systemtap-client-2.5-5.el6.x86_64.rpm。只要下載了這個rpm後執行rpm -ihv systemtap-client-2.5-5.el6.x86_64.rpm 安裝即可,如果缺少其他依賴,就先安裝其他依賴。
4、執行sudo yum install -y python-devel java-1.7.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel
5、安裝maven
構建Mesos:
# Change working directory.
$ cd mesos
# Configure and build.
$ mkdir build
$ cd build
$ ../configure
在執行到../configure時遇到了如下問題:
1) configure:error:cannot find libz. libz is required for mesos to build.
原因是缺少zlib-devel-1.2.3-25.el6.x86_64
需要下載相應的rpm,並安裝:
rpm -ihv zlib-devel-1.2.3-25.el6.x86_64.rpm
2) configure:error:cannot find libcurl. libcurl is required for mesos to build.
原因是缺少libcurl-devel-7.19.7-16.el6.x86_64
需要下載相應的rpm,並安裝:
rpm -ihv libidn-devel-1.18-2.el6.x86_64.rpm
rpm -ihv libcurl-devel-7.19.7-16.el6.x86_64.rpm
3) configure:error:cannot find libapr-1. libapr-1 is required for mesos to build.
原因是缺少apr-devel-1.3.9-3.el6.x86_64
需要下載相應的rpm,並安裝:
rpm -ihv apr-1.3.9-3.el6.x86_64.rpm
rpm -ihv apr-devel-1.3.9-3.el6.x86_64.rpm
rpm -ihv apr-util-1.3.9-3.el6.x86_64.rpm
4) configure:error:cannot find libsvn_subr-1 headers. libsubversion-1 is required for mesos to build.
從http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/找到下面的包,並安裝:
rpm -ihv serf-1.2.0-1.x86_64.rpm
rpm -ihv subversion-debuginfo-1.8.0-1.x86_64.rpm
rpm -ihv subversion-1.8.0-1.x86_64.rpm
rpm -ihv subversion-javahl-1.8.0-1.x86_64.rpm
rpm -ihv subversion-devel-1.8.0-1.x86_64.rpm
5) configure:error:cannot find libsasl2. We need libsasl2 for authentication!
安裝 rpm -ihv cyrus-sasl-devel-2.1.23-8.el6.x86_64.rpm
6) 如下圖的錯誤:
解決方法:rpm -ihv python-devel-2.6.5-3.el6.x86_64.rpm
$ make
遇到如下問題:
為了解決上面的問題,需要安裝下列包:
rpm -ihv apr-util-1.3.9-3.el6.x86_64.rpm
rpm -ihv apr-util-ldap-1.3.9-3.el6.x86_64.rpm
rpm -ihv db4-cxx-4.7.25-16.el6.x86_64.rpm
rpm -ihv db4-devel-4.7.25-16.el6.x86_64.rpm
rpm -ihv expat-devel-2.0.1-9.1.el6.x86_64.rpm
rpm -ihv openldap-devel-2.4.19-15.el6.x86_64.rpm
rpm -ihv apr-util-devel-1.3.9-3.el6.x86_64.rpm
# Run test suite.
$ make check
遇到如下問題:
意思是Linux的核心太舊了,這個錯誤我沒鳥它,直接跳到下一步了。
# Install (***Optional***).
$ make install
測試:
# Change into build directory.
$ cd build
# Start mesos master (***Ensure work directory exists and has proper permissions***).
$ ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos
# Start mesos slave.
$ ./bin/mesos-slave.sh --master=127.0.0.1:5050 (可以在最後加個&)
# Visit the mesos web page.
$ http://127.0.0.1:5050
# Run Java framework (***Exits after successfully running some tasks.***).
$ ./src/examples/java/test-framework 127.0.0.1:5050
測試步驟沒什麼大問題,可以訪問頁面,需要注意的是,使用jps並不能查到mesos的程序.