2012-09-19

周海漢 /文

2012.9.19

## 下載

[[email protected] ~]$ wget http://labs.mop.com/apache-mirror/hadoop/chukwa/stable/chukwa-0.4.0.tar.gz
[[email protected] ~]$ wget http://labs.mop.com/apache-mirror/hadoop/common/hadoop-2.0.1-alpha/hadoop-2.0.1-alpha.tar.gz
Length: 82726054 (79M)
[[email protected] yarn]$ wget http://labs.mop.com/apache-mirror/hadoop/common/hadoop-2.0.1-alpha/hadoop-2.0.1-alpha-src.tar.gz
[
 
[email protected] yarn]$ tar zxvf hadoop-2.0.1-alpha-src.tar.gz

環境：

• 類Unix系統
• JDK 1.6 以上
• Maven 3.0 (用於代替ant的編譯環境)
• Forrest 0.8 (Apache Forrest是一個平臺獨立的文件框架。把來自各種不同的輸入資料來源轉換成用一種或多種輸出格式(比如HTML,PDF等)來統一顯示的釋出系統。它基於Apache Cocoon並分離了內容與內容結構,不僅可以生成靜態的文件也可以當作一個動態的伺服器。如果要生成文件則需要。)
• Findbugs 1.3.9 (Java 語言靜態分析工具，用於查詢bug。非必須。)
• ProtocolBuffer 2.4.1+ ( MapReduce 和 HDFS 用google protocol buffer來壓縮和交換資料)
• Autotools (如果編譯 native code需要)
• 第一次編譯要用到網路連線 (用於獲取所有 Maven 和 Hadoop 依賴庫)

安裝maven

[[email protected] ~]$ wget http://labs.mop.com/apache-mirror/maven/maven-3/3.0.4/binaries/apache-maven-3.0.4-bin.tar.gz

[[email protected]

~]$ vi .bashrc export PATH=$PATH:/home/zhouhh/maven/bin

maven用法：

• Clean : mvn clean
• Compile : mvn compile [-Pnative]
• Run tests : mvn test [-Pnative]
• Create JAR : mvn package
• Run findbugs : mvn compile findbugs:findbugs
• Run checkstyle : mvn compile checkstyle:checkstyle
• Install JAR in M2 cache : mvn install
• Deploy JAR to Maven repo : mvn deploy
• Run clover : mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
• Run Rat : mvn apache-rat:check
• Build javadocs : mvn javadoc:javadoc
• Build distribution : mvn package [-Pdist][-Pdocs][-Psrc][-Pnative][-Dtar]
• Change Hadoop version : mvn versions:set -DnewVersion=NEWVERSION

Build 選項:

• Use -Pnative to compile/bundle native code
• Use -Dsnappy.prefix=(/usr/local) & -Dbundle.snappy=(false) to compile Snappy JNI bindings and to bundle Snappy SO files
• Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist)
• Use -Psrc to create a project source TAR.GZ
• Use -Dtar to create a TAR with the distribution (using -Pdist)

[[email protected] hadoop-2.0.1-alpha-src]$ mvn package -Pdist,native,docs -DskipTests … main: [exec] target/compile-proto.sh: line 17: protoc: command not found

安裝protobuf

下載protocol buffer [[email protected] yarn]$ wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.bz2 [[email protected] yarn]$ tar jxvf protobuf-2.4.1.tar.bz2 [[email protected] protobuf-2.4.1]$ ./configure [[email protected] protobuf-2.4.1]$ make [[email protected] protobuf-2.4.1]$ sudo make install

[[email protected] hadoop-2.0.1-alpha-src]$ mvn package -Pdist,native,docs -DskipTests

main: [exec] protoc: error while loading shared libraries: libprotobuf.so.7: cannot open shared object file: No such file or directory

[[email protected] local]$ protoc –version protoc: error while loading shared libraries: libprotobuf.so.7: cannot open shared object file: No such file or directory

[[email protected] lib]$ ls /usr/local/lib/libprotobuf-lite.so.7 /usr/local/lib/libprotobuf-lite.so.7

[[email protected] local]$ sudo vi /etc/profile export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib [[email protected] local]$ source /etc/profile [[email protected] local]$ protoc –version libprotoc 2.4.1

錯誤處理

[[email protected] hadoop-2.0.1-alpha-src]$ mvn package -Pdist,native,docs -DskipTests [ERROR] Failed to execute goal org.codehaus.mojo:make-maven-plugin:1.0-beta-1:autoreconf (compile) on project hadoop-common: autoreconf command returned an exit value != 0. Aborting build; see debug output for more information. -> [Help 1] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-common 這個是因為編譯的時候帶了 native 引數，但是沒裝autotool。

安裝autotool

[[email protected] hadoop-2.0.1-alpha-src]$ sudo yum install autoconf automake libtool

[[email protected] hadoop-2.0.1-alpha-src]$ mvn package -Pdist,native,docs -DskipTests -rf :hadoop-common

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (site) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program “${env.FORREST_HOME}/bin/forrest” (in directory “/home/zhouhh/yarn/hadoop-2.0.1-alpha-src/hadoop-common-project/hadoop-common/target/docs-src”): java.io.IOException: error=2, No such file or directory -> [Help 1]

沒有安裝forrest，沒法生成文件。

apache forrest 安裝

[[email protected] yarn]$ wget http://labs.mop.com/apache-mirror//forrest/apache-forrest-0.9-sources.tar.gz [[email protected] yarn]$ wget http://labs.mop.com/apache-mirror//forrest/apache-forrest-0.9-dependencies.tar.gz 解壓apache-forrest-0.9-sources.tar.gz

[[email protected] main]$ pwd /home/zhouhh/yarn/apache-forrest-0.9/main [[email protected] main]$ ./build.sh ./build.sh: line 39: /home/zhouhh/yarn/apache-forrest-0.9/main/../tools/ant/bin/ant: No such file or directory

Could not load definitions from resource net/sf/antcontrib/antlib.xml. It could not be found. … BUILD FAILED /home/zhouhh/yarn/apache-forrest-0.9/main/build.xml:522: /home/zhouhh/yarn/apache-forrest-0.9/tools/jetty not found … error: error reading /home/zhouhh/yarn/apache-forrest-0.9/lib/endorsed/xercesImpl-2.9.1.jar; error in opening zip file 解壓apache-forrest-0.9-dependencies.tar.gz到forrest目錄。tools裡面的ant等都是dependencies裡面帶的。 或者用svn去下載最新的版本svn co http://svn.apache.org/repos/asf/forrest/trunk/

修改forrest環境變數 [[email protected] ~]$ vi .bashrc export FORREST_HOME=/home/zhouhh/yarn/apache-forrest-0.9 export PATH=$PATH:$FORREST_HOME/bin [[email protected] ~]$ source .bashrc

繼續編譯除錯

[[email protected] hadoop-2.0.1-alpha-src]$ mvn package -Pdist,native,docs -DskipTests

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.0:site (default) on project hadoop-auth: SiteToolException: IOException: cannot interpolate environment properties: Cannot run program “env”: java.io.IOException: error=12, Cannot allocate memory -> [Help 1] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-auth 記憶體只有2G，不夠了，關掉一些程式。

[exec] Exception in thread “main” java.lang.InternalError: Can’t connect to X11 window server using ‘192.168.20.81:0’ as the value of the DISPLAY variable. 啟動了xwindow [[email protected] hadoop-2.0.1-alpha-src]$ mvn package -Pdist,native,docs -DskipTests -rf :hadoop-auth [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (site) on project hadoop-common: An Ant BuildException has occured: stylesheet /home/zhouhh/yarn/hadoop-2.0.1-alpha-src/hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn’t exist. -> [Help 1] findbugs 沒安裝

安裝findbugs

http://findbugs.sourceforge.net/downloads.html [[email protected] yarn]$ wget http://downloads.sourceforge.net/project/findbugs/findbugs/2.0.1/findbugs-2.0.1-source.zip?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Ffindbugs%2Ffiles%2Ffindbugs%2F2.0.1%2Ffindbugs-2.0.1-source.zip%2Fdownload%3Fuse_mirror%3Dcdnetworks-kr-1%26download%3D&ts=1347957365&use_mirror=cdnetworks-kr-1

[[email protected] yarn]$ unzip findbugs-2.0.1-source.zip

[[email protected] yarn]$ cd findbugs-2.0.1 [[email protected] findbugs-2.0.1]$ ant BUILD FAILED /home/zhouhh/yarn/findbugs-2.0.1/build.xml:166: Problem: failed to create task or type native2ascii Cause: the class org.apache.tools.ant.taskdefs.optional.Native2Ascii was not found. This looks like one of Ant’s optional components. Action: Check that the appropriate optional JAR exists in -/usr/share/ant/lib -/home/zhouhh/.ant/lib -a directory added on the command line with the -lib argument

ant沒安裝好，到其他地方將ant的庫拷貝到/usr/share/ant/lib。該類在ant-nodeps.jar裡面。http://labs.mop.com/apache-mirror//forrest/apache-forrest-0.9-dependencies.tar.gz 裡面就含有。 redhat和centos自帶的ant，在/usr/bin/ant 不能正確編譯Findbugs,必須到ant 網站http://ant.apache.org/下載。推薦下載二進位制版本。

重新安裝ant

[[email protected] yarn]$ wget http://mirror.bjtu.edu.cn/apache//ant/binaries/apache-ant-1.8.4-bin.zip Length: 8043520 (7.7M)

自帶ant版本 [[email protected] findbugs-2.0.1]$ ant -version Apache Ant version 1.7.1 compiled on September 26 2008

[[email protected] ~]$ vi .bashrc export ANT_HOME=/home/zhouhh/yarn/apache-ant-1.8.4 export PATH=$PATH:$ANT_HOME/bin [[email protected] ~]$ source .bashrc

新版本ant 1.8.4 [[email protected] ~]$ ant -version Apache Ant(TM) version 1.8.4 compiled on May 22 2012

[[email protected] findbugs-2.0.1]$ ant Buildfile: /home/zhouhh/yarn/findbugs-2.0.1/build.xml BUILD SUCCESSFUL

設定findbugs環境

[[email protected] ~]$ vi .bashrc export FINDBUGS_HOME=/home/zhouhh/yarn/findbugs-2.0.1 export PATH=$PATH:$FINDBUGS_HOME/bin [[email protected] ~]$ source .bashrc

[[email protected] hadoop-2.0.1-alpha-src]$ mvn package -Pdist,native,docs -DskipTests -rf :hadoop-common [INFO] Apache Hadoop Common ………………………… SUCCESS [7:57.297s] [INFO] Apache Hadoop Common Project …………………. SUCCESS [0.338s] [INFO] Apache Hadoop HDFS ………………………….. FAILURE [3:01.064s]

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (site) on project hadoop-hdfs: An Ant BuildException has occured: input file /home/zhouhh/yarn/hadoop-2.0.1-alpha-src/hadoop-hdfs-project/hadoop-hdfs/target/findbugsXml.xml does not exist -> [Help 1]

[ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-hdfs

將native和生成文件的選項去掉，編譯ok

[[email protected] hadoop-2.0.1-alpha-src]$ mvn package -Pdist -DskipTests -rf :hadoop-hdfs

ok

如非註明轉載, 均為原創. 本站遵循知識共享CC協議,轉載請註明來源