Linux下Hadoop Eclipse外掛編譯安裝
我的hadoop是1.2.1版本的,在網上也有一些編譯好的hadoop外掛,但是還是自己動手,豐衣足食,以後版本更新的時候能自己編譯外掛,而且這樣還能保證往eclipse裡安裝的時候不會出太多的問題。
主要是參考了這篇文章http://www.cnblogs.com/kinuxroot/archive/2013/05/06/linux_hadoop_eclipse_plugin.html
為方便檢視,還是將其中資訊貼上過來。宣告:我按照前面的解釋部分一點點新增程式碼的時候沒成功,使用了後面的整體檔案,稍加修改之後才成功的,果然胡亂往裡新增是不行的,畢竟版本也是不一樣,有些地方還是不同。
1.先要加入eclipse的設定和hadoop的版本設定,我的eclipse安裝在/home/kinuxroot/apps/eclipse下,所以我們要修改為:
<property name="eclipse.home" location="/home/kinuxroot/apps/eclipse"/> <property name="version" value="1.1.2"/>
這一步,location中的路徑請大家根據各自的實際路徑進行修改。
2.我們需要引用hadoop的一些包,但是預設的classpath沒有這些包(我們沒有從頭編譯)。所以需要修改classpath
定位<path id="classpath">,加入:
<fileset dir="${hadoop.root}"> <include name="**/*.jar" /> </fileset>
3.程式碼中使用了一些遺留功能,所以我們要修改deprecation的設定。
開啟hadoop根路徑下面的src/contrib/build-contrib.xml,定位
<property name="javac.deprecation" value="off"/>
然後修改成
<property name="javac.deprecation" value="on"/>
4.修改includeantruntime設定。定位compile的target,修改javac的設定,加入一個選項
includeantruntime="on"
也就是將javac修改成
<javac encoding="${build.encoding}" srcdir="${src.dir}" includes="**/*.java" destdir="${build.classes}" debug="${javac.debug}" deprecation="${javac.deprecation}" includeantruntime="on"> <classpath refid="classpath"/> </javac>
5.jar打包的時候需要hadoop的一些jar檔案,但是我們沒有編譯生成它,所以我們需要修改一下jar這個target。
另外,有幾個jar是我們需要用到,而build.xml裡面沒有自動包含的,如果不包含它們,Eclipse連線Hadoop會出現failure to login錯誤,其實就是找不到類
找到
<copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/> <copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>
我們修改成
<copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/> <copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar" tofile="${build.dir}/lib/commons-cli.jar" verbose="true"/> <copy file="${hadoop.root}/lib/commons-configuration-1.6.jar" tofile="${build.dir}/lib/commons-configuration.jar" verbose="true"/> <copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar" tofile="${build.dir}/lib/commons-httpclient.jar" verbose="true"/> <copy file="${hadoop.root}/lib/commons-lang-2.4.jar" tofile="${build.dir}/lib/commons-lang.jar" verbose="true"/> <copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar" tofile="${build.dir}/lib/jackson-core-asl.jar" verbose="true"/> <copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar" tofile="${build.dir}/lib/jackson-mapper-asl.jar" verbose="true"/>
6.但是這樣,我們的jar檔案還是不會自動部署到eclipse中,你可以手動複製,但是我們希望ant幫我們自動部署進去。
我們新建一個target,用來部署:
<target name="deploy" depends="jar" unless="skip.contrib"> <copy file="${build.dir}/hadoop-${name}-${version}.jar" todir="${eclipse.home}/plugins" verbose="true"/> </target>
然後修改project的預設target,也就是將project修改成:
<project default="deploy" name="eclipse-plugin">
7.接下來一步我們要修改Hadoop根目錄下的src/contrib/eclipse-plugin/META-INFO/MANIFEST.MF,修改這個jar的classpath。
找到這個檔案的Bundle-ClassPath這一行,然後,修改成
Bundle-ClassPath: classes/,lib/commons-cli.jar,lib/commons-httpclient.jar,lib/hadoop-core.jar,lib/jackson-mapper-asl.jar,lib/commons-configuration.jar,lib/commons-lang.jar,lib/jackson-core-asl.jar
8.執行ant,程式碼就會被編譯,外掛會被自動安裝到eclipse的plugins目錄中,開啟eclipse就可以使用了(如果沒有安裝ant,
請去apache觀望下載ant的二進位制編譯版)。另外編譯時會自動聯網下載需要的包,所以請保證網路通暢。
最後,我把我的build.xml貼在下面,大家可以參考:
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <project default="deploy" name="eclipse-plugin"> <import file="../build-contrib.xml"/> <property name="eclipse.home" location="/home/kinuxroot/apps/eclipse"/><!--這裡的location改成本機的eclipse安裝路徑--> <property name="version" value="1.1.2"/><!--這裡的版本資訊要改成和自己一樣的,我就在這裡出了錯--> <path id="eclipse-sdk-jars"> <fileset dir="${eclipse.home}/plugins/"> <include name="org.eclipse.ui*.jar"/> <include name="org.eclipse.jdt*.jar"/> <include name="org.eclipse.core*.jar"/> <include name="org.eclipse.equinox*.jar"/> <include name="org.eclipse.debug*.jar"/> <include name="org.eclipse.osgi*.jar"/> <include name="org.eclipse.swt*.jar"/> <include name="org.eclipse.jface*.jar"/> <include name="org.eclipse.team.cvs.ssh2*.jar"/> <include name="com.jcraft.jsch*.jar"/> </fileset> </path> <!-- Override classpath to include Eclipse SDK jars --> <path id="classpath"> <pathelement location="${build.classes}"/> <pathelement location="${hadoop.root}/build/classes"/> <fileset dir="${hadoop.root}"> <include name="**/*.jar" /> </fileset> <path refid="eclipse-sdk-jars"/> </path> <!-- Skip building if eclipse.home is unset. --> <target name="check-contrib" unless="eclipse.home"> <property name="skip.contrib" value="yes"/> <echo message="eclipse.home unset: skipping eclipse plugin"/> </target> <target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib"> <echo message="contrib: ${name}"/> <javac encoding="${build.encoding}" srcdir="${src.dir}" includes="**/*.java" destdir="${build.classes}" debug="${javac.debug}" deprecation="${javac.deprecation}" includeantruntime="on"> <classpath refid="classpath"/> </javac> </target> <!-- Override jar target to specify manifest --> <target name="jar" depends="compile" unless="skip.contrib"> <mkdir dir="${build.dir}/lib"/> <copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/> <copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar" tofile="${build.dir}/lib/commons-cli.jar" verbose="true"/> <copy file="${hadoop.root}/lib/commons-configuration-1.6.jar" tofile="${build.dir}/lib/commons-configuration.jar" verbose="true"/> <copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar" tofile="${build.dir}/lib/commons-httpclient.jar" verbose="true"/> <copy file="${hadoop.root}/lib/commons-lang-2.4.jar" tofile="${build.dir}/lib/commons-lang.jar" verbose="true"/> <copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar" tofile="${build.dir}/lib/jackson-core-asl.jar" verbose="true"/> <copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar" tofile="${build.dir}/lib/jackson-mapper-asl.jar" verbose="true"/> <echo message="${build.dir}"/> <echo message="${root}"/> <jar jarfile="${build.dir}/hadoop-${name}-${version}.jar" manifest="${root}/META-INF/MANIFEST.MF"> <fileset dir="${build.dir}" includes="classes/ lib/"/> <fileset dir="${root}" includes="resources/ plugin.xml"/> </jar> </target> <target name="deploy" depends="jar" unless="skip.contrib"> <copy file="${build.dir}/hadoop-${name}-${version}.jar" todir="${eclipse.home}/plugins" verbose="true"/> </target> </project>