hadoop叢集lzo的安裝
主要步驟:
1,安裝和更新gcc、ant(系統已經安裝的話,略去下面步驟)
yum -y install gcc gcc-c++ autoconf automake
tar -jxvf apache-ant-1.8.2-bin.tar.bz2
export ANT_HOME=/usr/local/apache-ant-1.8.2
vi /etc/profile
source /etc/profile
export PATH=$PATH:$ANT_HOME/bin
2在各個節點安裝lzo
tar -zxvf lzo-2.04.tar.gzcd lzo-2.04
./configure --enable-shared
make && make install
拷貝/usr/local/lib目錄下的lzo庫檔案到/usr/lib(32位平臺),或/usr/lib64(64位平臺)
3,安裝lzo編碼/解碼器
tar -zxvf kevinweil-hadoop-lzo-2ad6654.tar.gz
cd kevinweil-hadoop-lzo-2ad6654
ant compile-native tar
安裝編譯過程中有錯誤,請考慮安裝下面兩個rpm包
rpm -ivh lzo-2.04-1.el5.rf.i386.rpm
rpm -ivh lzo-devel-2.04-1.el5.rf.i386.rpm
(根據系統選擇)編譯成功後,拷貝kevinweil-hadoop-lzo-2ad6654/bulid/hadoop-lzo-0.4.14.jar到hadoop/lib下
拷貝build/native/Linux-amd64-64/lib下檔案到hadoop/lib/native/Linux-amd64-64下
4,修改配置檔案,並同步各節點的配置檔案
core-site.xml中增加
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,co
m.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
mapred-site.xml中增加
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
5,安裝lzop壓縮軟體,到http://www.lzop.org/的下面download部分下載,可以選擇選擇Precompiled binaries are available for下載,直接配置可用,或者下載source,./configure make&makeinstall即可這是就可以通過lzop檔名,來生成lzo壓縮檔案,傳到hadoop平臺,平臺支援檔案分割。