作業系統：Ubuntu 16.04 LTS

nutch版本：2.2.1

配置nutch之前，要先配置ant，不會的可以看我的另一篇文章UBUNTU環境配置ANT

然後去nutch官網下載nutch，不過2.3.1的版本編譯時有問題，切換maven2庫也沒用，會一直卡在以下介面：

[email protected]:/opt/apache-nutch-2.3.1# ant runtime
Buildfile: /opt/apache-nutch-2.3.1/build.xml
ivy-probe-antlib:
ivy-download:
ivy-download-unchecked:
ivy-init-antlib:
ivy-init:
init:
    [mkdir] Created dir: /opt/apache-nutch-2.3.1/build
    [mkdir] Created dir: /opt/apache-nutch-2.3.1/build/classes
    [mkdir] Created dir: /opt/apache-nutch-2.3.1/build/release
    [mkdir] Created dir: /opt/apache-nutch-2.3.1/build/test
    [mkdir] Created dir: /opt/apache-nutch-2.3.1/build/test/classes

clean-lib:

resolve-default:
[ivy:resolve] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::
[ivy:resolve] :: loading settings :: file = /opt/apache-nutch-2.3.1/ivy/ivysettings.xml
 

於是我放棄了，決定採用nutch2.2.1版本進行安裝，nutch2.2.1下載地址：http://archive.apache.org/dist/nutch/2.2.1/

Ubuntu環境下的firefox預設下載儲存路徑為~/Downloads

1、用命令cd ~/Downloads切換路徑，然後使用tar -xvf apache-nutch-2.2.1-src-tar-gz解壓檔案

然後移動到/opt目錄下，用命令sudo mv apache-nutch-2.2.1 /opt/移動到/opt資料夾下

2、配置nutch對mysql的支援，修改 ${NUTCH_HOME}/ivy/ivy.xml檔案

先取消以下行的註釋

<dependency org=”mysql” name=”mysql-connector-java” rev=”5.1.18″ conf=”*->default”/>

然後修改以下行，從預設的

<dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>

改成

<dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>

最後取消掉以下行的註釋

<dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />

3、資料庫連線配置編輯 ${NUTCH_HOME}/conf/gora.properties檔案，註釋掉預設的資料庫連線配置，同時新增以下配置內容：

###############################
# Default MySQL properties    #
###############################
gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true
gora.sqlstore.jdbc.user=xxxx（MySQL使用者名稱）
gora.sqlstore.jdbc.password=xxxx（MySQL密碼）

4、資料表對映配置

修改 ${NUTCH_HOME}/conf/gora-sql-mapping.xml 檔案

將primarykey 的長度從512修改成767，即 <primarykey column=”id” length=”767″/>

5、修改nutch-site.xml配置檔案

可直接將nutch-default.xml儲存為nutch-site.xml，使用命令sudo mv nutch-default-xml nutch-size.xml

然後sudo gedit nutch-site,在末尾的</configuration>前新增以下程式碼

<property>
  <name>http.agent.name</name>
  <value>YourNutchSpider</value>
</property>
<property>
  <name>http.accept.language</name>
  <value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>
  <description>Value of the Accept-Language request header field.
   This allows selecting non-English language as default one to retrieve.
   It is a useful setting for search engines build for certain national group.
  </description>
</property>
<property>
  <name>storage.data.store.class</name>
  <value>org.apache.gora.sql.store.SqlStore</value>
  <description>The Gora DataStore class for storing and retrieving data.
    Currently the following stores are available:.
  </description>
</property> 
<property>
  <name>parser.character.encoding.default</name>
  <value>utf-8</value>
  <description>The character encoding to fall back to when no other information
   is available</description>
 </property>
<property>
  <name>generate.batch.id</name>
  <value>*</value>
</property>

6、使用ant編譯

切換到NUTCH目錄

cd ${NUTCH_HOME}
ant runtime

可能遇到的問題：

1）許可權不足，建立資料夾例如build資料夾失敗，使用命令sudo -i切換到root許可權再進行ant編譯

2）提示：

Trying to override old definition of task javac [taskdef] 
      Could not load definitions from resource org/sonar/ant/antlib.xml. It could not be found.

先下載sonar-ant-task-2.2.jar,將其拷貝到 ${NUTCH_HOME}/lib 目錄下面

然後使用命令sudo gedit /${NUTCH_HOME}/build.xml

通過ctrl+F開啟搜尋功能，輸入antlib:org,sonar.ant定位到以下程式碼，新增紅色部分的程式碼即可

<span style="color:#4b4b4b;"><!-- Define the Sonar task if this hasn't been done in a common script -->
 <taskdef url="antlib:org.sonar.ant" resource="org/sonar/ant/antlib.xml">
  <classpath path="${ant.library.dir}" />
  <classpath path="${mysql.library.dir}" />
  </span><span style="color:#ff0000;"><classpath><fileset dir="lib/" includes="sonar*.jar" /></classpath></span><span style="color:#4b4b4b;">
 </taskdef></span>

3）build failed,提示如

[ivy:resolve]         :: com.google.code.findbugs#jsr305;1.3.9!jsr305.jar
[ivy:resolve]         ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] 
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

BUILD FAILED
/opt/apache-nutch-2.2.1/build.xml:444: impossible to resolve dependencies:
    resolve failed - see output for details

或者是其他的依賴性問題導致BUILD FAILED的，可通過修改maven中央庫地址來解決

sudo gedit ${NUTCH_HOME}/ivy/ivysettings.xml，找到以下程式碼

<property name="repo.maven.org"
    value="http://repo1.maven.org/maven2/"
    override="false"/>

將maven中央庫地址 http://repo1.maven.org/maven2/  替換成國內OSC提供的映象：http://maven.oschina.net/content/groups/public/

4）卡在以下介面

resolve-default:
[ivy:resolve] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::
[ivy:resolve] :: loading settings :: file = /opt/apache-nutch-2.3.1/ivy/ivysettings.xml

解決方案：耐心等待，載入需要時間，如果超過10分鐘沒反應就放棄吧，可以換個maven（見問題3）。

一般編譯時間為半個小時左右！上個我成功的介面截圖

7、網站抓取測試

7.1 設定抓取網站

cd ${NUTCH_HOME}/runtime/local
sudo mkdir -p urls

cd urls
sudo gedit seed.txt

在seed.txt輸入一個網站，例如http://blog.csdn.net/u010317005

然後輸入冒號：wq儲存

7.2 執行爬蟲操作

bin/nutch crawl urls -depth 3 -topN 5