golang-proxy v3.0

中文文件

Golang-Proxy -- 簡單高效的免費代理抓取工具通過抓取網路上公開的免費代理，來維護一個屬於自己的高匿代理池，用於網路爬蟲、資源下載等用途。

在 v3.0 有哪些新特性

1. 依舊提供了高度靈活的 API 介面，在啟動主程式後，即可通過在瀏覽器訪問localhost:9999/all 與 localhost:9999/random 直接獲取抓到的代理！甚至可以使用 localhost:9999/sql?query=來執行一些簡單的 SQL 語句來自定義代理篩選規則！
2. 支援自動對代理型別進行判斷, 可以通過 schemeType 判定代理對http
   和https的支援程度
3. 支援了MySQL資料庫, 詳情請見 Config
4. 支援單獨啟動服務, 在啟動編譯好的二進位制檔案時, 通過 -mode= 來指定是否單獨啟動 producer/consumer/assessor/service
5. 重新設計了資料表, 請注意, 這意味著 API 介面發生了變動
6. 重新設計了 源 的資料結構, 去除了 filter 等欄位, 請注意, 這意味著 v2.0 的源在直接提供給v3.0 使用時可能會出現一些問題
7. 更新了一些 源
8. 不再支援 -source 啟動引數

如何使用 golang-proxy

1. 使用開箱即用版本

Release 頁面 根據系統環境提供了一些壓縮包，將他們解壓後執行即可。

下載完成後, 將壓縮包中的二進位制檔案和 source 目錄解壓到同一個位置, 啟動二進位制檔案即可, 程式將會啟動下面這些服務:

1. producer : 週期性的抓取source目錄中定義的源, 將抓取到的代理寫入到 crude_proxy 表中
2. consumer : 週期性的從 crude_proxy 中讀取一定數量的代理, 判斷它們的代理型別以及可用性, 將它們寫入到 proxy表中
3. assessor : 週期性的從 proxy 表中讀取一定數量的代理, 評估它們的質量
4. service : golang-proxy 提供的 http api 介面, 使你可以通過 localhost:9999/all
   , localhost:9999/random, localhost:9999/sql?query= 這三個介面來篩選和獲取 crude_proxy和 proxy 表中的代理

當你啟動編譯好的二進位制檔案時, 預設這些服務會依次啟動, 但是在 v3.0 版本, 你可以通過新增 -mode 啟動引數來指定單獨啟動某個服務, 比如:

golang-proxy -mode=service

這樣執行, 將只會啟動 service 服務, 在啟動了 service 之後, 你可以在瀏覽器中訪問以下介面, 獲得相應的代理:

請注意, crude_proxy 只是抓取到的代理的臨時儲存表, 不能保證它們的質量, 而proxy 表中的代理將會不斷得到 assessor 的評估, proxy 表中的 score 欄位可以較為全面的反映一個代理的質量, 質量較低時會被刪除

介面示例: localhost:9999/sql

例如訪問 localhost:9999/sql?query=SELECT * FROM PROXY WHERE SCORE > 5 ORDER BY SCORE DESC, 將會返回 proxy 表中所有分數大於5的代理, 並按照分數從高到低返回

{
    "error": "",
    "message": [
        {
            "id": 2,
            "ip": "45.113.69.177",
            "port": "1080",
            // scheme_type 可以取以下值:
            // 0: 代理只支援 http
            // 1: 代理只支援 https
            // 2: 代理同時支援 http 和 https
            "scheme_type": 0,
            "content": "45.113.69.177:1080",
            // 評估次數
            "assess_times": 9,
            // 評估成功次數, 可以通過 success_times/assess_times獲得代理連線成功率
            "success_times": 9,
            // 平均響應時間
            "avg_response_time": 0.098,
            // 連續失敗次數
            "continuous_failed_times": 0,
            // 分數, 推薦使用 5 分以上的代理
            "score": 68.45106053570785,
            "insert_time": 1540793312,
            "update_time": 1540797880
        },
    ]
}

2. 使用原始碼編譯

go get -u github.com/storyicon/golang-proxy

進入到 golang-proxy 目錄，執行 go build main.go，執行生成的二進位制的執行程式即可。

注意：

專案根目錄下的 ./source 是專案執行必須的資料夾，裡面儲存了各類網站源，其他的資料夾儲存的均為專案原始碼。所以在編譯後得到二進位制程式 main 檔案後，即可將 main 檔案和 source 資料夾一同移動到任意地方，main 檔案可以任意命名。

為什麼要用 Golang-Proxy

1. 穩定、快速。 抓取模組，單核併發可以到達 1000 個頁面/秒。
2. 高可配置性、高拓展性。 你不需要寫任何程式碼，花一兩分鐘填寫一個配置檔案就可以新增一個新的網站源。
3. 評估功能。 通過 Assessor 評估模組，週期性測試代理質量，根據代理的測試成功率、高匿性、測試次數、突變性、響應速度等獨立影響因子進行綜合評分，演算法具有高度可配置性，可以根據專案的需要可以對因子的權重進行獨立調整。
4. 提供了高度靈活的 API 介面，在啟動主程式後，即可通過在瀏覽器訪問localhost:9999/all 與 localhost:9999/random 直接獲取抓到的代理！甚至可以使用 localhost:9999/sql?query=來執行 SQL 語句來自定義代理篩選規則！
5. 不依賴任何服務型資料庫，一鍵下載，開箱即用！

如何配置一個新的源

./source/下的所有 yml 格式的檔案都是源，你可以增加源，也可以通過在檔名前加上一個 . 來使程式忽略這個源，當然你也可以直接刪除，來讓一個源永遠的消失，下面進行 Source 引數介紹：

#Page配置項
page:
    entry: "https://xxx/1.html"
    template: "https://xxx/{page}.html"
    from: 2
    to: 10
#publisher將會首先抓取entry，即 https://xxx/1.html
#然後根據 template、from 和 to 依次抓取
#　　https://xxx/2.html
#　　https://xxx/3.html
#　　https://xxx/4.html
#　　...
#　　https://xxx/10.html

#Selector配置項
selector:
    iterator: ".table tbody tr"
    ip: "td:nth-child(1)"
    port: "td:nth-child(2)"
# 以上配置用於抓取下面這種 HTML 結構
# <table class="table">
#     <tbody>
#         <tr>
#             <td>187.3.0.1</td>
#             <td>8080</td>
#             <td>HTTP</td>
#         <tr>
#         <tr>
#             <td>164.23.1.2</td>
#             <td>80</td>
#             <td>HTTPS</td>
#         <tr>
#         <tr>
#             <td>131.9.2.3</td>
#             <td>8080</td>
#             <td>HTTP</td>
#         <tr>
#     <tbody>
# <table>
# 選擇器為通用的JQuery選擇器，iterator為迴圈物件，比如表格裡的行，每行一條代理，那這個行的選擇器就是iterator，而ip、port、protocal則是在iterator選擇器的基礎上進行子元素的查詢。

category:
    # 並行數
    parallelnumber: 1
    # 對於這個源，每抓取一個頁面
    # 將會隨機等待5~20s再抓下一個頁面
    delayRange: [5, 20]
    # 間隔多長時間啟用一次這個源
    # @every 10s ， @every 10h...
    interval: "@every 10m"
debug: true

徵求意見

1. 使用中任何問題提 issues 即可
2. 如果發現了新的好用的源，歡迎提交上來分享
3. 來都來了點個 Star 再走唄 : )

English Document

Golang-proxy is an efficient free proxy crawler that ensures that the captured proxies are highly anonymous and at the same time guarantee their quality. You can use these captured proxies to download network resources and ensure the privacy of your own identity.

1. Feature

• Very high speed of proxy crawler, which can download 1000 pages per second.
• You can customize the source of proxy crawler. The configuration file is extremely simple.
• Provide a compiled version, comes with a SQLite database, and supports mysql
• Comes with an API interface, all functions can be used with one click
• Proxy evaluation system to ensure the quality of the proxy pool

2. How to use

golang-proxy provides compiled binary files so that you do not need golang on the machine. Download binary compression pack to Release Page According to your system type, download the corresponding compression package, unzip it and run it. After a few minutes, you can access localhost:9999/all in the browser to see the proxy's crawl results.

Before I go into the detailed introduction of golang-proxy, I think it's best to tell you the most useful information first.

API interface

After you start the binary, you can access the following interface in the browser to get the proxy

Having mastered the above content, you have been able to use the 50% function of golang-proxy. But the last interface allows you to execute custom SQL statements, and you'll find that you need to know at least the structure of the tables. The following will tell you.

3. Advanced

golang-proxy consists of the following parts:

• two data tables
• one configuration file
• one source folder
• four modules

two data tables

1. Table Crude Proxy

In order to store temporary proxies, we designed the data table crude_proxy, the table is defined as follows.

table crude_proxy stores the proxies that are crawled out, and cannot guarantee their quality.

2. Table Proxy

When the agent in the crude_proxy table passes through pre assess ( pre assess roughly verifies the availability of the proxy and tests the proxy's support for https and http ), it will enter the proxy table.

The proxy in the proxy table will be evaluated periodically and their scores will be modified. Low scores will be deleted.

one configuration file

For convenience, the proxy in golang-proxy is stored in the portable database sqlite by default. You can make golang-proxy use the mysql database by adding the config.yml file in the executable directory.

For details, see Config page.

one source folder

golang-proxy needs source to define its crawling contents and rules. Therefore, the run directory of golang-proxy needs at least one source folder, and the source folder should have at least one source in yml format. The source is defined as follows:

page: 
    entry: "http://www.xxx.com/http/?page=1"
    template: "http://www.xxx.com/http/?page={page}"
    from: 1
    to: 2000
selector:
    iterator: ".list item"
    ip: ".ip"
    port: ".port"
category:
    parallelnumber: 3
    delayRange: [10, 30]
    interval: "@every 10m"
debug: true

In the definition above, producer will first crawl the entry page, then crawl:

http://www.xxx.com/http/?page=1      
http://www.xxx.com/http/?page=2      
http://www.xxx.com/http/?page=3      
...      
http://www.xxx.com/http/?page=2000     

This source definition page expects this format:

<html>
    ...
    <div class="list">
        <div class="item">
            <div class="ip"> 127.0.0.1 </div>
            <div class="port"> 80 </div>
            ...
        </div>
        <div class="item">
            <div class="ip"> 125.4.0.1 </div>
            <div class="port"> 8080 </div>
            ...
        </div>
        ...
    </div>
    ...
</html>

When producer parses a single page, it always traverses the nodes defined by iterator first, and then gets the elements defined by ip and port selectors from these nodes. The source definition above is still valid for the following HTML structure.

<html>
    ...
    <div class="list">
        <div class="item">
            <div class="ip"> 127.0.0.1:80 </div>
        </div>
        <div class="item">
            <div class="ip"> 125.4.0.1:8080</div>
        </div>
        ...
    </div>
    ...
</html>

Because when the port selector cannot get the content, it will try to parse the port from the text selected by the ip selector.

The source is stored in the source folder in yml format, and a source definition is completed. Golang-proxy will read it and crawl it the next time it starts. So you successfully define a source, store it in the source folder in YML format, and the next time you start golang-proxy, the source will enter the crawl list.

If a source file name starts with a . , the source will not be read.

four modules

golang-proxy consists of four modules, which cooperate to complete the task that golang-proxy wants to accomplish.

When you start the executable file of golang-proxy, you will start these module in turn. But you can add the -mode startup parameter after the golang-proxy executable to command golang-proxy to start only one module. Like below:

golang-proxy -mode=service

This will only start the HTTP API interface service.

At this point, you have mastered the 95% function of golang-proxy. If you want to find more, you can read the source code provided above, and improve them.

Request for comments

Welcome to submit issue. If you feel that golang-proxy is helping you, you can order a star or watch, thanks !