HTTP/FTP客戶端開發庫：libwww、libcurl、libfetch 以及更多
阿新 • • 發佈：2019-01-09
原文：http://hi.baidu.com/zkheartboy/blog/item/e40fc362f5d985dee6113ad9.html
網頁抓取和ftp訪問是目前很常見的一個應用需要，無論是搜尋引擎的爬蟲，分析程式，資源獲取程式，WebService等等都是需 要的，自己開發抓取庫當然是最好了，不過開發需要時間和週期，使用現有的Open source程式是個更好的選擇，一來別人已經寫的很好了，就近考驗，二來自己使用起來非常快速，三來自己還能夠學習一下別人程式的優點。
閒來無事，在網上瀏覽，就發現了這些好東西，特別抄來分享分享。主要就是libwww、libcurl、libfetch 這三個庫，當然，還有一些其他很多更優秀庫，文章後面會有簡單的介紹。
以下資料來源：http://9.douban.com/site/entry/15448100/，http://zh.wikipedia.org/wiki/Libwww
簡介：
Libwww 是一個高度模組化使用者端的網頁存取API ，用C語言寫成，可在 Unix 和 Windows 上執行。 It can be used for both large and small applications including: browsers/editors, robots and batch tools. There are pluggable modules provided with Libwww which include complete HTTP/1.1 with caching, pipelining, POST, Digest Authentication, deflate, etc. The purpose of libwww is to serve as a testbed for protocol experiments. 蒂姆·伯納斯－李 在 1992 年十一月創造出了 Libwww，用於展示網際網路的潛能。使用 Libwww 的應用程式，如被廣泛使用的命令列文字瀏覽器 Lynx 及 Mosaic web browser 即是用 Libwww 所寫成的。 Libwww 目前為一開放原始碼程式，並於日前移至 W3C 管理。基於其為開放原始碼的特性，任何人都能為 Libwww 付出一點心力，這也確保了 Libwww 能一直進步，成為更有用的軟體。
操作示例： 
這是在FreeBSD裡找到的一個庫：libfetch，原始碼在/usr/src/lib/libfetch裡，它對http和ftp協議進行了封裝，提供了一些很容易使用的函式，因為昨天剛看到，還沒仔細研究，我試了一個用http取網頁的函式，示例如下：
#include <stio.h>
#include
#include
#include "fetch.h"const char * myurl = "http://qjlemon:[email protected]:8080/test.html";
main()
{
FILE * fp;
char buf[1024];
fp = fetchGetURL(myurl, "";
if (!fp) {
printf("error: %s ", fetchLastErrString);
return 1;
}
while (!feof(fp)) {
memset(buf, 0, sizeof(buf));
fgets(buf, sizeof(buf), fp);
if (ferror(fp))
break;
if (buf[0])
printf("%s", buf);
elsebreak;
}
fclose(fp);
fp = NULL;
}
這裡最重要的就是fetchGetURL函式，它按指定的URL來取檔案，比如URL
是以http開頭的，這個函式就知道按http取檔案，如果是ftp://，就會按ftp取檔案，還可以指定使用者名稱和口令。
如果檔案被取到，它會返回一個FILE指標，可以象操作普通的檔案一樣把網頁的內容取出來。
另外這個庫還提供了一些函式，可以對網路操作進行更為精細的控制。
當然最有用的是還是幾個PUT函式，想要灌水就得用這個喲！哈哈哈！
Free Software and Open Source projects have a long tradition of forks and duplicate efforts. We enjoy "doing it ourselves", no matter if someone else has done something very similar already.
Free/open libraries that cover parts of libcurl's features:
這是在FreeBSD裡找到的一個庫：libfetch，原始碼在/usr/src/lib/libfetch裡，它對http和ftp協議進行了封裝，提供了一些很容易使用的函式，因為昨天剛看到，還沒仔細研究，我試了一個用http取網頁的函式，示例如下：
#include <stio.h>
#include
#include
#include "fetch.h"const char * myurl = "http://qjlemon:[email protected]:8080/test.html";
main()
{
FILE * fp;
char buf[1024];
fp = fetchGetURL(myurl, "";
if (!fp) {
printf("error: %s ", fetchLastErrString);
return 1;
}
while (!feof(fp)) {
memset(buf, 0, sizeof(buf));
fgets(buf, sizeof(buf), fp);
if (ferror(fp))
break;
if (buf[0])
printf("%s", buf);
elsebreak;
}
fclose(fp);
fp = NULL;
}
這裡最重要的就是fetchGetURL函式，它按指定的URL來取檔案，比如URL
是以http開頭的，這個函式就知道按http取檔案，如果是ftp://，就會按ftp取檔案，還可以指定使用者名稱和口令。
如果檔案被取到，它會返回一個FILE指標，可以象操作普通的檔案一樣把網頁的內容取出來。
另外這個庫還提供了一些函式，可以對網路操作進行更為精細的控制。
當然最有用的是還是幾個PUT函式，想要灌水就得用這個喲！哈哈哈！
Free Software and Open Source projects have a long tradition of forks and duplicate efforts. We enjoy "doing it ourselves", no matter if someone else has done something very similar already.
Free/open libraries that cover parts of libcurl's features:
- a highly portable and easy-to-use client-side URL transfer library, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TELNET, DICT, FILE, TFTP and LDAP. libcurl also supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, kerberos, HTTP form based upload, proxies, cookies, user+password authentication, file transfer resume, http proxy tunnelling and more!
- Having a glance at libghttp (a gnome http library), it looks as if it works rather similar to libcurl (for http). There's no web page for this and the person who's email is mentioned in the README of the latest release I found claims he has passed the leadership of the project to "eazel". Popular choice among GNOME projects.
- More complex, and and harder to use than libcurl is. Includes everything from multi-threading to HTML parsing. The most notable transfer-related feature that libcurl does not offer but libwww does, is caching.
- C++ library "for transferring files via http, ftp, gopher, proxy server". Based on 'snarf' 2.0.9-code (formerly known as libsnarf). Quote from freshmeat:
- An HTTP and WebDAV client library, with a C interface. I've mainly heard and seen people use this with WebDAV as their main interest.
- Part of glib (GNOME). Supports: HTTP 1.1, Persistent connections, Asynchronous DNS and transfers, Connection cache, Redirects, Basic, Digest, NTLM authentication, SSL with OpenSSL or Mozilla NSS, Proxy support including SSL, SOCKS support, POST data. Probably not very portable. Lacks: cookie support, NTLM for proxies, GSS, gzip encoding, trailers in chunked responses and more.
- Handles URLs, protocols, transports for the Mozilla browser.
- Minimal download library targeted to be much smaller than the above mentioned netlib. HTTP and FTP support.
- While not a library at all, I've been told that people sometimes extract the network code from it and base their own hacks from there.
- Does HTTP and FTP transfers (both ways), supports file: URLs, and an API for URL parsing. The utility
- "
- "
- (Windows) Provides client-side protocol support for communication with HTTP servers. A client computer can use the XMLHTTP object to send an arbitrary HTTP request, receive the response, and have the Microsoft® XML Document Object Model (DOM) parse that response.
- QHttp is a class in the Qt library from Troll Tech. Seems to be restricted to plain HTTP. Supports GET, POST and proxy. Asynchronous.
- "
- A C++ library for "easy FTP client functionality. It features resuming of up- and downloads, FXP support, SSL/TLS encryption, and logging functionality."
- Has a URLStream class. This C++ class allow you to download a file using HTTP. See demo/urlfetch.cpp in commoncpp2-1.3.19.tar.gz
- Java HTTP client library.
- A Java HTTP client library written by the Jakarta project.