使用libcurl庫獲取網頁資訊(C++ php)
這周要做一個小專案,需要獲得學生的賬戶名密碼後,去本校教務系統驗證是否正確並且抓一個網頁資訊下來。
查網上時候,可以通過傳送POST或GET請求來實現,然後嘗試用POST先。
首先安裝libcurl庫
sudo yum install libcurl
安裝好後之後程式碼中新增標頭檔案curl/curl.h就可以使用curl的函數了。
當時需要訪問的正方網頁(222.24.62.120)需要驗證碼,但是有另一個網站222.24.162.120/default4.aspx 可以用來驗證帳號密碼是否正確,從這個網站上拿到Cookie然後帶著Cookie去訪問正方網站就可以進去了。
然後我們來看看進入網頁傳送的請求包內容。
這個是帶著帳號密碼訪問該網站時傳送的請求包,圖中Request Headers是傳送包頭部資訊,Form Data下面還有兩個資訊分別是帳號密碼,它裡面是傳送包的資料部分,程式碼訪問網站就是模擬這個過程,將這些資訊手動加入,然後打包發出去。
過程中用到一共三個函式:
curl_slist_append();
curl_easy_init();
curl_easy_setopt();
curl_easy_perform();
curl_easy_cleanup();
1.curl_easy_init()
函式原型
CURL *curl_easy_init();
函式功能
初始化一個CURL指標,相應的末尾要清理這個指標。
2.curl_easy_setopt()
函式原型
CURLcode curl_easy_setopt(CURL *handle, CURLoption option, parameter);
函式功能
這個引數很重要,要通過它對訪問過程做很多設定。基本都是通過第二個引數在設定,所以第二個引數牽扯到很多巨集,下面列幾個我用到的。
重要巨集(option)
- CURLOPT_HTTPHEADER 用來設定HTTP頭部欄位,第三個引數是一個數組,包含了所有頭部資訊
- CURLOPT_COOKIEJAR 指定連線結束後儲存Cookie資訊的檔案
- CURLOPT_COOKIEFILE 指定讀取Cookie的檔案
- CURLOPT_URL 設定要訪問的URL
- CURLOPT_WRITEDATA 指定將網頁資訊寫進去的檔案
- CURLOPT_POSTFIELDS 全部資料使用HTTP協議中的POST操作來發送。第三個資料指定要傳送傳送資料。
- CURLOPT_RETURNTRANSFER 將curl_exec()獲取的資訊以檔案流的形式返回,而不是直接輸出。
3.curl_easy_perform()
函式原型
CURLcode curl_easy_perform( CURL *easy_handle);
該函式完成curl_easy_setopt指定的所指定的值,返回0意味一切ok,非0代表錯誤發生。主要錯誤碼說明:
1.CURLE_OK
任務完成一切都好
2.CURLE_UNSUPPORTED_PROTOCOL
不支援的協議,由URL的頭部指定
3.CURLE_COULDNT_CONNECT
不能連線到remote 主機或者代理
4.CURLE_REMOTE_ACCESS_DENIED
訪問被拒絕
5.CURLE_HTTP_RETURNED_ERROR
Http返回錯誤
6.CURLE_READ_ERROR
讀本地檔案錯誤
1.curl_easy_cleanup()
函式原型
void curl_easy_cleanup(CURL *handle);
函式功能
結束licurl使用時,清理掉curl_easy_init的指標。
6.curl_slist_append()
函式原型
struct curl_slist *curl_slist_append(struct curl_slist *list, const char *string);
函式功能
追加字串string到連結串列list尾,返回新增後的連結串列,在程式碼中用來將頭部資訊逐句新增到字串中。
這一段便可訪問到222.24.162.120/default4.aspx 的網站,然後獲取到Cookie資料存入檔案。
bool postUrl(const char *filename, std::string id, std::string passwd )
{
CURL *curl;
CURLcode res;
FILE *fp;
std::string mess = "__VIEWSTATE=dDwxMTE4MjQwNDc1Ozs%2BYofaNxf5dpXqcC3ZAqYdKfPCdbw%3D&TextBox1="
+ id + "&TextBox2=" + passwd + "&RadioButtonList1=%D1%A7%C9%FA&Button1=+%B5%C7+%C2%BC+";
if( (fp = fopen(filename, "w")) == NULL )
{
return false;
}
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" );
headers = curl_slist_append(headers, "Accept-Encoding:gzip, deflate");
headers = curl_slist_append(headers, "Accept-Language:zh-CN,zh;q=0.8");
headers = curl_slist_append(headers, "ache-Control:no-cache");
headers = curl_slist_append(headers, "Connection:keep-alive" );
headers = curl_slist_append(headers, "Content-Type:application/x-www-form-urlencoded" );
headers = curl_slist_append(headers, "Host:222.24.62.120");
headers = curl_slist_append(headers, "Origin:http://222.24.62.120");
headers = curl_slist_append(headers, "Pragma:no-cache");
headers = curl_slist_append(headers, "Referer:http://222.24.62.120/default4.aspx");
headers = curl_slist_append(headers, "Upgrade-Insecure-Requests:1");
headers = curl_slist_append(headers, "User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36");
curl = curl_easy_init();
if( curl )
{
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);// 改協議頭
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "/home/yunting/cookie.txt");
curl_easy_setopt(curl,CURLOPT_POSTFIELDS, mess.c_str());
curl_easy_setopt(curl, CURLOPT_URL, "http://222.24.62.120/default4.aspx");
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
fclose(fp);
return true;
}
然後用同樣的方法,在之前儲存Cookie的檔案中拿到Cookie資料,就能愉快的獲取資料了。
bool postUrl_two(const char *filename,std::string id)
{
CURL *curl;
CURLcode res;
FILE *fp;
std::string mess2 = "http://222.24.62.120/lw_xsxx.aspx?xh=" + id;
std::string mess1 = "Referer: "+mess2;
if((fp = fopen(filename,"w")) == NULL)
{
return false;
}
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" );
headers = curl_slist_append(headers, "Accept-Encoding:gzip, deflate,sdch");
headers = curl_slist_append(headers, "Accept-Language:zh-CN,zh;q=0.8");
headers = curl_slist_append(headers, "Cache-Control:no-cache");
headers = curl_slist_append(headers, "Connection:keep-alive" );
headers = curl_slist_append(headers, "Host:222.24.62.120");
headers = curl_slist_append(headers, "Pragma:no-cache");
headers = curl_slist_append(headers, mess1.c_str());
headers = curl_slist_append(headers, "Upgrade-Insecure-Requests:1");
headers = curl_slist_append(headers, "User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36");
curl = curl_easy_init();
if( curl )
{
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);// 改協議頭
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "/home/yunting/cookie.txt");
curl_easy_setopt(curl, CURLOPT_URL, mess2.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
fclose(fp);
return true;
}
因為當時還需要用從網頁中匹配出所需資訊,而網頁是gb2312編碼格式,所以在編碼轉換上奮戰一下午,未果,尋PHP終。。。
PHP實現
思路大概相同,就是函式名有點變化,用到函式有:
curl_init();
curl_setopt();
curl_exec();
curl_getinfo();
curl_close();
其中curl_getinfo()獲得一個CURL連線資源控制代碼的資訊,至於獲得哪個資訊,由引數決定。
//get請求得到id,passwd,前期判斷
<?php
header("content-type:text/html;charset=utf-8");
if(empty($_GET))
{
echo "{\"status\":-2,\"value\":\"GET請求為空\"}";
exit();
}
if(empty($_GET['sid']))
{
echo "{\"status\":-2,\"value\":\"學號不能為空\"}";
exit();
}
if(empty($_GET['passwd']))
{
echo "{\"status\":-2,\"value\":\"密碼不能為空\"}";
exit();
}
$sid = $_GET['sid'];
$passwd = $_GET['passwd'];
if( ((preg_match('/[0-2][0-9]1[2-6]\d{4}/', $sid)) == FALSE) || (strlen($sid) != 8) )
{
echo "{\"status\":-2,\"value\":\"輸入帳號長度有問題\"}";
}
if( strlen($passwd) < 6 || strlen($passwd) > 16 )
{
echo "{\"status\":-2,\"value\":\"輸入密碼長度有問題\"}";
}
//定義字串陣列$req_head,新增所有頭部資訊
"Accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding" => "gzip, deflate",
"Accept-Language" => "zh-CN,zh;q=0.8",
"ache-Control" => "no-cache",
"Connection" => "keep-alive",
"Content-Type" => "application/x-www-form-urlencoded",
"Host" => "222.24.62.120",
"Origin" => "http://222.24.62.120",
"Pragma" => "no-cache",
"Referer" => "http://222.24.62.120/default4.aspx",
"Upgrade-Insecure-Requests" => "1",
"User-Agent" => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36",
);
//定義請求的$form_data
$form_data = "__VIEWSTATE=dDwxMTE4MjQwNDc1Ozs%2BYofaNxf5dpXqcC3ZAqYdKfPCdbw%3D&TextBox1=".$sid."&TextBox2=".$passwd."&RadioButtonList1=%D1%A7%C9%FA&Button1=+%B5%C7+%C2%BC+";
$to_url = "http://222.24.62.120/default4.aspx";
$curl = curl_init();
$cookie = "cookie";
if($curl)
{
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_HTTPHEADER, $req_head);
curl_setopt($curl, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($curl, CURLOPT_POSTFIELDS, $form_data);
curl_setopt($curl, CURLOPT_URL, $to_url);
$res = curl_exec($curl);
$httpCode = curl_getinfo($curl, CURLINFO_HTTP_CODE);
curl_close($curl);
if($httpCode == 302)
{
$mes = "http://222.24.62.120/lw_xsxx.aspx?xh=".$sid;
$req_head2 = array(
"Accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding" => "gzip, deflate, sdch",
"Accept-Language" => "zh-CN,zh;q=0.8,en;q=0.6,ia;q=0.4",
"Cache-Control" => "no-cache",
"Connection" => "keep-alive",
"Host" => "222.24.62.120",
"Pragma" => "no-cache",
"Upgrade-Insecure-Requests" => "1",
"User-Agent" => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36",
);
$curl2 = curl_init();
if($curl2)
{
curl_setopt($curl2, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl2, CURLOPT_HEADER, false);
curl_setopt($curl2, CURLOPT_NOBODY, false);
curl_setopt($curl2, CURLOPT_HTTPGET, true);
curl_setopt($curl2, CURLOPT_HTTPHEADER, $req_head2);
curl_setopt($curl2, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($curl2, CURLOPT_REFERER, $mes);
curl_setopt($curl2, CURLOPT_URL, $mes);
$res2 = curl_exec($curl2);
$httpCode2 = curl_getinfo($curl2, CURLINFO_HTTP_CODE);
if($httpCode2 == 200)
{
$res2 = mb_convert_encoding($res2, "UTF-8","GB2312"); //將$res中的資訊從gb2312格式轉換為UTF-8
$tmp = htmlentities($res2, ENT_QUOTES, 'UTF-8');
echo "<pre>".$tmp."</pre><hr />";
$dom = new DOMDocument();
//建立DOM樹儲存資訊
$dom ->loadHTML("<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">".$res2); //頭部新增此資訊讓來查詢的覺得這是UTF-8編碼
$user = array();
$user['xh'] = $dom->getElementById('xh')->nodeValue;
$user['xm'] = $dom->getElementById('xm')->nodeValue;
$user['xy'] = $dom->getElementById('xy')->nodeValue;
$user['zy'] = $dom->getElementById('zy')->nodeValue;
$user['xzb'] = $dom->getElementById('xzb')->nodeValue; //查出對應ID的資訊
echo json_encode($user,JSON_UNESCAPED_UNICODE); //傳送json串
curl_close($curl2);
}
}
}
}
PHP中編碼轉換函式:
$res = mb_convert_encoding($res, "UTF-8","gb2312");
res指定要轉換的字串變數,第二個引數指定要轉換成什麼格式,第三個引數指定轉換前文字的格式。返回轉換結果。例如此函式將gb2312轉換為utf-8。
完了!!搞定!為建設中國特色社會主義道路加油奮鬥!