最近有個專案需要用到全國中學的資訊，自己整理肯定事很難可行了，在網上看到這一片文章：用HttpClient抓取人人網高校資料庫（http://www.iteye.com/topic/826988），就想能不能從人人網上把這部分資訊搞下來。下面是實現步驟：

1 首先firefox安裝httpfox外掛，用來監控http請求

2 登入人人網，修改個人基本資訊，點選修改學校資訊

3 開啟httpfox，點選start，開始監控http資訊，如下圖

4 點選學校資訊的高中，此時會彈出學校選擇對話方塊，然後注意觀察http請求

注意看選中的那一行，就是從伺服器傳回來的學校資料，content裡面是伺服器返回的資料，一段標準的html程式碼，拷貝連結到瀏覽器，出現如下介面，可以得出一個結論，這部分資源資訊，人人也沒有加session驗證，省去了不少麻煩。

多點選幾個其他的省看一下：

發現每個省請求的連結都不同，分析可以知道，每個市對應於一個html檔案，省和市這兩級的資料經過分析可以在cityArray.js這個檔案中找到，檔案的結構如下所示：

var _city_1=["110101:\u4e1c\u57ce\u533a","110102:\u897f\u57ce\u533a","110103:\u5d07\u6587\u533a","110104:\u5ba3\u6b66\u533a","110105:\u671d\u9633\u533a","110106:\u4e30\u53f0\u533a","110107:\u77f3\u666f\u5c71\u533a","110108:\u6d77\u6dc0\u533a","110109:\u95e8\u5934\u6c9f\u533a","110111:\u623f\u5c71\u533a","110112:\u901a\u5dde\u533a","110113:\u987a\u4e49\u533a","110114:\u660c\u5e73\u533a","110115:\u5927\u5174\u533a","110116:\u6000\u67d4\u533a","110117:\u5e73\u8c37\u533a","110228:\u5bc6\u4e91\u53bf","110229:\u5ef6\u5e86\u53bf"];
var _city_2=["310101:\u9ec4\u6d66\u533a","310103:\u5362\u6e7e\u533a","310104:\u5f90\u6c47\u533a","310105:\u957f\u5b81\u533a","310106:\u9759\u5b89\u533a","310107:\u666e\u9640\u533a","310108:\u95f8\u5317\u533a","310109:\u8679\u53e3\u533a","310110:\u6768\u6d66\u533a","310112:\u95f5\u884c\u533a","310113:\u5b9d\u5c71\u533a","310114:\u5609\u5b9a\u533a","310115:\u6d66\u4e1c\u65b0\u533a","310116:\u91d1\u5c71\u533a","310117:\u677e\u6c5f\u533a","310118:\u9752\u6d66\u533a","310119:\u5357\u6c47\u533a","310120:\u5949\u8d24\u533a","310230:\u5d07\u660e\u53bf"];
var _city_3=["120101:\u548c\u5e73\u533a","120102:\u6cb3\u4e1c\u533a","120103:\u6cb3\u897f\u533a","120104:\u5357\u5f00\u533a","120105:\u6cb3\u5317\u533a","120106:\u7ea2\u6865\u533a","120107:\u5858\u6cbd\u533a","120108:\u6c49\u6cbd\u533a","120109:\u5927\u6e2f\u533a","120110:\u4e1c\u4e3d\u533a","120111:\u897f\u9752\u533a","120112:\u6d25\u5357\u533a","120113:\u5317\u8fb0\u533a","120114:\u6b66\u6e05\u533a","120115:\u5b9d\u577b\u533a","120221:\u5b81\u6cb3\u53bf","120223:\u9759\u6d77\u53bf","120225:\u84df\u53bf"];
 

每一行代表一個省的城市資料，對中文進行了編碼，可以通過程式解碼出來看看，python中可以用下面的程式碼，解碼\u897f\u57ce格式的編碼

s="\u6cb3"
s.decode('unicode_escape')

由此檔案可以分析出全國省市兩級的結構，程式碼如下

def getProvinceData():
    content = open("/home/xiyang/workspace/school-data/cityArray.js")
    #分離出市級id和名稱
    partten = re.compile("(\d+):([\w\d\\\\]+)")
    provinceList = []
    for line in content.readlines():
        data = partten.findall(line)
        citys = []
        province = {} 
        for s in data:
            if len(s[0]) == 4:#城市
                #print s[0],s[1].decode('unicode_escape')
                citys.append({"id":s[0],"name":s[1].decode('unicode_escape')})
            
        province_id = len(data[0][0])==4 and data[0][0] or data[0][0][0:4]
    
        #只處理列表中的幾個省
        if provinceMap.has_key(int(province_id)):
            province['id'] = province_id
            province['name'] = provinceMap[int(province_id)]
            province['citys'] = citys
            provinceList.append(province)
        
    return provinceList
 

通過httpfox分析出來每個市對應一個html檔案，包含該市所有的縣區和學校的資料，請求的連結類似與http://support.renren.com/juniorschool/1101.html，返回的資料格式片段如下所示：

<ul id="schoolCityQuList" class="module-qulist"><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370102')">歷下區</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370103')">市中區</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370104')">槐蔭區</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370105')">天橋區</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370112')">歷城區</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370113')">長清區</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370124')">平陰縣</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370125')">濟陽縣</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370126')">商河縣</a></li><li><a href="#highschool_anchor"  onclick="SchoolComponent.tihuan('city_qu_370181')">章丘市</a></li></ul>

<ul id="city_qu_370102" style="display:none;">
<li><a onclick='if(SchoolComponent.cl_school){return SchoolComponent.cl_school(event,40019572)}' href="40019572">山師大附中</a></li>
<li><a onclick='if(SchoolComponent.cl_school){return SchoolComponent.cl_school(event,40033777)}' href="40033777">濟南二十四中</a></li>
<li><a onclick='if(SchoolComponent.cl_school){return SchoolComponent.cl_school(event,40033962)}' href="40033962">濟寧育才學校</a></li>

獲取這些資料可以使用urllib2方便的獲取

#獲得某個市級區域的學校列表，如果事直轄市，則是整個直轄市的學校
def getTownHtml(town_id):
    try:
        url = "http://support.renren.com/juniorschool/%s.html" % town_id
        print "請求網路資料：",url
        return urllib2.urlopen(url).read()
    except:
        print "網路錯誤！"
        pass

5 分析這部分資料，可以有多個思路：

• 直接使用jquery分析html，然後使用檔案相關api儲存到檔案，這種方式分析html是比較方便的，由於chrome和firefox的檔案操作比較麻煩，沒有繼續嘗試
• 使用正則表示式分析出資料，這種方式提取縣區還是比較簡單的，但是想要分析完整的資料還是不太好用
• 使用html解析工具，解析html的結構，提取資料，最後選用了這一種。

在python中，有多個解析html的工具，比如HTMLParser，sgmllib，htmllib，他們都是基於事件驅動的，對於這種結構的資料還是不怎麼好用，最後選用了BeautifulSoup，分析這個資料格式小菜一碟，下面是程式碼：

#獲得某個的市級區域所有縣區的學校
def getCitySchool(content):
    soup = BeautifulSoup(content)
    
    #某個城市的中學列表
    citySchoolData = []
    #縣區的列表
    townlist = soup.findAll('a',href="#highschool_anchor")    


    for town in townlist:
        d = {}
        d['name'] = getUnicodeStr(town.string)
        d['id'] = town['onclick'][24:38]
        townSchools = []
        #獲得每個縣的中學列表
        for school in soup.find('ul',id=d['id']).findChildren('a'):
            townSchools.append(getUnicodeStr(school.string))
        d['schoollist'] = townSchools
        
        citySchoolData.append(d)
    
    return citySchoolData

上面的函式，分析html的內容，並返回某個市所有的縣區和學校的資訊， 執行之後的結果：

這樣即可根據個人的需要完成進一步的操作了。 完整的程式碼如下，

#!/usr/bin/env python
#-*- coding:utf-8 -*-
#============================================
# Author:[email protected]
# date:2012-12-29
# description: 解析人人網全國中學數學資訊
# 思路：
# 1 首先獲取全國的省市的資料（下載cityarray.js並使用正則表示式解析）
# 2 每個市（包括直轄市）對應一個html檔案，包含了該市所有的縣區列表和學校列表，通過urllib2模組從網上下載資料
# 3 使用BeautifulSoup分析從網上抓取的資料，然後解析出資料內容
# 4 將資料存到mongodb中
#============================================
import urllib2
import re
from BeautifulSoup import BeautifulSoup
from pymongo import MongoClient

db_host = "127.0.0.1"
db_port = 27017
db_name = "openclass"

provinceMap = {
    "北京":1101,
    "上海":3101,
    "天津":1201,
    "重慶":5001,
    "黑龍江":2301,
    "吉林":2201,
    "遼寧":2101,
    "山東":3701,
    "山西":1401,
    "陝西":6101,
    "河北":1301,
    "河南":4101,
    "湖北":4201,
    "湖南":4301,
    "海南":4601,
    "江蘇":3201,
    "江西":3601,
    "廣東":4401,
    "廣西":4501,
    "雲南":5301,
    "貴州":5201,
    "四川":5101,
    "內蒙古":1501,
    "寧夏":6401,
    "甘肅":6201,
    "青海":6301,
    "西藏":5401,
    "新疆":6501,
    "安徽":3401,
    "浙江":3301,
    "福建":3501,
    "香港":8101,
}
provinceMap = dict([[v,k] for k,v in provinceMap.items()])

#解碼字串 北京
def getUnicodeStr(s):
    name = []
    for word in s.split(";"):
        try:
            name.append(unichr(int(word[2:])))
        except:
            pass    
    return "".join(name)

#獲得某個市級區域的學校列表，如果事直轄市，則是整個直轄市的學校
def getTownHtml(town_id):
    try:
        url = "http://support.renren.com/juniorschool/%s.html" % town_id
        print "請求網路資料：",url
        return urllib2.urlopen(url).read()
    except:
        print "網路錯誤！"
        pass
         
def getProvinceData():
    content = open("/home/xiyang/workspace/school-data/cityArray.js")
    #分離出市級id和名稱
    partten = re.compile("(\d+):([\w\d\\\\]+)")
    provinceList = []
    for line in content.readlines():
        data = partten.findall(line)
        citys = []
        province = {} 
        for s in data:
            if len(s[0]) == 4:#城市
                #print s[0],s[1].decode('unicode_escape')
                citys.append({"id":s[0],"name":s[1].decode('unicode_escape')})
            
        province_id = len(data[0][0])==4 and data[0][0] or data[0][0][0:4]
    
        #只處理列表中的幾個省
        if provinceMap.has_key(int(province_id)):
            province['id'] = province_id
            province['name'] = provinceMap[int(province_id)]
            province['citys'] = citys
            provinceList.append(province)
        
    return provinceList


#獲得某個的市級區域所有縣區的學校
def getCitySchool(content):
    soup = BeautifulSoup(content)
    
    #某個城市的中學列表
    citySchoolData = []
    #縣區的列表
    townlist = soup.findAll('a',href="#highschool_anchor")    

    for town in townlist:
        d = {}
        d['name'] = getUnicodeStr(town.string)
        d['id'] = town['onclick'][24:38]
        townSchools = []
        #獲得每個縣的中學列表
        for school in soup.find('ul',id=d['id']).findChildren('a'):
            townSchools.append(getUnicodeStr(school.string))
        d['schoollist'] = townSchools
        
        citySchoolData.append(d)
    
    return citySchoolData

conn = MongoClient(db_host)
db = conn.openclass
juniorschool = db.juniorschool

if __name__ == "__main__":
    provinceList = getProvinceData();
    print provinceList
    
    for province in provinceList:
        citys = province['citys']
        #print province
        if citys:#有城市，說明不是直轄市
            for city in citys:
                data = {"id":city['id'],"name":city['name'],"data":getCitySchool(getTownHtml(city['id']))}     
                print "insert into mongodb:",city['name']
                juniorschool.insert(data)          
        
        else:#直轄市
            data = {"id":province['id'],"name":province['name'],"data":getCitySchool(getTownHtml(province['id']))}   
            print "insert into mongodb:",province['name']           
            juniorschool.insert(data)

執行玩程式碼，會把相關的資料存到mongodb，如下所示：