1. 程式人生 > >40行代碼爬取金庸所有武俠小說

40行代碼爬取金庸所有武俠小說

col 鹿鼎記 image ext .text chap ror python代碼 gpo

  我們今天來用Python爬蟲爬取金庸所有的武俠小說,網址為:http://jinyong.zuopinj.com/,網頁如下:

技術分享圖片

  Python代碼如下:

 1 # -*- coding: utf-8 -*-
 2 import urllib.request  
 3 from bs4 import BeautifulSoup
 4 
 5 #get book‘s chapter
 6 def get_chapter(url):
 7     html = urllib.request.urlopen(url)  
 8     content = html.read().decode(utf8
) 9 html.close() 10 soup = BeautifulSoup(content, "lxml") 11 title = soup.find(h1).text #chapter title 12 text = soup.find(div, id=htmlContent) #chapter content 13 #processing the content to get nice style 14 content = text.get_text(\n,br/).replace(\n, \n
) 15 content = content.replace(  , \n  ) 16 return title, +content 17 18 def main(): 19 books = [射雕英雄傳,天龍八部,鹿鼎記,神雕俠侶,笑傲江湖,碧血劍,倚天屠龍記,20 飛狐外傳,書劍恩仇錄,連城訣,俠客行,越女劍,鴛鴦刀,白馬嘯西風,21 雪山飛狐] 22 order = [1,2,3,4,5,6,7,8,10,11,12,14,15,13,9] #
order of books to scrapy 23 #list to store each book‘s scrapying range 24 page_range = [1,43,94,145,185,225,248,289,309,329,341,362,363,364,375,385] 25 26 for i,book in enumerate(books): 27 for num in range(page_range[i],page_range[i+1]): 28 url = "http://jinyong.zuopinj.com/%s/%s.html"%(order[i],num) 29 try: 30 title, chapter = get_chapter(url) 31 with open(E://%s.txt%book, a, encoding=gb18030) as f: 32 print(book+:+title+-->寫入成功!) 33 f.write(title+\n\n\n) 34 f.write(chapter+\n\n\n) 35 except Exception as e: 36 print(e) 37 print(全部寫入完畢!) 38 39 main()

  運行結果如下:

技術分享圖片

上面的運行結果“HTTP Error 404: Not Found”是因為這個網頁不存在,並不影響書本內容的完整性。我們可以去E盤查看文件是否下載成功:

技術分享圖片

·  15本書都下載完畢了!整個過程才用了不到10分鐘!爬蟲的力量真是偉大啊~~

40行代碼爬取金庸所有武俠小說