2. 程式人生 > >定向爬蟲requests-bs4<2>

定向爬蟲requests-bs4<2>

阿新 • • 發佈：2018-12-15

Beautifulsoup4安裝

>>>Pip install beautifulsoup4

Beautifulsoup簡單使用 from BeautifulSoup(open(“index.html”)) soup = BeautifulSoup(“data”)

>>>from bs4 import BeautifulSoup
>>> html_doc = """
... <html><head><title>The Dormouse's story</title></head>
...     <body>
... <p class="title"><b>The Dormouse's story</b></p>
...
... <p class="story">Once upon a time there were three little sisters; and their names were
... <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
... <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
... <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
... and they lived at the bottom of a well.</p>
...
... <p class="story">...</p>
... """
>>> soup = BeautifulSoup(html_doc, 'html.parser')
>>> print(soup.prettify())
<html>
 <head>
  <title>
   The Dormouse's story
  </title>
 </head>
 <body>
  <p class="title">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    Elsie
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>

BeautifulSoup類的元素 將HTML文件轉換成標籤樹 **Tag：**標籤 **Name:**標籤的名字 .name **Attributes:**標籤的屬性 .attrs **NavigableString:**標籤內非屬性字串 .string **Comment：**標籤內字串註釋部分

‘’’
任何存在HTML語法中的標籤<tag>，都可以用soup.<tag>獲得
當HTML文件中存在多個相同<tag>對應內容時，soup.<tag>返回第一個
‘’’
>>> soup.a
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

#每一個<tag>的名字可以用<tag>.name獲取
>>> soup.a.name
'a'

#一個<tag>可以有零或多個屬性，字典型別
>>> t=soup.a
>>> t.attrs
{'href': 'http://example.com/elsie', 'class': ['sister'], 'id': 'link1'}
>>> t.attrs['href']
'http://example.com/elsie'

#
>>> soup.a.string
'Elsie'

遍歷文件樹 下行遍歷 .contents 將tag的子節點以列表的方式輸出 .children 子節點迭代型別 可以對所有tag的子節點進行遞迴迴圈 .descendants 子孫節點迭代型別 可以對所有tag的子孫節點進行遞迴迴圈 .string 若tag只有一個 NavigableString 型別子節點，可以使用 .string 得到子節點，否則，用.Strings或.stripped_strings迴圈遍歷

>>> soup.body.contents
 
['\n', <p class="title"><b>The Dormouse's story</b></p>, '\n', <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>, '\n', <p class="story">...</p>, '\n']
>>> for i in soup.body.children:
   print(i)

 


<p class="title"><b>The Dormouse's story</b></p>


<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>


<p class="story">...</p>


>>> for i in soup.body.descendants:
   print(i)

 


<p class="title"><b>The Dormouse's story</b></p>
<b>The Dormouse's story</b>
The Dormouse's story


<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
Once upon a time there were three little sisters; and their names were

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
Elsie
,

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
Lacie
 and

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
Tillie
;
and they lived at the bottom of a well.


<p class="story">...</p>
...


>>> for i in soup.body.strings:
   print(i)

 


The Dormouse's story


Once upon a time there were three little sisters; and their names were

Elsie
,

Lacie
 and

Tillie
;
and they lived at the bottom of a well.


...

#使用.strings輸出的字串中可能包含了很多空格或空行,使用 .stripped_strings 可以去除多餘空白內容
>>> for i in soup.body.stripped_strings:
   print(i)

 
The Dormouse's story
Once upon a time there were three little sisters; and their names were
Elsie
,
Lacie
and
Tillie
;
and they lived at the bottom of a well.
...

上行遍歷 .parent 獲取某元素的父節點 .parents 父輩節點迭代型別 可以對所有tag的父輩節點進行遞迴迴圈

>>> soup.a.parent
 
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
>>> for i in soup.a.parents:
   print(i)

 
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body>
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body></html>

<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body></html>

>>> for i in soup.a.parents:
   print(i.name)

 
p
body
html
[document]

>>> soup.a.string.parent
 
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

>>> for i in soup.a.string.parents:
   print(i.name)

 
a
p
body
html
[document]

平行遍歷 .next_sibling 按照HTML文字順序的下一個兄弟節點 .previous_sibling 按照HTML文字順序的上一個兄弟節點 .next_siblings 迭代型別 返回按照HTML文字順序後續所有兄弟節點 .previous_siblings 迭代型別 返回按照HTML文字順序前續所有兄弟節點

#如果以為第一個<a>標籤的 .next_sibling 結果是第二個<a>標籤,那就錯了,
真實結果是第一個<a>標籤和第二個<a>標籤之間的頓號和換行符
>>> soup.a.next_Sibling
 
',\n'

>>> soup.a.previous_sibling
 
'Once upon a time there were three little sisters; and their names were\n'

>>> for sibling in soup.p.next_siblings:
 print(sibling)

 


<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>


<p class="story">...</p>


>>> for sibling in soup.p.previous_siblings:
 print(sibling)



>>> for sibling in soup.a.previous_siblings:
 print(sibling)

 
Once upon a time there were three little sisters; and their names were
bs4庫的prettify()方法
  >>> print(soup.prettify())
 
<html>
 <head>
  <title>
   The Dormouse's story
  </title>
 </head>
 <body>
  <p class="title">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    Elsie
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>

相關推薦

定向爬蟲requests-bs4<2>

Beautifulsoup4安裝 >>>Pip install beautifulsoup4 Beautifulsoup簡單使用 from BeautifulSoup(open(“index.html”)) soup = Beautiful

定向爬蟲requests-bs4<1>

定向爬蟲requests-bs4<1> Requests庫的安裝 Win平臺：命令列cmd 執行 pip install requests 測試： >>>import requests >>>r = r.get(“

python爬蟲（五）：實戰 【2. 爬創客實驗室（requests + bs4）】

目標：爬取創科實驗室網站中講座的資訊， 輸出表：講座標題、報告人、單位、報告時間、講座內容、報告人簡介 技術：requests + bs4   檢視爬蟲協議： http://127.0.0.1/lab/robots.txt （創科實驗室是我自己寫的網址，不反爬蟲）

爬蟲3 requests基礎2 代理 證書 重定向 響應時間

9.1 取消 時間 span timeout rec https style res import requests # 代理 # proxy = { # ‘http‘:‘http://182.61.29.114.6868‘ # } # res = request

[定向爬蟲] 網絡爬蟲實例2-淘寶定向爬蟲

break 解析 tao 運行 title out 繼續 pri 字符串分割 import requests import re import time #獲取html頁面 def getHTMLText(url): try:

進程和線程的區別, 面相對象補充, 進程, 數據共享, 鎖, 進程池, 爬蟲模塊(requests, bs4(beautifulsoup))

request %s 功能 val 差異 http += 共享 str 一. 進程和線程的區別？ 第一： 進程是cpu資源分配的最小單元。 線程是cpu計算的最小單元。 第二： 一個進程中可以有多個線程。 第三：

網路爬蟲：Python+requests+bs4+xlwt 爬取京東商品存入Excel表

學了網路爬蟲兩週了，還是比較喜歡用網頁級庫requests，很靈活方便，scrapy網站級面向物件庫，還不熟悉，可能是原來c++學習面向物件就沒學好，對面向物件程式設計還沒理解好吧...兩週中爬了淘寶，京東，天貓（爬取失敗，反爬蟲把我這種新手難倒了，登入驗證就卡死），爬取搜狗

股票資料爬蟲（Scrapy框架與requests-bs4-re技術路線）

Scrapy中文名：抓取一個功能強大、快速、優秀的第三方庫它是軟體結構與功能元件的結合，可以幫助使用者快速實現爬蟲。Python開發的一個快速、高層次的螢幕抓取和web抓取框架，用於抓取web站點並從頁面中提取結構化的資料。Scrapy用途廣泛，可以用於資料探勘、監測和自動化

python爬蟲rp+bs4

每一個 使用 一個 進行 處理 爬蟲 字符 itl fin soup = BeautifulSoup(html_doc) soup 就是BeautifulSoup處理格式化後的字符串，soup.title 得到的是title標簽，soup.p 得到的是文檔中的第一個p標簽，

網絡爬蟲--requests庫中兩個重要的對象

resp head ppa except 代碼 http http響應 sts _for 當我們使用resquests.get（）時，返回的時response的對象，他包含服務器返回的所有信息，也包含請求的request的信息。 首先： response對象的屬性有以下幾個

爬蟲基本原理2

alt option mongo 種類型 瀏覽器 get 部分 json 頭部 什麽是爬? 請求?網站並提取數據的?自動化程序 爬蟲的基本流程 發起請求 通過HTTP庫向?目標站點發起請求，即發送?個Request，請求可以包含額外的headers等信息，等待服務器

中國大學排名定向爬蟲

展示 中國 判斷 點列 屏幕 .cn axu ise () 功能描述：輸入：大學排名URL鏈接輸出：大學排名信息的屏幕輸出（排名，大學名稱，總分）技術路線：requests+bs4定向爬蟲：僅對輸入URL進行爬取，不擴展爬取 程序的結構設計：步驟1：從網絡上獲取大學排名網頁

爬蟲-基於bs4庫的HTML內容查找方法

簽名 正則化 all 擴展 rev recursive title 參數 pre bs4有一個find_all(name,attrs,recursive,string,**kwargs)方法，返回一個列表類型，存儲查找的結果 name 對標簽名稱的檢索字符串 attrs 對

爬蟲系列（2）-----python爬取CSDN博客首頁所有文章

成功 -name 保存 eas attr eve lan url att 對於Python初學者來說，爬蟲技能是應該是最好入門，也是最能夠有讓自己有成就感的，今天在整理代碼時，整理了一下之前自己學習爬蟲的一些代碼，今天上第2個簡單的例子，python爬取CSDN博客首頁所有

關於linux中重定向 > 1> 2>&1 >>

改變 怎麽 引用 In 文件中 idt 技術分享 chm 很好 linux 中很多時候會用到重定向來完成一些工作下面就來談談我對重定向的理解 歡迎吐槽 1，0<或者< 表示標準輸入 代碼為0 或者 稱stdin

python爬蟲【第2篇】

multi 實現 from eat 進程 lun pid RR print 一、多進程 1.fork方法（os模塊,適用於Lunix系統） fork方法：調用1次，返回2次。原因：操作系統經當前進程（父進程）復制出一份進程（子進程），兩個進程幾乎完全相同，fork方法分別在

python 爬蟲 requests+BeautifulSoup 爬取巨潮資訊公司概況代碼實例

pan 字符 selenium 5.0 target 自我 color list tails 第一次寫一個算是比較完整的爬蟲，自我感覺極差啊，代碼low，效率差，也沒有保存到本地文件或者數據庫，強行使用了一波多線程導致數據順序發生了變化。。。 貼在這裏，引以為戒吧。 #

關於爬蟲的日常復習（13）—— 爬蟲requests的初級高級的基本用法

bubuko req src http ima 基本 爬蟲 用法 image 關於爬蟲的日常復習（13）—— 爬蟲requests的初級高級的基本用法

【Python爬蟲學習筆記2】urllib庫的基本使用

代理服務 cor proc 技術 origin car windows tpc -c urllib庫是python內置的實現HTTP請求的基本庫，通過它可以模擬瀏覽器的行為，向指定的服務器發送一個請求，並保存服務器返回的數據。 urlopen函數 函數原型：urlopen(

爬蟲 requests.post

爬蟲 requests.post 可以模擬網頁向伺服器傳送訊息，獲取想要的內容 1.無返回值 開啟並登陸豆瓣 模擬豆瓣登陸 import requests postUrl = 'https://www.douban.com/accounts/logi