Scrapy-css選擇器
阿新 • • 發佈:2018-12-25
和xpath選擇器比起來,感覺CSS選擇器容易一些,跟寫.css時方法基本一樣,就是在獲取內容時和xpath不同,這裡需要注意一下.
這裡介紹如何用css選擇器提取出一篇文章的資料
提取的資料跟xpath那篇文章內容相同
之前xpath中我們獲取元素是通過.entry-header h1::text
,如果是屬性則用.entry-header a::attr(href)
介紹一個常用的函式extract_first()
相當於extract()[0]
,但是extract()[0]當陣列沒有元素時,也就是沒有獲取到資料時會出錯,所以用extract_first()
方法,也可以加上需要返回的內容,比如空,extract_first("")
title = response.css(".entry-header h1::text").extract_first()
#p可以不加
create_date = response.css("p.entry-meta-hide-on-mobile::text").extract()[0].strip().replace('·','').strip()
#獲取點贊數
praise_nums = response.css('#110287votetotal::text').extract()[0]
#獲取收藏數
fav_nums = response.css('.btn-bluet-bigger.href-style.bookmark-btn .register-user-only::text ' ).extract()[0].strip()
match_re = re.match('.*?(\d+).*',fav_nums)
if match_re:
#獲取收藏數
fav_nums = match_re.group(1)
comment_nums = response.css('.btn-bluet-bigger.href-style.hide-on-480::text').extract()[0].strip()
match_re = re.match('.*?(\d+).*',fav_nums)
if match_re:
comment_nums = match_re.group(1 )
tag_list = response.css('.entry-meta-hide-on-mobile a::text').extract()
content = response.css('div.entry').extract()[0]
tag_list = [element for element in tag_list if not element.strip().endswith('評論')]
tag = ','.join(tag_list)
當我們要選擇的屬性名字有多個時比如下面:
這市在選擇時應該用
post_urls = response.css('#archive .post.floated-thumb .post-thumb a::attr(href)').extract()
也就是.post.floated-thumb應該連起來,或者只寫.floated-thumb
完整程式碼(準)
def parse_detail(self, response):
title = response.css(".entry-header h1::text").extract_first()
create_date = response.css("p.entry-meta-hide-on-mobile::text").extract()[0].strip().replace("·","").strip()
praise_nums = response.css(".vote-post-up h10::text").extract()[0]
fav_nums = response.css(".bookmark-btn::text").extract()[0]
match_re = re.match(".*?(\d+).*", fav_nums)
if match_re:
fav_nums = int(match_re.group(1))
else:
fav_nums = 0
comment_nums = response.css("a[href='#article-comment'] span::text").extract()[0]
match_re = re.match(".*?(\d+).*", comment_nums)
if match_re:
comment_nums = int(match_re.group(1))
else:
comment_nums = 0
content = response.css("div.entry").extract()[0]
tag_list = response.css("p.entry-meta-hide-on-mobile a::text").extract()
tag_list = [element for element in tag_list if not element.strip().endswith("評論")]
tags = ",".join(tag_list)
pass