爬蟲入門-4-3.爬取豆瓣電影
阿新 • • 發佈:2019-03-10
lose ike mlp requests one spa host alt gen
1 import requests 2 3 url = "https://movie.douban.com/cinema/nowplaying/changsha/" 4 proxy = { 5 ‘HTTP‘: ‘116.209.53.143:9999‘ 6 } 7 headers = { 8 ‘User-Agent‘: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36‘ 9 ‘ (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36抓取‘, 10 ‘Host‘: ‘movie.douban.com‘ 11 } 12 response = requests.get(url, headers=headers, proxies=proxy) 13 with open(‘movie.html‘, ‘w‘, encoding=‘utf-8‘) as f: 14 f.write(response.content.decode(‘utf-8‘))
1 from lxml import etree 2 3 parser = etree.HTMLParser(encoding=‘utf-8解析‘) 4 html = etree.parse(‘movie.html‘, parser=parser) 5 # 1.獲取ul等於lists的第一個ul標簽(正在上映的電影) 6 ul = html.xpath("//ul[@class=‘lists‘]")[0] 7 # 2.獲取ul=lists標簽下所有的li標簽 8 lis = ul.xpath("./li") 9 movies = list() 10 for li in lis: 11 # 3.獲取li標簽的各個屬性,返回的是一個列表,取第一個元素 12 title = li.xpath("@data-title")[0] 13 duration = li.xpath("@data-duration")[0] 14 region = li.xpath("@data-region")[0] 15 director = li.xpath("@data-director")[0] 16 actors = li.xpath("@data-actors")[0] 17 image = li.xpath(".//img/@src")[0] 18 movie = { 19 ‘片名‘: title, 20 ‘時長‘: duration, 21 ‘區域‘: region, 22 ‘導演‘: director, 23 ‘演員‘: actors, 24 ‘海報‘: image 25 } 26 movies.append(movie) 27 for m in movies: 28 print(m)
爬蟲入門-4-3.爬取豆瓣電影