音樂爬蟲

目的：爬取歌名，歌手，歌詞，歌曲url。

一、建立爬蟲專案

建立一個資料夾，進入資料夾，開啟cmd視窗，輸入：

scrapy startproject songSpider

依次輸入：

cd songSpider
scrapy genspider kuwo_spider kuwo.cn

在路徑songSpider\songSpider\spiders\下多了一個檔案kuwo_spider.py。

二、定義items.py中的欄位名

import scrapy

class SongspiderItem(scrapy.Item):
    # define the fields for your item here like:
 

    name = scrapy.Field()  # 歌名
    singer = scrapy.Field()  # 歌手
    lyric = scrapy.Field()  # 歌詞
    url = scrapy.Field()  # 能拿到音樂檔案的url
    originId = scrapy.Field()  # 來源

在Mysql資料庫中建立一個表，表中的欄位與上面的一樣，方便接下來把資料儲存進資料庫(持久化)。

三、修改settings.py中的內容

ROBOTSTXT_OBEY = False
#使用代理
DEFAULT_REQUEST_HEADERS = {
  'Accept'
 
: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en',
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',

}
#使用管道把item儲存進資料庫
ITEM_PIPELINES = {
   'songSpider.pipelines.SongspiderPipeline': 300,
}
 

四、修改pipelines.py

#儲存進MySQL資料庫

import pymysql

class SongspiderPipeline(object):
    def __init__(self):
        #連線資料庫
        self.conn = None
        #遊標
        self.cur = None

    # 開啟爬蟲時呼叫，只調用一次
    def open_spider(self,spider):
        self.conn = pymysql.connect(host='127.0.0.1',
                                    user='root',
                                    password="123456",
                                    database='songSearch',
                                    port=3306,
                                    charset='utf8')
        self.cur = self.conn.cursor()

    def process_item(self, item, spider):
        clos,value = zip(*item.items())

        # print('sql語句',sql,value)
        query_sql = f"""select sid from Song where  name='{value[0]}'
        and singer='{value[1]}'and originId={value[4]} """
        # 如果歌曲已存在就不執行sql
        if not self.cur.execute(query_sql):
            sql = "INSERT INTO `%s`(%s) VALUES (%s)" % ('Song',
                             ','.join(clos),
                             ','.join(['%s'] * len(value)))
            self.cur.execute(sql, value)
            self.conn.commit()

        return item

    def close_spider(self, spider):
        self.cur.close()
        self.conn.close()

五、修改kuwo_spider.py

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from songSpider.items import SongspiderItem

#使用多頁爬取
class KuwoSpider(CrawlSpider):
    name = 'kuwo_spider'
    allowed_domains = ['kuwo.cn']
    start_urls = ['http://yinyue.kuwo.cn/yy/category.htm',                
                  'http://www.kuwo.cn/mingxing',

                  ]
    # 設定爬取規則,可迭代物件，可設定多個規則

    rules = [
            Rule(LinkExtractor(allow=(
                                  "http://www.kuwo.cn/yinyue/(\d+)")),
             callback='get_song', follow=True),       
            Rule(LinkExtractor(allow=(".*"))),

             ]

    # 獲取歌曲資訊
    def get_song(self,response):

        item = SongspiderItem()
        item['name'] = response.xpath('//*[@id="lrcName"]/text()').extract()[0]
        item['singer'] = response.xpath('//*[@id="musiclrc"]/div[1]/p[2]/span/a/text()').extract()[0]
        # lyric = # 歌詞
        lyrics = response.xpath('//*[@id="llrcId"]/p')

        lyricstr = item['name']
        for lyric in lyrics:
            lyricstr += '\n'
            lyricstr += lyric.xpath('text()').extract()[0]
        item['lyric'] = lyricstr

        item['url'] = response.url

        item['originId'] = 2

        # with open(r'D:\Learn\pythonPro\qianfeng\MusicSearch\songSpider\songs.csv', 'a', newline='',encoding='utf-8')  as f:
        #     writer = csv.writer(f)
        #     # 按行寫入
        #     writer.writerow([item['name'],item['singer'],item['lyric'],item['url'],])
        return item

六、在專案目錄下新增start.py

主要使用來啟動爬蟲。

import scrapy.cmdline

def main():
    scrapy.cmdline.execute(['scrapy','crawl','kuwo_spider'])

if __name__ == '__main__':
    main()

七、使用Redis進行分散式爬蟲

1、修改kuwo_spider.py

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from songSpider.items import SongspiderItem
from scrapy_redis.spiders import RedisCrawlSpider

# 使用Redis爬取
class KuwoSpider(RedisCrawlSpider):
    name = 'kuwo_spider'
    allowed_domains = ['kuwo.cn']
    # 增加一個redis_key
    redis_key = "kuwo:start_url"
    # 設定爬取規則,可迭代物件，可設定多個規則
    rules = [
            Rule(LinkExtractor(allow=(
                                  "http://www.kuwo.cn/yinyue/(\d+)")),
             callback='get_song', follow=True),        
            Rule(LinkExtractor(allow=(".*"))),        
             ]

    # 獲取歌曲資訊
    def get_song(self,response):
        item = SongspiderItem()
        item['name'] = response.xpath('//*[@id="lrcName"]/text()').extract()[0]
        item['singer'] = response.xpath('//*[@id="musiclrc"]/div[1]/p[2]/span/a/text()').extract()[0]
        # lyric = # 歌詞
        lyrics = response.xpath('//*[@id="llrcId"]/p')

        lyricstr = item['name']
        for lyric in lyrics:
            lyricstr += '\n'
            lyricstr += lyric.xpath('text()').extract()[0]
        item['lyric'] = lyricstr
        item['url'] = response.url
        item['originId'] = 2

        return item

2、修改settings.py

ITEM_PIPELINES = {
   'songSpider.pipelines.SongspiderPipeline': 300,
   'scrapy_redis.pipelines.RedisPipeline': 400, # 通向redis
}

# 分散式配置
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST = True

REDIS_URL = "redis://:[email protected]:6379"

3、在CMD命令終端輸入

開啟redis服務。

然後在另一個cmd視窗輸入：

redis-cli
auth 123456   
lpush kuwo:start_url http://yinyue.kuwo.cn/yy/category.htm

4、在PyCharm中的Terminal中輸入：

scrapy runspider kuwo_spider.py

5、實現分散式爬取

把這個程式在另外一臺電腦執行，redis連線主機的伺服器，啟動爬蟲即可。

具體操作為：

5.1 主機設定

在主機上啟動redis服務：

redis-server redis.conf

開啟CMD視窗輸入redis_key:

redis-cli
auth 123456   
lpush kuwo:start_url http://yinyue.kuwo.cn/yy/category.htm

在主機上啟動爬蟲：

scrapy runspider kuwo_spider.py

5.2 從機設定(可以有多個從機)

在從機上連線主機redis伺服器：

# 在CMD視窗中輸入
# 連線註解reids伺服器
# 我當前的主機ip為10.3.141.228  埠為6379 密碼為123456
redis-cli -h 10.3.141.228 -p 6379 -a 123456

開啟CMD視窗輸入redis_key:

redis-cli
auth 123456   
lpush kuwo:start_url http://www.kuwo.cn/mingxing

不同的從機，kuwo:start_url的值可以設定不同的連結。

在從機上啟動爬蟲：

scrapy runspider kuwo_spider.py

八、使用IP代理池

1、修改settings.py，增加

DOWNLOADER_MIDDLEWARES = {
     'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware':None,
     'songSpider.middlewares.ProxyMiddleWare':125,
     'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware':None
}

2、修改middlewares.py，增加

from scrapy import signals
import random
import scrapy
from scrapy import log
import pymysql

# ip代理池
class ProxyMiddleWare(object):
    """docstring for ProxyMiddleWare"""

    def process_request(self,request, spider):
        '''對request物件加上proxy'''
        proxy = self.get_random_proxy()
        print("this is request ip:"+proxy)
        request.meta['proxy'] = proxy


    def process_response(self, request, response, spider):
        '''對返回的response處理'''
        # 如果返回的response狀態不是200，重新生成當前request物件
        if response.status != 200:
            proxy = self.get_random_proxy()
            print("this is response ip:"+proxy)
            # 對當前reque加上代理
            request.meta['proxy'] = proxy
            return request
        return response

    def get_random_proxy(self):
        '''隨機從IP資料庫中讀取proxy'''
        # 連線資料庫
        con = pymysql.connect(
            host='localhost',  # 資料庫所在地址URL
            user='root',  # 使用者名稱
            password='123456',  # 密碼
            database='proxypool',  # 資料庫名稱
            port=3306,  # 埠號
            charset='utf8'
        )
        # 拿到查詢遊標
        cursor = con.cursor()
        #獲取ip
        ip_ports = cursor.execute('select ip,ip_port from proxypool ')
        if ip_ports:
            # 從遊標中取出查詢結果
            ip_ports = cursor.fetchall()

        # 斷開連線
        cursor.close()
        con.close()
        proxies = []
        for ip,port in ip_ports:
            proxies.append("http://"+ip+':'+port)

        proxy = random.choice(proxies)
        return proxy