Linux下爬蟲環境的配置
阿新 • • 發佈:2019-01-28
sudo apt-get install python3.6-dev
4.安裝redis (Mac上已經安裝好了redis)
1)安裝一些庫
pip install redis
pip install scrapy-redis(分散式爬蟲需要使用此庫)
準備讓Mac上的Redis作Master,Ubuntu上的Redis作slave,需要修改redis.conf,但找不到此配置檔案,於是重新下載,sudo apt-get install redis,找到配置檔案中的 bind 127.0.0.1,註釋掉
2)測試連通性
通過配置檔案啟動Mac上的redis: redis-server /etc/redis/redis.conf redis-cli ,Mac IP為192.168.2.123
開啟Ubuntu上的redis:
redis-server /etc/redis/redis.conf
redis-cli -h 192.168.2.123
測試連通性,在Ubuntu上輸入 keys * ,報錯 Redis is running in protected mode because protected mode is enabled,no bind address was specified,no authentication password is requested to clients
4.安裝redis (Mac上已經安裝好了redis)
1)安裝一些庫
pip install redis
pip install scrapy-redis(分散式爬蟲需要使用此庫)
準備讓Mac上的Redis作Master,Ubuntu上的Redis作slave,需要修改redis.conf,但找不到此配置檔案,於是重新下載,sudo apt-get install redis,找到配置檔案中的 bind 127.0.0.1,註釋掉
2)測試連通性
通過配置檔案啟動Mac上的redis: redis-server /etc/redis/redis.conf redis-cli ,Mac IP為192.168.2.123
開啟Ubuntu上的redis:
redis-server /etc/redis/redis.conf
redis-cli -h 192.168.2.123
測試連通性,在Ubuntu上輸入 keys * ,報錯 Redis is running in protected mode because protected mode is enabled,no bind address was specified,no authentication password is requested to clients