python scrapy 儲存資料到mysql的坑(1064錯誤)
阿新 • • 發佈:2018-11-16
在將scrapy中的資料儲存到mysql中,出現部分資料插入不進去的問題, 一直報:
You have an error in your SQL syntax;
mysql相對於mongo來說,對資料要嚴格些,有時候,很多SQL
語句寫對了,還是插不進去資料。針對上面的問題,我使用了一個pymysql中的方法file_count = pymysql.escape_string(item['file_content'])
,這樣就成功的把資料插入到MySQL中了。
下面貼一下全部的程式碼:
class MysqlPipeline(object):
def __init__ (self):
self.host = settings.DB_HOST
self.port = settings.DB_PORT
self.user = settings.DB_USER
self.pwd = settings.DB_PWD
self.name = settings.DB_NAME
self.charset = settings.DB_CHARSET
self.connect()
def connect(self):
self.conn = pymysql.connect(host=self.host,
port=self.port,
user=self.user,
password=self.pwd,
db=self.name,
charset=self.charset)
self. cursor = self.conn.cursor()
def close_spider(self, spider):
self.conn.close()
self.cursor.close()
def process_item(self, item, spider):
try:
print('在插入 *************************************** ')
# sql = 'insert into zb_contest_data(craw_date, cd_filename, cd_content, create_time) values("%s", "%s", "%s", "%s")' % (
# item['craw_date'], item['file_name'], str(item['file_content']), item['create_time'])
file_count = pymysql.escape_string(item['file_content'])
sql = 'insert into zb_contest_data(craw_date, cd_filename, cd_content, create_time) values("%s", "%s", "%s", "%s")' % (
item['craw_date'], item['file_name'], file_count, item['create_time'])
# sql = 'insert into zb_contest_data(craw_date, cd_filename, cd_content, create_time) values("%s", "%s", "%s", "%s")' % (
# item['craw_date'], item['file_name'], item['file_content'], item['create_time'])
# 執行sql語句
self.cursor.execute(sql)
self.conn.commit()
pass
except Exception as e:
print(e)
pass
return item
個人微信:hll643435675(備註:部落格)
更多資源請訪問:
https://blog.csdn.net/xudailong_blog/article/details/78762262
慕課視訊教程:https://blog.csdn.net/xudailong_blog/article/details/82909611