爬虫3:如何反爬虫和控制爬虫的速度-setting设置
1.修改是否遵守爬虫协议为False# Obey robots.txt rulesROBOTSTXT_OBEY = False2.修改并发请求数,修改为1,或者2,越小爬取速度越慢,太快容易被识别到# Configure maximum concurrent requests performed by Scrapy (default: 16)CONCURRENT_REQUE...