scrapy设置随机请求头
为了更好的伪装浏览器,不被干掉,所以我们的设置随机请求头,免得被服务器发现干掉
中间件
# 好用的请求头网站 import random class UserAgentDownloadMiddleWare(object): USER_AGENTS = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36', 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/44.0.2403.155 Safari/537.36', 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; pl-PL; rv:1.0.1) Gecko/20021111 Chimera/0.6', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36', 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/418.8 (KHTML, like Gecko, Safari) Cheshire/1.0.UNOFFICIAL', 'Mozilla/5.0 (X11; U; Linux i686; nl; rv:1.8.1b2) Gecko/20060821 BonEcho/2.0b2 (Debian-1.99+2.0b2+dfsg-1)' ] def process_request(self, request, spider): user_agent = random.choice(self.USER_AGENTS) request.headers['User-Agent'] = user_agent
然后setting里面开启随机请求头中间件
DOWNLOADER_MIDDLEWARES = { 'bmw.middlewares.UserAgentDownloadMiddleWare': 543, }
- 版权申明:此文如未标注转载均为本站原创,自由转载请表明出处《龙行博客》。
- 本文网址:https://www.liaotaoo.cn/289.html
- 上篇文章:scrapy设置ip代理池
- 下篇文章:scrapy发送post请求