2024-03-15 21:38:44 [scrapy.robotstxt] WARNING: Failure while parsing robots.txt. File either contains garbage or is in an encoding other than UTF-8, treating it as an empty file.
Traceback (most recent call last):
File "c:\users\卡尔\mysoftward\python3.7.9\lib\site-packages\scrapy\robotstxt.py", line 15, in decode_robotstxt
robotstxt_body = robotstxt_body.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcc in position 62: invalid continuation byte
似乎是因为scrapy默认遵守robots协议,发生了错误
在settings.py中拒绝遵守robots协议即可
# Obey robots.txt rules
ROBOTSTXT_OBEY = False