关于settings.py文件一开始的处理
关于是否遵守robots.txt协议
代码:
# Obey robots.txt rules
ROBOTSTXT_OBEY = True//遵守
# Obey robots.txt rules
ROBOTSTXT_OBEY = True//不遵守
# Obey robots.txt rules
ROBOTSTXT_OBEY = True//遵守
# Obey robots.txt rules
ROBOTSTXT_OBEY = True//不遵守