sogou spider 抓取网站robots.txt 400问题?

首先,我要说,网站正常访问是没问题的。而且,百度,360 spider都访问ok。
但sogou站长工具测试没问题,后台日志显示,抓取的时候,就是400.
不过,确实看不出来400的与其他有什么差别。由于采用了https访问,所以,做了301转向调整。另外 panjishengwu.com 转向了www.panjisheng.com的转向跳转。都是301.
浏览器测试都是正常。

对于400错误,我一定办法没有。而且,只有这个文件是400.
但这个文件影响了我的收录。 我调整域名,调整nginx的robots.txt配置,都无用。
请求根本到不了后端。到目前为止问题依然没有解决。看到的200状态,都是我利用sogou的站长工具测试的。测试是没有问题的。
看了一些文章,有说是域名不对。我域名设置为所有。针对非我域名做301跳转。
但我收到还是400.

也有说是客户端问题。那这个我就无法验证了。具体怎么回事,如果有大拿清楚原因,还请赐教。

123.126.113.90 - - [02/Mar/2019:15:21:30 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.73 - - [02/Mar/2019:19:22:31 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [29/Jan/2019:08:58:34 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.82 - - [29/Jan/2019:23:21:35 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.82 - - [29/Jan/2019:23:22:57 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.90 - - [30/Jan/2019:00:21:36 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.90 - - [30/Jan/2019:00:21:51 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.131 - - [30/Jan/2019:03:22:19 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.131 - - [30/Jan/2019:03:24:23 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.158 - - [30/Jan/2019:07:20:45 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.158 - - [30/Jan/2019:07:21:54 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [30/Jan/2019:08:54:13 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [30/Jan/2019:21:06:15 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.132 - - [31/Jan/2019:00:22:14 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.132 - - [31/Jan/2019:00:22:14 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.38.241.121 - - [31/Jan/2019:03:24:24 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [31/Jan/2019:21:26:51 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [01/Feb/2019:09:14:04 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.133 - - [02/Feb/2019:00:17:31 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.133 - - [02/Feb/2019:00:18:01 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
111.202.101.250 - - [02/Feb/2019:08:30:09 +0800] “GET /robots.txt HTTP/1.1” 200 21 “-” “Mozilla/5.0 (Linux; Android 6.0.1) AppleWebKit/601.1 (KHTML,like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Sogou web spider/4.0; +http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [02/Feb/2019:09:37:55 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [02/Feb/2019:22:18:44 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
218.30.103.29 - - [03/Feb/2019:03:23:56 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [03/Feb/2019:17:53:33 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.104 - - [04/Feb/2019:00:22:41 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.104 - - [04/Feb/2019:00:23:02 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.38.241.111 - - [04/Feb/2019:03:21:36 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [04/Feb/2019:11:54:26 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 8
    评论
评论 8
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值