robots.txt

最新推荐文章于 2024-03-06 16:43:55 发布

易心意

最新推荐文章于 2024-03-06 16:43:55 发布

阅读量543

点赞数

分类专栏： python

本文链接：https://blog.csdn.net/qq_39984050/article/details/106768148

版权

python 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

什么网站可以爬，不违法。。遵循规则

https://www.taobao.com/robots.txt

User-agent: Baiduspider
Disallow: / 不允许

https://nt.zu.ke.com/robots.txt

User-agent: Baiduspider
Allow: ?utm_source=office 允许

User-agent: 360Spider
Allow: ?utm_source=360hz

User-agent: haosouspider
Allow: ?utm_source=360hz
Allow: ?utm_source=360fj

User-agent: Sogou News Spider
Allow: ?utm_source=sgzf
Allow: ?utm_source=sglz

User-agent: Sosospider
Allow: ?utm_source=sgzf
Allow: ?utm_source=sglz

User-agent: *
Sitemap:http://nt.zu.ke.com/sitemap/nt_zufang1.xml
Disallow: ?utm_source=office 不允许
Disallow: ?utm_source=sgzf
Disallow: ?utm_source=sglz
Disallow: ?utm_source=360hz
Disallow: ?utm_source=360fj
Disallow: /rs
Disallow: /user
Disallow: /login
Disallow: /userinfo
Disallow: /solr
Disallow: /test
Disallow: /tools
Disallow: /exchange
Disallow: /agent
Disallow: /owner
Disallow: /center
Disallow: /cmsmanage
Disallow: /compatible
Disallow: /admin
Disallow: /install
Disallow: /zufang/haiwai/
Disallow: /zufang/shanghaizhoubian/
Disallow: ?project_name=
Disallow: ?id=
Disallow: ?house_codes=
Disallow: ?sug=
Disallow: ?nav=