什么网站可以爬,不违法。。遵循规则
https://www.taobao.com/robots.txt
User-agent: Baiduspider
Disallow: / 不允许
https://nt.zu.ke.com/robots.txt
User-agent: Baiduspider
Allow: ?utm_source=office 允许
User-agent: 360Spider
Allow: ?utm_source=360hz
User-agent: haosouspider
Allow: ?utm_source=360hz
Allow: ?utm_source=360fj
User-agent: Sogou News Spider
Allow: ?utm_source=sgzf
Allow: ?utm_source=sglz
User-agent: Sosospider
Allow: ?utm_source=sgzf
Allow: ?utm_source=sglz
User-agent: *
Sitemap:http://nt.zu.ke.com/sitemap/nt_zufang1.xml
Disallow: ?utm_source=office 不允许
Disallow: ?utm_source=sgzf
Disallow: ?utm_source=sglz
Disallow: ?utm_source=360hz
Disallow: ?utm_source=360fj
Disallow: /rs
Disallow: /user
Disallow: /login
Disallow: /userinfo
Disallow: /solr
Disallow: /test
Disallow: /tools
Disallow: /exchange
Disallow: /agent
Disallow: /owner
Disallow: /center
Disallow: /cmsmanage
Disallow: /compatible
Disallow: /admin
Disallow: /install
Disallow: /zufang/haiwai/
Disallow: /zufang/shanghaizhoubian/
Disallow: ?project_name=
Disallow: ?id=
Disallow: ?house_codes=
Disallow: ?sug=
Disallow: ?nav=