关于 ROBOTS.TXT一些看不明白的东东

Other issues with ROBOTS.TXT
These are issues recommended for future standards discussion that could not be resolved within the scope of this workshop.

• Ambiguities in the current specification http://www.kollar.com/robots.html
  A means of canonicalizing sites, using: HTTP-EQUIV HOST ROBOTS.TXT ALIAS
  ways of supporting multiple robots.txt files per site ("robotsN.txt")
  ways of advertising content that should be indexed (rather than just restricting content that should not be indexed)
  Flow control information: retrieval interval or maximum connections open to server