wget -r -p -np -k http://www.freebuf.com/
忽视,避开robots.txt,加一个-
e robots
=
off
用wget避开robots.txt的下载限制
wget -r -p -np -k -e robots=off http://www.baidu.com/
参考网站
http://blog.csdn.net/sdlyhjq/article/details/8596303
http://blog.csdn.net/zhouxinyu1cp/article/details/72758588
http://blog.csdn.net/doupeihua/article/details/51860504