利用wget批量下载http目录下文件

因为网络特殊,连不上互联网,只好自己制作一个puppet本地源。因为自己的电脑是是可以连互联网的,所以就直接将puppetlabs.repo导入进来了

rpm -Uvh http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-7.noarch.rpm

进入/etc/yum.repos.d/看以看到puppetlabs.repo.


注意:因为我是X86-64的系统【在终端中输入arch命令可以知道$basearch】的值

进入http://yum.puppetlabs.com/el/6.0/products/x86_64/我们可以看到许多的包:要是用wget命令逐个包下载,繁琐程度难以想象。所以这时候就必须利用wget批量下载:

wget -nd -r -l1  -A.rpm --no-parent http://yum.puppetlabs.com/el/6.0/products/x86_64/




---------------------------------------------------------------------------------------------------------------------------------------------------------

使用 wget 完成批量下载
如果想下载一个网站上目录中的所有文件, 我需要执行一长串wget命令, 但这样做会更好:
wget -nd -r -l1 --no-parent http://www.foo.com/mp3/
这条命令可以执行的很好, 但有时会下载像 index.@xx 这样一些我不想要的文件. 如果你知道想要文件的格式, 可以用下面的命令来避免下载那些多余的文件:
wget -nd -r -l1 --no-parent -A.mp3 -A.wma http://www.foo.com/mp3/
我来简单的介绍一下命令中指定选项的作用.
-nd 不创建目录, wget默认会创建一个目录
-r 递归下载
-l1 (L one) 递归一层,只下载指定文件夹中的内容, 不下载下一级目录中的.
–no-parent 不下载父目录中的文件

转载地址:http://article.yeeyan.org/view/30993/19075

补充:http://yhj1065.blog.163.com/blog/static/19800217201392393033224/

---------------------------------------------------------------------------------------------------------------------------------------------------------


有时候下载整个目录下的包也不现实,那么可以吧自己需要的包的URL地址写入到一个文件中去。如我再../lixw/seeds.txt放入了一些URL地址,那么在该路径下,只要将使用命令也可以批量下载:

wget -i seeds.txt 




展开阅读全文

HTTP服务器和客户端下载多个文件

02-26
<div class="post-text" itemprop="text"> <p>I have tried setting up a download server and a download client for individual files. How can I modify them to serve/download all the files from a directory?</p> <p>Following are my server and client codes:</p> <pre><code>//server.go func main() { http.HandleFunc("/dlpath", handle) err := http.ListenAndServe(":10001", nil) if err != nil { log.Fatal("ListenAndServe: ", err) } } func handle(writer http.ResponseWriter, r *http.Request) { filename := "C:\\Users\\aarvi\\GolandProjects\\src\\Practice\\download\\serve\\send.txt" http.ServeFile(writer, r, filename) } //client.go func main() { downloadFile("res_out.txt", "http://localhost:10001/dlpath") } func downloadFile(dirname string, url string) error { // Create the file out, err := os.OpenFile(dirname, os.O_WRONLY | os.O_CREATE | os.O_APPEND, 0666) if err != nil { fmt.Println(err) } defer out.Close() // get data request, err := http.NewRequest("GET", url, nil) if err != nil { fmt.Println(err) } client := http.Client{} resp, err := client.Do(request) if err != nil { return err } defer resp.Body.Close() // Write the body to file _, err = io.Copy(out, resp.Body) if err != nil { return err } return nil } </code></pre> <p>I tried serving the directory in the handle function like so:</p> <pre><code>dirname := "C:\\Users\\aarvi\\GolandProjects\\src\\Practice\\download\\serve" http.FileServer(http.Dir(dirname)) </code></pre> <p>and tried to print out the response on the client side, but I got nothing. How can I serve all the files from the /serve directory, and download them in the client?</p> <p>EDIT: Following are the contents of the serve directory:</p> <p><code>serve ---sample.txt ---send.txt ---dir2 ------abc.txt</code></p> <p>How can I download all these files on the client side as separate files, with the directory structure intact?</p> <p><strong>Update:</strong> When I call the http.Handle function (as mentioned in the answer) directly in the main function, I am able to serve all the files, and the file within the inner directory too.</p> <p>However, when I call the same within the handle function, it doesn't serve anything. I am guessing this has something to do with the path? </p> </div>
©️2020 CSDN 皮肤主题: 大白 设计师: CSDN官方博客 返回首页
实付0元
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值