shell bash xargs 多进程处理文本文件

版权声明:本文为博主可克技术原创文章,未经博主允许不得转载。 https://blog.csdn.net/samxx8/article/details/85281333

实现思路:

使用xargs的-P 参数实现多进程运行curl处理数据;


cat ids.txt | xargs -P 10 -I X curl -sL -w "%{http_code} X\n" https://xxxxx.com/stock_news/X -I -o /dev/null >> idss.txt

 

查看进程数:

[songam@java-prd-web01-165-205 ~]$ ps aux | grep curl 

songam   11401  0.4  0.0 101368   680 pts/0    S+   14:31   0:00 xargs -P 10 -I X curl -sL -w %{http_code} X\n https://xxxxx.com/stock_news/X -I -o /dev/null

songam   14796  9.0  0.0 187996  5904 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308333\n https://xxxxx.com/stock_news/308333 -I -o /dev/null

songam   14799  8.0  0.0 187996  5908 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308334\n https://xxxxx.com/stock_news/308334 -I -o /dev/null

songam   14802  8.0  0.0 187996  5908 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308336\n https://xxxxx.com/stock_news/308336 -I -o /dev/null

songam   14805  8.0  0.0 187996  5904 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308337\n https://xxxxx.com/stock_news/308337 -I -o /dev/null

songam   14808  7.0  0.0 187996  5628 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308338\n https://xxxxx.com/stock_news/308338 -I -o /dev/null

songam   14811  4.0  0.0 182408  3392 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308340\n https://xxxxx.com/stock_news/308340 -I -o /dev/null

songam   14814  6.0  0.0 187944  5336 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308341\n https://xxxxx.com/stock_news/308341 -I -o /dev/null

songam   14815  5.0  0.0 182412  3392 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308344\n https://xxxxx.com/stock_news/308344 -I -o /dev/null

songam   14820  2.0  0.0 182404  3272 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308346\n https://xxxxx.com/stock_news/308346 -I -o /dev/null

songam   14821  2.0  0.0 180216  3048 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308347\n https://xxxxx.com/stock_news/308347 -I -o /dev/null

songam   14827  0.0  0.0 103348   872 pts/1    S+   14:31   0:00 grep curl

 

没有更多推荐了,返回首页

私密
私密原因:
请选择设置私密原因
  • 广告
  • 抄袭
  • 版权
  • 政治
  • 色情
  • 无意义
  • 其他
其他原因:
120
出错啦
系统繁忙,请稍后再试

关闭