shell bash xargs 多进程处理文本文件

实现思路：

使用xargs的-P 参数实现多进程运行curl处理数据；

cat ids.txt | xargs -P 10 -I X curl -sL -w "%{http_code} X\n" https://xxxxx.com/stock_news/X -I -o /dev/null >> idss.txt

 

查看进程数：

[songam@java-prd-web01-165-205 ~]$ ps aux | grep curl 

songam   11401  0.4  0.0 101368   680 pts/0    S+   14:31   0:00 xargs -P 10 -I X curl -sL -w %{http_code} X\n https://xxxxx.com/stock_news/X -I -o /dev/null

songam   14796  9.0  0.0 187996  5904 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308333\n https://xxxxx.com/stock_news/308333 -I -o /dev/null

songam   14799  8.0  0.0 187996  5908 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308334\n https://xxxxx.com/stock_news/308334 -I -o /dev/null

songam   14802  8.0  0.0 187996  5908 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308336\n https://xxxxx.com/stock_news/308336 -I -o /dev/null

songam   14805  8.0  0.0 187996  5904 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308337\n https://xxxxx.com/stock_news/308337 -I -o /dev/null

songam   14808  7.0  0.0 187996  5628 pts/0    S+   14:31   0:00 curl -sL -w %{http_code} 308338\n https://xxxxx.com/stock_news/308338 -I -o /dev/null

songam   14811  4.0  0.0 182408  3392 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308340\n https://xxxxx.com/stock_news/308340 -I -o /dev/null

songam   14814  6.0  0.0 187944  5336 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308341\n https://xxxxx.com/stock_news/308341 -I -o /dev/null

songam   14815  5.0  0.0 182412  3392 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308344\n https://xxxxx.com/stock_news/308344 -I -o /dev/null

songam   14820  2.0  0.0 182404  3272 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308346\n https://xxxxx.com/stock_news/308346 -I -o /dev/null

songam   14821  2.0  0.0 180216  3048 pts/0    R+   14:31   0:00 curl -sL -w %{http_code} 308347\n https://xxxxx.com/stock_news/308347 -I -o /dev/null

songam   14827  0.0  0.0 103348   872 pts/1    S+   14:31   0:00 grep curl