例如,假设我要计算10个BIG文件的行数并打印总数.
for f in files
do
#this creates a background process for each file
wc -l $f | awk '{print $1}' &
done
我正在尝试类似的东西:
for f in files
do
#this does not work :/
n=$( expr $(wc -l $f | awk '{print $1}') + $n ) &
done
echo $n
解决方法:
您可能应该使用gnu parallel:
find . -maxdepth 1 -type f | parallel --gnu 'wc -l' | awk 'BEGIN {n=0} {n += $1} END {print n}'
否则xargs处于并行模式:
find . -maxdepth 1 -type f | xargs -n1 -P4 wc -l | awk 'BEGIN {n=0} {n += $1} END {print n}'
如果这不符合您的需求,另一种选择是写入临时文件.如果您不想写入磁盘,只需写入/ dev / shm.在大多数Linux系统上,这是一个虚拟磁盘.
#!/bin/bash
declare -a temp_files
count=0
for f in *
do
if [[ -f "$f" ]]; then
temp_files[$count]="$(mktemp /dev/shm/${f}-XXXXXX)"
((count++))
fi
done
count=0
for f in *
do
if [[ -f "$f" ]]; then
cat "$f" | wc -l > "${temp_files[$count]}" &
((count++))
fi
done
wait
cat "${temp_files[@]}" | awk 'BEGIN {n=0} {n += $1} END {print n}'
for tf in "${temp_files[@]}"
do
rm "$tf"
done
顺便说一句,这可以通过map-reduce来实现,其中wc进行映射,awk进行归约.
标签:shell,command-line,linux
来源: https://codeday.me/bug/20191030/1968828.html