awk牛刀小试

1+收件人唯一：awk -F, '{if($2!=second) print;second=$2}' notsns_other_mails.txt

 

2+投诉去重：awk -F, 'NR==FNR{list[$1]=$1}NR>FNR{if(!($1 in list))print $0}' delete.list notsns_other_mails.txt

 

3+数据散列: awk 'BEGIN{srand()}{b[rand()NR]=$0}END{for(x in b)print b[x]}' notsns_other_mails.txt

 

4+数据分块：awk -F, 'NR==FNR{list[$1]=$1}NR>FNR{if(!($1 in list))print $0}' delete.list notsns_other_mails.txt | awk '{print $0 > "tmp"int(NR/4)}'