1. 在shell中可以利用uniq
sort unsorted.txt | uniq
注意:要先进行排序,才能调用uniq (uniq命令,只是把相邻的重复的数据行去掉)
缺点:会打乱原有文件的内容
2.利用Perl 实现
2.1 去除文件重複的行
unless (open FILE,"<D:/test.txt") {die "Can't open file:$!"}
unless (open OUT,">D:/test_uniq.txt") {die "Can't open file:$!"}
chomp(@temp=<FILE>);
my %seen;
foreach $t (@temp){
if($seen{$t}++){}
else{
print OUT "$t \n";
}
}
2.2 去除數組中的重複元素
%seen;
my @uniq_list = grep{ ! $seen{$_}++} @src_list;