linux命令之-uniq

uniq命令

uniq命令用于检查及删除文本文件中重复出现的行列。

[root@iZwz9bhan5nqzh979qokrkZ ~]# uniq --help
用法：uniq [选项]... [文件]

-c, --count           prefix lines by the number of occurrences   统计连续重复行的次数
-d, --repeated        only print duplicate lines, one for each group   只输出重复的行
-D, --all-repeated[=METHOD]  print all duplicate lines    只输出重复的行，不过是有几行就输出几行
-f, --skip-fields=N   avoid comparing the first N fields   f 忽略的段数，-f 1 忽略第一段
-i, --ignore-case     ignore differences in case when comparing  不区分大小写
-s, --skip-chars=N    avoid comparing the first N characters  跟-f有点像，不过-s是忽略，后面多少个字符 -s 5表示忽略后面5个字符
-u, --unique          only print unique lines  去除重复的行，其他全部打印出来
-z, --zero-terminated  end lines with 0 byte, not newline  不知道有啥用，没有过
-w, --check-chars=N 对每行第N 个字符以后的内容不作对照

总结：sort -u file 或 cat file | sort -u ，即可实现去重输出。

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt
Whatever is worth doing is worth doing well.
Whatever is worth doing is worth doing well.
Whatever is worth doing is worth doing well.
In love folly is always sweet.
In love folly is always sweet.
The hard part isn’t making the decision. It’s living with it.
Your happy passer-by all knows, my distressed there is no place hides.
I’ll think of you every step of the way.
I’ll think of you every step of the way.
Your happy passer-by all knows, my distressed there is no place hides.

实例一：重复的行只打印一次，其他行的内容照样打印
 

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt | uniq
Whatever is worth doing is worth doing well.
In love folly is always sweet.
The hard part isn’t making the decision. It’s living with it.
Your happy passer-by all knows, my distressed there is no place hides.
I’ll think of you every step of the way.
Your happy passer-by all knows, my distressed there is no place hides.

实例二：打印a.txt文件中重复出现的行及行数(-c参数)

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt | uniq -c
3 Whatever is worth doing is worth doing well.
2 In love folly is always sweet.
1 The hard part isn’t making the decision. It’s living with it.
1 Your happy passer-by all knows, my distressed there is no place hides.
2 I’ll think of you every step of the way.
1 Your happy passer-by all knows, my distressed there is no place hides.

实例三：只打印重复出现的行，且只打印一次(-d参数)

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt | uniq -d
Whatever is worth doing is worth doing well.
In love folly is always sweet.
I’ll think of you every step of the way.

实例四：打印所有重复出现的行，有多少行就打印多少行(-D参数)

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt | uniq -D
Whatever is worth doing is worth doing well.
Whatever is worth doing is worth doing well.
Whatever is worth doing is worth doing well.
In love folly is always sweet.
In love folly is always sweet.
I’ll think of you every step of the way.
I’ll think of you every step of the way.

实例五：去掉重复的行，打印其他没有重复的行(-u参数)

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt | uniq -u
The hard part isn’t making the decision. It’s living with it.
Your happy passer-by all knows, my distressed there is no place hides.
Your happy passer-by all knows, my distressed there is no place hides.

实例五：忽略大小写(-i参数)

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt
Whatever is worth doing is worth doing well. 一个大写
Whatever is worth doing is worth doing well. 一个大写
whatever is worth doing is worth doing well. 一个小写
in love folly is always sweet.  一个小写
In love folly is always sweet.  一个大写
The hard part isn’t making the decision. It’s living with it.
Your happy passer-by all knows, my distressed there is no place hides.
I’ll think of you every step of the way.
I’ll think of you every step of the way.
Your happy passer-by all knows, my distressed there is no place hides.

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt | uniq -c  #打印连续重复的行及行数，主要是验证下面的-i参数
1 whatever is worth doing is worth doing well.
2 Whatever is worth doing is worth doing well.
1 in love folly is always sweet.
1 In love folly is always sweet.
1 The hard part isn’t making the decision. It’s living with it.
1 Your happy passer-by all knows, my distressed there is no place hides.
2 I’ll think of you every step of the way.
1 Your happy passer-by all knows, my distressed there is no place hides.

注：在统计连续重复的行及行数时，大写和小写是区分的，-i参数则可以解决这个大小写区分的问题。

[root@iZwz9bhan5nqzh979qokrkZ ~]# cat a.txt | uniq -c -i  
3 whatever is worth doing is worth doing well.   一个小写两个大写
2 in love folly is always sweet.  一个小写一个大写
1 The hard part isn’t making the decision. It’s living with it.
1 Your happy passer-by all knows, my distressed there is no place hides.
2 I’ll think of you every step of the way.
1 Your happy passer-by all knows, my distressed there is no place hides.

总结：在这里我只总结常用的参数，像一些不常用的就没有去花时间去总结！

 

 

---------------------
作者：东城绝神
原文：https://blog.csdn.net/m0_37814112/article/details/80492094