List列表去重方法

---

一. 对列表元素为String类的去重

1. 法1：使用Jav8 Stream类的distinct()方法（推荐）

1. 概述
   • distinct()方法使用hashCode()和eqauls()方法获取不同元素。因此，需要去重的类必须实现这两个方法。
   • 因为String类已经重写了这两个方法，因此去重可以直接使用。
2. 实现示例

/**
 * 测试
 */
public void streamDeduplication1() {
    List<String> lists = Lists.newArrayList("a", "b", "f", "a", "z");
    System.out.println("去重前："+JSON.toJSON(lists));
    List<String> deDuplicationList = lists.stream().distinct().collect(Collectors.toList());
    System.out.println("去重后："+JSON.toJSON(deDuplicationList));
}

• 输出结果

去重前：["a","b","f","a","z"]
去重后：["a","b","f","z"]

2. 法2：使用临时List在添加元素时进行重复判断

public void streamDeduplication1_1(){
    List<String> lists = Lists.newArrayList("a", "b", "f", "a", "z");
    System.out.println("去重前："+JSON.toJSON(lists));
    List<String> tempList = Lists.newArrayList();
    for(String str:lists){
        if(!tempList.contains(str)){
            tempList.add(str);
        }
    }
    System.out.println("去重后："+JSON.toJSON(tempList));
}

• 输出结果

去重前：["a","b","f","a","z"]
去重后：["a","b","f","z"]

3. 法3：使用set去重

public void streamDeduplication1_2(){
    List<String> lists = Lists.newArrayList("a", "b", "f", "a", "z");
    System.out.println("去重前："+JSON.toJSON(lists));
    lists = Lists.newArrayList(Sets.newHashSet(lists));
    System.out.println("去重后："+JSON.toJSON(lists));
}

• 输出结果

去重前：["a","b","f","a","z"]
去重后：["a","b","f","z"]

4. 法4：使用Collections.frequency()方法

1. Collections.frequency()：统计每个元素的在集合中的个数。当集合中数据量特别大的时候，性能不好。

public void streamDeduplication1_3(){
   List<String> lists = Lists.newArrayList("a", "b", "f", "a", "z");
   System.out.println("去重前："+JSON.toJSON(lists));
   for(int i=0;i<lists.size();i++){
       String currentValue = lists.get(i);
       if(Collections.frequency(lists,currentValue)>1){
           lists.remove(currentValue);
       }
   }
   System.out.println("去重后："+JSON.toJSON(lists));
}

• 输出结果

去重前：["a","b","f","a","z"]
去重后：["a","b","f","z"]

2. 注意⚠️：在循环过程中删除集合中元素时，只能通过iterator.remove()方法和使用for循环方式，不能使用增强for循环和forEach()方法。

---

二. 对列表元素为实体类的去重

1. 推荐方法：使用Jav8 Stream类的distinct()方法。
   • 对于实体类，可以通过重写其hashCode()和equals()方法来达到去重。
   • 更快速的方法，可以使用Lombok插件的@Data注解，它自动重写了equals()和hashCode()方法。

1. 法1：使用Lombok插件的@Data注解

1. 示例代码

/**
 * 定义一个实体类
 */ 
@Data
public class StudentDTO {
    private Integer studentId;
    private String studentName;

    public StudentDTO(Integer studentId, String studentName) {
        this.studentId = studentId;
        this.studentName = studentName;
    }
}

/**
 * 测试
 */
public void streamDeduplication2() {
        List<StudentDTO> studentDTOS = Lists.newArrayList();
        studentDTOS.add(new StudentDTO(1,"xixi"));
        studentDTOS.add(new StudentDTO(2,"houhou"));
        studentDTOS.add(new StudentDTO(2,"houhou"));
        System.out.println("去重前："+JSON.toJSON(studentDTOS));

        List<StudentDTO> deDupDTOS = studentDTOS.stream().distinct().collect(Collectors.toList());
        System.out.println("去重后："+JSON.toJSON(deDupDTOS));
    }

• 输出结果

去重前：[{"studentId":1,"studentName":"xixi"},{"studentId":2,"studentName":"houhou"},{"studentId":2,"studentName":"houhou"}]
去重后：[{"studentId":1,"studentName":"xixi"},{"studentId":2,"studentName":"houhou"}]

2. 法2：重写equals和hashCode方法

/**
  * 定义实体类
  */
public class StuDTO {
    private Integer studentId;
    private String studentName;

    public StuDTO(Integer studentId, String studentName) {
        this.studentId = studentId;
        this.studentName = studentName;
    }

   // getter and setter

    @Override
    public boolean equals(Object obj) {
        if(this==obj){
            return true;
        }
        if(obj==null || getClass()!=obj.getClass()) {
            return false;
        }
        StuDTO stuDTO = (StuDTO)obj;
        return Objects.equals(getStudentId(),stuDTO.getStudentId()) &&
                Objects.equals(getStudentName(),stuDTO.getStudentName());
    }

    @Override
    public int hashCode() {
        return Objects.hash(getStudentId(),getStudentName());
    }
}

/**
  * 测试
  */
public void streamDeduplication2_1() {
    List<StuDTO> stuDTOS = Lists.newArrayList();
    stuDTOS.add(new StuDTO(1,"xixi"));
    stuDTOS.add(new StuDTO(2,"houhou"));
    stuDTOS.add(new StuDTO(2,"houhou"));
    System.out.println("去重前："+JSON.toJSON(stuDTOS));

    List<StuDTO> deDupDTOS = stuDTOS.stream().distinct().collect(Collectors.toList());
    System.out.println("去重后："+JSON.toJSON(deDupDTOS));
}

• 输出结果

去重前：[{"studentId":1,"studentName":"xixi"},{"studentId":2,"studentName":"houhou"},{"studentId":2,"studentName":"houhou"}]
去重后：[{"studentId":1,"studentName":"xixi"},{"studentId":2,"studentName":"houhou"}]

---

三. 对列表元素为实体类某属性的去重

1. 方法：首先创建一个方法作为 Stream.filter() 的参数，其返回类型为 Predicate，原理就是判断一个元素能否加入到 Set 中，代码如下：

private static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

2. 示例代码

public void streamDeduplication3() {
    List<StudentDTO> studentDTOS = Lists.newArrayList();
    studentDTOS.add(new StudentDTO(1,"xixi"));
    studentDTOS.add(new StudentDTO(2,"houhou"));
    studentDTOS.add(new StudentDTO(3,"houhou"));
    System.out.println("去重前："+JSON.toJSON(studentDTOS));

    studentDTOS = studentDTOS.stream().filter(distinctByKey(StudentDTO::getStudentName)).collect(Collectors.toList());
    System.out.println("去重后："+JSON.toJSON(studentDTOS));
}

• 输出结果

去重前：[{"studentId":1,"studentName":"xixi"},{"studentId":2,"studentName":"houhou"},{"studentId":3,"studentName":"houhou"}]
去重后：[{"studentId":1,"studentName":"xixi"},{"studentId":2,"studentName":"houhou"}]

---

参考资料

• 分享几种 Java8 中通过 Stream 对列表进行去重的方法