java8 Stream数据去重的方法

Java8 版本添加了流Stream操作特性，大幅度提升了代码书写的简单性，在工作的中我们遇到需要对数据去重的需求。比如说，读取Excel导入数据库，在导入之前会对数据进行合规性检查，重复数据处理。

1. java8 的Stream中提供了distinct方法，可以用来对数据进行处理，需要注意的是，该方法依赖对象的equals和hashCode方法。对自定义类需要手动实现这两个方法。

Stream(XXXXXX).distinct();
Stream("1","1","2","3").distinct();//1,2,3

2. 如果不方便去重写equals和hashCode方法，有没有方法去实现去重功能呢,答案是肯定的，需要两步：

第一步，定义一个Predicate返回值方法

public static <T> Predicate<T> distinctByKey(Function<? super T, Object> keyExtractor)
{
    Map<Object, Boolean> map = new ConcurrentHashMap<>();
    return t -> map.putIfAbsent(keyExtractor.apply(t), Boolean.TRUE) == null;
}

第二步，结合Stream的filter方法

mport java.util.Arrays;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Function;
import java.util.function.Predicate;
import java.util.stream.Collectors;
 
public class JavaStreamDistinctExamples
{
    public static void main(String[] args)
    {
        Person lokesh = new Person(1, "Lokesh", "Gupta");
        Person brian = new Person(2, "Brian", "Clooney");
        Person alex = new Person(3, "Alex", "Kolen");
         
        //Add some random persons
        Collection<Person> list = Arrays.asList(lokesh,brian,alex,lokesh,brian,lokesh);
 
        // Get distinct objects by key
        List<Person> distinctElements = list.stream()
                                            .filter( distinctByKey(p -> p.getId()) )
                                            .collect( Collectors.toList() );
 
        // Let's verify distinct elements
        System.out.println( distinctElements );
    }
 
        //Utility function
    public static <T> Predicate<T> distinctByKey(Function<? super T, Object> keyExtractor)
    {
        Map<Object, Boolean> map = new ConcurrentHashMap<>();
        return t -> map.putIfAbsent(keyExtractor.apply(t), Boolean.TRUE) == null;
    }
}
 
//Model class
class Person
{
    public Person(Integer id, String fname, String lname) {
        super();
        this.id = id;
        this.fname = fname;
        this.lname = lname;
    }
 
    private Integer id;
    private String fname;
    private String lname;
 
    //Getters and Setters
 
    @Override
    public String toString() {
        return "Person [id=" + id + ", fname=" + fname + ", lname=" + lname + "]";
    }
}

以上代码片段来自https://howtodoinjava.com/java8/java-stream-distinct-examples/