pandas dataframe去除重复数据pandas.DataFrame.drop_duplicates

例子：

        df2 = pd.DataFrame({'工参中没有的cgi': self.mismatchedcgis})
        # subset='工参中没有的cgi' 表示只考虑列名为：工参中没有的cgi 这一列的重复项，不设则需考虑全部列，也可以设成多列
        # inplace=True是直接在df2表中删除重复项，如果设成inplace=False则不修改原表df2，而是返回去重后的新表
        df2.drop_duplicates(subset='工参中没有的cgi', inplace=True)

官方文档：

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html#pandas.DataFrame.drop_duplicates

 

DataFrame.drop_duplicates(subset=None, keep='first', inplace=False)[source]

Return DataFrame with duplicate rows removed, optionally only considering certain columns

Parameters:

subset : column label or sequence of labels, optional Only consider certain columns for identifying duplicates, by default use all of the columns keep : {‘first’, ‘last’, False}, default ‘first’ first : Drop duplicates except for the first occurrence. last : Drop duplicates except for the last occurrence. False : Drop all duplicates. inplace : boolean, default False Whether to drop duplicates in place or to return a copy

Returns:

deduplicated : DataFrame