pandas新建dataframe_pandas数据处理

1.介绍几个DataFrame的小功能

检查数据类型 df.dtypes

df=DataFrame({'name':['zs,','ls','ww','zl'],'score':[19,29,39,20],'date':['2019-10-10','2019-10-11','2019-10-12','2019-10-13']})df.dtypesname objectscore int64date objectdtype: object

将'date'这行数据转换为时间数据类型

df.date=pd.to_datetime(df.date)df.dtypesname objectscore int64date datetime64[ns]dtype: object

将'date'设置为行索引

df.set_index('date',inplace=True)dfdate name score 2019-10-10 zs 192019-10-11 ls 292019-10-12 ww 392019-10-13 zl 20

可视化

import matplotlib.pyplot as plot%matplotlib inlinedf['score'].plot()

2.pandas数据处理

(1)删除重复元素

df.drop_duplicates(subset=None, keep='first', inplace=False)df = DataFrame({'color':['white','white','red','red','white'],'value':[2,1,3,3,2]})display(df,df.duplicated(),df.drop_duplicates())输出:colorvalue0white21white12red33red34white20 False1 False2 False3 True4 Truedtype: boolcolorvalue0white21white12red3

(2)映射

映射的含义：创建一个映射关系列表，把values元素和一个特定的标签或者字符串绑定

需要使用字典：

map = {'label1':'value1','label2':'value2', ... }

包含三种操作：

a. replace()函数：替换元素

df = DataFrame({'item':['ball','mug','pen'],'color':['white','rosso','verde'], 'price':[5.56,4.20,1.30]})newcolors = {'rosso':'red','verde':'green','pen':'pi'}df.replace(newcolors)itemcolorprice0ballwhite5.561mugred4.202pigreen1.30

b. 最重要：map()函数：新建一列

!!!map中返回的数据是一个具体值，不能迭代

使用map()函数，由已有的列生成一个新列,适合处理某一单独的列df3 = DataFrame({'color':['red','green','blue'],'project':['math','english','chemistry']})price = {'red':5.56,'green':3.14,'chemistry':2.79}df3['price'] = df3['color'].map(price)display(df3)输出：colorprojectprice0redmath5.561greenenglish3.142bluechemistryNaN

c. rename()函数：替换索引

df4 = DataFrame({'color':['white','gray','purple','blue','green'],'value':np.random.randint(10,size = 5)})new_index = {0:'first',1:'two',2:'three',3:'four',4:'five'}display(df4,df4.rename(new_index))输出：colorvalue0white21gray02purple93blue24green0colorvaluefirstwhite2twogray0threepurple9fourblue2fivegreen0

(3)异常值检查和过滤

使用describe()函数查看每一列的描述性统计量

np.random.seed(0)df = DataFrame(np.random.randint(10,size = 10))display(df.head(10),df.describe())输出：005102333475963758294 0count10.000000mean4.100000std2.558211min0.00000025%3.00000050%3.50000075%5.000000max9.000000

使用std()函数可以求得DataFrame对象每一列的标准差

df2=DataFrame(np.random.randint(10,100,size=(8,8)))df2.std()0 26.7177711 20.3118542 20.2608003 26.0463194 28.6602645 36.4025126 18.0158667 22.646349dtype: float64

根据每一列的标准差，对DataFrame元素进行过滤。借助any()函数，对每一列应用筛选条件

display(df.std(),np.abs(df)>(3*df.std()),df[(np.abs(df)>df.std()*3).any(axis = 1)])0 2.558211dtype: float64 00False1False2False3False4False5True6False7False8False9False059DataFrame.any功能说明DataFrame.any(self,axis=0,bool_only=None, skipna=True, level=None,**kwargs,)Return whether any element is True, potentially over an axis.

(4)排序

使用.take()函数排序,可以借助np.random.permutation()函数随机排序

df5 = DataFrame(np.arange(25).reshape(5,5))new_order = np.random.permutation(5)display(df5,new_order,df5.take(new_order))输出：01234001234156789210111213143151617181942021222324array([4, 2, 3, 1, 0]) 0 1 2 3442021222324210111213143151617181915 6 7 8900 1 2 34

随机抽样

当DataFrame规模足够大时，直接使用np.random.randint()函数，就配合take()函数实现随机抽样

sample = np.random.randint(0,len(df5),size = 3)df.take(sample)输出：012340012342101112131442021222324

(5)数据聚合

df = DataFrame({'color':['white','red','green','red'],'item':['ball','mug','pen','pencil'], 'price1':np.random.rand(4),'price2':np.random.rand(4)})g = df.groupby('color')['price1']display(df,g,g.groups,type(g))display(g.sum(),g.mean(),g.max())输出：coloritemprice1price20whiteball0.6527900.4143691redmug0.6350590.4746982greenpen0.9953000.6235103redpencil0.5818500.338008{'green': Int64Index([2], dtype='int64'), 'red': Int64Index([1, 3], dtype='int64'), 'white': Int64Index([0], dtype='int64')}pandas.core.groupby.SeriesGroupBycolorgreen 0.995300red 1.216909white 0.652790Name: price1, dtype: float64colorgreen 0.995300red 0.608455white 0.652790Name: price1, dtype: float64colorgreen 0.995300red 0.635059white 0.652790Name: price1, dtype: float64

(6)高级数据聚合

可以使用pd.merge()函数包聚合操作的计算结果添加到df的每一行

d1={'item':['luobo','baicai','lajiao','donggua','luobo','baicai','lajiao','donggua'], 'color':['white','white','red','green','white','white','red','green'], 'weight':np.random.randint(10,size = 8), 'price':np.random.randint(10,size = 8)}df = DataFrame(d1)sums = df.groupby('color').sum().add_prefix('total_')items = df.groupby('item')['price','weight'].sum()means = items['price']/items['weight']means = DataFrame(means,columns=['means_price'])df2 = pd.merge(df,sums,left_on = 'color',right_index = True)df3 = pd.merge(df2,means,left_on = 'item',right_index = True)display(df2,df3)输出：coloritempriceweight0whiteluobo921whitebaicai592redlajiao583greendonggua114whiteluobo745whitebaicai806redlajiao687greendonggua43total_pricetotal_weightcolorgreen54red1116white2915pandas.core.frame.DataFramepandas.core.frame.DataFrameOut[141]:coloritempriceweighttotal_pricetotal_weight0whiteluobo9229151whitebaicai5929154whiteluobo7429155whitebaicai8029152redlajiao5811166redlajiao6811163greendonggua11547greendonggua4354

可以使用transform和apply实现相同功能

transform

d1={'item':['luobo','baicai','lajiao','donggua','luobo','baicai','lajiao','donggua'], 'color':['white','white','red','green','white','white','red','green'], 'weight':np.random.randint(10,size = 8), 'price':np.random.randint(10,size = 8)}df = DataFrame(d1)sum1 = df.groupby('color')['price','weight'].sum().add_prefix("total_")sums2 = df.groupby('color')['price','weight'].transform(lambda x:x.sum()).add_prefix('total_')sums3 = df.groupby('color')['price','weight'].transform(sum).add_prefix('total_')display(sum,df,sum1,sums2,sums3)输出：coloritempriceweight0whiteluobo771whitebaicai772redlajiao273greendonggua664whiteluobo125whitebaicai366red lajiao707greendonggua02total_pricetotal_weightcolorgreen68red97white1822total_pricetotal_weight01822118222973684182251822697768total_pricetotal_weight01822118222973684182251822697768

apply

def sum_price(x): return x.sum()sums3 = df.groupby('color')['price','weight'].apply(lambda x:x.sum()).add_prefix('total_')sums4 = df.groupby('color')['price','weight'].apply(sum_price).add_prefix('total_')display(df,sums3,sums4)输出：coloritempriceweight0whiteluobo441whitebaicai032redlajiao043greendonggua754whiteluobo315whitebaicai336redlajiao067greendonggua07colortotal_pricetotal_weightgreen712red010white1011colortotals_pricetotals_weightgreen712red010white1011