第8章 分类数据

第8章 分类数据

在这里插入图片描述

import pandas as pd
import numpy as np
df = pd.read_csv('data/table.csv')
df.head()
SchoolClassIDGenderAddressHeightWeightMathPhysics
0S_1C_11101Mstreet_11736334.0A+
1S_1C_11102Fstreet_21927332.5B+
2S_1C_11103Mstreet_21868287.2B+
3S_1C_11104Fstreet_21678180.4B-
4S_1C_11105Fstreet_41596484.8B+

一、category的创建及其性质

1. 分类变量的创建

(a)用Series创建
pd.Series(["a", "b", "c", "a"], dtype="category")
0    a
1    b
2    c
3    a
dtype: category
Categories (3, object): [a, b, c]
(b)对DataFrame指定类型创建
temp_df = pd.DataFrame({'A':pd.Series(["a", "b", "c", "a"], dtype="category"),'B':list('abcd')})
temp_df.dtypes
A    category
B      object
dtype: object
(c)利用内置Categorical类型创建
cat = pd.Categorical(["a", "b", "c", "a"], categories=['a','b','c'])
pd.Series(cat)
0    a
1    b
2    c
3    a
dtype: category
Categories (3, object): [a, b, c]
(d)利用cut函数创建
默认使用区间类型为标签
pd.cut(np.random.randint(0,60,5), [0,10,30,60])
[(30, 60], (0, 10], (30, 60], (10, 30], (30, 60]]
Categories (3, interval[int64]): [(0, 10] < (10, 30] < (30, 60]]
可指定字符为标签
pd.cut(np.random.randint(0,60,5), [0,10,30,60], right=False, labels=['0-10','10-30','30-60'])
[30-60, 10-30, 10-30, 30-60, 30-60]
Categories (3, object): [0-10 < 10-30 < 30-60]

2. 分类变量的结构

一个分类变量包括三个部分,元素值(values)、分类类别(categories)、是否有序(order)
从上面可以看出,使用cut函数创建的分类变量默认为有序分类变量
下面介绍如何获取或修改这些属性
(a)describe方法
该方法描述了一个分类序列的情况,包括非缺失值个数、元素值类别数(不是分类类别数)、最多次出现的元素及其频数
s = pd.Series(pd.Categorical(["a", "b", "c", "a",np.nan], categories=['a','b','c','d']))
s.describe()
count     4
unique    3
top       a
freq      2
dtype: object
s.values
[a, b, c, a, NaN]
Categories (4, object): [a, b, c, d]
(b)categories和ordered属性
查看分类类别和是否排序
s.cat.categories
Index(['a', 'b', 'c', 'd'], dtype='object')
s.cat.ordered
False

3. 类别的修改

(a)利用set_categories修改
修改分类,但本身值不会变化
s = pd.Series(pd.Categorical(["a", "b", "c", "a",np.nan], categories=['a','b','c','d']))
s.cat.set_categories(['new_a','c'])
0    NaN
1    NaN
2      c
3    NaN
4    NaN
dtype: category
Categories (2, object): [new_a, c]
(b)利用rename_categories修改
需要注意的是该方法会把值和分类同时修改
s = pd.Series(pd.Categorical(["a", "b", "c", "a",np.nan], categories=['a','b','c','d']))
s.cat.rename_categories(['new_%s'%i for i in s.cat.categories])
0    new_a
1    new_b
2    new_c
3    new_a
4      NaN
dtype: category
Categories (4, object): [new_a, new_b, new_c, new_d]
利用字典修改值
s.cat.rename_categories({'a':'new_a','b':'new_b'})
0    new_a
1    new_b
2        c
3    new_a
4      NaN
dtype: category
Categories (4, object): [new_a, new_b, c, d]
(c)利用add_categories添加
s = pd.Series(pd.Categorical(["a", "b", "c", "a",np.nan], categories=['a','b','c','d']))
s.cat.add_categories(['e'])
0      a
1      b
2      c
3      a
4    NaN
dtype: category
Categories (5, object): [a, b, c, d, e]
s.cat.add_categories('e')
0      a
1      b
2      c
3      a
4    NaN
dtype: category
Categories (5, object): [a, b, c, d, e]
(d)利用remove_categories移除
s = pd.Series(pd.Categorical(["a", "b", "c", "a",np.nan], categories=['a','b','c','d']))
s.cat.remove_categories(['d'])
0      a
1      b
2      c
3      a
4    NaN
dtype: category
Categories (3, object): [a, b, c]
(e)删除元素值未出现的分类类型
s = pd.Series(pd.Categorical(["a", "b", "c", "a",np.nan], categories=['a','b','c','d']))
s.cat.remove_unused_categories()
0      a
1      b
2      c
3      a
4    NaN
dtype: category
Categories (3, object): [a, b, c]

二、分类变量的排序

前面提到,分类数据类型被分为有序和无序,这非常好理解,例如分数区间的高低是有序变量,考试科目的类别一般看做无序变量

1. 序的建立

(a)一般来说会将一个序列转为有序变量,可以利用as_ordered方法
s = pd.Series(["a", "d", "c", "a"]).astype('category').cat.as_ordered()
s
0    a
1    d
2    c
3    a
dtype: category
Categories (3, object): [a < c < d]
退化为无序变量,只需要使用as_unordered
s.cat.as_unordered()
0    a
1    d
2    c
3    a
dtype: category
Categories (3, object): [a, c, d]
(b)利用set_categories方法中的order参数
pd.Series(["a", "d", "c", "a"]).astype('category').cat.set_categories(['a','c','d'],ordered=True)
0    a
1    d
2    c
3    a
dtype: category
Categories (3, object): [a < c < d]
(c)利用reorder_categories方法
这个方法的特点在于,新设置的分类必须与原分类为同一集合
s = pd.Series(["a", "d", "c", "a"]).astype('category')
s.cat.reorder_categories(['a','c','d'],ordered=True)
0    a
1    d
2    c
3    a
dtype: category
Categories (3, object): [a < c < d]
#s.cat.reorder_categories(['a','c'],ordered=True) #报错
#s.cat.reorder_categories(['a','c','d','e'],ordered=True) #报错

2. 排序

先前在第1章介绍的值排序和索引排序都是适用的
s = pd.Series(np.random.choice(['perfect','good','fair','bad','awful'],50)).astype('category')
s.cat.set_categories(['perfect','good','fair','bad','awful'][::-1],ordered=True).head()
0    good
1     bad
2     bad
3    good
4    good
dtype: category
Categories (5, object): [awful < bad < fair < good < perfect]
s.sort_values(ascending=False).head()
46    perfect
43    perfect
9     perfect
11    perfect
13    perfect
dtype: category
Categories (5, object): [awful, bad, fair, good, perfect]
s.sort_values(ascending=True).head()
39    awful
47    awful
6     awful
23    awful
8     awful
dtype: category
Categories (5, object): [awful, bad, fair, good, perfect]
df_sort = pd.DataFrame({'cat':s.values,'value':np.random.randn(50)}).set_index('cat')
df_sort.head()
value
cat
good0.371380
bad0.799006
bad-0.039575
good0.631024
good0.905758
df_sort.sort_index().head()
value
cat
awful0.713834
awful0.108047
awful-0.290348
awful2.646020
awful-0.212087

三、分类变量的比较操作

1. 与标量或等长序列的比较

(a)标量比较
s = pd.Series(["a", "d", "c", "a"]).astype('category')
s == 'a'
0     True
1    False
2    False
3     True
dtype: bool
(b)等长序列比较
s == list('abcd')
0     True
1    False
2     True
3    False
dtype: bool

2. 与另一分类变量的比较

(a)等式判别(包含等号和不等号)
两个分类变量的等式判别需要满足分类完全相同
s = pd.Series(["a", "d", "c", "a"]).astype('category')
s == s
0    True
1    True
2    True
3    True
dtype: bool
s != s
0    False
1    False
2    False
3    False
dtype: bool
s_new = s.cat.set_categories(['a','d','e'])
# s == s_new #报错
s_new
0      a
1      d
2    NaN
3      a
dtype: category
Categories (3, object): [a, d, e]
(b)不等式判别(包含>=,<=,<,>)
两个分类变量的不等式判别需要满足两个条件:① 分类完全相同 ② 排序完全相同
s = pd.Series(["a", "d", "c", "a"]).astype('category')
#s >= s #报错
s = pd.Series(["a", "d", "c", "a"]).astype('category').cat.reorder_categories(['a','c','d'],ordered=True)
s >= s
0    True
1    True
2    True
3    True
dtype: bool

四、问题与练习

【问题一】 如何使用union_categoricals方法?它的作用是什么?

from https://juejin.im/post/5b680a7af265da0f7a1d1de4

blood_type1 = pd.Categorical(["A", "AB"])
blood_type2 = pd.Categorical(["B", "O"])
pd.concat([pd.Series(blood_type1), pd.Series(blood_type2)])
0     A
1    AB
0     B
1     O
dtype: object
  • 可以发现,分类数据经过 pd.concat 合并后类型转为了 object 类型。如果想要保持分类类型的话,可以借助 union_categoricals 来完成。
from pandas.api.types import union_categoricals

pd.Series(union_categoricals([blood_type1, blood_type2]))
0     A
1    AB
2     B
3     O
dtype: category
Categories (4, object): [A, AB, B, O]
【问题二】 利用concat方法将两个序列纵向拼接,它的结果一定是分类变量吗?什么情况下不是?
blood_type1 = pd.Categorical(["A", "AB"])
blood_type2 = pd.Categorical(["B", "O"])
pd.concat([pd.Series(blood_type1), pd.Series(blood_type2)])
0     A
1    AB
0     B
1     O
dtype: object
  • 可以发现,分类数据经过 pd.concat 合并后类型转为了 object 类型。
blood_type1 = pd.Categorical([1,2])
blood_type2 = pd.Categorical([2,3])
pd.concat([pd.Series(blood_type1), pd.Series(blood_type2)])
0    1
1    2
0    2
1    3
dtype: int64
  • 类型转为了int64
【问题三】 当使用groupby方法或者value_counts方法时,分类变量的统计结果和普通变量有什么区别?
df = pd.read_csv('data/table.csv')
df.head()
SchoolClassIDGenderAddressHeightWeightMathPhysics
0S_1C_11101Mstreet_11736334.0A+
1S_1C_11102Fstreet_21927332.5B+
2S_1C_11103Mstreet_21868287.2B+
3S_1C_11104Fstreet_21678180.4B-
4S_1C_11105Fstreet_41596484.8B+
df['Physics'].value_counts()
B+    9
B     8
B-    6
A     4
A-    3
A+    3
C     2
Name: Physics, dtype: int64
df['Physics'].astype('category').value_counts()
B+    9
B     8
B-    6
A     4
A-    3
A+    3
C     2
Name: Physics, dtype: int64
df['Physics']=df['Physics'].astype('category')
for name ,group in df.groupby('Physics'):
    print(name)
    display(group)
A
SchoolClassIDGenderAddressHeightWeightMathPhysics
13S_1C_31304Mstreet_21957085.2A
19S_2C_12105Mstreet_41708134.2A
26S_2C_32302Mstreet_51718832.7A
30S_2C_42401Fstreet_21926245.3A
A+
SchoolClassIDGenderAddressHeightWeightMathPhysics
0S_1C_11101Mstreet_11736334.0A+
7S_1C_21203Mstreet_61605358.8A+
22S_2C_22203Mstreet_41559173.8A+
A-
SchoolClassIDGenderAddressHeightWeightMathPhysics
5S_1C_21201Mstreet_51886897.0A-
11S_1C_31302Fstreet_11755787.7A-
28S_2C_32304Fstreet_61648195.5A-
B
SchoolClassIDGenderAddressHeightWeightMathPhysics
8S_1C_21204Fstreet_51626333.8B
12S_1C_31303Mstreet_71888249.7B
20S_2C_22201Mstreet_519310039.1B
24S_2C_22205Fstreet_71837685.4B
29S_2C_32305Mstreet_41877348.9B
31S_2C_42402Mstreet_71668248.7B
33S_2C_42404Fstreet_21608467.7B
34S_2C_42405Fstreet_61935447.6B
B+
SchoolClassIDGenderAddressHeightWeightMathPhysics
1S_1C_11102Fstreet_21927332.5B+
2S_1C_11103Mstreet_21868287.2B+
4S_1C_11105Fstreet_41596484.8B+
10S_1C_31301Mstreet_41616831.5B+
16S_2C_12102Fstreet_61616150.6B+
18S_2C_12104Fstreet_51599772.2B+
21S_2C_22202Fstreet_71947768.5B+
25S_2C_32301Fstreet_41577872.3B+
32S_2C_42403Fstreet_61586059.7B+
B-
SchoolClassIDGenderAddressHeightWeightMathPhysics
3S_1C_11104Fstreet_21678180.4B-
6S_1C_21202Fstreet_41769463.5B-
9S_1C_21205Fstreet_61676368.4B-
14S_1C_31305Fstreet_51876961.7B-
17S_2C_12103Mstreet_41576152.5B-
23S_2C_22204Mstreet_11757447.2B-
C
SchoolClassIDGenderAddressHeightWeightMathPhysics
15S_2C_12101Mstreet_71748483.3C
27S_2C_32303Fstreet_71909965.9C
  • 好像没啥区别
【问题四】 下面的代码说明了Series创建分类变量的什么“缺陷”?如何避免?(提示:使用Series中的copy参数)
cat = pd.Categorical([1, 2, 3, 10], categories=[1, 2, 3, 4, 10])
s = pd.Series(cat, name="cat")
cat
[1, 2, 3, 10]
Categories (5, int64): [1, 2, 3, 4, 10]
s.iloc[0:2] = 10
cat
[10, 10, 3, 10]
Categories (5, int64): [1, 2, 3, 4, 10]
  • 解答

问题应该是我们对于Series里面的内容进行改变时,Categories中相对应的内容也进行了改变,避免的方式时在创建Series时copy 参数设为True这样一来就会自动拷贝一份输入的Categories,这时候我i们再对Series进行改变,Categories中相对应的内容不变。

copy:一个布尔值。如果为True,则拷贝输入数据data

cat = pd.Categorical([1, 2, 3, 10], categories=[1, 2, 3, 4, 10])
s = pd.Series(cat, name="cat",copy=True)
cat
[1, 2, 3, 10]
Categories (5, int64): [1, 2, 3, 4, 10]
s.iloc[0:2] = 10
cat
[1, 2, 3, 10]
Categories (5, int64): [1, 2, 3, 4, 10]
【练习一】 现继续使用第四章中的地震数据集,请解决以下问题:
(a)现在将深度分为七个等级:[0,5,10,15,20,30,50,np.inf],请以深度等级Ⅰ,Ⅱ,Ⅲ,Ⅳ,Ⅴ,Ⅵ,Ⅶ为索引并按照由浅到深的顺序进行排序。
pd.read_csv('data/Earthquake.csv').head()
日期时间维度经度方向距离深度烈度
02003.05.2012:17:44 AM39.0440.38west0.110.00.0
12007.08.0112:03:08 AM40.7930.09west0.15.24.0
21978.05.0712:41:37 AM38.5827.61south_west0.10.00.0
31997.03.2212:31:45 AM39.4736.44south_west0.110.00.0
42000.04.0212:57:38 AM40.8030.24south_west0.17.00.0
a=pd.read_csv('data/Earthquake.csv')
a['深度']=pd.cut(a['深度'], [0,5,10,15,20,30,50,np.inf], right=False, labels=['Ⅰ','Ⅱ','Ⅲ','Ⅳ','Ⅴ','Ⅵ','Ⅶ'])
a.sort_values('深度')
日期时间维度经度方向距离深度烈度
11821999.07.0512:57:31 AM41.2432.78north_west0.90.0
14861994.12.2412:40:49 AM37.4828.35north_east1.00.0
70922001.12.1212:59:55 AM37.4237.28north_east3.20.0
28102017.04.2112:25:56 AM38.7829.06south_east1.53.8
14831992.08.2112:10:52 AM37.6127.48north_east1.00.0
...........................
77711981.08.0412:52:19 AM38.9037.00north3.70.0
5251968.03.2112:42:51 AM38.8027.60south0.64.3
39211929.04.2712:18:06 AM40.5131.43south_west1.94.8
38962008.03.2612:16:15 AM37.0430.27east1.80.0
12141956.07.1812:46:53 AM39.9627.30north_east0.94.6

10062 rows × 8 columns

参考答案
df = pd.read_csv('data/Earthquake.csv')
df.head()
df_a = df.copy()
df_a['深度'] = pd.cut(df_a['深度'], [-1e-10,5,10,15,20,30,50,np.inf],labels=['Ⅰ','Ⅱ','Ⅲ','Ⅳ','Ⅴ','Ⅵ','Ⅶ'])
df_a.set_index('深度').sort_index().head()
日期时间维度经度方向距离烈度
深度
2009.09.0912:54:13 AM42.4243.03north_east95.40.0
1997.06.1612:18:04 AM37.9229.17north_east3.20.0
2011.10.2512:29:45 AM38.9643.64south_east1.63.9
1995.07.2312:05:04 AM37.6129.29north_east3.20.0
2013.06.1012:39:19 AM38.5343.85south_east1.63.7
(b)在(a)的基础上,将烈度分为4个等级:[0,3,4,5,np.inf],依次对南部地区的深度和烈度等级建立多级索引排序。

a['烈度']=pd.cut(a['烈度'],[0,3,4,5,np.inf],right=False)
a.set_index(['深度','烈度'])
# a.sort_values(['深度','烈度'])
日期时间维度经度方向距离
深度烈度
[0.0, 3.0)2003.05.2012:17:44 AM39.0440.38west0.1
[4.0, 5.0)2007.08.0112:03:08 AM40.7930.09west0.1
[0.0, 3.0)1978.05.0712:41:37 AM38.5827.61south_west0.1
[0.0, 3.0)1997.03.2212:31:45 AM39.4736.44south_west0.1
[0.0, 3.0)2000.04.0212:57:38 AM40.8030.24south_west0.1
.....................
[3.0, 4.0)2015.11.1812:17:48 AM42.3142.94north81.6
[0.0, 3.0)1990.01.2812:22:43 AM42.7026.20north_west89.5
[0.0, 3.0)2001.08.0912:58:14 AM42.7726.47north90.6
[0.0, 3.0)1994.06.0512:20:03 AM42.4143.06north_east94.3
[0.0, 3.0)2009.09.0912:54:13 AM42.4243.03north_east95.4

10062 rows × 6 columns

a[a['方向']=='south'].set_index(['深度','烈度']).sort_index()
日期时间维度经度方向距离
深度烈度
[0.0, 3.0)2000.05.1412:18:59 AM38.6039.92south0.3
[0.0, 3.0)1982.01.0512:53:17 AM39.6128.50south0.5
[0.0, 3.0)1986.07.2012:01:18 AM37.8135.91south0.5
[0.0, 3.0)1996.08.1412:14:37 AM40.7035.43south0.5
[0.0, 3.0)2002.09.1612:54:15 AM41.1934.16south0.5
........................
[4.0, 5.0)1933.01.0212:56:58 AM38.0138.24south2.6
[4.0, 5.0)1936.08.0212:21:09 AM37.8829.70south2.9
[4.0, 5.0)1971.04.3012:10:04 AM37.7636.18south3.5
[4.0, 5.0)1964.11.2012:59:19 AM40.2028.06south3.6
[4.0, 5.0)1970.03.3012:15:44 AM38.9629.73south4.0

605 rows × 6 columns

参考答案
df_a['烈度'] = pd.cut(df_a['烈度'], [-1e-10,3,4,5,np.inf],labels=['Ⅰ','Ⅱ','Ⅲ','Ⅳ'])
df_a.set_index(['深度','烈度']).sort_index().head()
日期时间维度经度方向距离
深度烈度
1978.05.0712:41:37 AM38.5827.61south_west0.1
2000.02.0712:11:45 AM40.0534.07south_east0.1
1971.05.2012:08:46 AM37.7230.00north_east0.1
1985.01.2812:20:56 AM38.8529.06north_east0.1
1990.07.0512:43:04 AM37.8729.18east0.1
【练习二】 对于分类变量而言,调用第4章中的变形函数会出现一个BUG(目前的版本下还未修复):例如对于crosstab函数,按照官方文档的说法,即使没有出现的变量也会在变形后的汇总结果中出现,但事实上并不是这样,比如下面的例子就缺少了原本应该出现的行’c’和列’f’。基于这一问题,请尝试设计my_crosstab函数,在功能上能够返回正确的结果。
foo = pd.Categorical(['a', 'b'], categories=['a', 'b', 'c'])
bar = pd.Categorical(['d', 'e'], categories=['d', 'e', 'f'])
pd.crosstab(foo, bar)
col_0de
row_0
a10
b01
# pd.crosstab(foo.categories)
pd.crosstab(index=foo.categories,columns=bar.categories,values=1,aggfunc='count')
def my_crosstab(foo,bar):
    df=pd.DataFrame({i:[0]*len(foo.categories)for i in bar.categories},index=foo.categories)
    for i in range(len(foo.categories)):
        df.iat[i,i]=1
    df.rename_axis(index={None:'row_0'},columns={None:'col_0'},inplace=True)
    return df
my_crosstab(foo ,bar )
col_0def
row_0
a100
b010
c001
查看答案后改进
pd.crosstab(index=foo.categories,columns=bar.categories,values=1,aggfunc='count')
def my_crosstab(foo,bar):
    col=bar.categories.union(bar)
    row=foo.categories.union(foo)
    df=pd.DataFrame({i:[0]*len(row)for i in col},index=row)
    for i in range(len(row)):
        df.iat[i,i]=1
    df.rename_axis(index={None:'row_0'},columns={None:'col_0'},inplace=True)
    return df
my_crosstab(foo ,bar )
col_0def
row_0
a100
b010
c001
参考答案
bar.categories.union(set(bar))
Index(['d', 'e', 'f'], dtype='object')
def my_crosstab(foo,bar):
    num = len(foo)
    s1 = pd.Series([i for i in list(foo.categories.union(set(foo)))],name='1nd var')
    print('s1')
    display(s1)
    s2 = [i for i in list(bar.categories.union(set(bar)))]
    print('s2')
    display(s2)
    df = pd.DataFrame({i:[0]*len(s1) for i in s2},index=s1)
    for i in range(num):
        df.at[foo[i],bar[i]] += 1
    return df.rename_axis('2st var',axis=1)
my_crosstab(foo,bar)
s1



0    a
1    b
2    c
Name: 1nd var, dtype: object


s2



['d', 'e', 'f']
2st vardef
1nd var
a100
b010
c000

  • 1
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值