Dataquest Data Scientist Path 整理笔记（2）

在Dataquest中学习Data Scientist方向的知识要点整理笔记。

Step 2: Data Analysis And Visualization

• ndarray : The core data structure in NumPy, stands for N-dimensional array.

• 将csv文件读取为numpy array格式

nfl = numpy.genfromtxt("nfl.csv", delimiter = ",", dtype = "U75", skip_header = 1)#以逗号为分隔符，格式统一为U75，跳过标题行

• 通过numpy.array()建立数组，ndarray.shape读取数组规模，ndarray.dtype读取元素类型。numpy中的元素为统一的一种格式，一些基本类型有bool，int，float，string。

vector = numpy.array([5, 10, 15, 20])
matrix = numpy.array([[5, 10, 15], 
                      [20, 25, 30], 
                      [35, 40, 45]])
print(vector.shape)
print(matrix.shape)
vector.dtype

• Numpy中使用nan表示Not a Number，使用na表示Not Available

• 分别读取list of list和numpy matrix创建矩阵，并读取某个元素

>>list_of_lists = [[5, 10, 15], 
                   [20, 25, 30]]
>>list_of_list[1][2]
30
>>matrix = np.array([[5, 10, 15], 
                     [20, 25, 30]])
>>matrix[1, 2]
30

• Slicing array

print(vector[1:3])#返回一个vector，即第2到4个元素
print(matrix[:, 1])#返回一个vector，即第2列
print(matrix[:, 1:3])#返回一个子matrix，即第2到4列

• Array Comparisons

>>vector = numpy.array([5, 10, 15, 20])
>>vector == 10
array([False, False, True, False], dtype=bool)

>>(vector == 10)&(vector == 15)
array([False, False, False, False], dtype=bool)

>>(vector == 10)|(vector == 15)
array([False, True, True, False], dtype=bool)

>>matrix = numpy.array([[5, 10, 15], 
                        [20, 25, 30],
                        [35, 40, 45]])
>>matrix == 25
array([[False, False, False],
       [False,  True, False],
       [False, False, False]], dtype=bool)

• Selecting Elements

matrix = numpy.array([[5, 10, 15], 
                      [20, 25, 30],
                      [35, 40, 45]])
second_column_25 = (matrix[:,1] == 25)
print(matrix[second_column_25, :])

• Replacing Values

>>vector = numpy.array([5, 10, 15, 20])
>>equal_to_ten_or_five = (vector == 10) | (vector == 5)
>>vector[equal_to_ten_or_five] = 50
>>print(vector)
[50 50 15 20]

>>matrix = numpy.array([[5, 10, 15],
                        [20, 25, 30],
                        [35, 40, 45]])
>>second_column_25 = matrix[:,1] == 25
>>matrix[second_column_25, 1] = 10
>>matrix
array([[ 5, 10, 15],
       [20, 10, 30],
       [35, 40, 45]])

• 转换元素格式 astype()

vector = numpy.array(["1", "2", "3"])
vector = vector.astype(float)

• Computing With NumPy

>>vector = numpy.array([5, 10, 15, 20])
>>vector.sum()
50
>>matrix = numpy.array([[5, 10, 15], 
                        [20, 25, 30],
                        [35, 40, 45]])
>>matrix.sum(axis=1)#axis=1为对每行运算，0为对每列运算
array([30, 75, 120])

• 例
  Year WHO region Country Beverage Types Display Value
  1986 Western Viet Nam Wine 0
  1986 Americas Uruguay Other 0.5
  1985 AfricaCte Cte d’Ivoire Wine 1.62
  1986 Americas Colombia Beer 4.27
  1987 Americas Saint Kitts Beer 1.98

totals = {}#新建字典
is_year = world_alcohol[:,0] == "1985"#
year = world_alcohol[is_year,:]

for country in countries:
    is_country = year[:,2] == country
    country_consumption = year[is_country,:]
    alcohol_column = country_consumption[:,4]
    is_empty = alcohol_column == ''
    alcohol_column[is_empty] = "0"
    alcohol_column = alcohol_column.astype(float)
    totals[country] = alcohol_column.sum()

• 使用pandas读取csv文件

crime_rates = pandas.read_csv("crime_rates.csv")#读取为pandas的通用数据格式 dataframe

• 对数据进行分析

first_rows = food_info.head()#前5行
column_names = food_info.columns#读取列名称
dimensions = food_info.shape#读取矩阵大小
num_rows = dimensions[0]#读取行数
num_cols = dimensions[1]#读取列数

• 选择行和列
  pandas使用第一行为列标，行数为行标

food_info.loc[0]#选择第1行
food_info.loc[2:5]#选择第3至6行
food_info.loc[[2,4,6]]#选择第3、5、7行

food_info.iloc[0]#重新排序后选择第1行
food_info.iloc[0:4]#重新排序后选择1至5行

food_info["NDB_No"]#选择列标为"NDB_No"的1列
food_info[["Zinc", "Copper"]]#选择列标为"Zinc"和"Copper"的2列，返回的列顺序与输入的列标顺序一致

• endswith() 选择以某个字符串结束的元素

for c in col_names:
    if c.endswith("(g)"):
        gram_columns.append(c)

• 对DataFrame进行数学运算

max_Iron = food_info["Iron_(mg)"].max()#取最大值
mean_Iron = food_info["Iron_(mg)"].mean()#取平均值
iron_grams = food_info["Iron_(mg)"] / 1000
food_info["Iron_(g)"] = iron_grams#新增1列"Iron_(g)"

• 排序

food_info.sort_values("Sodium_(mg)", inplace=True, ascending=False)#降序排序，对food_info直接更新

• Python中使用None表示“no value”，Pandas中使用NaN表示“not a number”，即缺失值，None和NaN均称为null 值，可使用pandas.isnull()筛选

sex = titanic_survival["sex"]
sex_is_null = pandas.isnull(sex)#返回Ture/False数组，矩阵也适用

• 例

passenger_classes = [1, 2, 3]
fares_by_class = {}
for this_class in passenger_classes:
    pclass_rows = titanic_survival[titanic_survival["pclass"] == this_class]
    pclass_fares = pclass_rows["fare"]
    fare_for_class = pclass_fares.mean()
    fares_by_class[this_class] = fare_for_class

• Pivot tables

passenger_class_fares = titanic_survival.pivot_table(index="pclass", values="fare", aggfunc=np.mean)#起到与上例中相同的作用，aggfunc默认为mean,可省略
port_stats = titanic_survival.pivot_table(index="embarked", values=["fare","survived"], aggfunc=np.sum)#可对values中的值进行运算

• Drop Missing Values

drop_na_rows = titanic_survival.dropna(axis=0)#axis=0或axis='index'会删除所有含null值的行，axis=1或axis='columns'会删除所有含null值的列

• iloc[]

first_row_first_column = new_titanic_survival.iloc[0,0]
all_rows_first_three_columns = new_titanic_survival.iloc[:,0:3]
row_index_83_age = new_titanic_survival.loc[83,"age"]
row_index_766_pclass = new_titanic_survival.loc[766,"pclass"]

• 对行标重新排序

titanic_reindexed = new_titanic_survival.reset_index(drop=True)#将旧的行标删除

• DataFrame.apply()：定义一个函数，对每一列/行执行这一函数

#计算每列的null值的个数
def not_null_count(column):
    column_null = pd.isnull(column)
    null = column[column_null]
    return len(null)

column_null_count = titanic_survival.apply(not_null_count)

#根据年龄进行分类
def generate_age_label(row):
    age = row["age"]
    if pd.isnull(age):
        return "unknown"
    elif age < 18:
        return "minor"
    else:
        return "adult"

age_labels = titanic_survival.apply(generate_age_label, axis=1)#axis=1表示对每行执行该函数

• Series：core data structure that pandas uses to represent rows and columns.
  Series与NumPy vector相似，不同的是Series可以使用非整数的标签

from pandas import Series
series_custom = Series(rt_scores, index = film_names)
fiveten = series_custom[5:11]#Series同样可以用integer index

• Series中的排序

sorted_by_index = series_custom.reindex(sorted_index)#根"sorted_index"的顺序对series_custom进行重新排列
sc2 = series_custom.sort_index()#根据index重新排序
sc3 = series_custom.sort_values()#根据values重新排序

• Series运算

np.add(series_custom, series_custom)#values翻倍
np.sin(series_custom)#取sin
np.max(series_custom)#取最大值

• Comparing And Filtering

series_custom > 50#返回True/False数组
series_greater_than_50 = series_custom[series_custom > 50]#可用"|"、"or"、"&"、"and"组合

• 选择行
  iloc[]可填入以下内容
  An integer, e.g. 5.
  A list or array of integers, e.g. [4, 3, 0].
  A slice object with ints, e.g. 1:7.
  A boolean array.

fandango[0:5]#选择前5行
fandango[140:]#选择140行之后
fandango.iloc[50]#选择第50行
fandango.iloc[[45,90]]#选择第45行和90行

• set_index()
  可输入一列作为行标
  参数：inplace = True，表示直接对dataframe进行替换；
  　　　drop = False，表示不会删除新增的作为行标的那一列。

fandango_films = fandango.set_index('FILM', drop = False)

• pandas.Series.value_counts()

data["Do you celebrate Thanksgiving?"].value_counts()

• 排序

food_info.sort_values("Sodium_(mg)", inplace=True, ascending=False)#降序排序，对food_info直接更新