Task11综合练习

Task11综合练习

学习参考：http://datawhale.club/t/topic/579/7

【任务四】显卡日志

下面给出了3090显卡的性能测评日志结果，每一条日志有如下结构：

Benchmarking #2# #4# precision type #1#
#1# model average #2# time : #3# ms

其中#1#代表的是模型名称，#2#的值为train(ing)或inference，表示训练状态或推断状态，#3#表示耗时，#4#表示精度，其中包含了float, half, double三种类型，下面是一个具体的例子：

Benchmarking Inference float precision type resnet50 resnet50 model
average inference time : 13.426570892333984 ms

请把日志结果进行整理，变换成如下状态，model_i用相应模型名称填充，按照字母顺序排序，数值保留三位小数：

import numpy as np
import pandas as pd
# 从文件中读取关键字，利用列表存储关键字
model_name_list = []  #1
type_list = []  #2:Train(ing)/Inference
time_list = []  #3
accuracy_list = []  #4:float/half/double

file = '../data/benchmark.txt'
with open(file) as f:  
    for line in f:
        line = line.strip()
        if line.startswith("Benchmarking"):
            line_list = line.split()
            if line_list[1] == "Training":
                type_list.append("Train")
            else:
                type_list.append("Inference")
            accuracy_list.append(line_list[2])
            model_name_list.append(line_list[-1])
        elif line.endswith("ms"):
            line_list = line.split()
            time = round(float(line_list[-2]),3)
            time_list.append(time)
        else:
            continue

df = pd.DataFrame([model_name_list,type_list,accuracy_list,time_list]).T
df.columns = ["model_name","type","accuracy","time"]
df.head()

df['type_accuracy'] = df['type']+'_'+df['accuracy']
#print(df.shape)  #(192, 5)
df1 = df.pivot(index='model_name', columns='type_accuracy', values='time').reset_index().rename_axis(columns={'type_accuracy':''})
df1 = df1.set_index('model_name')
df1 = df1.sort_index(level = 0)
print(df1.shape)
df1.head()

【任务五】水压站点的特征工程

df1和df2中分别给出了18年和19年各个站点的数据，其中列中的H0至H23分别代表当天0点至23点；df3中记录了18-19年的每日该地区的天气情况，请完成如下的任务：

df1 = pd.read_csv('../data/yali18.csv')
df2 = pd.read_csv('../data/yali19.csv')
df3 = pd.read_csv('../data/qx1819.csv')

#1.构造df，把时间设为索引，第一列为站点编号，第二列为对应时刻的压力大小
df = pd.concat([df1,df2],axis = 0)
df= df.melt(id_vars = df.columns[:2],
                    value_vars = df.columns[2:],
                    var_name = 'Hours',
                    value_name = '压力')
print(df.shape)  # (262800, 4)
df.head()

df['站点'] = df['MeasName'].str.strip("站点").astype('int')
df['Hours'] = df['Hours'].str.strip("H")
df['Time'] = df['Time'] +'-'+ df['Hours']
df['Time'] = pd.to_datetime(df['Time'],format = '%Y-%m-%d-%H')
df.drop(['Hours','MeasName'], axis=1, inplace=True)
df.sort_values(['Time','站点'],inplace=True)
df.set_index('Time',inplace=True)
df = df.loc[:,['站点','压力']]
df

#2.当天最高温、最低温和它们的温差
weather = df3['气温'].str.split("～", expand=True)
weather['最高温'] = pd.to_numeric(weather[0].str.extract('(-?\d+)')[0])
weather['最低温'] = pd.to_numeric(weather[1].str.extract('(-?\d+)')[0])
weather['温差'] = weather['最高温'] - weather['最低温']
weather = weather.drop([0,1],axis=1)
weather

#3.当天是否有沙暴、是否有雾、是否有雨、是否有雪、是否为晴天
weather['是否沙暴'] = df3['天气'].str.contains('沙')
weather['是否有雾'] = df3['天气'].str.contains('雾')
weather['是否有雨'] = df3['天气'].str.contains('雨')
weather['是否有雪'] = df3['天气'].str.contains('雪')
weather['是否晴天'] = df3['天气'].str.contains('晴')
weather

#4.选择一种合适的方法度量雨量/下雪量的大小（构造两个序列分别表示二者大小）
#不清楚描述方法？

#5.限制只用4列，对风向进行0-1编码（只考虑风向，不考虑大小）
weather['东风'] = df3['风向'].str.contains('东')
weather['南风'] = df3['风向'].str.contains('南')
weather['西风'] = df3['风向'].str.contains('西')
weather['北风'] = df3['风向'].str.contains('北')
weather['日期'] = pd.to_datetime(df3['日期'])
weather = weather.set_index('日期')
weather

#剩下的问题还没搞定，需要后面再想