Datawhale组队学习(Pandas) task10-时序数据

时间戳

  • 单独时间点即为时间戳,pandas中称为Timestamp,一系列时间戳可以组成DatetimeIndex

Timestamp的构造与属性

import numpy as np
import pandas as pd
ts = pd.Timestamp('2020-1-1 08:10:30')
ts
Timestamp('2020-01-01 08:10:30')
# 通过 year, month, day, hour, min, second 可以获取具体的数值
ts.year  # 2020
ts.month  # 1
ts.day  # 1
ts.hour  # 8
ts.minute  # 10
ts.second  # 30

Datetime序列的生成

to_datetime

# to_datetime 能将一系列时间戳格式的对象转化为datetime64[ns] 类型的时间序列
pd.to_datetime(['2020-1-1', '2020-1-3', '2020-1-6'])
pd.to_datetime(['2020-1-1', '2020-1-3', '2020-1-6']).year  # Int64Index([2020, 2020, 2020], dtype='int64')
Int64Index([2020, 2020, 2020], dtype='int64')
df = pd.read_csv('data/learn_pandas.csv')
df.head()
SchoolGradeNameGenderHeightWeightTransferTest_NumberTest_DateTime_Record
0Shanghai Jiao Tong UniversityFreshmanGaopeng YangFemale158.946.0N12019/10/50:04:34
1Peking UniversityFreshmanChangqiang YouMale166.570.0N12019/9/40:04:20
2Shanghai Jiao Tong UniversitySeniorMei SunMale188.989.0N22019/9/120:05:22
3Fudan UniversitySophomoreXiaojuan SunFemaleNaN41.0N22020/1/30:04:08
4Fudan UniversitySophomoreGaojuan YouMale174.074.0N22019/11/60:05:22
s = pd.to_datetime(df.Test_Date)
s.head()
0   2019-10-05
1   2019-09-04
2   2019-09-12
3   2020-01-03
4   2019-11-06
Name: Test_Date, dtype: datetime64[ns]
# 极少数情况下,时间戳不满足转换时,可以强制使用format进行匹配
temp = pd.to_datetime(['2020\\1\\1','2020\\1\\3'], format='%Y\\%m\\%d')
temp
DatetimeIndex(['2020-01-01', '2020-01-03'], dtype='datetime64[ns]', freq=None)
# DatetimeIndex → datetime64[ns]
# 显式用Series转化
pd.Series(temp).head()
0   2020-01-01
1   2020-01-03
dtype: datetime64[ns]
# 把表的多列时间属性拼接转为时间序列的to_datetime操作
df_date_cols = pd.DataFrame({'year':[2020, 2020], 'month':[1,1,], 'day': [1, 2], 'hour': [10, 20], 'minute': [30, 50], 'second': [20, 40]})
df_date_cols
yearmonthdayhourminutesecond
0202011103020
1202012205040
pd.to_datetime(df_date_cols)
0   2020-01-01 10:30:20
1   2020-01-02 20:50:40
dtype: datetime64[ns]

date_range

是一种生成连续间隔时间的方法,注意开始或结束日期如果作为端点则会被包含

  • start 开始时间
  • end 结束时间
  • freq 时间间隔
  • periods 时间戳个数
pd.date_range('2020-1-1','2020-1-21', freq='10D') # 包含两端
DatetimeIndex(['2020-01-01', '2020-01-11', '2020-01-21'], dtype='datetime64[ns]', freq='10D')
pd.date_range('2020-1-1','2020-2-28', freq='10D')  # 结束日期无法取到,freq不为10天
DatetimeIndex(['2020-01-01', '2020-01-11', '2020-01-21', '2020-01-31',
               '2020-02-10', '2020-02-20'],
              dtype='datetime64[ns]', freq='10D')
s = pd.Series(np.random.rand(5), index=pd.to_datetime(['2020-1-%d'%i for i in range(1,10,2)]))
# 2020-01-01    0.068769
# 2020-01-03    0.096169
# 2020-01-05    0.309931
# 2020-01-07    0.176197
# 2020-01-09    0.038457
# dtype: float64

# 改变序列采样频率
s.asfreq('D')
# 2020-01-01    0.068769
# 2020-01-02         NaN
# 2020-01-03    0.096169
# 2020-01-04         NaN
# 2020-01-05    0.309931
# 2020-01-06         NaN
# 2020-01-07    0.176197
# 2020-01-08         NaN
# 2020-01-09    0.038457
# Freq: D, dtype: float64

dt对象

  • 取出时间相关的属性
  • 判断时间戳是否满足条件
  • 取整操作
# 取出时间相关属性
# date, time, year, month, day, hour, minute, second, microsecond, nanosecond, dayofweek, dayofyear, weekofyear, daysinmonth(月中第几天), quarter(季度)
s = pd.Series(pd.date_range('2020-1-1','2020-1-3', freq='D'))

# 日期
s.dt.date
# 0    2020-01-01
# 1    2020-01-02
# 2    2020-01-03
# dtype: object

# 时间
s.dt.time
# 0    00:00:00
# 1    00:00:00
# 2    00:00:00
# dtype: object

# 月中有几天
s.dt.daysinmonth
# 0    31
# 1    31
# 2    31
# dtype: int64

# 周几
s.dt.dayofweek
# 0    2
# 1    3
# 2    4
# dtype: int64

# 月名(方法)
s.dt.month_name()
# 0    January
# 1    January
# 2    January
# dtype: object

# 星期名(方法)
s.dt.day_name()
# 0    Wednesday
# 1     Thursday
# 2       Friday
# dtype: object
# 判断操作主要用于测试是否为月/季/年的第一天或者最后一天
# 还可选 is_quarter/month_start/month_end
s.dt.is_year_start
s.dt.is_year_end
0    False
1    False
2    False
dtype: bool
# 取整操作包含round, ceil, floor
s = pd.Series(pd.date_range('2020-1-1 20:35:00', '2020-1-1 22:35:00', freq='45min'))
# 0   2020-01-01 20:35:00
# 1   2020-01-01 21:20:00
# 2   2020-01-01 22:05:00
# dtype: datetime64[ns]

s.dt.round('1H')
# 0   2020-01-01 21:00:00
# 1   2020-01-01 21:00:00
# 2   2020-01-01 22:00:00
# dtype: datetime64[ns]

s.dt.ceil('1H')
# 0   2020-01-01 21:00:00
# 1   2020-01-01 22:00:00
# 2   2020-01-01 23:00:00
# dtype: datetime64[ns]

s.dt.floor('1H')
# 0   2020-01-01 20:00:00
# 1   2020-01-01 21:00:00
# 2   2020-01-01 22:00:00
# dtype: datetime64[ns]
0   2020-01-01 20:00:00
1   2020-01-01 21:00:00
2   2020-01-01 22:00:00
dtype: datetime64[ns]

时间戳的切片与索引

  • 利用dt对象和布尔条件联合使用
  • 利用切片,常用于连续时间戳
s = pd.Series(np.random.randint(2,size=366), index=pd.date_range('2020-01-01','2020-12-31'))
idx = pd.Series(s.index).dt
s.head()
# 2020-01-01    1
# 2020-01-02    1
# 2020-01-03    1
# 2020-01-04    0
# 2020-01-05    1
# Freq: D, dtype: int32
2020-01-01    0
2020-01-02    1
2020-01-03    1
2020-01-04    0
2020-01-05    0
Freq: D, dtype: int32
# 每月的第一天或者最后一天
s[(idx.is_month_start|idx.is_month_end).values].head()
2020-01-01    0
2020-01-31    1
2020-02-01    1
2020-02-29    1
2020-03-01    0
dtype: int32
# 双休日
s[idx.dayofweek.isin([5,6]).values].head()
2020-01-04    0
2020-01-05    0
2020-01-11    0
2020-01-12    1
2020-01-18    1
dtype: int32
# 取单日
s['2020-01-01']
# s['20200101']  可自动转换标准格式
0
# 取出七月
s['2020-07'].head()
2020-07-01    0
2020-07-02    0
2020-07-03    0
2020-07-04    0
2020-07-05    1
Freq: D, dtype: int32
# 取出5月初至7月15日
s['2020-05':'2020-7-15']
2020-05-01    0
2020-05-02    1
2020-05-03    1
2020-05-04    0
2020-05-05    0
             ..
2020-07-11    0
2020-07-12    1
2020-07-13    1
2020-07-14    0
2020-07-15    0
Freq: D, Length: 76, dtype: int32

时间差

可以理解为两个时间戳的差

Timedelta的生成

pd.Timestamp('20200102 08:00:00')-pd.Timestamp('20200101 07:35:00')
Timedelta('1 days 00:25:00')
pd.Timedelta(days=1, minutes=25)
Timedelta('1 days 00:25:00')
# 字符串生成
pd.Timedelta('1 days 00:25:00')
Timedelta('1 days 00:25:00')
# 通过to_timedelta生成时间差序列
s = pd.to_timedelta(df.Time_Record)
s.head()
0   0 days 00:04:34
1   0 days 00:04:20
2   0 days 00:05:22
3   0 days 00:04:08
4   0 days 00:05:22
Name: Time_Record, dtype: timedelta64[ns]
# 通过timedelta_range生成时间差序列
pd.timedelta_range('0s','1000s',freq='6min')
TimedeltaIndex(['0 days 00:00:00', '0 days 00:06:00', '0 days 00:12:00'], dtype='timedelta64[ns]', freq='6T')
# dt对象
# days, seconds(对天数取余后剩余的秒数), mircroseconds, nanoseconds
s.dt.seconds.head()
0    274
1    260
2    322
3    248
4    322
Name: Time_Record, dtype: int64
s.dt.total_seconds().head()
0    274.0
1    260.0
2    322.0
3    248.0
4    322.0
Name: Time_Record, dtype: float64
s.head()
0   2021-01-09 00:04:34
1   2021-01-09 00:04:20
2   2021-01-09 00:05:22
3   2021-01-09 00:04:08
4   2021-01-09 00:05:22
Name: Time_Record, dtype: datetime64[ns]
# 取整函数在dt对象上使用
pd.to_timedelta(df.Time_Record).dt.round('min')
0     0 days 00:05:00
1     0 days 00:04:00
2     0 days 00:05:00
3     0 days 00:04:00
4     0 days 00:05:00
            ...      
195   0 days 00:05:00
196   0 days 00:04:00
197   0 days 00:05:00
198   0 days 00:05:00
199   0 days 00:05:00
Name: Time_Record, Length: 200, dtype: timedelta64[ns]

Timedelta的运算

时间差支持的常用运算:

  • 与标量的乘法运算
  • 与时间戳的加减法运算
  • 与时间差的加减法与除法运算
td1 = pd.Timedelta(days=1)  # Timedelta('1 days 00:00:00')
td2 = pd.Timedelta(days=3)  # Timedelta('3 days 00:00:00')
ts = pd.Timestamp('20200101')
# 与标量的乘法运算
td1 * 2
Timedelta('2 days 00:00:00')
# 与时间差加减法
td2 - td1
Timedelta('2 days 00:00:00')
# 与时间戳加减法
ts + td1
Timestamp('2020-01-02 00:00:00')
# 与时间戳加减法
ts - td1
Timestamp('2019-12-31 00:00:00')
td1 = pd.timedelta_range(start='1 days', periods=5)
td1  # TimedeltaIndex(['1 days', '2 days', '3 days', '4 days', '5 days'], dtype='timedelta64[ns]', freq='D')
td1 * 5
TimedeltaIndex(['5 days', '10 days', '15 days', '20 days', '25 days'], dtype='timedelta64[ns]', freq='5D')
# 逐个相乘
td1 * pd.Series(list(range(5)))
0    0 days
1    2 days
2    6 days
3   12 days
4   20 days
dtype: timedelta64[ns]
td2 = pd.timedelta_range(start='12 hours', freq='2H', periods=5)
td2
TimedeltaIndex(['0 days 12:00:00', '0 days 14:00:00', '0 days 16:00:00',
                '0 days 18:00:00', '0 days 20:00:00'],
               dtype='timedelta64[ns]', freq='2H')
td1 - td2
TimedeltaIndex(['0 days 12:00:00', '1 days 10:00:00', '2 days 08:00:00',
                '3 days 06:00:00', '4 days 04:00:00'],
               dtype='timedelta64[ns]', freq=None)

日期偏置

Offset对象

  • + 表示获取离其最近的下一个日期
  • - 表示获取离其最近的上一个日期
# 求2020年9月第一个周一的日期
# 2020.08.31是周一
pd.Timestamp('20200831') + pd.offsets.WeekOfMonth(week=0, weekday=0)
Timestamp('2020-09-07 00:00:00')
# 求2020年9月7日后的第30个工作日
pd.Timestamp('20200907') + pd.offsets.BDay(30)
Timestamp('2020-10-19 00:00:00')

offsets.CDay

  • holidays参数对自定义的日期进行过滤,传入需要过滤的日期列表
  • weekmask参数对自定义的星期进行过滤,传入星期缩写字符(3个字母),只保留字符串中出现的日期
dr = pd.date_range('20200108', '20200111')
# DatetimeIndex(['2020-01-08', '2020-01-09', '2020-01-10', '2020-01-11'], dtype='datetime64[ns]', freq='D')
dr.to_series().dt.dayofweek
2020-01-08    2
2020-01-09    3
2020-01-10    4
2020-01-11    5
Freq: D, dtype: int64
# n=1表示增加一天
# holidays=['20200109']表示排除掉这天
# weekmask='Wed Fri'表示只留周三周五
my_filter = pd.offsets.CDay(n=1,weekmask='Wed Fri',holidays=['20200109'])
[i + my_filter for i in dr]
[Timestamp('2020-01-10 00:00:00'),
 Timestamp('2020-01-10 00:00:00'),
 Timestamp('2020-01-15 00:00:00'),
 Timestamp('2020-01-15 00:00:00')]

偏置字符串

# 月初
pd.date_range('20200101','20200331', freq='MS')
pd.date_range('20200101','20200331', freq=pd.offsets.MonthBegin())
# DatetimeIndex(['2020-01-01', '2020-02-01', '2020-03-01'], dtype='datetime64[ns]', freq='MS')

# 月末
pd.date_range('20200101','20200331', freq='M')
pd.date_range('20200101','20200331', freq=pd.offsets.MonthEnd())
# DatetimeIndex(['2020-01-31', '2020-02-29', '2020-03-31'], dtype='datetime64[ns]', freq='M')
DatetimeIndex(['2020-01-31', '2020-02-29', '2020-03-31'], dtype='datetime64[ns]', freq='M')
# 工作日
pd.date_range('20200101','20200110', freq='B')
pd.date_range('20200101','20200110', freq=pd.offsets.BDay())

# 周一
pd.date_range('20200101','20200201', freq='W-MON')
pd.date_range('20200101','20200201', freq=pd.offsets.CDay(weekmask='Mon'))

# 每月第一个周一
pd.date_range('20200101','20200201', freq='WOM-1MON')
pd.date_range('20200101','20200201', freq=pd.offsets.WeekOfMonth(week=0,weekday=0))
DatetimeIndex(['2020-01-06'], dtype='datetime64[ns]', freq='WOM-1MON')

时序中的滑窗与分组

滑动窗口

import matplotlib.pyplot as plt
idx = pd.date_range('20200101', '20201231', freq='B')
np.random.seed(2020)
data = np.random.randint(-1,2,len(idx)).cumsum() # 随机游动构造模拟序列
s = pd.Series(data,index=idx)
s.head()
2020-01-01   -1
2020-01-02   -2
2020-01-03   -1
2020-01-06   -1
2020-01-07   -2
Freq: B, dtype: int32
r = s.rolling('30D')
plt.plot(s)
plt.title('BOLL LINES')
Text(0.5, 1.0, 'BOLL LINES')

在这里插入图片描述

plt.plot(r.mean())
plt.plot(r.mean()+r.std()*2)
plt.plot(r.mean()-r.std()*2)

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-PNLH3EXo-1610278521418)(output_59_1.png)]

s.head()
2020-01-01   -1
2020-01-02   -2
2020-01-03   -1
2020-01-06   -1
2020-01-07   -2
Freq: B, dtype: int32
s.shift(freq='50D').head()
2020-02-20   -1
2020-02-21   -2
2020-02-22   -1
2020-02-25   -1
2020-02-26   -2
dtype: int32

重采样

# 对上面的序列计算每10天的均值
s.resample('10D').mean().head()
2020-01-01   -2.000000
2020-01-11   -3.166667
2020-01-21   -3.625000
2020-01-31   -4.000000
2020-02-10   -0.375000
Freq: 10D, dtype: float64
# 计算极差
# 自定义函数
s.resample('10D').apply(lambda x:x.max()-x.min()).head()
2020-01-01    3
2020-01-11    4
2020-01-21    4
2020-01-31    2
2020-02-10    4
Freq: 10D, dtype: int32

练习

Ex1:太阳辐射数据集

df = pd.read_csv('./data/solar.csv', usecols=['Data','Time','Radiation','Temperature'])
df.head(5)
DataTimeRadiationTemperature
09/29/2016 12:00:00 AM23:55:261.2148
19/29/2016 12:00:00 AM23:50:231.2148
29/29/2016 12:00:00 AM23:45:261.2348
39/29/2016 12:00:00 AM23:40:211.2148
49/29/2016 12:00:00 AM23:35:241.1748
# 将Datetime, Time合并为一个时间列Datetime,同时把它作为索引后排序
df['Datetime'] = df['Data'].str.split(' ')
df['Data'] = df['Data'].str.split(' ', expand=True).loc[:,0]
df['Datetime'] = df['Data']+' '+df['Time']
df['Datetime'] = pd.to_datetime(df['Datetime'].values)
df=df.set_index('Datetime')
df = df.sort_index()
df = df.drop(['Data','Time'],1)
df.head()
RadiationTemperature
Datetime
2016-09-01 00:00:082.5851
2016-09-01 00:05:102.8351
2016-09-01 00:20:062.1651
2016-09-01 00:25:052.2151
2016-09-01 00:30:092.2551
# 找出间隔时间的前三个最大值所对应的三组时间戳
time_diff = df.index.to_series().diff().sort_values(ascending=False).dt.total_seconds()
time_diff
Datetime
2016-12-08 11:10:42    224689.0
2016-12-01 00:00:02    104100.0
2016-10-01 00:00:19     86693.0
2016-09-16 07:40:15     53108.0
2016-09-09 18:45:22     26099.0
                         ...   
2016-09-29 15:05:23       193.0
2016-11-03 17:00:22       192.0
2016-09-17 13:20:34         6.0
2016-09-16 07:40:16         1.0
2016-09-01 00:00:08         NaN
Name: Datetime, Length: 32686, dtype: float64
# 这部分不知道怎么写才能返回上一个时间戳
# 参考答案
s = df.index.to_series().reset_index(drop=True).diff().dt.total_seconds()
max_3 = s.nlargest(3).index  # 返回普通索引值
df.index[max_3.union(max_3-1)]  # 找到当前索引和对应的上一个索引

# 是否存在一个大致的范围,使得绝大多数的间隔时间都落在这个区间中?如果存在,请对此范围内的样本间隔秒数画出柱状图,设置bins=50
res = s.mask((s>s.quantile(0.99))|(s<s.quantile(0.01)))
_ = plt.hist(res, bins=50)

在这里插入图片描述

# 温度与辐射量的6小时滑动相关系数
res = df.Radiation.rolling('6H').corr(df.Temperature)
res

# 以三点、九点、十五点、二十一点为分割,该观测所在时间区间的温度均值序列
# 本题6小时一分割,考察重采样的知识点
res = df.Temperature.resample('6H', origin='03:00:00').mean()

# 每个观测6小时前的辐射量(一般而言不会恰好取到,此时取最近时间戳对应的辐射量)
# 参考答案
my_dt = df.index.shift(freq='-6H')
int_loc = [df.index.get_loc(i, method='nearest') for i in my_dt]
res = df.Radiation.iloc[int_loc]
res.tail(3)

习题二没有做完,后面一定会补齐的(连同上一节落下的那道题)。最近杂七杂八事情实在是太多 抱歉。

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值