目录
时间戳
- 单独时间点即为时间戳,pandas中称为
Timestamp
,一系列时间戳可以组成DatetimeIndex
Timestamp的构造与属性
import numpy as np
import pandas as pd
ts = pd.Timestamp('2020-1-1 08:10:30')
ts
Timestamp('2020-01-01 08:10:30')
# 通过 year, month, day, hour, min, second 可以获取具体的数值
ts.year # 2020
ts.month # 1
ts.day # 1
ts.hour # 8
ts.minute # 10
ts.second # 30
Datetime序列的生成
to_datetime
# to_datetime 能将一系列时间戳格式的对象转化为datetime64[ns] 类型的时间序列
pd.to_datetime(['2020-1-1', '2020-1-3', '2020-1-6'])
pd.to_datetime(['2020-1-1', '2020-1-3', '2020-1-6']).year # Int64Index([2020, 2020, 2020], dtype='int64')
Int64Index([2020, 2020, 2020], dtype='int64')
df = pd.read_csv('data/learn_pandas.csv')
df.head()
School | Grade | Name | Gender | Height | Weight | Transfer | Test_Number | Test_Date | Time_Record | |
---|---|---|---|---|---|---|---|---|---|---|
0 | Shanghai Jiao Tong University | Freshman | Gaopeng Yang | Female | 158.9 | 46.0 | N | 1 | 2019/10/5 | 0:04:34 |
1 | Peking University | Freshman | Changqiang You | Male | 166.5 | 70.0 | N | 1 | 2019/9/4 | 0:04:20 |
2 | Shanghai Jiao Tong University | Senior | Mei Sun | Male | 188.9 | 89.0 | N | 2 | 2019/9/12 | 0:05:22 |
3 | Fudan University | Sophomore | Xiaojuan Sun | Female | NaN | 41.0 | N | 2 | 2020/1/3 | 0:04:08 |
4 | Fudan University | Sophomore | Gaojuan You | Male | 174.0 | 74.0 | N | 2 | 2019/11/6 | 0:05:22 |
s = pd.to_datetime(df.Test_Date)
s.head()
0 2019-10-05
1 2019-09-04
2 2019-09-12
3 2020-01-03
4 2019-11-06
Name: Test_Date, dtype: datetime64[ns]
# 极少数情况下,时间戳不满足转换时,可以强制使用format进行匹配
temp = pd.to_datetime(['2020\\1\\1','2020\\1\\3'], format='%Y\\%m\\%d')
temp
DatetimeIndex(['2020-01-01', '2020-01-03'], dtype='datetime64[ns]', freq=None)
# DatetimeIndex → datetime64[ns]
# 显式用Series转化
pd.Series(temp).head()
0 2020-01-01
1 2020-01-03
dtype: datetime64[ns]
# 把表的多列时间属性拼接转为时间序列的to_datetime操作
df_date_cols = pd.DataFrame({'year':[2020, 2020], 'month':[1,1,], 'day': [1, 2], 'hour': [10, 20], 'minute': [30, 50], 'second': [20, 40]})
df_date_cols
year | month | day | hour | minute | second | |
---|---|---|---|---|---|---|
0 | 2020 | 1 | 1 | 10 | 30 | 20 |
1 | 2020 | 1 | 2 | 20 | 50 | 40 |
pd.to_datetime(df_date_cols)
0 2020-01-01 10:30:20
1 2020-01-02 20:50:40
dtype: datetime64[ns]
date_range
是一种生成连续间隔时间的方法,注意开始或结束日期如果作为端点则会被包含
start
开始时间end
结束时间freq
时间间隔periods
时间戳个数
pd.date_range('2020-1-1','2020-1-21', freq='10D') # 包含两端
DatetimeIndex(['2020-01-01', '2020-01-11', '2020-01-21'], dtype='datetime64[ns]', freq='10D')
pd.date_range('2020-1-1','2020-2-28', freq='10D') # 结束日期无法取到,freq不为10天
DatetimeIndex(['2020-01-01', '2020-01-11', '2020-01-21', '2020-01-31',
'2020-02-10', '2020-02-20'],
dtype='datetime64[ns]', freq='10D')
s = pd.Series(np.random.rand(5), index=pd.to_datetime(['2020-1-%d'%i for i in range(1,10,2)]))
# 2020-01-01 0.068769
# 2020-01-03 0.096169
# 2020-01-05 0.309931
# 2020-01-07 0.176197
# 2020-01-09 0.038457
# dtype: float64
# 改变序列采样频率
s.asfreq('D')
# 2020-01-01 0.068769
# 2020-01-02 NaN
# 2020-01-03 0.096169
# 2020-01-04 NaN
# 2020-01-05 0.309931
# 2020-01-06 NaN
# 2020-01-07 0.176197
# 2020-01-08 NaN
# 2020-01-09 0.038457
# Freq: D, dtype: float64
dt对象
- 取出时间相关的属性
- 判断时间戳是否满足条件
- 取整操作
# 取出时间相关属性
# date, time, year, month, day, hour, minute, second, microsecond, nanosecond, dayofweek, dayofyear, weekofyear, daysinmonth(月中第几天), quarter(季度)
s = pd.Series(pd.date_range('2020-1-1','2020-1-3', freq='D'))
# 日期
s.dt.date
# 0 2020-01-01
# 1 2020-01-02
# 2 2020-01-03
# dtype: object
# 时间
s.dt.time
# 0 00:00:00
# 1 00:00:00
# 2 00:00:00
# dtype: object
# 月中有几天
s.dt.daysinmonth
# 0 31
# 1 31
# 2 31
# dtype: int64
# 周几
s.dt.dayofweek
# 0 2
# 1 3
# 2 4
# dtype: int64
# 月名(方法)
s.dt.month_name()
# 0 January
# 1 January
# 2 January
# dtype: object
# 星期名(方法)
s.dt.day_name()
# 0 Wednesday
# 1 Thursday
# 2 Friday
# dtype: object
# 判断操作主要用于测试是否为月/季/年的第一天或者最后一天
# 还可选 is_quarter/month_start/month_end
s.dt.is_year_start
s.dt.is_year_end
0 False
1 False
2 False
dtype: bool
# 取整操作包含round, ceil, floor
s = pd.Series(pd.date_range('2020-1-1 20:35:00', '2020-1-1 22:35:00', freq='45min'))
# 0 2020-01-01 20:35:00
# 1 2020-01-01 21:20:00
# 2 2020-01-01 22:05:00
# dtype: datetime64[ns]
s.dt.round('1H')
# 0 2020-01-01 21:00:00
# 1 2020-01-01 21:00:00
# 2 2020-01-01 22:00:00
# dtype: datetime64[ns]
s.dt.ceil('1H')
# 0 2020-01-01 21:00:00
# 1 2020-01-01 22:00:00
# 2 2020-01-01 23:00:00
# dtype: datetime64[ns]
s.dt.floor('1H')
# 0 2020-01-01 20:00:00
# 1 2020-01-01 21:00:00
# 2 2020-01-01 22:00:00
# dtype: datetime64[ns]
0 2020-01-01 20:00:00
1 2020-01-01 21:00:00
2 2020-01-01 22:00:00
dtype: datetime64[ns]
时间戳的切片与索引
- 利用
dt
对象和布尔条件联合使用 - 利用切片,常用于连续时间戳
s = pd.Series(np.random.randint(2,size=366), index=pd.date_range('2020-01-01','2020-12-31'))
idx = pd.Series(s.index).dt
s.head()
# 2020-01-01 1
# 2020-01-02 1
# 2020-01-03 1
# 2020-01-04 0
# 2020-01-05 1
# Freq: D, dtype: int32
2020-01-01 0
2020-01-02 1
2020-01-03 1
2020-01-04 0
2020-01-05 0
Freq: D, dtype: int32
# 每月的第一天或者最后一天
s[(idx.is_month_start|idx.is_month_end).values].head()
2020-01-01 0
2020-01-31 1
2020-02-01 1
2020-02-29 1
2020-03-01 0
dtype: int32
# 双休日
s[idx.dayofweek.isin([5,6]).values].head()
2020-01-04 0
2020-01-05 0
2020-01-11 0
2020-01-12 1
2020-01-18 1
dtype: int32
# 取单日
s['2020-01-01']
# s['20200101'] 可自动转换标准格式
0
# 取出七月
s['2020-07'].head()
2020-07-01 0
2020-07-02 0
2020-07-03 0
2020-07-04 0
2020-07-05 1
Freq: D, dtype: int32
# 取出5月初至7月15日
s['2020-05':'2020-7-15']
2020-05-01 0
2020-05-02 1
2020-05-03 1
2020-05-04 0
2020-05-05 0
..
2020-07-11 0
2020-07-12 1
2020-07-13 1
2020-07-14 0
2020-07-15 0
Freq: D, Length: 76, dtype: int32
时间差
可以理解为两个时间戳的差
Timedelta的生成
pd.Timestamp('20200102 08:00:00')-pd.Timestamp('20200101 07:35:00')
Timedelta('1 days 00:25:00')
pd.Timedelta(days=1, minutes=25)
Timedelta('1 days 00:25:00')
# 字符串生成
pd.Timedelta('1 days 00:25:00')
Timedelta('1 days 00:25:00')
# 通过to_timedelta生成时间差序列
s = pd.to_timedelta(df.Time_Record)
s.head()
0 0 days 00:04:34
1 0 days 00:04:20
2 0 days 00:05:22
3 0 days 00:04:08
4 0 days 00:05:22
Name: Time_Record, dtype: timedelta64[ns]
# 通过timedelta_range生成时间差序列
pd.timedelta_range('0s','1000s',freq='6min')
TimedeltaIndex(['0 days 00:00:00', '0 days 00:06:00', '0 days 00:12:00'], dtype='timedelta64[ns]', freq='6T')
# dt对象
# days, seconds(对天数取余后剩余的秒数), mircroseconds, nanoseconds
s.dt.seconds.head()
0 274
1 260
2 322
3 248
4 322
Name: Time_Record, dtype: int64
s.dt.total_seconds().head()
0 274.0
1 260.0
2 322.0
3 248.0
4 322.0
Name: Time_Record, dtype: float64
s.head()
0 2021-01-09 00:04:34
1 2021-01-09 00:04:20
2 2021-01-09 00:05:22
3 2021-01-09 00:04:08
4 2021-01-09 00:05:22
Name: Time_Record, dtype: datetime64[ns]
# 取整函数在dt对象上使用
pd.to_timedelta(df.Time_Record).dt.round('min')
0 0 days 00:05:00
1 0 days 00:04:00
2 0 days 00:05:00
3 0 days 00:04:00
4 0 days 00:05:00
...
195 0 days 00:05:00
196 0 days 00:04:00
197 0 days 00:05:00
198 0 days 00:05:00
199 0 days 00:05:00
Name: Time_Record, Length: 200, dtype: timedelta64[ns]
Timedelta的运算
时间差支持的常用运算:
- 与标量的乘法运算
- 与时间戳的加减法运算
- 与时间差的加减法与除法运算
td1 = pd.Timedelta(days=1) # Timedelta('1 days 00:00:00')
td2 = pd.Timedelta(days=3) # Timedelta('3 days 00:00:00')
ts = pd.Timestamp('20200101')
# 与标量的乘法运算
td1 * 2
Timedelta('2 days 00:00:00')
# 与时间差加减法
td2 - td1
Timedelta('2 days 00:00:00')
# 与时间戳加减法
ts + td1
Timestamp('2020-01-02 00:00:00')
# 与时间戳加减法
ts - td1
Timestamp('2019-12-31 00:00:00')
td1 = pd.timedelta_range(start='1 days', periods=5)
td1 # TimedeltaIndex(['1 days', '2 days', '3 days', '4 days', '5 days'], dtype='timedelta64[ns]', freq='D')
td1 * 5
TimedeltaIndex(['5 days', '10 days', '15 days', '20 days', '25 days'], dtype='timedelta64[ns]', freq='5D')
# 逐个相乘
td1 * pd.Series(list(range(5)))
0 0 days
1 2 days
2 6 days
3 12 days
4 20 days
dtype: timedelta64[ns]
td2 = pd.timedelta_range(start='12 hours', freq='2H', periods=5)
td2
TimedeltaIndex(['0 days 12:00:00', '0 days 14:00:00', '0 days 16:00:00',
'0 days 18:00:00', '0 days 20:00:00'],
dtype='timedelta64[ns]', freq='2H')
td1 - td2
TimedeltaIndex(['0 days 12:00:00', '1 days 10:00:00', '2 days 08:00:00',
'3 days 06:00:00', '4 days 04:00:00'],
dtype='timedelta64[ns]', freq=None)
日期偏置
Offset对象
+
表示获取离其最近的下一个日期-
表示获取离其最近的上一个日期
# 求2020年9月第一个周一的日期
# 2020.08.31是周一
pd.Timestamp('20200831') + pd.offsets.WeekOfMonth(week=0, weekday=0)
Timestamp('2020-09-07 00:00:00')
# 求2020年9月7日后的第30个工作日
pd.Timestamp('20200907') + pd.offsets.BDay(30)
Timestamp('2020-10-19 00:00:00')
offsets.CDay
holidays
参数对自定义的日期进行过滤,传入需要过滤的日期列表weekmask
参数对自定义的星期进行过滤,传入星期缩写字符(3个字母),只保留字符串中出现的日期
dr = pd.date_range('20200108', '20200111')
# DatetimeIndex(['2020-01-08', '2020-01-09', '2020-01-10', '2020-01-11'], dtype='datetime64[ns]', freq='D')
dr.to_series().dt.dayofweek
2020-01-08 2
2020-01-09 3
2020-01-10 4
2020-01-11 5
Freq: D, dtype: int64
# n=1表示增加一天
# holidays=['20200109']表示排除掉这天
# weekmask='Wed Fri'表示只留周三周五
my_filter = pd.offsets.CDay(n=1,weekmask='Wed Fri',holidays=['20200109'])
[i + my_filter for i in dr]
[Timestamp('2020-01-10 00:00:00'),
Timestamp('2020-01-10 00:00:00'),
Timestamp('2020-01-15 00:00:00'),
Timestamp('2020-01-15 00:00:00')]
偏置字符串
# 月初
pd.date_range('20200101','20200331', freq='MS')
pd.date_range('20200101','20200331', freq=pd.offsets.MonthBegin())
# DatetimeIndex(['2020-01-01', '2020-02-01', '2020-03-01'], dtype='datetime64[ns]', freq='MS')
# 月末
pd.date_range('20200101','20200331', freq='M')
pd.date_range('20200101','20200331', freq=pd.offsets.MonthEnd())
# DatetimeIndex(['2020-01-31', '2020-02-29', '2020-03-31'], dtype='datetime64[ns]', freq='M')
DatetimeIndex(['2020-01-31', '2020-02-29', '2020-03-31'], dtype='datetime64[ns]', freq='M')
# 工作日
pd.date_range('20200101','20200110', freq='B')
pd.date_range('20200101','20200110', freq=pd.offsets.BDay())
# 周一
pd.date_range('20200101','20200201', freq='W-MON')
pd.date_range('20200101','20200201', freq=pd.offsets.CDay(weekmask='Mon'))
# 每月第一个周一
pd.date_range('20200101','20200201', freq='WOM-1MON')
pd.date_range('20200101','20200201', freq=pd.offsets.WeekOfMonth(week=0,weekday=0))
DatetimeIndex(['2020-01-06'], dtype='datetime64[ns]', freq='WOM-1MON')
时序中的滑窗与分组
滑动窗口
import matplotlib.pyplot as plt
idx = pd.date_range('20200101', '20201231', freq='B')
np.random.seed(2020)
data = np.random.randint(-1,2,len(idx)).cumsum() # 随机游动构造模拟序列
s = pd.Series(data,index=idx)
s.head()
2020-01-01 -1
2020-01-02 -2
2020-01-03 -1
2020-01-06 -1
2020-01-07 -2
Freq: B, dtype: int32
r = s.rolling('30D')
plt.plot(s)
plt.title('BOLL LINES')
Text(0.5, 1.0, 'BOLL LINES')
plt.plot(r.mean())
plt.plot(r.mean()+r.std()*2)
plt.plot(r.mean()-r.std()*2)
s.head()
2020-01-01 -1
2020-01-02 -2
2020-01-03 -1
2020-01-06 -1
2020-01-07 -2
Freq: B, dtype: int32
s.shift(freq='50D').head()
2020-02-20 -1
2020-02-21 -2
2020-02-22 -1
2020-02-25 -1
2020-02-26 -2
dtype: int32
重采样
# 对上面的序列计算每10天的均值
s.resample('10D').mean().head()
2020-01-01 -2.000000
2020-01-11 -3.166667
2020-01-21 -3.625000
2020-01-31 -4.000000
2020-02-10 -0.375000
Freq: 10D, dtype: float64
# 计算极差
# 自定义函数
s.resample('10D').apply(lambda x:x.max()-x.min()).head()
2020-01-01 3
2020-01-11 4
2020-01-21 4
2020-01-31 2
2020-02-10 4
Freq: 10D, dtype: int32
练习
Ex1:太阳辐射数据集
df = pd.read_csv('./data/solar.csv', usecols=['Data','Time','Radiation','Temperature'])
df.head(5)
Data | Time | Radiation | Temperature | |
---|---|---|---|---|
0 | 9/29/2016 12:00:00 AM | 23:55:26 | 1.21 | 48 |
1 | 9/29/2016 12:00:00 AM | 23:50:23 | 1.21 | 48 |
2 | 9/29/2016 12:00:00 AM | 23:45:26 | 1.23 | 48 |
3 | 9/29/2016 12:00:00 AM | 23:40:21 | 1.21 | 48 |
4 | 9/29/2016 12:00:00 AM | 23:35:24 | 1.17 | 48 |
# 将Datetime, Time合并为一个时间列Datetime,同时把它作为索引后排序
df['Datetime'] = df['Data'].str.split(' ')
df['Data'] = df['Data'].str.split(' ', expand=True).loc[:,0]
df['Datetime'] = df['Data']+' '+df['Time']
df['Datetime'] = pd.to_datetime(df['Datetime'].values)
df=df.set_index('Datetime')
df = df.sort_index()
df = df.drop(['Data','Time'],1)
df.head()
Radiation | Temperature | |
---|---|---|
Datetime | ||
2016-09-01 00:00:08 | 2.58 | 51 |
2016-09-01 00:05:10 | 2.83 | 51 |
2016-09-01 00:20:06 | 2.16 | 51 |
2016-09-01 00:25:05 | 2.21 | 51 |
2016-09-01 00:30:09 | 2.25 | 51 |
# 找出间隔时间的前三个最大值所对应的三组时间戳
time_diff = df.index.to_series().diff().sort_values(ascending=False).dt.total_seconds()
time_diff
Datetime
2016-12-08 11:10:42 224689.0
2016-12-01 00:00:02 104100.0
2016-10-01 00:00:19 86693.0
2016-09-16 07:40:15 53108.0
2016-09-09 18:45:22 26099.0
...
2016-09-29 15:05:23 193.0
2016-11-03 17:00:22 192.0
2016-09-17 13:20:34 6.0
2016-09-16 07:40:16 1.0
2016-09-01 00:00:08 NaN
Name: Datetime, Length: 32686, dtype: float64
# 这部分不知道怎么写才能返回上一个时间戳
# 参考答案
s = df.index.to_series().reset_index(drop=True).diff().dt.total_seconds()
max_3 = s.nlargest(3).index # 返回普通索引值
df.index[max_3.union(max_3-1)] # 找到当前索引和对应的上一个索引
# 是否存在一个大致的范围,使得绝大多数的间隔时间都落在这个区间中?如果存在,请对此范围内的样本间隔秒数画出柱状图,设置bins=50
res = s.mask((s>s.quantile(0.99))|(s<s.quantile(0.01)))
_ = plt.hist(res, bins=50)
# 温度与辐射量的6小时滑动相关系数
res = df.Radiation.rolling('6H').corr(df.Temperature)
res
# 以三点、九点、十五点、二十一点为分割,该观测所在时间区间的温度均值序列
# 本题6小时一分割,考察重采样的知识点
res = df.Temperature.resample('6H', origin='03:00:00').mean()
# 每个观测6小时前的辐射量(一般而言不会恰好取到,此时取最近时间戳对应的辐射量)
# 参考答案
my_dt = df.index.shift(freq='-6H')
int_loc = [df.index.get_loc(i, method='nearest') for i in my_dt]
res = df.Radiation.iloc[int_loc]
res.tail(3)
习题二没有做完,后面一定会补齐的(连同上一节落下的那道题)。最近杂七杂八事情实在是太多 抱歉。