import numpy as np
import pandas as pd
1.时间戳-Timestamp ,一系列的时间戳可以组成DatetimeIndex,把它放到Series中变成datetime64[ns], 涉及时区 datetime64[ns, tz] tz-timezone
2.时间差 两个Timestamp做差就得到时间差,用Timedelta表示。 一系列的时间差组成TimedeltaIndex,放到Series中,Series类型就变为了timedelta64[ns]
3.时间段 在时间区间持续性地动作, Period, 一系列的时间段组成PeriodIndex, 把它放到Series中,Series类型就变成了Period
4. 日期偏置 DataOffset,pandas没有为一列时间偏置专门涉及存储类型10.2.1 Timestamp构造和属性
单个时间戳的生成利用pd.Timestamp实现
ts = pd.Timestamp('2020/1/1')
ts
Timestamp('2020-01-01 00:00:00')
ts = pd.Timestamp('2020/1/1 08:10:30')
ts
Timestamp('2020-01-01 08:10:30')
ts.year
2020
ts.month
1
ts.day
1
ts.hour
8
ts.minute
10
ts.second
30
# 获取时间戳表示的范围
pd.Timestamp.max
Timestamp('2262-04-11 23:47:16.854775807')
pd.Timestamp.min
Timestamp('1677-09-21 00:12:43.145225')
pd.Timestamp.max.year - pd.Timestamp.min.year
585
10.2.2 Datetime 序列的生成
一组时间戳可以组成时间序列,可用to_datetime和date_range来生成,
to_datetime可以把一列时间戳格式的对象转换成datetime64[ns]类型的时间序列
pd.to_datetime(['2020-1-1','2020-1-3', '2020-1-6'])
DatetimeIndex(['2020-01-01', '2020-01-03', '2020-01-06'], dtype='datetime64[ns]', freq=None)
df = pd.read_csv('learn_pandas.csv')
s = pd.to_datetime(df.Test_Date)
s.head()
0 2019-10-05
1 2019-09-04
2 2019-09-12
3 2020-01-03
4 2019-11-06
Name: Test_Date, dtype: datetime64[ns]
# 时间戳格式不满足转换时,可以强制使用format进行匹配
temp = pd.to_datetime(['2020\\1\\1', '2020\\1\\3'], format='%Y\\%m\\%d')
temp
DatetimeIndex(['2020-01-01', '2020-01-03'], dtype='datetime64[ns]', freq=None)
注意上面传入的是列表,而非pandas内部的Series,因此返回的是DatetimeIndex, 如果想转为datetime64[ns]的序列,需要显式用Series转化:
pd.Series(temp).head()
0 2020-01-01
1 2020-01-03
dtype: datetime64[ns]
还存在一种把表的多列时间属性拼接转为时间序列的to_datetime操作,此时的列名必须和以下给定的时间关键词列名一致:
df_date_cols = pd.DataFrame({'year':[2020,2020],
'month':[1,1],
'day':[1,2],
'hour':[10,20],
'minute':[30,50],
'second':[20,40]})
pd.to_datetime(df_date_cols)
0 2020-01-01 10:30:20
1 2020-01-02 20:50:40
dtype: datetime64[ns]
date_range是一种生成连续间隔时间的一种方法,参数有start,end,freq,periods,它们分别表示开始时间,结束时间,时间间隔,时间戳个数。其中3个参数确定了,剩下一个参数就确定了,
pd.date_range('2020-1-1', '2020-1-21', freq='10D')
DatetimeIndex(['2020-01-01', '2020-01-11', '2020-01-21'], dtype='datetime64[ns]', freq='10D')
pd.date_range('2020-1-1', '2020-2-28', freq='10D')
DatetimeIndex(['2020-01-01', '2020-01-11', '2020-01-21', '2020-01-31',
'2020-02-10', '2020-02-20'],
dtype='datetime64[ns]', freq='10D')
pd.date_range('2020-1-1', '2020-2-28', periods=6)
DatetimeIndex(['2020-01-01 00:00:00', '2020-01-12 14:24:00',
'2020-01-24 04:48:00', '2020-02-04 19:12:00',
'2020-02-16 09:36:00', '2020-02-28 00:00:00'],
dtype='datetime64[ns]', freq=None)
改变序列采样频率的方法asfreq,它能够根据给定的freq对序列进行类似于reindex的操作
s = pd.Series(np.random.rand(5),
index=pd.to_datetime(['2020-1-%d'%i for i in range(1,10,2)]))
s.head()
2020-01-01 0.270221
2020-01-03 0.604583
2020-01-05 0.721043
2020-01-07 0.162300
2020-01-09 0.372285
dtype: float64
s.asfreq('D').head()
2020-01-01 0.270221
2020-01-02 NaN
2020-01-03 0.604583
2020-01-04 NaN
2020-01-05 0.721043
Freq: D, dtype: float64
s.asfreq('12H').head()
2020-01-01 00:00:00 0.270221
2020-01-01 12:00:00 NaN
2020-01-02 00:00:00 NaN
2020-01-02 12:00:00 NaN
2020-01-03 00:00:00 0.604583
Freq: 12H, dtype: float64
10.2.3 dt对象
在时序类型的序列上定义dt对象来 完成许多时间序列的相关操作,对于datetime64[ns]类型而言,可以分为:取出时间相关的属性、判断时间戳是否满足条件、取整操作
s = pd.Series(pd.date_range('2020-1-1', '2020-1-3', freq='D'))
s.dt.date
0 2020-01-01
1 2020-01-02
2 2020-01-03
dtype: object
s.dt.time
0 00:00:00
1 00:00:00
2 00:00:00
dtype: object
s.dt.day
0 1
1 2
2 3
dtype: int64
# 表示月中的第几天
s.dt.daysinmonth
0 31
1 31
2 31
dtype: int64
# 返回了周中的星期情况,
s.dt.dayofweek
0 2
1 3
2 4
dtype: int64
# 返回了周中的星期情况,周一为0、周二为1
s.dt.dayofweek
0 2
1 3
2 4
dtype: int64
# 返回英文的月名
s.dt.month_name()
0 January
1 January
2 January
dtype: object
# 返回英文的星期名
s.dt.day_name()
0 Wednesday
1 Thursday
2 Friday
dtype: object
# 判断操作是用于测试是否为月/季/年的第一天或者最后一天
s.dt.is_year_start
0 True
1 False
2 False
dtype: bool
s.dt.is_year_end
0 False
1 False
2 False
dtype: bool
# 第三类取整包含round, ceil, floor,它们的公共参数为freq,常用H,min,S(小时、分钟、秒)
s = pd.Series(pd.date_range('2020-1-1 20:35:00','2020-1-1 22:35:00', freq='45min'))
s
0 2020-01-01 20:35:00
1 2020-01-01 21:20:00
2 2020-01-01 22:05:00
dtype: datetime64[ns]
# 对于分钟四舍五入
s.dt.round('1H')
0 2020-01-01 21:00:00
1 2020-01-01 21:00:00
2 2020-01-01 22:00:00
dtype: datetime64[ns]
# +0.5取整
s.dt.ceil('1H')
0 2020-01-01 21:00:00
1 2020-01-01 22:00:00
2 2020-01-01 23:00:00
dtype: datetime64[ns]
# -0.5取整
s.dt.floor('1H')
0 2020-01-01 20:00:00
1 2020-01-01 21:00:00
2 2020-01-01 22:00:00
dtype: datetime64[ns]
10.2.4 时间戳的切片与索引
时间戳序列作为索引使用,如果想要选出某个子时间戳序列,第一类方法是利用dt对象和布尔条件联合使用,另一种方式是利用切片,后者而常用于连续时间戳
s = pd.Series(np.random.randint(2, size=366),
index=pd.date_range('2020-01-01', '2020-12-31'))
s.index
DatetimeIndex(['2020-01-01', '2020-01-02', '2020-01-03', '2020-01-04',
'2020-01-05', '2020-01-06', '2020-01-07', '2020-01-08',
'2020-01-09', '2020-01-10',
...
'2020-12-22', '2020-12-23', '2020-12-24', '2020-12-25',
'2020-12-26', '2020-12-27', '2020-12-28', '2020-12-29',
'2020-12-30', '2020-12-31'],
dtype='datetime64[ns]', length=366, freq='D')
idx = pd.Series(s.index).dt
s.head()
2020-01-01 1
2020-01-02 1
2020-01-03 1
2020-01-04 1
2020-01-05 1
Freq: D, dtype: int32
# 每月的第一天或者最后一天
s[(idx.is_month_start| idx.is_month_end).values].head()
2020-01-01 1
2020-01-31 0
2020-02-01 1
2020-02-29 0
2020-03-01 0
dtype: int32
# 双休日
s[idx.dayofweek.isin([5,6]).values].head()
2020-01-04 1
2020-01-05 1
2020-01-11 0
2020-01-12 1
2020-01-18 1
dtype: int32
# 取出单日值
s['2020-01-01']
1
# 自动转换为标准格式
s['20200101']
1
# 取出7月
s['2020-07'].head()
2020-07-01 1
2020-07-02 0
2020-07-03 1
2020-07-04 0
2020-07-05 1
Freq: D, dtype: int32
# 取出5月初至7月15日
s['2020-05': '2020-7-15'].head()
2020-05-01 0
2020-05-02 1
2020-05-03 1
2020-05-04 0
2020-05-05 0
Freq: D, dtype: int32
s['2020-05': '2020-7-15'].tail()
2020-07-11 1
2020-07-12 0
2020-07-13 0
2020-07-14 0
2020-07-15 1
Freq: D, dtype: int32
10.3 时间差
10.3.1 Timedelta的生成
时间差可以理解为两个时间戳的差,通过 pd.Timedelta来构造
# 大时间-小时间
pd.Timestamp('20200102 08:00:00')-pd.Timestamp('20200101 07:35:00')
Timedelta('1 days 00:25:00')
# 注意加day和minute都要加s
pd.Timedelta(days=1, minutes=25)
Timedelta('1 days 00:25:00')
# 字符串生成
pd.Timedelta('1 days 25 minutes')
Timedelta('1 days 00:25:00')
生成时间差序列的主要方式是pd.to_timedelta,其类型为timedelta64[ns]:
s=pd.to_timedelta(df.Time_Record)
s.head()
0 0 days 00:04:34
1 0 days 00:04:20
2 0 days 00:05:22
3 0 days 00:04:08
4 0 days 00:05:22
Name: Time_Record, dtype: timedelta64[ns]
与 date_range一样,时间差序列也可以用timedelta_range来生成,它们两者具有一致的参数
# 每个6min也就是360s,所以有在1000s中有3个值
pd.timedelta_range('0s', '1000s', freq='6min')
TimedeltaIndex(['0 days 00:00:00', '0 days 00:06:00', '0 days 00:12:00'], dtype='timedelta64[ns]', freq='6T')
# 1000s中有3个区间
pd.timedelta_range('0s', '1000s', periods=3)
TimedeltaIndex(['0 days 00:00:00', '0 days 00:08:20', '0 days 00:16:40'], dtype='timedelta64[ns]', freq=None)
对于Timedelta序列,同样定义了dt对象
而seconds不是指单纯的秒,而是对天数取余后剩余的秒数
# 如对于0来说, 0 days 00:04:34,天数取余为0,4*60+34=274
s.dt.seconds.head()
0 274
1 260
2 322
3 248
4 322
Name: Time_Record, dtype: int64
# 如果不想对天数取余而直接对应秒数,可以使用total_seconds
s.dt.total_seconds().head()
0 274.0
1 260.0
2 322.0
3 248.0
4 322.0
Name: Time_Record, dtype: float64
# 取整函数可以在dt对象上使用,四舍五入
pd.to_timedelta(df.Time_Record).dt.round('min').head()
0 0 days 00:05:00
1 0 days 00:04:00
2 0 days 00:05:00
3 0 days 00:04:00
4 0 days 00:05:00
Name: Time_Record, dtype: timedelta64[ns]
10.3.2 Timedelta的运算
时间差支持的运算有:1.与标量的乘法运算、与时间戳的加减法运算、与时间差的加减法与除法运算
td1 = pd.Timedelta(days=1)
td2 = pd.Timedelta(days=3)
ts = pd.Timestamp('20200101')
td1 * 2
Timedelta('2 days 00:00:00')
td2 - td1
Timedelta('2 days 00:00:00')
# 时间点+时间段
ts + td1
Timestamp('2020-01-02 00:00:00')
ts - td1
Timestamp('2019-12-31 00:00:00')
# 这些运算可以移植到时间差的序列上
td1 = pd.timedelta_range(start = '1 days', periods=5)
td2 = pd.timedelta_range(start = '12 hours', freq = '2H',periods=5)
ts = pd.date_range('20200101', '20200105')
td1 * 5
TimedeltaIndex(['5 days', '10 days', '15 days', '20 days', '25 days'], dtype='timedelta64[ns]', freq='5D')
td1 * pd.Series(list(range(5))) # 逐个相乘
0 0 days
1 2 days
2 6 days
3 12 days
4 20 days
dtype: timedelta64[ns]
# td1: 1d,2d,3d,4d,5d;
# td2: 12:00, 14:00, 16:00, 18:00, 20:00
td1 - td2
TimedeltaIndex(['0 days 12:00:00', '1 days 10:00:00', '2 days 08:00:00',
'3 days 06:00:00', '4 days 04:00:00'],
dtype='timedelta64[ns]', freq=None)
td1 + pd.Timestamp('20200101')
DatetimeIndex(['2020-01-02', '2020-01-03', '2020-01-04', '2020-01-05',
'2020-01-06'],
dtype='datetime64[ns]', freq='D')
# ts为20200101 - 20200105
td1 + ts
DatetimeIndex(['2020-01-02', '2020-01-04', '2020-01-06', '2020-01-08',
'2020-01-10'],
dtype='datetime64[ns]', freq=None)
10.4 日期偏置
10.4.1 offset对象
是一种和日历相关的特殊时间差
# 求2020年9月第一个周一的日期
pd.Timestamp('20200831') + pd.offsets.WeekOfMonth(week=0, weekday=0)
Timestamp('2020-09-07 00:00:00')
# 2020年9月7日后的第30个工作日是哪一天
pd.Timestamp('20200907') + pd.offsets.BDay(30)
Timestamp('2020-10-19 00:00:00')
offset对象在pd.offsets中被定义,使用+时获取离其最近的下一个日期,当使用-时获取离其最近的上一个日期
# 距离2020年8月31日当前月第一个周一
pd.Timestamp('20200831') - pd.offsets.WeekOfMonth(week=0, weekday=0)
Timestamp('2020-08-03 00:00:00')
# 距离2020年9月7日往前数30个工作日(不包含周末)
pd.Timestamp('20200907') - pd.offsets.BDay(30)
Timestamp('2020-07-27 00:00:00')
# 2020年9月最后一天
pd.Timestamp('20200907') + pd.offsets.MonthEnd()
Timestamp('2020-09-30 00:00:00')
一个特殊的Offset对象CDay, 其中holidays, weekmask参数能够分别对自定义的日期和星期进行过滤,前者传入了需要过滤的日期列表,后者传入的是三个字母的星期缩写构成的星期字符串,作用是只保留字符串中出现的星期
# 注意holidays传入的是需要过滤的日期列表, weekmask是三个字母的星期缩写的字符串,用空格隔开
my_filter = pd.offsets.CDay(n=1,weekmask='Wed Fri', holidays=['20200109'])
dr = pd.date_range('20200108','20200111')
# 取dr日期范围内日期,保留字符串中的出现的星期:周三~周六
dr.to_series().dt.dayofweek
2020-01-08 2
2020-01-09 3
2020-01-10 4
2020-01-11 5
Freq: D, dtype: int64
# 1=1 表示增加一天CDay,dr中第一天为2020108,下一天的20200109被排除了,且20200110是合法的周五,因此转为周五
# 20200115是合法筛选的周三
[i + my_filter for i in dr]
[Timestamp('2020-01-10 00:00:00'),
Timestamp('2020-01-10 00:00:00'),
Timestamp('2020-01-15 00:00:00'),
Timestamp('2020-01-15 00:00:00')]
10.4.2 偏置字符串
date_range的freq取值可用offset对象,offset对象绑定了日期偏置字符串(frequencies strings/ offset aliases),可以指定offset对应的字符串来替代使用
# 月初MS
pd.date_range('20200101', '20200331', freq='MS')
DatetimeIndex(['2020-01-01', '2020-02-01', '2020-03-01'], dtype='datetime64[ns]', freq='MS')
# 月末M
pd.date_range('20200101', '20200331', freq='M')
DatetimeIndex(['2020-01-31', '2020-02-29', '2020-03-31'], dtype='datetime64[ns]', freq='M')
# 工作日B
pd.date_range('20200101', '20200331', freq='B')
DatetimeIndex(['2020-01-01', '2020-01-02', '2020-01-03', '2020-01-06',
'2020-01-07', '2020-01-08', '2020-01-09', '2020-01-10',
'2020-01-13', '2020-01-14', '2020-01-15', '2020-01-16',
'2020-01-17', '2020-01-20', '2020-01-21', '2020-01-22',
'2020-01-23', '2020-01-24', '2020-01-27', '2020-01-28',
'2020-01-29', '2020-01-30', '2020-01-31', '2020-02-03',
'2020-02-04', '2020-02-05', '2020-02-06', '2020-02-07',
'2020-02-10', '2020-02-11', '2020-02-12', '2020-02-13',
'2020-02-14', '2020-02-17', '2020-02-18', '2020-02-19',
'2020-02-20', '2020-02-21', '2020-02-24', '2020-02-25',
'2020-02-26', '2020-02-27', '2020-02-28', '2020-03-02',
'2020-03-03', '2020-03-04', '2020-03-05', '2020-03-06',
'2020-03-09', '2020-03-10', '2020-03-11', '2020-03-12',
'2020-03-13', '2020-03-16', '2020-03-17', '2020-03-18',
'2020-03-19', '2020-03-20', '2020-03-23', '2020-03-24',
'2020-03-25', '2020-03-26', '2020-03-27', '2020-03-30',
'2020-03-31'],
dtype='datetime64[ns]', freq='B')
# 周日W-MON
pd.date_range('20200101', '20200331', freq='W-MON')
DatetimeIndex(['2020-01-06', '2020-01-13', '2020-01-20', '2020-01-27',
'2020-02-03', '2020-02-10', '2020-02-17', '2020-02-24',
'2020-03-02', '2020-03-09', '2020-03-16', '2020-03-23',
'2020-03-30'],
dtype='datetime64[ns]', freq='W-MON')
# 每月第一个周一 WOM-1MON
pd.date_range('20200101', '20200331', freq='WOM-1MON')
DatetimeIndex(['2020-01-06', '2020-02-03', '2020-03-02'], dtype='datetime64[ns]', freq='WOM-1MON')
# 以上方法等价于使用如下的Offset对象
# 月初
pd.date_range('20200101', '20200331',
freq=pd.offsets.MonthBegin())
DatetimeIndex(['2020-01-01', '2020-02-01', '2020-03-01'], dtype='datetime64[ns]', freq='MS')
# 月末
pd.date_range('20200101', '20200331',
freq=pd.offsets.MonthEnd())
DatetimeIndex(['2020-01-31', '2020-02-29', '2020-03-31'], dtype='datetime64[ns]', freq='M')
# 工作日
pd.date_range('20200101', '20200331',
freq=pd.offsets.BDay())
DatetimeIndex(['2020-01-01', '2020-01-02', '2020-01-03', '2020-01-06',
'2020-01-07', '2020-01-08', '2020-01-09', '2020-01-10',
'2020-01-13', '2020-01-14', '2020-01-15', '2020-01-16',
'2020-01-17', '2020-01-20', '2020-01-21', '2020-01-22',
'2020-01-23', '2020-01-24', '2020-01-27', '2020-01-28',
'2020-01-29', '2020-01-30', '2020-01-31', '2020-02-03',
'2020-02-04', '2020-02-05', '2020-02-06', '2020-02-07',
'2020-02-10', '2020-02-11', '2020-02-12', '2020-02-13',
'2020-02-14', '2020-02-17', '2020-02-18', '2020-02-19',
'2020-02-20', '2020-02-21', '2020-02-24', '2020-02-25',
'2020-02-26', '2020-02-27', '2020-02-28', '2020-03-02',
'2020-03-03', '2020-03-04', '2020-03-05', '2020-03-06',
'2020-03-09', '2020-03-10', '2020-03-11', '2020-03-12',
'2020-03-13', '2020-03-16', '2020-03-17', '2020-03-18',
'2020-03-19', '2020-03-20', '2020-03-23', '2020-03-24',
'2020-03-25', '2020-03-26', '2020-03-27', '2020-03-30',
'2020-03-31'],
dtype='datetime64[ns]', freq='B')
# 周一
pd.date_range('20200101', '20200331',
freq=pd.offsets.CDay(weekmask='Mon'))
DatetimeIndex(['2020-01-06', '2020-01-13', '2020-01-20', '2020-01-27',
'2020-02-03', '2020-02-10', '2020-02-17', '2020-02-24',
'2020-03-02', '2020-03-09', '2020-03-16', '2020-03-23',
'2020-03-30'],
dtype='datetime64[ns]', freq='C')
# 每月第一个周一
pd.date_range('20200101', '20200331',
freq=pd.offsets.WeekOfMonth(week=0, weekday=0))
DatetimeIndex(['2020-01-06', '2020-02-03', '2020-03-02'], dtype='datetime64[ns]', freq='WOM-1MON')
10.5 时序中的滑窗与分组
10.5.1 滑动窗口
时序的滑动窗口就是把滑动窗口用freq关键词代替,
在股票市场中有一个指标为BOLL 指标,它由中轨线、上轨线、下轨线这三根线构成,具体的计算方法分别是N 日均值线、N日均值加两倍N 日标准差线、N 日均值减两倍N 日标准差线。利用rolling 对象计算N=30 的BOLL 指标可以如下写出:
import matplotlib.pyplot as plt
idx = pd.date_range('20200101', '20201231', freq='B')
np.random.seed(2020)
# 随机游动构造模拟序列
data = np.random.randint(-1,2,len(idx)).cumsum()
s=pd.Series(data, index=idx)
s.head()
2020-01-01 -1
2020-01-02 -2
2020-01-03 -1
2020-01-06 -1
2020-01-07 -2
Freq: B, dtype: int32
r = s.rolling('30D')
plt.plot(s)
plt.title('BOLL LINES')
plt.plot(r.mean())
plt.plot(r.mean() + r.std()*2)
plt.plot(r.mean() - r.std()*2)
[<matplotlib.lines.Line2D at 0x245792dcca0>]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-oOlyR5YU-1610292030169)(output_104_1.png)]
# 在datetime64为索引的序列上使用shift函数,可以指定freq单位进行滑动
s.shift(freq='50D').head()
2020-02-20 -1
2020-02-21 -2
2020-02-22 -1
2020-02-25 -1
2020-02-26 -2
dtype: int32
datetime64[ns]的序列进行diff后就能够得到timedelta64[ns]的序列,这能够使用户方便地观察有序时间序列的间隔
my_series = pd.Series(s.index)
my_series.head()
0 2020-01-01
1 2020-01-02
2 2020-01-03
3 2020-01-06
4 2020-01-07
dtype: datetime64[ns]
# 后一个相对于前一个的时间之差
my_series.diff(1).head()
0 NaT
1 1 days
2 1 days
3 3 days
4 1 days
dtype: timedelta64[ns]
10.6.1 EX1:太阳辐射数据集1. 将Datetime, Time 合并为一个时间列Datetime ,同时把它作为索引后排序。2. 每条记录时间的间隔显然并不一致,请解决如下问题:
(a) 找出间隔时间的前三个最大值所对应的三组时间戳。
(b) 是否存在一个大致的范围,使得绝大多数的间隔时间都落在这个区间中?如果存在,请对此范围内的样本间隔秒数画出柱状图,设置bins=50。3. 求如下指标对应的Series :
(a) 温度与辐射量的6 小时滑动相关系数
(b) 以三点、九点、十五点、二十一点为分割,该观测所在时间区间的温度均值序列
© 每个观测6 小时前的辐射量(一般而言不会恰好取到,此时取最近时间戳对应的辐射量)
df = pd.read_csv('solar.csv', usecols=['Data','Time','Radiation','Temperature'])
solar_date = df.Data.str.extract('([/|\w]+\s).+')[0]
df['Data'] = pd.to_datetime(solar_date + df.Time)
df = df.drop(columns='Time').rename(columns={'Data':'Datetime'}).set_index('Datetime').sort_index()
df.head(3)
Radiation | Temperature | |
---|---|---|
Datetime | ||
2016-09-01 00:00:08 | 2.58 | 51 |
2016-09-01 00:05:10 | 2.83 | 51 |
2016-09-01 00:20:06 | 2.16 | 51 |
s = df.index.to_series().reset_index(drop=True).diff().dt.total_seconds()
max_3 = s.nlargest(3).index
df.index[max_3.union(max_3-1)]
DatetimeIndex(['2016-09-29 23:55:26', '2016-10-01 00:00:19',
'2016-11-29 19:05:02', '2016-12-01 00:00:02',
'2016-12-05 20:45:53', '2016-12-08 11:10:42'],
dtype='datetime64[ns]', name='Datetime', freq=None)
res = s.mask((s>s.quantile(0.99))|(s<s.quantile(0.01)))
_ = plt.hist(res, bins=50)
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-2llpwIQn-1610292030171)(output_115_0.png)]
res = df.Radiation.rolling('6H').corr(df.Temperature)
res.tail(3)
Datetime
2016-12-31 23:45:04 0.328574
2016-12-31 23:50:03 0.261883
2016-12-31 23:55:01 0.262406
dtype: float64
res = df.Temperature.resample('6H', origin='03:00:00').mean()
res.head(3)
Datetime
2016-08-31 21:00:00 51.218750
2016-09-01 03:00:00 50.033333
2016-09-01 09:00:00 59.379310
Freq: 6H, Name: Temperature, dtype: float64
my_dt = df.index.shift(freq='-6H')
int_loc = [df.index.get_loc(i, method='nearest') for i in my_dt]
res = df.Radiation.iloc[int_loc]
res.tail(3)
Datetime
2016-12-31 17:45:02 9.33
2016-12-31 17:50:01 8.49
2016-12-31 17:55:02 5.84
Name: Radiation, dtype: float64
- 统计如下指标:
(a) 每月上半月(15 号及之前)与下半月葡萄销量的比值
(b) 每月最后一天的生梨销量总和
© 每月最后一天工作日的生梨销量总和
(d) 每月最后五天的苹果销量均值
df = pd.read_csv('fruit.csv')
df.Date = pd.to_datetime(df.Date)
df_grape = df.query("Fruit == 'Grape'")
res = df_grape.groupby([np.where(df_grape.Date.dt.day<=15,'First', 'Second'),df_grape.Date.dt.month])['Sale'].mean().to_frame().unstack(0 ).droplevel(0,axis=1)
res = (res.First/res.Second).rename_axis('Month')
res.head()
Month
1 1.174998
2 0.968890
3 0.951351
4 1.020797
5 0.931061
dtype: float64
df[df.Date.dt.is_month_end].query("Fruit == 'Pear'").groupby('Date').Sale.sum().head()
Date
2019-01-31 847
2019-02-28 774
2019-03-31 761
2019-04-30 648
2019-05-31 616
Name: Sale, dtype: int64
df[df.Date.isin(pd.date_range('20190101', '20191231',freq='BM'))].query("Fruit == 'Pear'" ).groupby('Date').Sale.mean().head()
Date
2019-01-31 60.500000
2019-02-28 59.538462
2019-03-29 56.666667
2019-04-30 64.800000
2019-05-31 61.600000
Name: Sale, dtype: float64
target_dt = df.drop_duplicates().groupby(df.Date.drop_duplicates().dt.month)['Date'].nlargest(5).reset_index(drop=True)
res = df.set_index('Date').loc[target_dt].reset_index().query("Fruit == 'Apple'")
res = res.groupby(res.Date.dt.month)['Sale'].mean( ).rename_axis('Month')
res.head()
Month
1 65.313725
2 54.061538
3 59.325581
4 65.795455
5 57.465116
Name: Sale, dtype: float64
- 按月计算周一至周日各品种水果的平均记录条数,行索引外层为水果名称,内层为月份,列索引为星期。
month_order = ['January','February','March','April', 'May','June','July','August','September', 'October','November','December']
week_order = ['Mon','Tue','Wed','Thu','Fri','Sat','Sum']
group1 = df.Date.dt.month_name().astype('category').cat.reorder_categories( month_order, ordered=True)
group2 = df.Fruit
group3 = df.Date.dt.dayofweek.replace(dict(zip(range(7),week_order))).astype('category').cat.reorder_categories( week_order, ordered=True)
res = df.groupby([group1, group2,group3])['Sale'].count().to_frame( ).unstack(0).droplevel(0,axis=1)
res.head()
Date | January | February | March | April | May | June | July | August | September | October | November | December | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Fruit | Date | ||||||||||||
Apple | Mon | 46 | 43 | 43 | 47 | 43 | 40 | 41 | 38 | 59 | 42 | 39 | 45 |
Tue | 50 | 40 | 44 | 52 | 46 | 39 | 50 | 42 | 40 | 57 | 47 | 47 | |
Wed | 50 | 47 | 37 | 43 | 39 | 39 | 58 | 43 | 35 | 46 | 47 | 38 | |
Thu | 45 | 35 | 31 | 47 | 58 | 33 | 52 | 44 | 36 | 63 | 37 | 40 | |
Fri | 32 | 33 | 52 | 31 | 46 | 38 | 37 | 48 | 34 | 37 | 46 | 41 |
- 按天计算向前10 个工作日窗口的苹果销量均值序列,非工作日的值用上一个工作日的结果填充。
df_apple = df[(df.Fruit=='Apple')&( ~df.Date.dt.dayofweek.isin([5,6]))]
s = pd.Series(df_apple.Sale.values,index=df_apple.Date).groupby('Date').sum()
res = s.rolling('10D').mean().reindex(pd.date_range('20190101','20191231')).fillna(method='ffill')
res.head()
2019-01-01 189.000000
2019-01-02 335.500000
2019-01-03 520.333333
2019-01-04 527.750000
2019-01-05 527.750000
Freq: D, dtype: float64