在我最后一篇关于查找日期/时间与周末的距离的文章之后,Hadley Wickham建议我可以通过向量化来改善功能……
@markhneedham使用pmin(pmax(dateToLookup – before,0),pmax(after – dateToLookup,0))/ dhours(1)进行矢量化
— Hadley Wickham(@hadleywickham) 2014年12月14日
…所以我想我会尝试向量化我最近编写的其他一些功能,并显示这两个版本。
我发现以下文章对解释向量化以及为什么要进行向量化很有用:
让我们开始吧。
距离周末
我们想找出距周末(即最近的周六/周日)有几个小时的特定日期/时间。 我们将使用以下库和日期/时间集:
library(dplyr)
library(lubridate)
library(geosphere)
options("scipen"=100, "digits"=4)
times = ymd_hms("2002-01-01 17:00:00") + c(0:99) * hours(1)
data = data.frame(time = times)
> data %>% head()
time
1 2002-01-01 17:00:00
2 2002-01-01 18:00:00
3 2002-01-01 19:00:00
4 2002-01-01 20:00:00
5 2002-01-01 21:00:00
6 2002-01-01 22:00:00
首先让我们看一下非矢量化版本:
distanceFromWeekend = function(dateToLookup) {
before = floor_date(dateToLookup, "week") + hours(23) + minutes(59) + seconds(59)
after = ceiling_date(dateToLookup, "week") - days(1)
timeToBefore = dateToLookup - before
timeToAfter = after - dateToLookup
if(timeToBefore < 0 || timeToAfter < 0) {
0
} else {
if(timeToBefore < timeToAfter) {
timeToBefore / dhours(1)
} else {
timeToAfter / dhours(1)
}
}
}
现在,让我们针对我们的数据框运行它:
> system.time(
data %>% mutate(ind = row_number()) %>% group_by(ind) %>% mutate(dist = distanceFromWeekend(time))
)
user system elapsed
1.837 0.020 1.884
现在,对于Hadley的矢量化版本:
distanceFromWeekendVectorised = function(dateToLookup) {
before = floor_date(dateToLookup, "week") + hours(23) + minutes(59) + seconds(59)
after = ceiling_date(dateToLookup, "week") - days(1)
pmin(pmax(dateToLookup - before, 0), pmax(after - dateToLookup, 0)) / dhours(1)
}
> system.time(data %>% mutate(dist = distanceFromWeekendVectorised(time)))
user system elapsed
0.020 0.001 0.023
提取开始日期
我的下一个示例是清理Google趋势数据并从CSV文件中的单元格中提取开始日期。
我们将使用以下数据框:
googleTrends = read.csv("/Users/markneedham/Downloads/report.csv", row.names=NULL)
names(googleTrends) = c("week", "score")
> googleTrends %>% head(10)
week score
1 Worldwide; 2004 - present
2 Interest over time
3 Week neo4j
4 2004-01-04 - 2004-01-10 0
5 2004-01-11 - 2004-01-17 0
6 2004-01-18 - 2004-01-24 0
7 2004-01-25 - 2004-01-31 0
8 2004-02-01 - 2004-02-07 0
9 2004-02-08 - 2004-02-14 0
10 2004-02-15 - 2004-02-21 0
非矢量化版本如下所示:
> system.time(
googleTrends %>%
mutate(ind = row_number()) %>%
group_by(ind) %>%
mutate(dates = strsplit(week, " - "),
start = dates[[1]][1] %>% strptime("%Y-%m-%d") %>% as.character())
)
user system elapsed
0.215 0.000 0.214
在这种情况下,实际上不可能使用strsplit对代码进行矢量化处理,因此我们需要使用其他方法。 Antonios向我展示了如何使用substr做到这一点:
> system.time(googleTrends %>% mutate(start = substr(week, 1, 10) %>% ymd()))
user system elapsed
0.018 0.000 0.017
计算haversine距离
我想计算出从一系列场所到伦敦市中心的巨大圆距离。 我从这个数据框开始:
centre = c(-0.129581, 51.516578)
venues = read.csv("/tmp/venues.csv")
> venues %>% head()
venue lat lon
1 Skills Matter 51.52 -0.09911
2 Skinkers 51.50 -0.08387
3 Theodore Bullfrog 51.51 -0.12375
4 The Skills Matter eXchange 51.52 -0.09923
5 The Guardian 51.53 -0.12234
6 White Bear Yard 51.52 -0.10980
我的非向量化版本如下所示:
> system.time(venues %>%
mutate(distanceFromCentre = by(venues, 1:nrow(venues), function(row) { distHaversine(c(row$lon, row$lat), centre) }))
)
user system elapsed
0.034 0.000 0.033
它非常快,但我们可以做得更好–如果第一个参数lon / lat值的矩阵而不是向量,则distHaversine函数允许我们计算多个距离:
> system.time(
venues %>% mutate(distanceFromCentre = distHaversine(cbind(venues$lon, venues$lat), centre))
)
user system elapsed
0.001 0.000 0.001
一个我不知道的...
最后,我有一个我无法弄清楚如何向量化的函数,但是也许某个R技术水平比我高的人?
我有一个数据框,其中包含各个NoSQL London组的累积成员计数 :
cumulativeMeetupMembers = read.csv("/tmp/cumulativeMeetupMembers.csv")
> cumulativeMeetupMembers %>% sample_n(10)
g.name dayMonthYear n
4734 Hadoop Users Group UK 2013-10-26 1144
4668 Hadoop Users Group UK 2013-08-03 979
4936 Hadoop Users Group UK 2014-07-31 1644
5150 Hive London 2012-10-15 109
8020 Neo4j - London User Group 2014-03-15 826
7666 Neo4j - London User Group 2012-08-06 78
1030 Big Data London 2013-03-01 1416
6500 London MongoDB User Group 2013-09-21 952
8290 Oracle Big Data 4 the Enterprise 2012-06-04 61
2584 Data Science London 2012-03-20 285
我想找出某个特定日期的小组成员人数。 例如,鉴于以下数据...
> cumulativeMeetupMembers %>% head(10)
g.name dayMonthYear n
1 Big Data / Data Science / Data Analytics Jobs 2013-01-29 1
2 Big Data / Data Science / Data Analytics Jobs 2013-02-06 15
3 Big Data / Data Science / Data Analytics Jobs 2013-02-07 28
4 Big Data / Data Science / Data Analytics Jobs 2013-02-10 31
5 Big Data / Data Science / Data Analytics Jobs 2013-02-18 33
6 Big Data / Data Science / Data Analytics Jobs 2013-03-27 38
7 Big Data / Data Science / Data Analytics Jobs 2013-04-16 41
8 Big Data / Data Science / Data Analytics Jobs 2013-07-17 53
9 Big Data / Data Science / Data Analytics Jobs 2013-08-28 58
10 Big Data / Data Science / Data Analytics Jobs 2013-11-11 63
…2013年11月10日,“大数据/数据科学/数据分析工作”组的成员人数应为58。
我创建了包含组和随机日期的数据框:
dates = ymd("2014-09-01") + c(0:9) * weeks(1)
groups = cumulativeMeetupMembers %>% distinct(g.name) %>% select(g.name)
groupsOnDate = merge(dates, groups)
names(groupsOnDate) = c('date', 'name')
> groupsOnDate %>% sample_n(10)
date name
156 2014-10-06 GridGain London
153 2014-09-15 GridGain London
70 2014-11-03 Couchbase London
185 2014-09-29 Hadoop Users Group UK
105 2014-09-29 Data Science London
137 2014-10-13 Equal Experts Technical Meetup Group
360 2014-11-03 Scale Warriors of London
82 2014-09-08 Data Science & Business Analytics London Meetup
233 2014-09-15 London ElasticSearch User Group
84 2014-09-22 Data Science & Business Analytics London Meetup
非矢量化版本如下所示:
memberCount = function(meetupMembers) {
function(groupName, date) {
(meetupMembers %>%
filter(g.name == groupName & dayMonthYear < date) %>% do(tail(., 1)))$n
}
}
findMemberCount = memberCount(cumulativeMeetupMembers)
> system.time(groupsOnDate %>% mutate(groupMembers = by(groupsOnDate, 1:nrow(groupsOnDate), function(row) {
findMemberCount(row$name, as.character(row$date))
}) %>%
cbind() %>%
as.vector() ))
user system elapsed
2.259 0.005 2.269
输出看起来像这样:
date name groupMembers
116 2014-10-06 DeNormalised London 157
322 2014-09-08 OpenCredo Tech Workshops 7
71 2014-09-01 Data Enthusiasts London
233 2014-09-15 London ElasticSearch User Group 614
171 2014-09-01 HPC & GPU Supercomputing Group of London 80
109 2014-10-27 Data Science London 3632
20 2014-11-03 Big Data Developers in London 708
42 2014-09-08 Big Data Week London Meetup 96
127 2014-10-13 Enterprise Search London Meetup 575
409 2014-10-27 Women in Data 548
我尝试了许多不同的方法,但是还无法提供一个版本,该版本无法让我将所有行都传递给memberCount并一次性计算每一行的计数。
欢迎任何想法/建议/提示!
翻译自: https://www.javacodegeeks.com/2014/12/r-vectorising-all-the-things.html