R Programming - Reading Data

Reading Tabular Data

There are a few principle functions reading data into R

• read.table, read.csv, for reading tabular data
• readLines, for reading lines of a text file
• source, for reading in R code files(inverse of dump)
• dget, for reading in R code files (inverse of dput)
• load, for reading in saved workspaces
• unserialize, for reading single R objects in binary form

Writing Data

There are analogous functions for writing data to files

• write.table
• writeLines
• dump
• dput
• save
• serialize

Reading Data Files with read.table

The read.table function is one of the most commonly used functions for reading data. It has a few important arguments:

• file, the name of a file, or a connection
• header, logical indicating if the file has a header line,导入的数据第一行是否是列名
• sep, a string indicating how the columns are separated,分隔符，csv文件如果没有分隔符就会乱码
• colClasses, a character vector indicating the class of each column in the dataset
• nrows, the number of rows in the dataset,读取数据前多少行
• comment.char, a character string indicating the comment character
• skip, the number of lines to skip from the beginning,跳过前几行数据开始读取（有的数据前几行会有注释信息，借此跳过）
• stringAsFactors, should character variables be coded as factors?,读入的字符串是否转换成因子，不需要设置成false就行
• na.strings，告诉系统缺失值是什么符号表示的，遇到缺失值系统自动转换成NA
  For small to moderately sized datasets, you can usually call read.table without specifying any other arguments

> data <- read.table("foo.txt")

R will automatically

• skip lines that begin with a #
• figure out how many rows there are (and how much memory needs to be allocated)
• figure what type of variable is in each column of the table Telling R all these things directly makes R run faster and more efficiently
• read.csv is identical to read.table expect that the default separator is a comma

Reading in Larger Datasets with read.table

with much larger datasets, doing the following things will make your file easier and will prevent R from choking.

• Read the help page for read.table, which contains many hints
• Make a rough calculation of the memory required to store your dataset. If the dataset is larger than the amount of RAM on your computer, you can probably stop right here
• Set comment.cahr = "" if there are no commented lines in your life
• Use the colClasses argument. Specifying this option instead of using the default can make ‘read.table’ run MUCH faster, often twice as fast. In order to use this option, you have to know the class of each column in your data frame. If all of the columns are “numeric”, for example, then you can just set colClasses = "numeric". A quick an dirty way to figure out the classes of each column is the following:

initial <- read.table("datatable.txt", nrows=100)
classes <- aspply(initial, class)
tabAll <- read.table("datatable.txt", colClasses=classes)

• Set nrows. This doesn’t make R run faster but it helps with memory usage. A mild overestimate is okay. You can use the Unix tool wc to calculate the number of lines in a file.

Know Thy System

In general, when using R with larger datasets, it’s useful to know a few things about your system.

• How much memory is availble?
• What other applications are in use?
• Are there other users logged into the same system?
• What operating system?
• Is the OS 32 or 64 bit?

Calculating Memory Requirements

I have a data frame with 1,500,000 rows and 120 columns, all of which are numeric data. Roughly, how much memory is required to store this data frame?

1,500,000 x 120 x 8 bytes/numeric
=1440000000 bytes
=1440000000/ 2^20 bytes/MB
= 1,373.29 MB
= 1.34 GB

Textual Data Formats

• dumping and deputing are useful because the resulting textual format is edit-able, and in the case of corruption, potentially recoverable
• Unlike writing out a table or csv file, dump and put preserve the metadata (sacrificing some readability), so that another user doesn’t have to specify it all over again
• Textual formats can work much better with version control programs like subversion or git which can only track changes meaningfully in text files
• Textual formats can be longer-lived; if there is corruption somewhere in the file, it can be easier to fix the problem
• Textual formats adhere to the “Unix philosophy”
• Downside: The format is noe very space-efficient

dput-ting R Objects

Another way to pass data around is by departing the R object with spot and reading it back in using dget.

> y <- data.frame(a=1, b="a")
> dput(y)
structure(list(a = 1, b = "a"), class = "data.frame", row.names = c(NA, 
-1L))
> dput(y, file="y.R")
> new.y <- dget("y.R")
> new.y
  a b
1 1 a

dumping R Objects

Multiple objects can be deparsed using the dump function and read back in using source

> x <- "foo"
> y <- data.frame(a=1, b="a")
> dump(c("x","y"), file="data.R")
> rm(x,y)
> source("data.R")
> y
  a b
1 1 a
> x
[1] "foo"

Connections: Interfaces to the Outside World

Data are read in using connection interfaces. Connections can be made to files (most common) or to other more exotic things.

• file, opens a connection to a file
• gafile, opens a connection to a file compressed with gzip
• bzfile, opens a connection to a file compressed with bzip2
• url, opens a connection to a webpage

File Connections

> str(file)
function (description = "", open = "", blocking = TRUE, encoding = getOption("encoding"), 
    raw = FALSE, method = getOption("url.method", "default"))  

• description is the name of the file
• open is a code indication
  • “r” read only
  • “w” writing (and initializing a new file)
  • “a” appending
  • “rb”, “wb”, “ab” reading, writing, or appending in binary mode(windows)

In general, connections are powerful tools that let you navigate files or other external objects. In practice, we often don’t need to deal with the connection interface directly

con <- file("foo.txt", 'r')
data <- read.csv(con)
close(con)

is the same as

data <- read.csv("foo.txt")

Reading Lines of a Text File

> con <- gzfile("words.gz")
> x <- readLines(con, 10)
> x
[1] "1080"      "10-point"    "10th"      "11-point"
[5] "12-point"  "16-point"    "18-point"  "1st" 
[9] "2"         "20-point"

writeLines takes a character vector and writes each element one line at a time to a text file

readLines(con)

## This might take time
con <- url("http://www.jhsph.edu", "r")
x <- readLines(con)
> head(x)
[1] "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\">"
[2] ""
[3] "<html>"
[4] "<head>"
[5] "\t<meta http-euqiv=\"Content-Type\" content=\"text/html;charset=utf-8  

Subsetting - Basics

There are a number of operators that can be used to extract subsets of R objects

• [ always returns an object of the same class as the original; can be used to select more than one element (there is one expection)
• [[ is used to extract elements of a list or a data frame; it can only be extract a single element and the class of the returned object will not necessarily be a list or data frame
• $ is used to extract elements of a list or data frame by name; semantics are similar to has of [[

> x <- c("a", "b", "c", "c", "d", "a")
> x[1]
[1] "a"
> x[2]
[1] "b"
> x[1:4]
[1] "a" "b" "c" "c"
> x[x > "a"]
[1] "b" "c" "c" "d"
> u <- x > "a"
> u
[1] FALSE  TRUE  TRUE  TRUE  TRUE FALSE

Subsetting Lists

> x <- list(foo=1:4, bar=0.6)
> x[1]
$foo
[1] 1 2 3 4

> x[[1]]
[1] 1 2 3 4
> x$bar
[1] 0.6
> x["bar"]
$bar
[1] 0.6

The [[ operator can be uded with computed indices; $ can only be used with literal names

> x <- list(foo=1:4, bar=0.6, baz="hello")
> name <- "foo"
> x[[name]] ## computed index for "foo"
[1] 1 2 3 4
> x$name    ## element 'name' doesn't exist!
NULL
> x$foo
[1] 1 2 3 4

Subsetting Nested Elements of a list

The [[ can take an integer sequence

> x <- list(a = list(10,12,14), b=c(3.14, 2.81))
> x[[c(1, 3)]]
[1] 14
> x[[1]][[3]]
[1] 14
> x[[c(2, 1)]]
[1] 3.14

Subsetting-Matrices

Matrices can be subsetted in the usual way with (i,j) type indices

> x <- matrix(1:6, 2,3)
> x[1,2]
[1] 3
> x[2,1]
[1] 2

Indices can also be missing

> x[1,]
[1] 1 3 5
> x[,2]
[1] 3 4

By default, when a single element of a matrix is retrieved, it is returned as a vector of length 1 rather than a 1 x 1 matrix. This behavior can be turned off by setting drop = False.

> x <- matrix(1:6, 2, 3)
> x[1,2]
[1] 3
> x[1,2,drop=FALSE]
     [,1]
[1,]    3

Similarly, subsetting a single column or a single row will give you a vector, not a matrix (by default)

> x <- matrix(1:6, 2, 3)
> x[1, ]
[1] 1 3 5
> x[1, , drop=FALSE]
     [,1] [,2] [,3]
[1,]    1    3    5

Subsetting - Partial Matching

Partial marching or names is allowed with [[ and $

> x <- list(aardvark = 1:5)
> x$a
[1] 1 2 3 4 5
> x[["a"]]
NULL
> x[["a", exact=FALSE]]
[1] 1 2 3 4 5

Subsetting - Removing Missing Values

A common task is to remove missing values (NAS)

> x <- c(1, 2, NA, 4, NA, 5)
> bad <- is.na(x)
> x[!bad]
[1] 1 2 4 5

What if there are multiple things and you want to take subset with no missing values?

> x <- c(1, 2, NA, 4, NA, 5)
> y <- c("a", "b", NA, "d", NA, "f")
> good <- complete.cases(x,y)
> good
[1]  TRUE  TRUE FALSE  TRUE FALSE  TRUE
> x[good]
[1] 1 2 4 5
> y[good]
[1] "a" "b" "d" "f"

Vectorized Operations

Many operations in R are vectorized making code more efficient, concise, and easier to read

> x <- 1:4; y <- 6:9
> x + y
[1]  7  9 11 13
> x > 2
[1] FALSE FALSE  TRUE  TRUE
> x >= 2
[1] FALSE  TRUE  TRUE  TRUE
> y == 8
[1] FALSE FALSE  TRUE FALSE
> x * y
[1]  6 14 24 36
> x / y
[1] 0.1666667 0.2857143 0.3750000 0.4444444

> x <- 1:4; y <- 2:3
> x + y
[1] 3 5 5 7

Vectorized Matrix Operations

> x <- matrix(1:4, 2, 2); y <- matrix(rep(10, 4), 2, 2)
> x * y    ## element-wise multiplication
     [,1] [,2]
[1,]   10   30
[2,]   20   40
> x / y
     [,1] [,2]
[1,]  0.1  0.3
[2,]  0.2  0.4
> x %*% y  ## true matrix multiplication
     [,1] [,2]
[1,]   40   40
[2,]   60   60

max(a)
min(a)
mean(a)
median(a)
var(a):方差
sd(a):标准差
cor(a, log(a)):相关系数
cov(a, log(a)):协方差

NA 的处理：

> mean(a)
[1] 5.333333
> a <- c(8,4,7,6,NA,2,5)
> mean(a)
[1] NA
> mean(a,na.rm=TRUE)
[1] 5.333333

> a <- c(8,4,7,6,NA,2,5)
> mean(a)
[1] NA
> is.na(a)
[1] FALSE FALSE FALSE FALSE  TRUE FALSE FALSE
> a[is.na(a)]
[1] NA
> a[is.na(a)] <- 0
> a
[1] 8 4 7 6 0 2 5
> mean(a)
[1] 4.571429

example

read a csv file from desktop:

> data <- read.table("Desktop/hw1_data.csv", header=TRUE, sep=",")
> data <- read.table("Desktop/hw1_data.csv", header=TRUE, sep=",", nrows=2) #提取前两行
> data[152:153,] #提取最后两行
> data[47, "Ozone"] #提取47行，Ozone列
> x <- c(data[, "Ozone"]) #提取指定列
> good <- complete.cases(x) 
> x[good] # 删除NA
> length <- length(x[good])
> y <- length(x) - length(x[good]) # NA的行数
> mean(x[good]) 
> data_1 <- data[, c("Ozone", "Temp", "Solar.R")] #提取指定列
> col <- c("Ozone", "Temp", "Solar.R")
> data_1 = data(, col) #提取指定列
> data_3 <- data_1[data_1$Ozone>31 & data_1$Temp>90, ] #提取指定行
> list <- data_3[["Solar.R"]] #使用[[]]
> rm <- is.na(list)
> list[!rm] #删除NA
> mean(list[!rm]) 
> data_4 <- data[, c("Temp", "Month")]
> data_5 <- data_4[data_4$Month==6, ]
> list <- c(data_5[["Temp"]])
> mean(list)
> new_list <- data[, c("Ozone", "Month")]
> May <- new_list[new_list$Month==5,]
> lis <- c(May[["Ozone"]])
> new_rm <- is.na(lis)
> lis[!new_rm]
> max(lis[!new_rm])