Ian Gow iangow

Benchmarking WRDS data retrieval

Summary

The following are some benchmarks that emerged naturally from some work I was doing to get Stata data from WRDS. In essence, there are two steps. First, retrieve data from WRDS's PostgreSQL database. Second, do some analysis of the retrieved data (here a simple summary query). Below I have tabulated the elapsed times in seconds for each of these two tasks for a number of different approaches.

	library(dplyr, warn.conflicts = FALSE)
	library(tidyr)
	library(readr) # For read_lines(), read_fwf(), etc.
	library(stringr) # For str_c(), str_detect()
	library(pdftools) # For pdf_text()
	library(lubridate) # For ymd()
	library(ggplot2)

	url <- paste0("https://aaahq.org/portals/0/documents/meetings/2023/RC/",
	"2023%20Rookie%20Camp%20Alphabetical%20Presentation%20Schedule.pdf")

	library(dplyr, warn.conflicts = FALSE)
	library(dbplyr, warn.conflicts = FALSE)
	library(DBI)
	set.seed(2023)
	n <- 100
	betas <- tibble(id = 1:n, beta = runif(n, min = 0.7, max = 1.3))

	start_date <- as.Date("1980-01-01")
	end_date <- as.Date("2023-01-01")

	library(dplyr, warn.conflicts = FALSE)
	library(tidyr)
	library(readr) # For read_lines(), read_fwf(), etc.
	library(stringr) # For str_c(), str_detect()
	library(pdftools) # For pdf_text()
	library(ggplot2)

	url <- "https://www.newtonma.gov/home/showpublisheddocument/97990/638140435866000000"

	col_names <- c("street_name", "length_mi", "length_ft",