Skip to content

Instantly share code, notes, and snippets.

@benmarwick
benmarwick / scraping-pnas-titles.R
Last active October 6, 2022 06:42
scrape PNAS archaeology articles and basic text analysis of abstracts
# JB says she searched for "archaeolog*" and "archeolog*", these return 257 results for me,
# much less than the 1002 we get from searching the archaeology 'keyword', e.g.
# https://www.pnas.org/action/doSearch?Concept=500376&Concept=500375&startPage=0&sortBy=Earliest
# in any case, let's start with "archaeolog*" and "archeolog*", I've copied the URL of the search
# results page and edited the URL to return 500 items on the first page, so we can get all results without
# having to scrape multiple pages of results, just to simplify the process
library(tidyverse)
@benmarwick
benmarwick / gist:57d1c1ba265a2e5ab6c5f33b729b8fdd
Last active May 10, 2022 18:56
Analyse text reuse using minhash and locality-sensitive hashing (LSH)
library(tidyverse)
library(textreuse)
# one row per student, to get the data, go to canvas -> quiz -> 'quiz stats' -> 'student analysis'
cnvs <- read_csv("quiz-responses-downloaded-from-canvas.csv")
# select only the column with the text we want to compare
cnvs_q5 <-
cnvs %>%
select( q5 = contains("peers' reflections"))
@benmarwick
benmarwick / tektite-maps-sea-vietnam.R
Created October 16, 2021 06:33
Tektite maps for southeast Asia & Vietnam
library(tidyverse)
library(sf)
library(googlesheets4)
# get our data from google sheets, we need to:
# - 'publish to web'
# - adjust sharing settings to share with anyone
# I've done these, now, so it should just work:
my_key <- "1xUqRGnb9kwBi128cERiHkV5w4uREwl-mqAVf0jmQhhg"
# also at https://gist.github.com/benmarwick/f11ae49ab9afde0071b133012ff76cbc
ctv <- "https://raw.githubusercontent.com/benmarwick/ctv-archaeology/master/README.md"
library(tidyverse)
library(glue)
archy_ctv_readme <- readLines(ctv)
# get just the articles
@benmarwick
benmarwick / viralarchive.Rmd
Created June 1, 2021 16:51
Object recognition in Images in #viralarchive tweets
I used the Python library GetOldTweets3 to get the tweets because the rtweet package cannot get tweets older than 6-9 days. Details about this Python library are here: https://github.com/Mottl/GetOldTweets3
I used this line in the shell to get tweets using the #viralarchive hashtag:
```{bash, engine.opts="-l", eval = F}
GetOldTweets3 --querysearch 'viralarchive' --maxtweets 10000
```
@benmarwick
benmarwick / ggplot-to-jpg-set-dpi.R
Created December 19, 2020 08:16
How to save a ggplot as a JPG file with specific dimensions and a high dpi
# How to save a ggplot as a JPG file with a
# specific dpi and dimensions, for example,
# because a publisher requires it
library(ggplot2)
p <-
ggplot(mtcars) +
aes(mpg,
disp) +
library(tidyverse)
library(rvest)
# what quarter and year do we want to check for the availability of DSM courses?
qrt_year <- "AUT2020"
# this is the URL to our canonical list of DSM courses
webpage <- "https://www.washington.edu/uaa/advising/single-pages/data-science-minor/"
@benmarwick
benmarwick / .github-workflows-main.yml
Last active November 7, 2022 23:22
GitHub workflow to render all Rmd files in a GitHub repo, e.g. for testing student assignments
# from https://github.com/cboettig/compendium/blob/master/.github/workflows/main.yml
on: [push]
name: render all R Markdown documents
jobs:
render:
name: render all R Markdown documents
runs-on: macOS-latest
steps:
@benmarwick
benmarwick / inspect-rmd-diff-from-last-commit.R
Last active May 23, 2025 03:30
GitHub doesn't show rich diffs for Rmd files. That can make collaborative writing tough. Here's how to see rich diffs of two commits of a single R Markdown document on a GitHub repo or local Git repo
# another method
# remotes::install_github("ropenscilabs/reviewer")
browseURL(reviewer::diff_rmd("analysis/paper/paper.qmd",
# this gets the sha of the previous commit
git2r::commits(n=2)[[2]]$sha)$raw)
@benmarwick
benmarwick / Tehrani_Collard_2002.R
Created December 30, 2019 21:32
Replicating Tehrani & Collard 2002
# paste from http://www.ceacb.ucl.ac.uk/ceacb_files/misc/Tehrani_Collard_2002.pdf
# edit to get each textile design on one line
raw_input <-
c("Ersari 1 0 1 0 1 1 0 1 0 1 1 0 1 1 100000011010000001011001111101011000011111100110111011011011011011110000000 0
Saryk 1 0 1 0 1 1 1 1 1 1 1 0 1 0 100000011010111101010100001111100000010101010110011011100000001011100000000 0
Salor 1 0 1 1 0 1 0 1 0 1 0 1 0 0 011110110101000001011000001011110111010101000100111101111101001101100000101 0
PSDP Tekke 1 1 0 0 0 1 0 1 1 0 1 0 0 0 011000011111111111011010001010100000010101101101000000000000000000001101000 0
SDP Tekke 1 1 0 0 0 1 0 1 1 0 0 1 0 0 000011111001110011111010000000000010100001100000000000001100101010001011111 1
Yomut 0 0 0 0 0 1 0 1 1 0 0 0 1 1 000000000000101100000001100000011000010100000000000000000000000000000000000 0")