erictleung

Spark, R, and Databricks Notes

Here are some notes on using R on Databricks and making use of the underlying Spark tooling.

Contents

• SparkR vs sparklyr
• sparklyr function types
• Connect to Snowflake
• Error messages

erictleung

erictleung

library(tidyverse)
library(rvest)
library(ggtext)
library(extrafont)

rt <- read_html("https://editorial.rottentomatoes.com/guide/best-christmas-movies/")

film_regex <- regex(
  "^([A-Za-z0-9.' ,-:&]* \\([0-9]{4,4}\\))\\n\\s+([0-9]{0,4})\\%\\n\\s+([0-9]{0,4})\\%"
)

erictleung

#!/usr/bin/env python3

"""
Help audit, remove, and update musician infoboxes.

https://en.wikipedia.org/wiki/Category:Pages_using_infobox_musical_artist_with_associated_acts
"""

import re
import requests

erictleung

#!/usr/bin/env bash

# Description: Extract unique URLs from the plain text output from Zoom chat and
#   save links to new file.
# Usage: bash extract_zoom_urls.sh

# Set path to where Zoom chat is, or `cd` to the directory itself
FILE=meeting_saved_chat.txt

# Get list of unique links shared in Zoom chat

erictleung

# Follow instructions here for more:
#   https://exercism.io/cli-walkthrough
# Releases:
#   https://github.com/exercism/cli/releases

# Download and move around
# Change URL as needed for new versions
wget https://github.com/exercism/cli/releases/download/v3.1.0/exercism-3.1.0-linux-x86_64.tar.gz
mkdir bin
mv exercism bin/

erictleung

# Make logo
library(hexSticker)
library(showtext)

# Add Google Font
font_add_google(name = "Open Sans", family = "Open Sans")
showtext_auto() # Use this font in all rendering

# Picture of book that was quickly screenshot
imgurl <- "~/Downloads/Screenshot 2022-06-13 092720.png"

erictleung

library(dplyr)
library(stringr)
library(ggplot2)

d <- rtweet::search_tweets("from:stats_feed", n = 30) %>%
  filter(status_id == "1521871339908263937") %>%
  pull(text) %>%
  str_split("\n") %>%
  as_tibble(.name_repair = "universal") %>%
  rename("data" = "...1") %>%

erictleung

erictleung

Category: All categories

Month,Data scientist: (United States)
2004-01,0
2004-02,0
2004-03,0
2004-04,0
2004-05,0
2004-06,0
2004-07,0