philshem’s gists

philshem / _get_musical_names.py

Created June 6, 2021 12:43

musical names (yob2020.txt comes from https://www.ssa.gov/oact/babynames/limits.html)

	#!/usr/bin/python3

	musical_notes = list('CDEFGAB')
	#musical_notes = list('CDEFGAH')

	fp = 'yob2020.txt'
	with open(fp, 'r') as f:
	lines = f.read().split('\n')

	names = [x.split(',')[0].upper() for x in lines if len(x) > 0]

philshem / get.py

Created May 6, 2021 05:03

s&p index + weights scraping https://www.slickcharts.com/sp500

	import requests

	df = pd.read_html(requests.get('https://www.slickcharts.com/sp500',headers={'User-agent': 'Mozilla/5.0'}).text)[0]

	print(df)
	df.to_csv('sp.csv',index=False)

philshem / publibike_zuerich_elevation.csv

Created February 19, 2021 19:19

Elevation of Publibike stations around Zürich

station_id	latitude	longitude	elevation	elevation_above_lake
pb_110	47.396401	8.4860017	399.0173645	-6.982635498
pb_149	47.392336	8.491061	399.4473877	-6.552612305
pb_286	47.3985553	8.493822	401.0709534	-4.929046631
pb_90	47.3917964	8.5056978	402.3087158	-3.69128418
pb_395	47.3669233	8.5420304	402.3295898	-3.670410156
pb_152	47.389823	8.49163	402.6643982	-3.335601807
pb_155	47.374711	8.543042	402.9417419	-3.058258057
pb_450	47.350578	8.560587	403.0110779	-2.988922119
pb_130	47.391366	8.488108	403.0958862	-2.90411377

philshem / Collect Parler Metadata.ipynb

Created January 12, 2021 07:25 — forked from kylemcdonald/Collect Parler Metadata.ipynb

Collect video URLs and GPS data for Parler videos.

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

philshem / forbes400.csv

Last active December 7, 2020 10:11

scraper: Forbes' richest 400, 30 years

We can't make this file beautiful and searchable because it's too large.

	name,lastName,uri,imageUri,worthChange,age,source,industry,gender,timestamp,headquarters,state,realTimeWorth,date,realTimeRank,realTimePosition,squareImage,title,category,government,position,rank,worth,year,source_url,salary,managementAssets,pay
	Abigail Johnson,Johnson,abigail-johnson,abigail-johnson,0.0,58.0,money management,Finance & Investments,F,2020-12-07 08:50:01.435,"Milton, MA",Massachusetts,14974.7,1993-01-01 00:00:00,39.0,39.0,//specials-images.forbesimg.com/imageserve/5f15ddd274642d00070464eb/416x416.jpg?background=000000&cropX1=62&cropX2=1355&cropY1=144&cropY2=1437,,,,,,,1990,http://www.forbes.com/ajax/list/data?year=1990&uri=forbes-400&type=person,,,
	Aerin Lauder,Lauder,aerin-lauder,aerin-lauder,0.0,50.0,cosmetics,Fashion & Retail,F,2020-12-07 08:50:01.435,"New York, NY",New York,3215.528,1993-01-01 00:00:00,285.0,285.0,https://specials-images.forbesimg.com/imageserve/5f46b424808a289780083f05/416x416.jpg?background=000000&cropX1=895&cropX2=2304&cropY1=287&cropY2=1696,,,,,,,1990,http://www.forbes.

philshem / deming.csv

Created November 28, 2020 12:10

deming.csv - https://opendata.stackexchange.com/a/18277/1511

x	y	y_ideal
10	10.9277728455939	10
10.25	10.5933490961181	10.25
10.5	14.0852031368451	10.5
10.75	12.4232922691896	10.75
11	14.7683460256395	11
11.25	9.93532391180117	11.25
11.5	12.4777201750146	11.5
11.75	11.3343548513887	11.75
12	10.2801940515687	12

philshem / at.20200927.csv

Last active September 27, 2020 18:44

scrape individual state data from https://coronadata.at used for this answer https://opendata.stackexchange.com/a/18094/1511

We can make this file beautiful and searchable if this error is corrected: It looks like row 7 should actually have 11 columns, instead of 10 in line 6.

	Datum,Tests,Fälle (Gesamt),Fälle (Aktiv),/ 100.000 Einwohner,Genesene,Verstorbene,Hospitalisiert,Intensivstation,state,url
	27.09.2020 20:00,46.351 +476 (1%),758 +9 (1%),141 0 (0%),48,606 +9 (2%),11 +0 (0%),8 +1 (14%),1 0 (0%),burgenland,https://www.coronatracker.at/burgenland
	26.09.2020 20:00,45.875 +592 (1%),749 +20 (3%),141 +6 (4%),48,597 +14 (2%),11 +0 (0%),7 +4 (133%),1 +1 (100%),burgenland,https://www.coronatracker.at/burgenland
	25.09.2020 20:00,45.283 +908 (2%),729 +8 (1%),135 -5 (-4%),46,583 +13 (2%),11 +0 (0%),3 -4 (-57%),0 0 (0%),burgenland,https://www.coronatracker.at/burgenland
	24.09.2020 20:00,44.375 +528 (1%),721 +4 (1%),140 -1 (-1%),48,570 +5 (1%),11 +0 (0%),7 +3 (75%),0 0 (0%),burgenland,https://www.coronatracker.at/burgenland
	23.09.2020 20:00,43.847 +999 (2%),717 +33 (5%),141 +23 (19%),48,565 +10 (2%),11 +0 (0%),4 -3 (-43%),0 -1 (-100%),burgenland,https://www.coronatracker.at/burgenland
	22.09.2020 20:00,42.848 +303 (1%),684 +14 (2%),118 +2 (2%),40,555 +12 (2%),11 +0 (0%),7 -2 (-22%),1 0 (0%),b

philshem / 53827775.py

Created September 1, 2020 07:41

refactor stackoverflow answer to read unread emails and also fix for loop https://stackoverflow.com/a/53827775/2327328

	import imaplib
	import email

	# adapted from: https://stackoverflow.com/a/53827775/2327328

	def read_email_from_gmail():
	mail = imaplib.IMAP4_SSL('imap.gmail.com')
	mail.login('MYEMAIL','MYSECRET')
	mail.select('inbox')

philshem / 1298632074849587204.py

Last active August 26, 2020 15:07

https://twitter.com/dancow/status/1298632074849587204 reply to dancow

	import pandas as pd
	# https://burntsushi.net/stuff/worldcitiespop.csv
	df = pd.read_csv('worldcitiespop.csv',low_memory=False)
	df = df.query('Country == "us"')
	print(len(df))
	# prints 141989

philshem / data.json

Last active August 24, 2020 03:26

sample data

	{
	"1960":316.91,
	"1961":317.64,
	"1962":318.45,
	"1963":318.99,
	"1964":319.62,
	"1965":320.04,
	"1966":321.38,
	"1967":322.16,
	"1968":323.04,

@philshem philshem