Skip to content

Instantly share code, notes, and snippets.

@gitdagray
gitdagray / kanview_complete.py
Last active June 10, 2020 18:22
Full Python code for web scraping Kanview utilizing Selenium, Beautiful Soup, and pandas
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import re
import pandas as pd
from tabulate import tabulate
import os
#launch url
url = "http://kanview.ks.gov/PayRates/PayRates_Agency.aspx"
@xiaostrong
xiaostrong / qqmessage.py
Created February 21, 2018 08:57
QQ空间留言脚本
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import requests
import time
import re
import json
import time
def login():
options = webdriver.ChromeOptions()
@WillKoehrsen
WillKoehrsen / submit_assignment.py
Created March 10, 2018 20:26
Automation of Assignment Submisison
# selenium for web driving
import selenium
from selenium import webdriver
# time for pausing between navigation
import time
# Datetime for recording time of submission
import datetime
@rdinse
rdinse / Google_Colaboratory_backup.py
Created March 13, 2018 22:55
Simple Google Drive backup script with automatic authentication for Google Colaboratory (Python 3)
# Simple Google Drive backup script with automatic authentication
# for Google Colaboratory (Python 3)
# Instructions:
# 1. Run this cell and authenticate via the link and text box.
# 2. Copy the JSON output below this cell into the `mycreds_file_contents`
# variable. Authentication will occur automatically from now on.
# 3. Create a new folder in Google Drive and copy the ID of this folder
# from the URL bar to the `folder_id` variable.
# 4. Specify the directory to be backed up in `dir_to_backup`.
from selenium import webdriver
from selenium.webdriver.chrome.options import DesiredCapabilities
from selenium.webdriver.common.proxy import Proxy, ProxyType
import time
co = webdriver.ChromeOptions()
co.add_argument("log-level=3")
co.add_argument("--headless")
@liquidgenius
liquidgenius / dataframe_dataset_dataframe_roundtrip.py
Last active April 24, 2018 04:14
In Python, save a Pandas DataFrame to Sqlite using Dataset module and retrieve into Dataframe utilizing only Dataset function all().
import dataset
import pandas as pd
# create dataframe
df = pd.DataFrame()
names = ['Bob', 'Jane', 'Alice', 'Ricky']
ages = [31, 30, 31, 30]
df['names'] = names
df['ages'] = ages
@lobstrio
lobstrio / linkedin_extract.py
Created April 24, 2018 11:05
Scraping Linkedin profiles information through Selenium Python
# python package
import csv
import time
import random
import sys
import os
# selenium package
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
@npearce
npearce / install-docker.md
Last active April 26, 2025 04:27
Amazon Linux 2 - install docker & docker-compose using 'sudo amazon-linux-extras' command

UPDATE (March 2020, thanks @ic): I don't know the exact AMI version but yum install docker now works on the latest Amazon Linux 2. The instructions below may still be relevant depending on the vintage AMI you are using.

Amazon changed the install in Linux 2. One no-longer using 'yum' See: https://aws.amazon.com/amazon-linux-2/release-notes/

Docker CE Install

sudo amazon-linux-extras install docker
sudo service docker start
@liquidgenius
liquidgenius / Query.py
Last active May 15, 2019 01:51
A simple class that utilizes chained methods to query a MySQL table for designated columns. All records are returned. Inspired by pandas.
import pymysql
from pprint import pprint as pprint
__author__ = "Liquidgenius"
class Query:
def __init__(self, host, port, user, password, schema):
self.host = host
self.port = port
self.user = user
def low_pass_filter_anomaly_detection(df,
column_name,
number_of_stdevs_away_from_mean):
"""
Implement a low-pass filter to detect anomalies in a time series, and save the filter outputs
(True/False) to a new column in the dataframe.
Arguments:
df: Pandas dataframe
column_name: string. Name of the column that we want to detect anomalies in
number_of_stdevs_away_from_mean: float. Number of standard deviations away from