This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"first names": { | |
"Oliver": 4, | |
"Kennedy": 4, | |
"Adkins": 4, | |
"Wyatt": 4, | |
"Theodore": 4, | |
"Nicholas": 4, | |
"Alston": 3, | |
"Donia": 3, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pandas as pd | |
from pyproj import Transformer | |
def towgs84(row, lat_col, lon_col): | |
"""An apply function for a pyproj transformer from 4326/WGS84 to 3857/web mercator | |
:param row: a row of a pandas dataframe as supplied by df.apply(). | |
:param lat_col: str of the lat col | |
:param lon_col: str of the lon col |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import geopandas as gpd | |
from shapely.ops import nearest_points | |
def nearest(row, df1, df2, geom1_col='geometry', geom2_col='geometry', src_column=None): | |
"""Find the nearest point and return the corresponding value from specified column.""" | |
# Construct a multipoint object | |
geom_union = df2.unary_union | |
# Find the geometry that is closest | |
nearest = df2[geom2_col] == nearest_points(row[geom1_col], geom_union)[1] | |
# Get the corresponding value from df2 (matching is based on the geometry) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
import tempfile | |
import uuid | |
import argparse | |
import boto3 | |
import azure.storage.blob as blob | |
import tqdm | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
import csv | |
import argparse | |
from tqdm import tqdm | |
def split(filehandler, delimiter=',', row_limit=10000, | |
output_name_template='output_%s.csv', output_path='.', keep_headers=True): | |
""" | |
Splits a CSV file into multiple pieces. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import foo | |
import argparse | |
def main(quux, quuz, **kwargs): | |
try: | |
if quux is True: | |
foo.bar() | |
else: | |
foo.baz() | |
except Exception as ex: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import boto3 | |
import botocore | |
from joblib import Parallel, delayed | |
import os | |
bucket_name = "bucket_name" | |
s3 = boto3.resource('s3') | |
bucket = s3.Bucket(bucketname) | |
exists = True | |
try: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from collections import defaultdict | |
r_dd = lambda: defaultdict(r_dd) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
reports = {} | |
for name, clf in zip(classifier_names, classifiers): | |
clf.fit(X_train, Y_train) | |
Y_predictions = clf.predict(X_test) | |
Y_true = Y_test | |
reports[name] = classification_report(Y_true, Y_predictions, output_dict=True) | |
reporting_df = pd.DataFrame.from_dict({(i,j): reports[i][j] | |
for i in reports.keys() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from sklearn.neural_network import MLPClassifier | |
from sklearn.neighbors import KNeighborsClassifier | |
from sklearn.svm import SVC | |
from sklearn.gaussian_process import GaussianProcessClassifier | |
from sklearn.gaussian_process.kernels import RBF | |
from sklearn.tree import DecisionTreeClassifier | |
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier | |
from sklearn.naive_bayes import GaussianNB | |
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis |
NewerOlder