Skip to content

Instantly share code, notes, and snippets.

View exy02's full-sized avatar

Exy02 exy02

1 follower · 0 following
View GitHub Profile
@exy02
exy02 / awsLambda-DownloadAnyPythonPackage.py
Last active February 28, 2022 09:13
import subprocess
import shutil
import boto3
import json
import os
def lambda_handler(event, context):
#Parameters =============================
pipPackage = "requests"
packageVersion = "2.26.0"
@exy02
exy02 / airmap-demo-mapData.csv
Created October 9, 2021 21:55
Resulting mapped data output from AirMap
Line Service ID ServiceProviderID Date of Service PurchaserDOB PurchaseAmount Document
20394822 1938472929 10/09/2021 7/4/1776 $103.20 AJ3fLN32jS1SK2
20428284 1293018083 10/07/2021 12/30/1899 $120.12 J2sjldw3nSk4S2l
19428292 9420410832 10/08/2021 1/26/1980 $84.23 2Pn28dh1lsdD0Q
@exy02
exy02 / airmap-demo-viewMap.csv
Created October 9, 2021 21:51
sample map table output from AirMap
OutputColumn InputColumn InputSource MaxLength ColumnPosition DataType Format Active Description Required
Date of Service ServiceDate Internal_Import_Format1 10 1 Date mm-dd-yyyy Yes Date of services rendered Y
PurchaserDOB ClientDOB Internal_Import_Format1 10 2 Date mm-dd-yyyy Yes Client's date of birth Y
ServiceProviderID ProviderID Internal_Import_Format1 10 3 String ########## Yes Service providers ID Y
PurchaseAmount PurchaseAmount Internal_Import_Format1 20 4 String $dd.cc Yes Client's purchase Y
Line Service ID ServiceLineID Internal_Table_Format1 8 5 String ######## Yes Service providers ID Y
Document DocumentID Internal_Import_Format2 10 6 String NaN Yes DocumentID N
@exy02
exy02 / airmap-demo-view_map.py
Created October 9, 2021 20:15
view the details of a chosen data map
airMap.findMappingView("Project A - Client Summary").viewMap()
@exy02
exy02 / airmap-demo-map_data.py
Created October 9, 2021 19:57
map data using AirMap
Internal_Import_Format1 = pd.read_csv("Internal_Import_Format1.csv",dtype=str)
Internal_Import_Format2 = pd.read_csv("Internal_Import_Format2.csv",dtype=str)
Internal_Table_Format1 = pd.read_csv("Internal_Table_Format1.csv",dtype=str)
data = {'Internal_Import_Format1' : Internal_Import_Format1,
'Internal_Import_Format2' : Internal_Import_Format2,
'Internal_Table_Format1' : Internal_Table_Format1 }
airMap = airtableDataMapper(base_key,airAPI_key)
airMap.findMappingView("Project A - Client Summary").mapData(data)
@exy02
exy02 / xml_interpreter_converter.py
Last active October 5, 2021 19:48
xml_interpreter_converter
import pandas as pd
import requests
import pickle
import base64
import getpass
import boto3
import json
def get_access_token():
user_successful_login = False
@exy02
exy02 / simple_pgres_query_5.py
Last active October 10, 2020 20:26
Simplified PostgreSQL querying (with or without SSH) - General Queries
sql_statement = """
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'ey_test_table'
;
"""
query_df = pgres.query(db='database_name', query=sql_statement)
query_df
#returns the results of an sql statement as a pandas dataframe.
#This example returns the column names and data types of table 'ey_test_table'.
@exy02
exy02 / simple_pgres_query_4.py
Created September 26, 2020 19:11
Simplified PostgreSQL querying (with or without SSH) - Return Tables
pgres.tables(db='database_name',schema='schema_name')
#returns the number of tables and all table names within the specified schema as a pandas dataframe
@exy02
exy02 / simple_pgres_query_3.py
Created September 26, 2020 19:09
Simplified PostgreSQL querying (with or without SSH) - Return Schemas
pgres.schemas(db='database_name')
#returns the number of schemas and all schema names within the specified database as a pandas dataframe
@exy02
exy02 / simple_pgres_query_2.py
Last active September 26, 2020 20:36
Simplified PostgreSQL querying (with or without SSH) - Class Initialization
p_host = '0.0.0.0'
p_port = 5432
db = 'database_name'
ssh = True
ssh_user = 'ssh_user'
ssh_host = 'ip address or web address'
ssh_pkey = '/path/to/user_authentication.pem'
pgres = Postgresql_connect(pgres_host=p_host, pgres_port=p_port, db=db, ssh=ssh, ssh_user=ssh_user, ssh_host=ssh_host, ssh_pkey=ssh_pkey)
#initiates a connection to the PostgreSQL database. In this instance we use ssh and must specify our ssh credentials.