Connor Ameres cameres
cameres
/ compute_correlation_matrix.py
Last active
November 22, 2022 14:19
Compute Pandas Correlation Matrix of a Spark Data Frame
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from pyspark.mllib.stat import Statistics | |
| import pandas as pd | |
| # result can be used w/ seaborn's heatmap | |
| def compute_correlation_matrix(df, method='pearson'): | |
| # wrapper around | |
| # https://forums.databricks.com/questions/3092/how-to-calculate-correlation-matrix-with-all-colum.html | |
| df_rdd = df.rdd.map(lambda row: row[0:]) | |
| corr_mat = Statistics.corr(df_rdd, method=method) | |
| corr_mat_df = pd.DataFrame(corr_mat, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [ | |
| { | |
| "date": "2016-01-01", | |
| "line_1": 55.7460227373914, | |
| "line_3": 65.55089867445142, | |
| "line_2": 116.05179943545978 | |
| }, | |
| { | |
| "date": "2016-01-02", | |
| "line_1": 71.7460227373914, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # these aliases can be appended to your .profile, .bashrc, .zshrc, etc. | |
| ## replace version w/ version downloaded in brew | |
| ## since I installed hadoop 2.7.3, I replace <version> w/ 2.7.3 | |
| alias hstart="/usr/local/Cellar/hadoop/<version>/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/<version>/sbin/start-yarn.sh" | |
| alias hstop="/usr/local/Cellar/hadoop/<version>/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/<version>/sbin/stop-dfs.sh" |
cameres
/ Jupyter Notebook Clustering.ipynb
Created
November 28, 2016 23:00
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
cameres
/ Test Notebook.ipynb
Created
November 2, 2016 17:25
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <meta charset="utf-8"> | |
| Hello, world! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <meta charset="utf-8"> | |
| Hello, world! |
cameres
/ AudibleiTunesJavascript.scpt
Created
March 22, 2016 06:21
A simple script that I created to allow audible to correctly detect audio books.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| iTunes = Application('iTunes') | |
| songs = iTunes.selection() | |
| count = 0 | |
| for(i = 0; i < songs.length; i++){ | |
| songs[i].name = "Part " + count++; | |
| } |