hsm207 hsm207
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # from https://unix.stackexchange.com/questions/69314/automated-ssh-keygen-without-passphrase-how | |
| cat /dev/zero | | |
| ssh-keygen -q -N "" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # from: https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html | |
| BUCKET=amazon-eks | |
| LATEST_KUBECTL_VERSION=`aws s3 ls s3://$BUCKET/ | | |
| grep --invert-match "cloudformation\|manifests" | | |
| awk '{print $2}' | | |
| cut -d/ -f 1 | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # from: https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html | |
| # download and extract the latest version of eksctl (including prerelease) | |
| # from: https://gist.github.com/steinwaywhw/a4cd19cda655b8249d908261a62687f8 | |
| LATEST_EKSCTL=`curl -s https://api.github.com/repos/weaveworks/eksctl/releases | | |
| jq ".[0].assets | map(select(.name == \"eksctl_Linux_amd64.tar.gz\")) | .[0].browser_download_url" | | |
| tr -d '"'` |
hsm207
/ find_host_ip.sh
Created
March 8, 2020 14:04
Command to find the host's ip address from inside a container
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Thanks to: https://forums.docker.com/t/accessing-host-machine-from-within-docker-container/14248/10 | |
| docker run --network host \ | |
| --rm \ | |
| openjdk:8 bash -c "apt update; apt install net-tools; route | awk '/^default/ { print \$2 }' | grep -v 0.0.0.0" |
hsm207
/ create_build_env.sh
Created
March 8, 2020 13:20
Script to create a docker container to build Spark
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| docker run -it \ | |
| -v $(pwd):/spark \ | |
| -v /c/path/to/.m2:/root/.m2 \ | |
| -v /c/path/to/.sbt:/root/.sbt \ | |
| -v /c/path/to/.ivy2:/root/.ivy2 \ | |
| -w /spark \ | |
| --network host \ | |
| openjdk:8 bash |
hsm207
/ copy_redis.sh
Last active
February 5, 2020 17:33
Script to migrate all keys inside a redis instance into another redis instance
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # source: https://stackoverflow.com/questions/37166947/copying-all-keys-in-redis-database-using-migrate | |
| # make sure the redis in SOURCE_HOST and DESTINATION_HOST are identical! | |
| SOURCE_HOST=127.0.0.1 | |
| SOURCE_PORT=6381 | |
| DESTINATION_HOST=foo.bar | |
| DESTINATION_PORT=6379 |
hsm207
/ working_with_random_samples.html
Last active
December 24, 2019 12:43
Working with random samples on Spark
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html> | |
| <head> | |
| <meta name="databricks-html-version" content="1"> | |
| <title>working_with_random_samples - Databricks</title> | |
| <meta charset="utf-8"> | |
| <meta name="google" content="notranslate"> | |
| <meta name="robots" content="nofollow"> | |
| <meta http-equiv="Content-Language" content="en"> |
hsm207
/ keras_estimator_feature_column_bug.ipynb
Created
October 22, 2019 15:56
keras_estimator_feature_column_bug.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def getAnalytics(bucketName: String, brand: String, bucketYear: String, bucketMonth: String, bucketDay: String, candidate_field: String=candidateField, groups: String=Groups): DataFrame = { | |
| var rankingOrderedIds = Window.partitionBy("c12").orderBy("id") | |
| val s3PathAnalytics = getS3Path(bucketName, brand, bucketFolder, year=bucketYear, month=bucketMonth, day=bucketDay) | |
| readJSON(s3PathAnalytics) | |
| .distinct | |
| .withColumn("x", explode($"payload")) | |
| // a few more calls to withColumn to create columns | |
| .withColumn("c10", explode(when(size(col("x1")) > 0, col("x1")).otherwise(array(lit(null).cast("string"))))) | |
| // a few more calls to withColumn to create columns | |
| .withColumn("id", monotonically_increasing_id) |