EMR has components running on internal ip's such as ip-172-23-53-101.ec2.internal. To resolve to the implied public/vpn IP's, setup dnsmasq
$ brew install dnsmasq
Michael Spector spektom
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| version: '3' | |
| services: | |
| zookeeper: | |
| image: zookeeper | |
| kafka: | |
| image: wurstmeister/kafka | |
| environment: | |
| KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 | |
| KAFKA_ADVERTISED_HOST_NAME: kafka | |
| bash: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Add the following lines to /usr/local/etc/dnsmasq.conf: | |
| address=/ec2.internal/127.0.0.1 | |
| synth-domain=ec2.internal,0.0.0.0,255.255.255.255,ip- | |
| address=/us-west-2.compute.internal/127.0.0.1 | |
| synth-domain=us-west-2.compute.internal,0.0.0.0,255.255.255.255,ip- | |
| address=/eu-west-1.compute.internal/127.0.0.1 | |
| synth-domain=eu-west-1.compute.internal,0.0.0.0,255.255.255.255,ip- |
spektom
/ awsips-dnsmasq.md
Created
August 20, 2018 05:08
— forked from ashee/awsips-dnsmasq.md
dnsmasq in osx (macSierra) to resolve aws internal ip's
spektom
/ generate_hive_schema.scala
Created
May 3, 2018 13:47
Generate Hive schema from Spark Dataframe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark.sql.DataFrame | |
| def dataFrameToDDL(dataFrame: DataFrame, tableName: String): String = { | |
| val columns = dataFrame.schema.map { field => | |
| " " + field.name + " " + field.dataType.simpleString.toUpperCase | |
| } | |
| s"CREATE TABLE $tableName (\n${columns.mkString(",\n")}\n)" | |
| } | |
| import spark.sqlContext.implicits._ |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
spektom
/ postgresql-json-fts.sql
Created
April 12, 2018 10:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| CREATE TABLE bookdata ( | |
| id serial NOT NULL PRIMARY KEY, | |
| info json NOT NULL | |
| ) | |
| CREATE INDEX bookdata_fts ON bookdata USING gin((to_tsvector('english',info->'title'))); | |
| INSERT INTO bookdata (info) | |
| VALUES | |
| ( '{ "title": "The Tattooed Duke", "items": {"product": "Diaper","qty": 24}}'), |
spektom
/ pgo_env.sh
Created
April 12, 2018 10:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| export COROOT=$GOPATH/src/github.com/crunchydata/postgres-operator | |
| export PATH=$PATH:$GOPATH/bin | |
| export PGO_CA_CERT=$COROOT/conf/apiserver/server.crt | |
| export PGO_CLIENT_CERT=$COROOT/conf/apiserver/server.crt | |
| export PGO_CLIENT_KEY=$COROOT/conf/apiserver/server.key | |
| export CO_NAMESPACE=demo | |
| export CO_CMD=kubectl | |
| export CO_APISERVER_URL=https://127.0.0.1:8443 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash -eu | |
| export GOPATH=$HOME/pgcluster-test | |
| export PATH=$PATH:$GOPATH/bin | |
| mkdir -p $GOPATH/src $GOPATH/bin $GOPATH/pkg | |
| mkdir -p $GOPATH/src/github.com/crunchydata/ | |
| # Download PostgreSQL Operator source: | |
| cd $GOPATH/src/github.com/crunchydata | |
| git clone https://github.com/CrunchyData/postgres-operator.git |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM rabbitmq:management | |
| RUN rabbitmq-plugins enable --offline rabbitmq_shovel && rabbitmq-plugins enable --offline rabbitmq_shovel_management |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"log":"time=\"2017-09-22T17:50:58Z\" level=info msg=\"Establishing connection with Rethinkdb\" \n","stream":"stderr","time":"2017-09-22T17:50:58.428872932Z"} | |
| {"log":"time=\"2017-09-22T17:50:58Z\" level=info msg=\"writing out configs and watching for changes\" \n","stream":"stderr","time":"2017-09-22T17:50:58.48144991Z"} | |
| {"log":"time=\"2017-09-22T17:50:58Z\" level=info msg=\"watching configs\" \n","stream":"stderr","time":"2017-09-22T17:50:58.481480667Z"} | |
| {"log":"time=\"2017-09-22T17:50:58Z\" level=info msg=\"watching for changes to configtracker.configSpec{src:\\\"certs/postgres_ca/cert.pem\\\", writer:(configtracker.WriterFunc)(0x4863a0), templateFunc:(configtracker.TemplateFunc)(0x484cb0), cacheKey:\\\"21f500d9-8562-48d6-98db-65b3792665eb\\\"}\" \n","stream":"stderr","time":"2017-09-22T17:50:58.481520965Z"} | |
| {"log":"time=\"2017-09-22T17:50:58Z\" level=info msg=\"watching for changes to configtracker.configSpec{src:\\\"certs/postgres_server/cert.pem/a2382528a00b\\\", writer:(configtracker.WriterFunc)(0x4863a |
NewerOlder