John Hammink Jammink2
Jammink2
/ gist:6fcf0ca97fc241d611f3
Last active
August 29, 2015 14:11
— forked from wrburgess/gist:2170970
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| require 'csv' | |
| require 'json' | |
| csv_string = CSV.generate do |csv| | |
| JSON.parse(File.open("filename.json").read).each do |hash| | |
| data = hash[1] | |
| theline = theline + " #{data}," | |
| end | |
| end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import urllib, urllib2, base64, time, json, sys | |
| import dateutil, dateutil.parser, datetime | |
| td_database = 'kinesis' | |
| td_table = 'events' | |
| td_master_key = '<YOUR_TD_MASTER_API_KEY>' | |
| td_endpoint = 'https://in.treasuredata.com/js/v3/event' | |
| def upload_td(records): | |
| # https://docs.treasuredata.com/articles/javascript-sdk#appendix-api-endpoint |
Jammink2
/ lambda.json
Created
April 4, 2016 23:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "Records": [ | |
| { | |
| "eventID": "shardId-000000000000:49545115243490985018280067714973144582180062593244200961", | |
| "eventVersion": "1.0", | |
| "kinesis": { | |
| "partitionKey": "partitionKey-3", | |
| "data": "eyJmaWVsZDEiOiAyLCAiZmllbGQyIjogImIifQ==", | |
| "kinesisSchemaVersion": "1.0", | |
| "sequenceNumber": "49545115243490985018280067714973144582180062593244200961" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-2.3.1.deb | |
| sudo dpkg -i elasticsearch-2.3.1.deb |
Jammink2
/ gist:65b75afef30f1e3824fa81751fed545f
Last active
April 12, 2016 23:17
Edit Elasticsearch.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| sudo nano /etc/elasticsearch/elasticsearch.yml |
Jammink2
/ gist:2ececd79f85b97e42fbb2b3180e2892e
Last active
April 8, 2016 22:05
Elasticsearch Settings to edit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cluster.name: elasticsearch | |
| node.name: "integrations" | |
| node.master: true | |
| node.data: true | |
| index.number_of_shards: 5 | |
| Index.number_of_replicas: 1 | |
| network.bind_host: <your_instance's_internal_IP> #from EC2 console | |
| transport.tcp.port: 9300 | |
| http.port: 9200 |
Jammink2
/ gist:7a71f4b891e5d5c7a2e6379af4ad37ed
Last active
April 12, 2016 23:17
Tailing Elasticsearch log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| tail -f /var/log/elasticsearch/elasticsearch.log |
Jammink2
/ gist:855dc4577bd54f3c130365f59cc5abf4
Last active
April 12, 2016 23:22
Restarting Elasticsearch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| sudo /etc/init.d/elasticsearch restart |
Jammink2
/ accounts.yml
Created
April 8, 2016 21:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| in: | |
| type: sfdc | |
| username: "USERNAME" # your salesforce username | |
| password: "PASSWORD" # your Salesforce password | |
| security_token: "SECURITY_TOKEN" # your Security Token | |
| client_id: "CLIENT_ID" # your app's consumer key | |
| client_secret: "CLIENT_SECRET" # your app's consumer secret | |
| login_url: https://login.salesforce.com/ | |
| incremental: false # incremental = full data dump | |
| target: Account # Salesforce Object to import |
Jammink2
/ gist:ed5798b9a0f248496ec99e5628e8f70a
Last active
April 12, 2016 23:18
Create DB and Table on Treasure Data
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| td db:create sfdc | |
| td table:create sfdc account |
OlderNewer