wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.1.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.3.1-darwin-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.1.zip
Mark Walkom markwalkom
markwalkom
/ BOM-json.py
Created
December 15, 2018 07:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from ftplib import FTP | |
| import os | |
| import xml.etree.ElementTree as ET | |
| import xmltodict | |
| import json | |
| url = 'ftp.bom.gov.au' | |
| filename = 'IDN65068.xml' | |
| def writeline(data): |
markwalkom
/ Elasticsearch-5.6.2.xml
Last active
September 20, 2019 15:50
Elasticsearch+Kibana 5.6.2 on unRAID 6.3.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <?xml version="1.0"?> | |
| <Container version="2"> | |
| <Name>Elasticsearch-5.6.2</Name> | |
| <Repository>59b11c02b218</Repository> | |
| <Registry>https://docker.elastic.co/</Registry> | |
| <Network>bridge</Network> | |
| <Privileged>false</Privileged> | |
| <Support>https://discuss.elastic.co/c/elasticsearch</Support> | |
| <Overview>Elasticsearch is a open source, distributed, RESTful search and analytics engine.</Overview> | |
| <Category>Tools:</Category> |
markwalkom
/ paypal-transactions.conf
Created
July 22, 2017 02:07
Logstash config to process PayPal transaction history (downloaded in CSV)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| input { | |
| stdin {} | |
| } | |
| filter { | |
| csv { | |
| columns => ["Date","Time","Time Zone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Counterparty Status","Shipping address","Address Status","Item Title","Item ID","Shipping and Handling Amount","Compensation Amount","GST","Option 1 Name","Option 1 Value","Option 2 Name","Option 2 Value","Auction Site","Buyer ID","Item URL","Closing Date","Escrow ID","Invoice ID","Reference Txn ID","Invoice Number","Custom Number","Quantity","Receipt ID","Balance","Contact Phone Number"] | |
| add_field => [ "timestamp", "%{Date} %{Time}" ] | |
| remove_field => [ "Date", "Time", "Time Zone" ] | |
| } | |
| date { |
markwalkom
/ Step 1 - Elasticsearch and Kibana.md
Last active
April 26, 2017 23:51
Monitoring Your Elastic Stack, with Beats
markwalkom
/ Step 1 - Elasticsearch and Kibana.md
Last active
April 26, 2017 06:46
Monitoring Your Elastic Stack with Beats
markwalkom
/ missing-fields-query.json
Created
May 18, 2016 05:13
Via Kibana, only show documents that have a missing field
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Via https://smelloworld.wordpress.com/2016/05/17/missing-fields-search-in-elasticsearch/ | |
| {“query”:{“filtered”:{“query”:{“match_all”:{}},”filter”:{“missing”:{“field”:”FIELDNAME”}}}}} |
cat nodes_stats.json|jq '.nodes[].indices.docs.count'|awk '{s+=$0} END {print s}'
cat nodes_stats.json|jq '.nodes[].indices.store.size_in_bytes'|awk '{s+=$0} END {print s}'
cat people.json | jq -r '.[]|"\"\(.pk)\"" + ": " + "\"\(.fields.name)\""'
markwalkom
/ gist:3766250c5f6b6206bafcc6c23562b3fc
Created
May 4, 2016 08:39
Logstash Grok Pattern for Windows DNS Log files - via https://discuss.elastic.co/t/windows-dns-incompatible-character-encodings/48961/5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| WINDNS %{NUMBER:log_date} %{TIME:log_time} %{WORD:dns_thread_id} %{WORD:dns_context}%{SPACE}%{WORD:dns_packet_id} %{WORD:dns_ip_protocol} %{WORD:dns_direction} %{IP:dns_client_address}%{SPACE}%{WORD:dns_xid}%{SPACE}(?:Q|R|U) ?(Q|R|U)?%{SPACE}[%{GREEDYDATA:dns_hex_flags}%{SPACE}%{WORD:dns_response}]%{SPACE}%{WORD:dns_recordtype}%{SPACE}([1-9][0-9]?)%{GREEDYDATA:dns_query_name} |
markwalkom
/ Logstash all
Created
February 5, 2016 03:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ logstash-2.2.0/bin/plugin list | |
| logstash-codec-avro | |
| logstash-codec-cef | |
| logstash-codec-cloudfront | |
| logstash-codec-cloudtrail | |
| logstash-codec-collectd | |
| logstash-codec-compress_spooler | |
| logstash-codec-dots | |
| logstash-codec-edn | |
| logstash-codec-edn_lines |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| F2B_DATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[ ]%{HOUR}:?%{MINUTE}(?::?%{SECOND}) | |
| F2B_ACTION (\w+)\.(?:\w+)(\s+)?\: | |
| F2B_JAIL \[(?<jail>\w+\-?\w+?)\] | |
| F2B_LEVEL (?<level>\w+)\s+ |
NewerOlder