Skip to content

Instantly share code, notes, and snippets.

View prestonmcgowan's full-sized avatar

Preston McGowan prestonmcgowan

View GitHub Profile
-- Priority
-- 1. Grab the number between lt/gt brackets (https://stackoverflow.com/questions/1454913/regular-expression-to-find-a-string-included-between-two-characters-while-exclud)
-- Message
-- 1. HIDE the spaces in a quoted string (https://stackoverflow.com/questions/36705436/regular-expression-to-select-all-whitespace-that-is-in-quotes)
-- 2. Convert spaces to commas
-- 3. Revert the HIDDEN spaces to spaces
-- Prepare the parts
-- 1. Concat the extracted priority with a key and add a comma before concat'ing with the rest of the Message
-- Finalize
-- 1. Split_to_map to get the key-value pairs
#!/bin/bash
FILE=/var/log/speedtest/speedtest-`date +"%Y%m"`.results
printf -- '-%.0s' {1..80} >> $FILE
echo >> $FILE
date >> $FILE
/usr/local/bin/speedtest --progress=no >> $FILE
printf -- '-%.0s' {1..80} >> $FILE
echo >> $FILE
IFS=','; for c in `curl -k -s https://localhost:8083/connectors | sed 's/\"//g; s/\[//g; s/\]//g'`; do echo $c; curl -k -s https://localhost:8083/connectors/$c/config > $c.config; done
SELECT
CONCAT (
REGEXP_EXTRACT(
'(<(?<priority>\\d+)>)\\d+: (?<host>([a-z0-9\\-]+)): ]: (?<date>([a-zA-Z]{3}\\s+\\d+\\s+\\d+:\\d+:\\d+)|([0-9T:Z\\-]+)): (?<message>.+)',
`rawMessage`, 1
),
REGEXP_EXTRACT(
'(<(?<priority>\\d+)>)\\d+: (?<host>([a-z0-9\\-]+)): ]: (?<date>([a-zA-Z]{3}\\s+\\d+\\s+\\d+:\\d+:\\d+)|([0-9T:Z\\-]+)): (?<message>.+)',
`rawMessage`, 5
),
@prestonmcgowan
prestonmcgowan / confluent--shutdown_all.sh
Created June 10, 2021 12:32
Using the cp-ansible inventory file, shutdown Confluent Services in order
#!/bin/bash
echo "Shutting down control_center"
ansible all -i $1 --limit control_center -m service -a "name=confluent-control-center state=stopped"
sleep 60
echo "Shutting down ksql"
ansible all -i $1 --limit ksql -m service -a "name=confluent-ksql state=stopped"
sleep 60
echo "Shutting down schema_registry"
ansible all -i $1 --limit schema_registry -m service -a "name=confluent-schema-registry state=stopped"
sleep 60
@prestonmcgowan
prestonmcgowan / send-directory-of-syslog.sh
Last active January 19, 2021 14:58
Loop through all the files in a directory, and make use of the send-syslog-data.sh to send the syslog data to Confluent Connect.
#!/bin/bash
## The ./send-syslog-data.sh command below runs as a background task to maximize the cores and ram.
## This script may make the system usable. Change the "&" to a ";" if you want to run it one file at
## a time.
for x in `find ./$1 -type f`; do ./send-syslog-data.sh $x & done
@prestonmcgowan
prestonmcgowan / send-syslog-data.sh
Last active January 19, 2021 14:55
Send data from a file of line delimited syslog to the Confluent Connect Syslog Connector
#!/bin/bash
echo "Sending $1"
while read line; do printf "."; echo $line | nc SYSLOG-CONNECT-FQDN_GOES-HERE 1514; done < $1
echo
@prestonmcgowan
prestonmcgowan / marklogic-password-checker-with-history.xqy
Last active October 30, 2019 19:32
MarkLogic Password Plugin for complexity and history
(: Password rules
:
: at least 12 characters
: has not been used in the previous 24 passwords
: has not been changed in the last day
: does not contain your account or full name
: contains at least three of the the four character groups
: - lower case
: - upper case
: - numbers
@prestonmcgowan
prestonmcgowan / install-valid-tde.xqy
Created October 29, 2019 13:46
Install valid TDE or show validation error
xquery version "1.0-ml";
import module namespace tde = "http://marklogic.com/xdmp/tde"
at "/MarkLogic/tde.xqy";
let $template-uri := "/gds_sample_GeoJSON.tde"
let $template :=
<template xmlns="http://marklogic.com/xdmp/tde">
<description>Example template for GeoJSON documents</description>
<context>/envelope[header/type="geojson"]</context>
@prestonmcgowan
prestonmcgowan / uris-with-element.xqy
Created October 10, 2019 18:55
Find MarkLogic URIs with some element
cts:uris('/sources/', "document", cts:element-query(xs:QName('myElement'),cts:and-query(())))