RickCogley

Current

I am using the fantastic jq to manipulate a REST API's json, into a csv for upsertting into another database system, via its API. Once in the target system, I'm doing some date math, including rounding time stamps to 30 min intervals to allow me to do group them, for standard deviation calculation. The problem is, there is a lot of data and the database chokes when it has to round every record and then do std dev calculations on each.

• Sample.json shows a small sample of what the input data looks like, but actually it's just stdout from the curl command in the shell script.
• example-initial.sh shows the important bit of the initial shell script, that uses curl to authenticate against the data location's API, and then use jq to add a couple of columns and export to csv. It works like a charm.
• ACME-X1-A.csv is an example of the output CSV file, that's then upsertted into the target db.

Goal

ipbastola

JQ to filter JSON by value

Syntax: cat <filename> | jq -c '.[] | select( .<key> | contains("<value>"))'

Example: To get json record having _id equal 611

cat my.json | jq -c '.[] | select( ._id | contains(611))'

Remember: if JSON value has no double quotes (eg. for numeric) to do not supply in filter i.e. in contains(611)

asfaltboy

Log cronjob output to syslog and send errors to sentry

The wonderful [cron-sentry][1] takes care of sending an event for program errors to sentry. To uniformy log everything outputted, we pipe both stderr and stdout to [logger][2] (should be available on most linux distros)

How to use

Download script, add execution perm and move to a directory on PATH

$ wget https://gist.githubusercontent.com/asfaltboy/23a7f963bbfcb48ce510307ec8cb8737/raw/49ebca20004211a16c85a61a77db7d0883ddf31c/log-and-sentry

$ chmod +x log-and-sentry

kennwhite

# Based on CentOS7 fork of @smartmadsoft: https://gist.github.com/moneytoo/ab3f34e4fddc2110675952f8280f49c5

# "6" for CentOS6 or Amazon Linux, "7" for CentOS7
CENTVER="6"

OPENSSL="openssl-1.1.0-pre5"
NGINX="nginx-1.11.0-1"

yum clean all
# Install epel packages (required for GeoIP-devel)

zoni

[program:errbot]
command = /srv/errbot/virtualenv/bin/errbot --config /srv/errbot/config.py
user = errbot
stdout_logfile = /var/log/supervisor/errbot.log
stderr_logfile = NONE
redirect_stderr = true
directory = /srv/errbot/
startsecs = 3
stopsignal = INT
environment = LC_ALL="en_US.UTF-8"

moneytoo

### No longer needed as of nginx-1.13.6-1.el7_4.ngx.x86_64.rpm from nginx.org
### it was compiled against OpenSSL 1.0.2 from CentoOS 7.4 so it supports ALPN (HTTP2 works)

yum -y groupinstall 'Development Tools'
yum -y install wget openssl-devel libxml2-devel libxslt-devel gd-devel perl-ExtUtils-Embed GeoIP-devel rpmdevtools

OPENSSL="openssl-1.0.2l"
NGINX_VERSION="1.13.5-1"
NJS_VERSION="1.13.5.0.1.13-1"

lethee

# Save this file
#   mac   - "~/Library/Application Support/Sublime Text 3/Packages/"
#   linux - "~/.config/sublime-text-3/Packages/"

# NOTE!
#   ~/.nvm/alias/default must contain full version name.
#   $ nvm alias default v4.3.5

import os

crittelmeyer

Vim Cheatsheet

As configured in my dotfiles

Intro Resources

• Your problem with Vim is that you don't grok vi (on StackOverflow)
• Coming Home to Vim
• VimCasts.org
• Vim Galore
• Vim Adventures!

gene1wood

{
    "APIGatewayServiceRolePolicy": {
        "Arn": "arn:aws:iam::aws:policy/aws-service-role/APIGatewayServiceRolePolicy",
        "AttachmentCount": 0,
        "CreateDate": "2019-10-22T18:22:01+00:00",
        "DefaultVersionId": "v6",
        "Document": {
            "Statement": [
                {

brock

# This is just a cheat sheet:

# On production
sudo -u postgres pg_dump database | gzip -9 > database.sql.gz

# On local
scp -C production:~/database.sql.gz
dropdb database && createdb database
gunzip < database.sql.gz | psql database