Helpful Shell Commands

helpful_shell_commands.md

Helpful Command Cheet Sheet

These are commands that I forget regularly and need to look up.

Format JSON output as pretty

cat boring.json | python3 -m json.tool > pretty.json

Remove blank line from a file

cat 20210614.raw |awk 'NF'

Convert DOS to UNIX Text Formatting

tr -d '\r' < a.txt > b.txt

Create a good SSH KEy

ssh-keygen -t rsa -m PEM -b 4096 -C "mylogin@HOST"

Rename a list of files to be lowercase

The ls -1 is the selector of which files are impacted so be careful.

for f in `ls -1`; do mv -v "$f" "`echo $f | tr '[A-Z]' '[a-z]'`"; done

Create a missing directory

Say you want a sub directory called output_data

cd dt_modules/sample
pwd
if [ ! -d "output_data" ]; then
  mkdir output_data
fi

Determine if a file is Zero length

You might have a file that exists but has nothing in it.

file_length=$(cat ${json_content_object_file} | wc -l)

if [[ ${file_length} -eq 0 ]]; then
		echo "WARNING: NO DATA FOR "${json_content_object_file}
else
    echo "INFO   : "${json_content_object_file}" has "${file_length}" characters in it"
fi

Fix Jetbrains Rider Console Log

When you copy/paste console logs from Rider, they end up with a newline where you have a word wrap. To fix this, the following command can be adapted. In this example, we are expecting that each log entry ends in a ']' and starts with a date/time. In one case some lines did end with a ']' but where not complete. Those had "Batch: [GUID]" at the end so they are treated as a special case.

klanger@laptop:~/logs$ cat Rider_RAW.log | \
        grep -v "^NewWorld" | \
        grep -v "^$" | \
        sed ':a;/[a-zA-Z0-9]$/{N;s/\n//;ba}'| \
        sed ':a;/-$/{N;s/\n//;ba}'| \
        sed ':a;/[\:\[]$/{N;s/\n//;ba}' | 
        sed ':a;/Batch\: \[[a-z0-9\-]*\]$/{N;s/\n//;ba}'

The goal of the above regex sed statements do the following:

1. Filter out any line starting with NewWorld
2. Filter out blank lines
3. Any line ENDING with a a-z, A-Z, or 0-9 should be combined with next line
4. Any line ENDING with a "-" should be combined with next line
5. Any line ENDING with ":" or "[" should be combined with next line
6. Any line ENDING with "Batch: [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]" where "x" could be a-z, A-Z, 0-9, or "-". Lines like this should be combined with the next line.

END