Breaking down a one-liner I'm using to backup files to S3 via crontab

backup-breakdown.md

Breaking down bash one-liner

I use the following one-liner to backup a file to AWS S3 daily as a cronjob

EXT=<extension here>; DIR="/home/pi/SERVER-SHARES"; FILENAME=$(ls -t "$DIR" | head -n 1 | grep ".$EXT$"); PRE=$(date -d "$(echo "$FILENAME" | awk '{print $4,$5}' | sed 's/,/, /g')" +"%Y%m%d"); RES=$(/home/pi/.pyenv/versions/awscli/bin/aws s3 cp --no-progress --storage-class DEEP_ARCHIVE "$DIR"/"$FILENAME" s3://<bucket-here>/"$PRE"--"$FILENAME" 2>&1); DATE=$(date +'%Y-%m-%d %T'); echo "$DATE | $RES" | sed -e :a -e '$!N;s/\n/^$DATE | /;ta' | tr '^' '\n' >> /home/pi/SERVER-SHARES/aws.log

---

Breakdown

• Setup directory/file names

  EXT=<extension here>
  DIR="/home/pi/SERVER-SHARES"
  FILENAME=$(ls -t "$DIR" | head -n 1 | grep ".$EXT$")

• Get the date from the filename (format e.g. Apr 15,2021) converted to format yyyymmdd. Last sed is to add a space after comma for date parsing

  Note: this expects a fixed format of filename for the awk command and breaks if a non-date string gets picked up at positions $4 and $5

  PRE=$(\
    date -d "\
      $(echo "$FILENAME" | awk '{print $4,$5}' | \
      sed 's/,/, /g')\
    " \
    +"%Y%m%d"\
  )

• Do the upload and save any outputs to a variable

  RES=$(\
    /home/pi/.pyenv/versions/awscli/bin/aws s3 cp \
      --no-progress \
      --storage-class DEEP_ARCHIVE "$DIR"/"$FILENAME" \
      s3://<bucket-here>/"$PRE"--"$FILENAME" \
        2>&1 \
  )

• Get current date in readable format

  DATE=$(date +'%Y-%m-%d %T')

• Print the results of the s3 command to a logfile with timestamps for each line. sed is to replace \n characters with a signal character (^) and the timestamp, tr is to replace the signal character (^) back with a \n since sed doesn't directly support \n characters (see here)

  echo "$DATE | $RES" | \
    sed -e :a -e '$!N;s/\n/^Date | /;ta' | tr '^' '\n' \
    >> /home/pi/SERVER-SHARES/aws.log