nicksan2c

System Design Cheatsheet

Picking the right architecture = Picking the right battles + Managing trade-offs

Basic Steps

Clarify and agree on the scope of the system

User cases (description of sequences of events that, taken together, lead to a system doing something useful)
- Who is going to use it?
- How are they going to use it?

A curated list of AWS resources to prepare for the AWS Certifications

A curated list of awesome AWS resources you need to prepare for the all 5 AWS Certifications. This gist will include: open source repos, blogs & blogposts, ebooks, PDF, whitepapers, video courses, free lecture, slides, sample test and many other resources.

This document is a list with video playlists from recent conferences on DevOps, WebOps, SRE, Systems administration, Monitoring, Building and Scaling Highly available sytems, apps and websites.

Available

How to backup data from sherlock to google drive

Download and install rclone to your $HOME/bin directory. Setup up rclone to connect to your google drive. Suppose you call your remote google drive drive.
Create a bash script called backup_gdrive.sh with the following information

#!/bin/sh

STUDY_DIR=STUDY
LOCAL_DIR=${SCRATCH}/${STUDY_DIR}

	--[[

	SLURM job submit filter for QOS

	Some code and ideas pulled from https://github.com/edf-hpc/slurm-llnl-misc-plugins/blob/master/job_submit/job_submit.lua

	--]]

	--########################################################################--
	--

	FROM docker.io/centos:latest
	MAINTAINER Bowen Denning <[email protected]>

	ENV PATH=$PATH:/opt/splunk/bin

	RUN yum update -y

	## Download and install Splunk
	RUN curl https://download.splunk.com/products/splunk/releases/6.4.1/linux/splunk-6.4.1-debde650d26e-linux-2.6-x86_64.rpm -O && \
	yum install -y splunk*.rpm && \

	# -- mode: ruby --
	# vi: set ft=ruby :
	#TODO
	Vagrant.configure(2) do \|config\|

	hosts = {
	'splunk01' => { 'ip' => '192.168.124.201', 'cpus' => 1, 'memory' => 2048, 'autostart' => true },
	# 'splunk02' => { 'ip' => '192.168.124.202', 'cpus' => 1, 'memory' => 2048, 'autostart' => true },
	}

	[debug_log_transform]
	REGEX=DEBUG\s[
	DEST_KEY = queue
	FORMAT = nullQueue

	Link https://answers.splunk.com/answers/11617/route-unwanted-logs-to-a-null-queue.html

	# install vagrant
	macmini:vagrant tru$ VERS=1.9.1
	macmini:vagrant tru$ curl https://releases.hashicorp.com/vagrant/${VERS}/vagrant_${VERS}.dmg > vagrant_${VERS}.dmg
	macmini:vagrant tru$ curl https://releases.hashicorp.com/vagrant/${VERS}/vagrant_${VERS}_SHA256SUMS > vagrant_${VERS}_SHA256SUMS
	macmini:vagrant tru$ grep vagrant_${VERS}.dmg vagrant_${VERS}_SHA256SUMS \| shasum -a 256 -c && \
	macmini:vagrant tru$ hdiutil mount vagrant_${VERS}.dmg && \
	macmini:vagrant tru$ sudo installer -pkg /Volumes/Vagrant/Vagrant.pkg -target /
	macmini:vagrant tru$ hdiutil unmount /Volumes/Vagrant
	# installing VirtualBox
	macmini:vagrant tru$ VERSVB=5.1.14

	---
	# SSH server settings, in line with https://stribika.github.io/2015/01/04/secure-secure-shell.html
	# Before using, change myhosts to your hosts' nickname and myuser to your username (two instances! make sure you replace both or you'll be locked out of ssh!)
	- hosts: myhosts
	become: true
	remote_user: myuser
	tasks:
	# Key exchange, ciphers and MACs
	- lineinfile: dest=/etc/ssh/sshd_config regexp='^KexAlgorithms' line='KexAlgorithms [email protected],diffie-hellman-group-exchange-sha256'