Skip to content

Instantly share code, notes, and snippets.

@zeddee

zeddee/first-time-sysadmin-braindump.md

Last active October 13, 2019 19:44
Show Gist options
  • Save zeddee/853f6568ff940b8be64c7fb4c5a0b333 to your computer and use it in GitHub Desktop.
Save zeddee/853f6568ff940b8be64c7fb4c5a0b333 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
first-time-sysadmin-braindump.md

First Time Sysadmin Braindump

Simple Grepping Access Logs

From: https://blog.sucuri.net/2015/08/ask-sucuri-how-did-my-wordpress-website-get-hacked-a-tutorial.html

Useful for identifying attacker IP and what was done.

# Exclude access log lines for files with these extensions
cat access-log |grep -Ev "\.(js|css|png|jpg|jpeg) HTTP/1"| less

# Show only lines that show POST requests to wp-admin and wp-login
cat access-log |grep -E "wp-admin|wp-login|POST /" | less

# Exclude known (e.g. 1.2.3.4, 1.2.3.5) ip addresses
cat access-log |grep -E "wp-admin|wp-login|POST /" |grep -v "^1.2.3.4|1.2.3.5" | more

Watch a log file as it updates

Use tail -f <filename> or tailf to watch a file as it updates.

yum

List installed packages:

yum list installed | grep <pkg>

cPanel/WHM management

Creating cpmove files

Create cpmove files by running:

/scripts/pkgacct <username>

This creates an archive of <username>'s files for moving to another cPanel/WHM instance.

Backing up to S3 bucket with restic

restic is a backup tool that creates incremental backups by creating versioned snapshots that are managed with git.

You can set restic to back up to an S3 bucket or a compatible API (e.g. Digital Ocean Spaces) by following the instructions here: https://restic.readthedocs.io/en/latest/030\_preparing\_a\_new\_repo.html#amazon-s3

For posterity:

# set env variables containing S3 credentials
export AWS_ACCESS_KEY_ID=<key>
export AWS_SECRET_ACCESS_KEY=<secret>
export RESTIC_REPOSITORY=s3:sgp1.digitaloceanspaces.com/<bucket_name>
restic -r $RESTIC_REPOSITORY init
restic backup -r $RESTIC_REPOSITORY -p <password_file> -v <backup_src_dir>

Command to run restic in detached mode:

nohup restic backup \
	-r $RESTIC_REPOSITORY \
	-p ./passwd-restic \
	-v /home/thetrav1/public_html > restic-log-$(date +%Y-%m-%dT%H%M%S%z).log 2>&1 &

Use s3fs to mount an S3 bucket as volume

You can mount as S3 bucket as a local volume using s3fs, allowing you to read and write to the bucket as if it were a disk attached to your system.

Because S3 is an object storage system, reading and writing to the volume is slow and involves a separate REST API request sent to the object storage server. This makes it best suited for reading and writing a small number of large files e.g. tarballs rather than many small files.

However, because object storage is cheap, the tradeoff makes sense if your goal is, for instance, to store and serve static content e.g. have it act as a CDN, or to store backups as tarballs.

To install s3fs:

sudo yum install epel-release
sudo yum install s3fs-fuse

# or

sudo apt-get install s3fs

Or build from source. You'll need the following dependencies:

fuse-devel
libxml2-devel
libcurl-devel

Some VPS hosting providers remove fuse from the kernel, causing a device not found: fuse; run modprobe fuse error. This usually should not be the case because fuse ships with most modern Linux distributions. You have to insert the fuse.ko kernel module with insmod /path_to/fuse.ko, or simply switch to another hosting provider.

sshfs also depends on fuse.

Misc:

Check memory usage on Linux

free -m
/proc/meminfo
vmstat
dmidecode # this prints RAM hardware info

Using journalctl

Use journalctl to output system logs.

journalctl --no-pager # pipes everything to stdoud
journalctl -o json # output to json. Useful if sending to SIEM
journalctl -n # displays 10 most recent entries
journalctl -n 20 # displays 20 most recent entries
journalctl --disk-usage # displays how much disk space the journals take up
journalctl --vacuum-size=1G # removes old journal entries to trim journal to specified size
journalctl --vacuum-time=1years # trims journal to fit within given timeframe

More:

Managing Multi-PHP

Multi-PHP is an absolutely killer app that allows you to switch between different versions of PHP on the same server, without having to restart the webserver (typically EasyApache).

You may find certain PHP extensions missing for a given version of PHP when working with Multi-PHP, e.g. the curl or the mysqlnd extensions.

In systems running a single PHP server, you'd usually remedy this by running:

yum install php-curl mysqlnd

But when dealing with Multi-PHP, you'll need to install the package that targets the specific Multi-PHP installations. To find out which packages these are, run an apt-get search <pkg> or yum search <pkg>. For example, to find out which mysqlnd package to install, run:

yum search mysqlnd
============================================================================================================ N/S matched: mysqlnd ============================================================================================================
jetphp71-mysqlnd.x86_64 : jetphp71-mysqlnd Package
ea-php54-php-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
ea-php55-php-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
ea-php56-php-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
ea-php70-php-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
ea-php71-php-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
ea-php72-php-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
ea-php73-php-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases

Then, install the package that patches the Multi-PHP version you want to use.

User management

View current user sessions

  • w command displays users currently signed into system.
  • who command does the same, but displays different information.

Add new user

adduser userland
passwd userland # not required in debian
usermod -aG sudo userland # debian
usermod -aG wheel userland # RHEL, Fedora, CentOS

Network management

netstat

https://www.binarytides.com/linux-netstat-command-examples/

# netstat options:

-a # lists all current connections
-t # lists all TCP connections
-u # lists all UDP connections
-n # suppress DNS resolution i.e. don't resolve IP addresses to hostnames
-l # show only listening connections
-p # show PID
-r # show route information
-c # get netstat to run continuously

# Examples:
netstat -atnp | grep ESTA # Get all ESTABLISHED connections
watch -d -n0 "netstat -atnp | grep ESTA"
netstat -aple | grep ntp # check if a service is running

Firewalls

Configure firewall for cPanel/WHM

https://documentation.cpanel.net/display/CKB/How+to+Configure+Your+Firewall+for+cPanel+Services#HowtoConfigureYourFirewallforcPanelServices-cent7CentOS7,CloudLinux7,andRHEL7firewallmanagement

Download and install ConfigureServer Firewall (csf):

cd /usr/src
rm -fv csf.tgz
wget https://download.configserver.com/csf.tgz
tar -xzf csf.tgz
cd csf && ./install.sh

Run the following script to replace your current iptables ruleset with the contents of /etc/firewalld/services/cpanel.xml:

/usr/local/cpanel/scripts/configure_firewall_for_cpanel

Extended instructions:

To replace your existing iptables rules with the rules in the /etc/firewalld/services/cpanel.xml file, perform the following steps:

Run the yum install firewalld command to ensure that your system has firewalld installed.
Run the systemctl start firewalld.service command to start the firewalld service.
Run the systemctl enable firewalld command to start the firewalld service when the server starts.
Run the iptables-save > backupfile command to save your existing firewall rules.
Run the /usr/local/cpanel/scripts/configure_firewall_for_cpanel script.
Run the iptables-restore < backupfile command to incorporate your old firewall rules into the new firewall rules file.

firewalld

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment