marshki marshki

Automate Data Transfers Between CNS and the HPC at NYU (Synology use case)

Scope:

This doc covers how to set up an environment that allows one to Rsync data to/from CNS and the HPC using: a shell script, cron job, SSH keys, and Sendmail.

For reference:

source = local (hostname.cns.nyu.edu)

Transfer Data to NYU's HPC 🚀

Scope: Transfer data to NYU's high performance computing (HPC) cluster.

Summary of access nodes on "Greene":

Fully-qualified domain name (FQDN)	Purpose
gdtn.hpc.nyu.edu	Data transfer node (DTN)
greene.hpc.nyu.edu	Login node

	#!/usr/bin/env bash
	#
	# disk_check
	#
	# Use Broadcom's MegaCLI to check array's disk health. Log results.
	# Emit email if failed disk(s) found.
	#
	# Author: M. Krinitz <mjk235 [at] nyu [dot] edu>
	# Date: 7-JUL-2025
	# License: MIT

	#!/usr/bin/env bash
	#
	# mixer
	#
	# Download, unzip, and install BigFix in GNU/Linux.
	# 'Computer Name' is set to serial number in this script.
	#
	# Author: M. Krinitz <mjk235 [at] nyu [dot] edu>
	# Date: 2024-10-28
	# License: MIT

	#!/usr/bin/env bash
	#
	# fixer
	#
	# Download, unzip, and install BigFix in macOS.
	# A device's unique identifier in the BigFix console is its serial number.
	# 'Computer name' is set to serial number in this script.
	#
	# Author: M. Krinitz <mjk235 [at] nyu [dot] edu>
	# Date: 2024-10-10

	#!/usr/bin/env bash
	#
	# smarty_pants
	#
	# Parse SMART Monitor test results for bad sector count, drive info.
	#
	# Author: M. Krinitz <mjk235 [at] nyu [dot] edu>
	# Date: 04-AUG-2025
	# License: MIT
	host="$(hostname --fqdn)"

	#!/usr/bin/env bash
	#
	# inode_eraser
	#
	# Nuke inodes on: /dev/md0 partition (Synology) if capacity >= 90%.
	# Place file in: /usr/local/bin.
	# Set permisisons: chmod 644.
	# Set cronjob by editing: /etc/crontab, & restart service: systemctl restart crond
	#
	# Author: M. Krinitz <mjk235 [at] nyu [dot] edu>

	#!/usr/bin/env bash
	#
	# capacitOR
	#
	# Emit email if disk capacity is >= 90%.
	#
	# Author: M. Krinitz <mjk235 [at] nyu [dot] edu>
	# Date: 26-JUL-2025
	# License: MIT

	#!/usr/bin/env bash
	#
	# extractor
	#
	# Create text files with parsed data, suitable for import into Google Groups.
	#
	# Author: M. Krinitz <mjk235 [at] nyu [dot] edu>
	# Date: 2021.05.15
	# License: MIT

	#!/usr/bin/env bash
	#
	# boilerplate
	#
	# A boilerplate script for retrieving and installing .deb files, using TeamViewer as an example. Easy to modify!
	#
	# Author: M. Krinitz
	# Date: 2021.07.31
	# License: MIT