smhr

Building a Singularity Container for Machine Learning, Data Science, & Chemistry

Learning Objectives

Build a Linux based Singularity container.
- First build a writable sandbox with essential elements.
- Inspect the container.
- Install additional software.
- Convert the sandbox to a read-only SquashFS container image.
Install software & packages from multiple sources.

Using apt-get package management system.

Setting up Qemu with a tap interface

There are two parts to networking within QEMU:

The virtual network device that is provided to the guest (e.g. a PCI network card).
The network backend that interacts with the emulated NIC (e.g. puts packets onto the host's network).

Example: User mode network

Set up remotes

setup local repo

mkdir myrepository
cd myrepository
git init

add bitbucket remote as "origin"

Installing and running Slurm on greyworm and other clusters

This page will specifically detail how to install and run Slurm on the greyworm cluster at LLNL. These instructions can be adapted to work on other clusters with other nodes.

In the current configuration, aims4.llnl.gov is the head, while greyworm[1-8].llnl.gov are the compute nodes. Note: The steps here are for the current configuration of aims4 and greyworm[1-8]. The configuration might change in the future.

A Brief Intro to Slurm

Slurm is a job scheduler used to run jobs on supercomputers and clusters.

Common Options

-#, --progress-bar Make curl display a simple progress bar instead of the more informational standard meter.

-b, --cookie <name=data> Supply cookie with request. If no =, then specifies the cookie file to use (see -c).

-c, --cookie-jar <file name> File to save response cookies to.

	import requests
	import h5py
	import yt
	print(yt.__version__)

	# First download a test halo from the TNG project

	baseUrl = 'http://www.tng-project.org/api/'

	def get(path, params=None):

	# Let networkd manage all devices on this system
	network:
	version: 2
	renderer: networkd
	ethernets:
	#declare the interface used y the bridge device
	enp7s0:
	dhcp4: no
	dhcp6: no

	FROM ubuntu
	RUN apt update \
	&& apt install -y firefox \
	openssh-server \
	xauth \
	&& mkdir /var/run/sshd \
	&& mkdir /root/.ssh \
	&& chmod 700 /root/.ssh \
	&& ssh-keygen -A \
	&& sed -i "s/^.PasswordAuthentication.$/PasswordAuthentication no/" /etc/ssh/sshd_config \

	# Install x-related to compile x11vnc from source code.
	sudo apt-get update
	sudo apt-get install -y libxtst-dev libssl-dev libjpeg-dev

	# Grep source code.
	wget http://x11vnc.sourceforge.net/dev/x11vnc-0.9.14-dev.tar.gz
	gzip -dc x11vnc-0.9.14-dev.tar.gz \| tar -xvf -
	cd x11vnc-0.9.14/
	./configure --prefix=/usr/local CFLAGS='-g -O2 -fno-stack-protector -Wall'