Create a Cluster with Spack Binary Cache

Spack_Binary_cache.md

Create a Cluster with Spack Binary Cache

package	install command	working?
openfoam	spack install --no-check-signature --cache-only openfoam	✅
gromacs	spack install --no-check-signature --cache-only gromacs	✅
ior	spack install --no-check-signature --cache-only ior	✅
osu-micro-benchmarks	spack install --no-check-signature --cache-only osu-micro-benchmarks	✅

1. Create a Cluster, I used the following config:

Important You must have s3_read_resource = arn:aws:s3:::*

[aws]
aws_region_name = ${AWS_DEFAULT_REGION}

[global]
cluster_template = default
update_check = false
sanity_check = true

[cluster default]
key_name = ${AWS_DEFAULT_REGION}
vpc_settings = public
base_os = alinux2
ebs_settings = myebs
compute_instance_type = c5.18xlarge
master_instance_type = c5n.2xlarge
cluster_type = ondemand
placement_group = DYNAMIC
placement = compute
max_queue_size = 8
initial_queue_size = 0
disable_hyperthreading = true
scheduler = slurm
s3_read_resource = arn:aws:s3:::*

[vpc public]
vpc_id = ${vpc_id}
master_subnet_id = ${master_subnet_id}
compute_subnet_id = ${compute_subnet_id}

[ebs myebs]
shared_dir = /shared
volume_type = gp2
volume_size = 20

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}

2. Install Spack

sudo su 
export SPACK_ROOT=/shared/spack
mkdir -p $SPACK_ROOT
git clone https://github.com/spack/spack $SPACK_ROOT
cd $SPACK_ROOT
git checkout releases/v0.15
echo "export SPACK_ROOT=$SPACK_ROOT" > /etc/profile.d/spack.sh
echo "source $SPACK_ROOT/share/spack/setup-env.sh" >> /etc/profile.d/spack.sh
exit
source /etc/profile.d/spack.sh
sudo chown -R $USER:$USER $SPACK_ROOT

Verify the install:

$ spack -V
0.15.4

4. Add the environment

$ sudo su
$ mkdir -p $SPACK_ROOT/var/spack/environments/aws
$ cat <<EOF >> $SPACK_ROOT/var/spack/environments/aws/spack.yaml
spack:
  view: false
  concretization: separately

  config:
    install_tree: /shared/spack/opt/spack
    build_jobs: 4

    packages:
      all:
        providers:
          blas:
          - openblas
          mpi:
          - mpich
        variants: +mpi
      mpich:
        variants: ~wrapperrpath netmod=ofi
        version:
          - 3.2.1
      binutils:
        variants: +gold+headers+libiberty~nls
        version:
          - 2.33.1
      cmake:
        version:
          - 3.17.3
      openfoam:
        variants: +paraview
        version:
          - 2006
      paraview:
        variants: +qt+python3
        version:
          - 5.8.0
      qt:
          variants: +opengl
      ncurses:
          variants: +termlib
      sqlite:
          variants: +column_metadata
      hdf5:
          variants: +hl
      mesa:
        variants: swr=avx,avx2
        version:
          - 18.3.6
      llvm:
        version:
          - 6.0.1
      hwloc:
        version:
          - 1.11.11

  mirrors: { "mirror": "s3://spack-mirrors/amzn2-e4s" }
EOF
exit

5. Activate the environment

$ spack env list
aws
$ spack env activate aws
$ spack concretize
$ spack find

6. Install Python 3 and Boto3

sudo yum install -y python3
sudo pip3 install boto3

7. Install packages!

spack install --no-check-signature ior

FAQs

1. My packages are installing from source - help!

Note: patchelf always installs from source - this is because it's a spack dependency and not a package dependency.

There's 2 reasons why this may happen:

a. No access to the S3 mirror. Run:

$ aws s3 ls s3://spack-mirrors/amzn2-e4s
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

If you see "Access Denied" then add the AmazonS3ReadOnlyAccess to your instance's iam profile, or add s3_read_resource = arn:aws:s3:::* to your cluster's config and run update.

b. Environment isn't activated

Check spack env list and make sure aws is in green, if not run spack env activate aws

Hi, on step 2. the exit seems to be before the
source /etc/profile.d/spack.sh

Also, git installation will be pre requisite.

DM me f you want to test something, jcuencar@

kindest Regards