DaisukeMiyamoto

#!/bin/bash -xe
# Setting up Slurm Accounting feature with slurmdbd/MySQL for AWS ParallelCluster
# Daisuke Miyamoto [email protected]
#
# Test Condition:
# - 20200312
# - ParallelCluster 2.6.0
# - CentOS7/Slurm
#
# ParallelCluster config example:

DaisukeMiyamoto

#  
#  -- High Performance Computing Linpack Benchmark (HPL)                
#     HPL - 2.3 - December 2, 2018                          
#     Antoine P. Petitet                                                
#     University of Tennessee, Knoxville                                
#     Innovative Computing Laboratory                                 
#     (C) Copyright 2000-2008 All Rights Reserved                       
#                                                                       
#  -- Copyright notice and Licensing terms:                             
#                                                                       

DaisukeMiyamoto

#!/bin/bash

CLIENT_ID=
USERNAME=
PASSWORD=

aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id ${CLIENT_ID} --auth-parameters USERNAME=${USERNAME},PASSWORD=${PASSWORD}

DaisukeMiyamoto

#!/bin/bash -xe

mkdir 1m_10000
mkdir 10m_1000
mkdir 100m_100

for i in {0000..9999}; do dd if=/dev/urandom of=1m_10000/1m_$i.dat count=1024 bs=1K; done
for i in {0000..999}; do dd if=/dev/urandom of=10m_1000/10m_$i.dat count=10240 bs=1K; done
for i in {000..99}; do dd if=/dev/urandom of=100m_100/100m_$i.dat count=102400 bs=1K; done

DaisukeMiyamoto

AWS ParallelCluster + AWS Batch

Today I'm going to demonstrate running High Performance Conjucate Grandients (HPCG) in a containerized workload. This takes advantage of AWS ParallelCluster, AWS Batch, and OpenMPI.

First install aws-parallelcluster:

$ pip install aws-parallelcluster

DaisukeMiyamoto

#include <mpi.h>
#include <stdio.h>

int main() {
    int size;
    int rank;

    MPI_Init(NULL, NULL);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

DaisukeMiyamoto

# coding: utf-8

import json
import os
import boto3
import pandas as pd


def main():
    price_list = []

DaisukeMiyamoto

a1.large

========================================================================
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: ip-172-31-35-85.us-west-2.compute.internal: GNU/Linux
   OS: GNU/Linux -- 4.14.77-81.59.amzn2.aarch64 -- #1 SMP Mon Nov 12 21:28:57 UTC 2018
   Machine: aarch64 (aarch64)

DaisukeMiyamoto

import boto3
import json
import logging
import os
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError


SLACK_CHANNEL = os.environ['slackChannel']
HOOK_URL = os.environ['slackUrl']

DaisukeMiyamoto

#!/bin/bash                                                                                                                                                 

dd if=/dev/zero of=abcd10000mb_1 bs=1M count=10000
dd if=/dev/zero of=efgh10000mb_2 bs=1M count=10000
dd if=/dev/zero of=ijkl10000mb_3 bs=1M count=10000
dd if=/dev/zero of=mnop10000mb_4 bs=1M count=10000
dd if=/dev/zero of=qrst10000mb_5 bs=1M count=10000
dd if=/dev/zero of=uvwx10000mb_6 bs=1M count=10000
dd if=/dev/zero of=yzab10000mb_7 bs=1M count=10000
dd if=/dev/zero of=cdef10000mb_8 bs=1M count=10000