vuiseng9

Quick Guide - Bash Scripting

Mar'19, vuiseng9

The Shebang

#!/usr/bin/env bash

/bin/bash # for docker entrypoint

vuiseng9

Commandlines and Tricks

Your sync script between two servers, skipping existing files

#!/usr/bin/env bash

DESTHOST=destination
LOGDIR=rsync.log
TS=$(date +%F_%H-%M-%S)

mkdir -p $LOGDIR/$TS

vuiseng9

Docker Basics

Contributors:

• Lee, Dennis
• Chua, Vui Seng

Contents

• Docker Setup
• Running Containers: docker run
  • Understand entrypoint
  • Override entrypoint

vuiseng9

# Extended python -m http.serve with --username and --password parameters for
# basic auth, based on https://gist.github.com/fxsjy/5465353

from functools import partial
from http.server import SimpleHTTPRequestHandler, test
import base64
import os


class AuthHTTPRequestHandler(SimpleHTTPRequestHandler):

vuiseng9

#!/usr/bin/env python3
# coding: utf-8
# author: github.com/vuiseng9
# dependency ubt1804: graphviz libgraphviz-dev libgl1-mesa-glx

import os, sys
import pydotplus
from glob import glob
from natsort import natsorted

vuiseng9

Spin Up NNCF (PyTorch)

• Create a conda environment named nncf

conda create -n nncf python=3.8 # <your choice of python version, 3.8 is recommended>

# always activate your environment before following steps
conda activate nncf

vuiseng9

Pytest in vscode

settings.json

{
    "python.pythonPath": "/home/vchua/miniconda3/envs/AutoQPrecInit/bin/python",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.autoTestDiscoverOnSaveEnabled": false,
 "python.testing.pytestArgs": [

vuiseng9

# Ensure git-lfs is installed.

REPO_ID=<label>

# create a repo with your account in model hub
huggingface-cli repo create $REPO_ID

# clone the newly-created repo (git clone command will be printed)
git clone https://huggingface.co//

vuiseng9

export CUDA_VISIBLE_DEVICES=0
cd transformers/examples/pytorch/translation/
python run_translation.py  \
    --model_name_or_path Helsinki-NLP/opus-mt-de-en  \
    --dataset_name wmt16  \
    --dataset_config_name de-en  \
    --source_lang de  \
    --target_lang en  \
 --run_name opus-mt-de-en-test \

vuiseng9

docker pull openvino/ubuntu18_dev:2021.4.2 

all_onnx=$(ls /data1/vchua/tld-poc/repo/*/*.onnx)

for onnx in $all_onnx;
do
 base=$(basename $onnx .onnx)