shubhamagarwal92

import pandas as pd


def read_json_to_df(file_path):
    # df = pd.read_json(path_or_buf=file_path,orient='records',lines=True)
    df = pd.read_json(path_or_buf=file_path, orient='records')
    return df

def read_json_list_to_df(json_list):
    df = pd.DataFrame.from_records(json_list)

shubhamagarwal92

# Thanks to https://github.com/andreavanzo for figuring this out

# device should be integer here and not torch.device
# do 
torch.cuda.set_device(device)
# before
torch.cuda.empty_cache()

# Also do empty cache after every epoch

shubhamagarwal92

# Taken from https://datascience.stackexchange.com/a/46451

# https://stackoverflow.com/questions/57334052/how-to-fix-pandas-compat-has-no-attribute-string-types-error-in-python3
# https://github.com/pydata/pandas-datareader/issues/655
# pip install --upgrade pandas==0.24.2
# pip install xlswriter


import pandas as pd

shubhamagarwal92

# Check number of cpus
grep -c ^processor /proc/cpuinfo     

sudo apt-get install tmux 
sudo apt-get install htop
sudo apt install git

# Conda
sudo mkdir -p /scratch/shubham/packages
sudo chown -R shubham /scratch/shubham

shubhamagarwal92

pip install --no-cache-dir <pkg>

# Currently conda doesnt allow ignoring cache
# https://github.com/conda/conda/issues/7741
conda clean --all --force-pkgs-dirs


# https://stackoverflow.com/a/57994079/3776827

shubhamagarwal92

Installing Git – the easy way

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

– The Git website

Choose one of the following options.

• Instructions for Windows
• Instructions for Mac
• Instructions for Linux

shubhamagarwal92

Pre-requisite: You should be able to do ssh without entering password. See this

Steps (Fig 1 and 2)

1. Open project in PyCharm
2. Make directory on server
3. Test you can ssh without entering password. (Save your rsa key on server)
4. Preferences -> Deployment
5. Enable automatic upload https://www.jetbrains.com/help/pycharm/deployment-in-pycharm.html#upload-to-default-server
6. First time upload your directory. Right click project folder -> deployment -> deploy to <machine_name>
7. Add multiple directories if want to deploy on different servers.

shubhamagarwal92

This started as an introspection of the open-source contribs that I did during my PhD.

Creating a list of PRs and contribs in nice projects:

1. GSoC

Link: https://github.com/shubhamagarwal92/pecan

Medium: https://medium.com/@shubhamagarwal92/google-summer-of-code-2017-pecan-daa2fd11755a

shubhamagarwal92

# https://cmdlinetips.com/2016/03/how-to-join-pdf-files-in-mac-on-terminal/
# On mac
/System/Library/Automator/Combine\ PDF\ Pages.action/Contents/Resources/join.py -o pages.pdf 1.pdf 2.pdf


# Another command
gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=combine.pdf -dBATCH 1.pdf 2.pdf

# sed 's/>>> //g ' combinePDF.py >> combinePDF.py

shubhamagarwal92

# Following https://stackoverflow.com/questions/38080407/how-can-i-install-the-latest-anaconda-with-wget

latest=$(wget -qO- https://repo.anaconda.com/archive/  | 
grep -Eo "(href=\")(Anaconda3-.*-Linux-x86_64.sh)*\"" | 
sed 's/href=//g' | sed 's/\"//g' | head -n 1); wget "https://repo.anaconda.com/archive/$latest"

bash $latest -b

rm $latest
echo 'export PATH="~/anaconda3/bin:$PATH"' >> ~/.bashrc