How I manage my python environment in NFS

Step 1: install pyenv on NFS

```bash
export PYENV_ROOT="/net/nfs.cirrascale/allennlp/$(whoami)/pyenv"
curl https://pyenv.run | bash
```


Step 2: add the following to your `~/.bashrc`. Now your python environments are managed by pyenv
```
export PYENV_ROOT="/net/nfs.cirrascale/allennlp/$(whoami)/pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
source $(pyenv root)/completions/pyenv.bash
```

Step 3: install conda
```
source ~/.bashrc
pyenv install  miniconda3-3.9-24.1.2-0
```

Step 4: create a virtual env per project

```
conda create -n open-instruct python=3.10
```

Then you can run `pyenv versions` which should give you something like this

```
costah@allennlp-cirrascale-01:/net/nfs/allennlp/costa$ pyenv versions
  system
  mambaforge-23.1.0-0
* miniconda3-3.9-24.1.2-0 (set by /net/nfs.cirrascale/allennlp/costa/pyenv/version)
  miniconda3-3.9-24.1.2-0/envs/costa-utils
  miniconda3-3.9-24.1.2-0/envs/open-instruct
  miniconda3-3.9-24.1.2-0/envs/open-instruct-2
  miniconda3-3.9-24.1.2-0/envs/trl
```

You can then run `pyenv local miniconda3-3.9-24.1.2-0/envs/open-instruct` on the folder in which your 
project resides, and it creates a `.python-version` file in that folder. So every time you go to that folder
`miniconda3-3.9-24.1.2-0/envs/open-instruct` environment will be activated for you

NOTE on running on different nodes:

Everytime you run on a different node, you need to do Step 2 again, so that pyenv is in the `PATH`, and then it can find your python.