Checking how device_map="auto"
lays out model TheBloke/guanaco-65B-HF in memory on 96GB VRAM workstation (2x 48GB) and ~500GB RAM.
Deps:
pip install -q -U bitsandbytes
pip install -q -U git+https://github.com/huggingface/transformers.git
pip install -q -U git+https://github.com/huggingface/peft.git
pip install -q -U git+https://github.com/huggingface/accelerate.git
pip install scipy