mrseanryan
/ install_text_gen.sh
Last active
January 22, 2024 17:59
Basic INTERACTIVE install of text-generation-webui to host LLM on Ubuntu - Launch for 24GB GPU such as AWS EC2 G4 or G5 box
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| GPU_GB=${1:-24} | |
| #sudo apt update | |
| #sudo apt install python3-pip | |
| git clone https://github.com/oobabooga/text-generation-webui.git | |
| pushd text-generation-webui |
mrseanryan
/ gist:54a411dd1849578724b0157fb0e83b40
Last active
January 21, 2024 10:46
[blocks instance booting!] AWS EC2 User Data script to launch text-generation-webui on Instance Launch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #cloud-boothook | |
| #!/bin/bash | |
| cd /home/ubuntu/text-generation-webui | |
| GPU_CHOICE=A USE_CUDA118=FALSE LAUNCH_AFTER_INSTALL=TRUE INSTALL_EXTENSIONS=TRUE ./start_linux.sh --public-api --extensions openai --api-port 5000 --auto-devices --gpu-memory 24 | |
| # --share |
mrseanryan
/ calc-gpu-layers.py
Created
October 22, 2023 16:06
Rough PoC using binary search to find the optimal number of model layers to offload to the GPU, for this LLM and this hardware.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Rough PoC using binary search to find the optimal number of model layers to offload to the GPU, for this LLM and this hardware. | |
| """ | |
| import time | |
| def call_llm(prompt, gpu_layers): | |
| # TODO fill in the actual call to LLM here | |
| # dummy GPU memory limit | |
| test_best_layers = 60 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <link rel="import" href="../google-map/google-map.html"> | |
| <link rel="import" href="../cool-clock/cool-clock.html"> | |
| <link rel="import" href="../smoothie-chart/smoothie-chart.html"> | |
| <link rel="import" href="../paper-button/paper-button.html"> | |
| <link rel="import" href="../topeka-elements/theme.html"> | |
| <link rel="import" href="../topeka-elements/topeka-resources.html"> | |
| <link rel="import" href="../topeka-elements/topeka-app.html"> | |
| <link rel="import" href="../topeka-elements/topeka-datasource.html"> | |
| <link rel="import" href="../paper-calculator/paper-calculator.html"> | |
| <link rel="import" href="../topeka-elements/category-icons.html"> |
NewerOlder