llama-cpp-python to host local LLM running inside no-code visual programming chat flow

vignette_06_JUN_2023.md

Part 1/2 : Set up an OpenAI compatible CPU-based local LLM model

1. What is llama-cpp-python

   • llama-cpp-python offers a web server which aims to act as a drop-in replacement for the OpenAI API. This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).
   • https://github.com/abetlen/llama-cpp-python

2. Installation

pip install 'llama-cpp-python[server]'

3. Download the model from Huggingface: https://huggingface.co/TheBloke/LLaMa-13B-GGML/blob/main/llama-13b.ggmlv3.q4_0.bin and save it in ./models

4. Start the server:

python3 -m llama_cpp.server --model models/llama-13b.ggmlv3.q4_0.bin

5. Navigate to http://localhost:8000/docs to see the OpenAPI documentation.

6. Try it !

   • Use the swagger interface at http://localhost:8000/docs

7. Observation:

   • See 8000/docs showing functional emulation of an OpenAI model (input and output)
   • Measure #tokens / sec on your machine

8. Next:

   • Try with other models
   • Try with GPU support
   • Dockerize

Part 2/2 : Use it in a Langchain pipeline with Flowise

1. What is Flowise

   • Open source No-Code visual programming tool to build LLM apps

2. Setting up Flowise

   • Follow the instructions at: https://github.com/FlowiseAI/Flowise#-developers

3. Start the app:

yarn start

4. Access the app at http://localhost:3000

5. Design a simple conversational application

6. Python

7. Curl