LLM

llm-resources.md

Tools

Ollama is a go utility to run LLM locally. Either in the terminal as a command line or through a REST API
Ollama : https://github.com/jmorganca/ollama

To use Mistral with Ollama:
Ollama + mistral : https://www.reddit.com/r/LangChain/comments/16vwilm/mistralai_llm_with_langchin/

Langchain is a python library to interface with various LLMs:
https://www.langchain.com/

To use Ollama with LangChain:
LangChain + Ollama : https://python.langchain.com/docs/integrations/llms/ollama

A CLI utility and Python library for interacting with Large Language Models, both via remote APIs and models that can be installed and run on your own machine:
https://llm.datasette.io/en/stable/

Blog entries

Embeddings: What they are and why they matter: https://simonwillison.net/2023/Oct/23/embeddings/
HN Post about this blog entry: https://news.ycombinator.com/item?id=37985489

HN Post about Mistral where you will find info on LLM tooling in the comments: https://news.ycombinator.com/item?id=37675496

Using LLaMA 2.0, FAISS and LangChain for Question-Answering on Your Own Data: https://medium.com/@murtuza753/using-llama-2-0-faiss-and-langchain-for-question-answering-on-your-own-data-682241488476

Other gists

Use whisper to transcribe podcasts: https://gist.github.com/jdmichaud/28503f16c52e7907931c1b3bd6c1817f

To have a look

How to do Question/Answering using LangChain: langchain-ai/langchain#9121

talk.js

// Access Ollama LLM through a telnet connection.
//
// This script assumes `ollama` is running on port 11434 which is the default
// port when you run:
// ```
// ollama serve
// ```
// It also assumes that the llama2 and mistral model are installed:
// ```
// ollama pull llama2
// ollama pull mistral
// ```

const net = require('net');

function listen(port, model, stream) {
  const utf8Decoder = new TextDecoder("utf-8");
  const prmpt = `\n${model} >>> `;
  // Create a new TCP server
  const server = net.createServer((socket) => {
    // Handle incoming connections
    console.log(`New client connected: ${socket.remoteAddress}:${socket.remotePort}`);

    socket.write(prmpt);
    socket.on('data', async (data) => {
      const response = await fetch("http://localhost:11434/api/generate", {
        method: "POST",
        body: JSON.stringify({
          model,
          stream,
          prompt: data.toString().replace('\r', '').replace('\n', ''),
        }),
      });
      if (stream) {
        const reader = response.body.getReader();
      
        let chunk = { done: false };
        while (!chunk.done) {
          chunk = await reader.read();
          if (!chunk.done) {
            const responseData = JSON.parse(utf8Decoder.decode(chunk.value, { stream: true }));
            socket.write(responseData.response ?? responseData.error);
          }
        }
      } else {
        const responseData = await response.json();
        socket.write(responseData.response ?? responseData.error);
      }
      socket.write(prmpt);
    });
  
    socket.on('end', () => {
      console.log(`Client disconnected: ${socket.remoteAddress}:${socket.remotePort}`);
    });
  });

  // Start listening for incoming connections
  server.listen(port, 'localhost', () => {
    console.log(`${model} on port ${port}`);
  });
}

try { listen(13000, 'llama2', true); } catch (e) { console.error(e); }
try { listen(13001, 'mistral', true); } catch (e) { console.error(e); }