Local LLM

local-llm.md

Ollama exposes an OpenAI-compatible API on port 11434, so most OpenAI-compatible tools can talk to it by simply changing the base URL. I am gonna try Qwen3-Coder 14B, light enough for a much nicer developer experience while still having enough resources left for Docker, VS Code, PostgreSQL, browsers, etc.

Ollama supports OpenAI-style /v1/chat/completions, you can test it:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-coder:14b",
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      }
    ]
  }'

OpenCode

Open ~/.config/opencode/opencode.json and paste the following to it.

{
  "provider": "openai",
  "model": "qwen3-coder:14b",
  "baseURL": "http://localhost:11434/v1",
  "apiKey": "ollama"
}

How to Utilize LLMs/Agents to Develop Software

1. Do a deep research first (Gemeni 3):
   • To understand the problem at hand.
   • Know better the problem and potential issues.
2. Do a small PoC/prototyping.
   • Vibe code, e.g. when you are not familiar with that particular programming language.
   • Pair program to build a solid app.
   • Be sure that vibe coding stuff ain't considered as a feature for customer. The reason for this is that this vibe coded feature might breaks other parts of our app.
   • Use it as a rapid solution for looking at a system which works, and catch potential issues, corner cases and where it might interfier with other things (e.g. trashing RAM).
   • Convey to others that this is just showing the capability to gauge whether this is generating value for the customers. For PoC we do not need to write unit tests.
   • These cannot be deployed to prod!
3. Go over what LLM did, then ask another model for second opinion. Keep asking question from the model constantly.
4. Give context to the LLM:
   • Integrate the PoC in your main app.
   • Write the missing tests to catch corner cases.

     ⚠️ Warning

     Sometimes LLM does something stupid like exposing a public API which e.g. deletes the database.

   • Give ChatGPT the success criteria and then ask it to write unit test based on those criteria (this way it won't write some tests which only are testing everything except what you intend it to test 😉).
   • If you iterate a few times you should be able to get the things right as you want.

     ⚠️ Warning

     But also do NOT trust LLM blindly. Sometimes it is way faster to make a change instead of asking LLM to do that (it is a contried example but sometimes it is really happening. You might write a prompt like: change the color of XYZ div to red 😆).

Tip

• You can use AI assistance to have a high level overview of what is happening, instead of indulging in nitty gritty details of code. But there is a catch to that. I always have a better understanding of what LLM did if I read the code it wrote for me.
• Investigation: ask LLM to explain what a project does for you, and if you need to modify something in that project ask it to tell you where and how you should change it.

Consider what LLM is generating cyber-security-wise:

• Sonarcube might be helpful.
• Do NOT run LLMs on codes which are NOT trusted (3rd party or cloned from GitHub) since they might send your secrets to the attacker or you might run into prompt injection attacks.

local-llm.yml

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped

    ports:
      - "11434:11434"

    volumes:
      - ollama:/root/.ollama

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama: