mlx_lm_openai.md

MLX LM with the OpenAI Python Package

1. Install

Install MLX LM and openai:

pip install mlx-lm openai

2. Run the MLX LM server

Run the MLX LM server with:

mlx_lm.server

3. Make the HTTP request:

Make a Python script (like test.py). And include the following:

import openai

openai_client = openai.OpenAI(
  api_key="placeholder-api", base_url="http://localhost:8080"
)

response = openai_client.chat.completions.create(
    model='mlx-community/Meta-Llama-3-8B-Instruct-4bit',
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": f"Say this is a test!"},
    ],
)

# Process response.

Run the script python test.py.

Try updating MLX LM pip install -U mlx-lm, It should work.