Chat Model Architecture: OpenRouter → LiteLLM → Agent Zero with Qwen 3.5 Plus

chat-model-architecture.md

Chat Model Architecture: OpenRouter → LiteLLM → Agent Zero

Overview

This document details the cost-effective Chat Model architecture replacing Claude Sonnet 4.6 with Qwen 3.5 Plus, routed through OpenRouter and a local LiteLLM server for optimal performance and cost savings.

Architecture Flow:

Agent Zero → LiteLLM Server (local) → OpenRouter API → Qwen 3.5 Plus

Why This Architecture?

Cost Comparison

Model	Input Cost (/1M)	Output Cost (/1M)	Savings
Claude Sonnet 4.6	~$3.00	~$15.00	baseline
Qwen 3.5 Plus	~$0.30	~$0.60	~90% reduction

Performance Characteristics

Qwen 3.5 Plus strengths:

• ✅ Code generation and refactoring (near-parity with Sonnet)
• ✅ Tool calling and function execution (excellent)
• ✅ Long context understanding (256K tokens)
• ✅ Multi-step reasoning (strong)
• ⚠️ Creative writing (good, but different style)
• ⚠️ Nuanced instruction following (95% of Sonnet)

Architecture Components

1. OpenRouter (Model Gateway)

OpenRouter provides unified access to 100+ LLM providers with:

• Single API interface
• Automatic fallbacks
• Usage tracking and cost optimization
• No vendor lock-in

Base URL: https://openrouter.ai/api/v1

2. LiteLLM Server (Local Proxy)

LiteLLM acts as a local OpenAI-compatible proxy:

• Standardizes API format for Agent Zero
• Handles authentication and rate limiting
• Provides caching and retry logic
• Enables model switching without code changes

Local Endpoint: http://localhost:4000

3. Agent Zero Integration

Agent Zero connects to LiteLLM using standard OpenAI SDK:

• No framework modifications required
• Seamless model swapping
• Consistent tool calling interface

Configuration Files

LiteLLM Config (config.yaml)

model_list:
  - model_name: chat-model
    litellm_params:
      model: openrouter/qwen/qwen-3.5-plus-02-15
      api_key: os.environ/OPENROUTER_API_KEY
      api_base: https://openrouter.ai/api/v1
      timeout: 120
      max_retries: 3

  - model_name: chat-model-fallback
    litellm_params:
      model: openrouter/anthropic/claude-3-5-sonnet
      api_key: os.environ/OPENROUTER_API_KEY
      api_base: https://openrouter.ai/api/v1
      timeout: 120

litellm_settings:
  drop_params: true
  set_verbose: false
  cache: true
  cache_params:
    type: redis
    host: localhost
    port: 6379

Agent Zero Environment (.env)

# LiteLLM Server Configuration
LITELLM_SERVER_URL=http://localhost:4000
LITELLM_API_KEY=sk-1234  # Can be any value for local

# OpenRouter Configuration
OPENROUTER_API_KEY=sk-or-v1-xxxxxxxxxxxxx
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1

# Model Configuration
CHAT_MODEL=chat-model
CHAT_MODEL_TEMPERATURE=0.7
CHAT_MODEL_MAX_TOKENS=4096
CHAT_MODEL_CONTEXT_WINDOW=256000

Agent Zero Model Config (settings.json)

{
  "chat": {
    "provider": "openai-compatible",
    "model": "chat-model",
    "base_url": "http://localhost:4000/v1",
    "api_key": "sk-1234",
    "temperature": 0.7,
    "max_tokens": 4096,
    "context_window": 256000,
    "top_p": 0.9,
    "frequency_penalty": 0.0,
    "presence_penalty": 0.0,
    "stop_sequences": [],
    "tool_choice": "auto",
    "parallel_tool_calls": true
  }
}

Optimal Parameter Settings

For Coding Tasks

{
  "temperature": 0.3,
  "top_p": 0.9,
  "max_tokens": 8192,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0
}

Rationale:

• Lower temperature (0.3) for deterministic code output
• High max_tokens for complex file operations
• No penalty settings to allow necessary repetition in code

For General Chat/Reasoning

{
  "temperature": 0.7,
  "top_p": 0.95,
  "max_tokens": 4096,
  "frequency_penalty": 0.1,
  "presence_penalty": 0.1
}

Rationale:

• Moderate temperature for balanced creativity/coherence
• Slight penalties to reduce repetition
• Standard context window for most conversations

For Tool Calling

{
  "temperature": 0.1,
  "top_p": 0.5,
  "max_tokens": 2048,
  "tool_choice": "auto",
  "parallel_tool_calls": true
}

Rationale:

• Very low temperature for precise tool selection
• Low top_p to focus on most likely tool calls
• Parallel execution for efficiency

Deployment Steps

1. Install LiteLLM

pip install litellm[proxy]
pip install redis  # Optional: for caching

2. Start LiteLLM Server

litellm --config config.yaml --port 4000

3. Verify Connection

curl http://localhost:4000/v1/models   -H "Authorization: Bearer sk-1234"

4. Configure Agent Zero

Update Agent Zero's model configuration to point to LiteLLM:

• Set base_url to http://localhost:4000/v1
• Use any API key (LiteLLM handles actual auth)
• Set model name to match LiteLLM config (chat-model)

5. Test Chat Functionality

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000/v1",
    api_key="sk-1234"
)

response = client.chat.completions.create(
    model="chat-model",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Monitoring & Observability

LiteLLM Dashboard

Access at http://localhost:4000/ui:

• Real-time request tracking
• Cost analytics
• Error rates and latency
• Model usage breakdown

Key Metrics to Track

Metric	Target	Alert Threshold
Latency (p95)	<2s	>5s
Error Rate	<1%	>5%
Cost/Day	<$5	>$10
Cache Hit Rate	>40%	<20%

Troubleshooting

Common Issues

1. Connection Refused

# Check LiteLLM is running
ps aux | grep litellm

# Restart if needed
litellm --config config.yaml --port 4000 --detailed_debug

2. Model Not Found

# Verify model name in LiteLLM config
curl http://localhost:4000/v1/models   -H "Authorization: Bearer sk-1234"

3. Rate Limiting

# Add retry logic to LiteLLM config
litellm_settings:
  num_retries: 3
  request_timeout: 120
  fallbacks: [{"chat-model": ["chat-model-fallback"]}]

4. High Latency

# Enable caching
litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: localhost
    port: 6379

Cost Tracking

Monthly Projection (Based on Current Usage)

Usage Tier	Sonnet 4.6	Qwen 3.5 Plus	Savings
Light (100K tokens/day)	~$54/mo	~$5/mo	$49
Medium (500K tokens/day)	~$270/mo	~$27/mo	$243
Heavy (2M tokens/day)	~$1,080/mo	~$108/mo	$972

OpenRouter Dashboard

Track usage at: https://openrouter.ai/activity

• Real-time token consumption
• Cost breakdown by model
• API call history

Security Considerations

1. API Key Management

   • Store OpenRouter key in environment variables
   • Never commit keys to version control
   • Rotate keys quarterly

2. Local Network Security

   • Bind LiteLLM to localhost only (default)
   • Use firewall rules to restrict access
   • Enable authentication for production

3. Data Privacy

   • LiteLLM doesn't log request content by default
   • Disable verbose logging in production
   • Review OpenRouter's data policy

Future Enhancements

1. Multi-Model Routing

   • Route simple queries to cheaper models (Qwen 2.5)
   • Escalate complex tasks to Qwen 3.5 Plus
   • Automatic fallback to Sonnet if needed

2. Advanced Caching

   • Semantic caching with embeddings
   • Cross-session cache persistence
   • Cache invalidation strategies

3. Load Balancing

   • Multiple LiteLLM instances
   • Round-robin model routing
   • Health check monitoring

References

• OpenRouter Documentation
• LiteLLM Proxy Docs
• Qwen 3.5 Model Card
• Agent Zero Framework