Last active
June 14, 2026 11:03
-
-
Save eSlider/63438d88549c37bfdf2bb258fb3f1ac0 to your computer and use it in GitHub Desktop.
Intel Arc @ vllm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <title>grok Chat Export</title> | |
| <style> | |
| body { | |
| font-family: Arial, sans-serif; | |
| max-width: 800px; | |
| margin: 0 auto; | |
| padding: 20px; | |
| line-height: 1.6; | |
| } | |
| .message { | |
| margin-bottom: 20px; | |
| padding: 15px; | |
| border-radius: 8px; | |
| } | |
| .user { | |
| background-color: #f5f5f5; | |
| border-left: 4px solid #4a90e2; | |
| } | |
| .assistant { | |
| background-color: #f8f9fa; | |
| border-left: 4px solid #6c757d; | |
| } | |
| .role { | |
| font-weight: bold; | |
| margin-bottom: 8px; | |
| color: #333; | |
| } | |
| .content { | |
| white-space: pre-wrap; | |
| color: #2c3e50; | |
| } | |
| pre { | |
| background-color: #f8f9fa; | |
| padding: 10px; | |
| border-radius: 4px; | |
| overflow-x: auto; | |
| } | |
| code { | |
| font-family: 'Courier New', Courier, monospace; | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <div class="message user"> | |
| <div class="role">User</div> | |
| <div class="content">gfx1150 performance vs intel arc b570 llama.ccom</div> | |
| </div> | |
| <div class="message assistant"> | |
| <div class="role">Grok AI</div> | |
| <div class="content">Thought for 7sGFX1150 (AMD Radeon 890M iGPU) vs. Intel Arc B570 for llama.cpp inference.Rocm.docs.amd | |
| Hardware Overview | |
| GFX1150 / Radeon 890M: Integrated GPU (iGPU) in AMD Ryzen AI 300-series "Strix Point" APUs (e.g., Ryzen AI 9 HX 370). It's an RDNA 3.5 GPU with 16 Compute Units (~5.9–6 TFLOPS FP32), sharing system memory (typically DDR5/LPDDR5X, ~80–100+ GB/s effective depending on config and allocation). Power envelope is part of the APU (often 15–54W total).Chipsandcheese | |
| Intel Arc B570: Discrete Battlemage (Xe2) GPU, ~$220 MSRP. Significantly higher specs: more execution units, dedicated GDDR6 VRAM (likely 8–10GB class, higher bandwidth ~300+ GB/s), higher clocks, and full discrete TDP (around 100–150W range based on similar B580). It's positioned as a strong budget discrete card, competitive with or beating RTX 4060 / RX 7600 in many gaming scenarios.Reddit | |
| Raw compute advantage: The B570 is substantially more powerful (roughly 1.7–2x+ in TFLOPS and memory bandwidth for dedicated workloads). Discrete vs. iGPU is a big gap in sustained performance.Gpu.userbenchmark | |
| llama.cpp Performance Comparison | |
| llama.cpp supports both via backends like Vulkan (broad compatibility), SYCL/oneAPI (strong on Intel), HIP/ROCm (AMD, but limited/unofficial for gfx1150), and others. | |
| Radeon 890M (gfx1150): | |
| Uses Vulkan or limited ROCm/HIP (full ROCm support for gfx1150 is patchy/unofficial; often requires workarounds like HSA_OVERRIDE).GitHub | |
| Example (7B Q4_0 model): ~18–19 t/s token generation, ~110–120 t/s prompt processing on Linux with ROCm where it works.GitHub | |
| Vulkan on Linux works but can be 35–60% of potential "native" performance; good for smaller models that fit in allocated shared VRAM. Larger models spill to system RAM and slow down.Dev | |
| Strengths: Low power, integrated (no extra hardware), decent for 7–13B models. Weak for bigger ones due to memory sharing/bandwidth. | |
| Intel Arc B570 (and similar Battlemage like B580): | |
| Excellent SYCL backend support in llama.cpp (Intel's native path) + Vulkan. Recent optimizations (e.g., Q8_0 reorder kernels) have boosted performance significantly.GitHub | |
| B580/B570 class: Strong results, e.g., 25–35+ t/s on mid-sized models (like DeepSeek variants) with optimizations/overclocking; scales well for larger models thanks to dedicated VRAM. Prompt processing is very competitive.YouTube | |
| B570 should be close to B580 (typically 85–90% performance at lower power/clock). Much better for offloading larger models without system RAM penalties. | |
| Winner for llama.cpp: Arc B570 is significantly faster overall (often 1.5–2.5x+ in token generation for comparable models, especially quantized ones) due to dedicated VRAM, higher bandwidth, and mature SYCL optimizations. The 890M is respectable for an iGPU but limited by shared memory and less optimized compute paths for this workload.Phoronix | |
| Other Factors | |
| Power/Efficiency: 890M wins for ultra-low power laptops/mini-PCs. B570 draws more but delivers far better perf/W in sustained inference. | |
| VRAM & Model Size: Discrete B570 handles much larger models (e.g., 27B+ quantized) comfortably. 890M is capped by system RAM allocation. | |
| Software: Intel has stronger out-of-box llama.cpp integration (SYCL). AMD relies more on Vulkan or workarounds for Strix Point. | |
| Use Case: For a dedicated LLM rig, get the B570. For portable/light integrated use, 890M is solid but not a match. | |
| In short, the Intel Arc B570 dominates in raw llama.cpp performance, while the GFX1150/890M is a capable iGPU option for lighter workloads or power-constrained systems. Benchmarks vary by model/quantization/backend—test with your specific setup (e.g., via llama-bench).</div> | |
| </div> | |
| <div class="message user"> | |
| <div class="role">User</div> | |
| <div class="content">vs mac mini?</div> | |
| </div> | |
| <div class="message assistant"> | |
| <div class="role">Grok AI</div> | |
| <div class="content">Thought for 6sGFX1150 (Radeon 890M iGPU) vs. Intel Arc B570 vs. Mac Mini (M4 / M4 Pro) for llama.cpp / local LLMs. | |
| Quick Hardware Summary | |
| GFX1150 / Radeon 890M: AMD Ryzen AI 300 iGPU (16 CUs, ~6 TFLOPS FP32). Shared system memory (LPDDR5X/DDR5, high allocation possible, e.g., 64–96GB configs). Low power, integrated. | |
| Intel Arc B570: Discrete Battlemage (Xe2) GPU. Dedicated GDDR6 VRAM (likely 8–10GB+ class), much higher bandwidth (~300+ GB/s), stronger raw compute. ~100–150W TDP range. Strong budget discrete option. | |
| Mac Mini M4 / M4 Pro: Apple Silicon with unified memory (up to 32GB base M4, 48–64GB+ on Pro). Excellent memory bandwidth (e.g., ~273 GB/s on M4 Pro). Optimized Metal / MLX backends. Very efficient, silent, low power. | |
| llama.cpp Performance (Token Generation Focus) | |
| Performance varies heavily by model size, quantization (Q4/Q5/Q8), backend, and RAM. Numbers are approximate real-world ranges from benchmarks (mostly 7B–32B models): | |
| Radeon 890M (gfx1150): | |
| ~15–27 t/s on 7B–20B models (Vulkan or patched ROCm/HIP). | |
| Strong for its class as an iGPU, especially with 64GB+ shared RAM. Prompt eval is decent but generation is memory-bandwidth limited. Larger models (>20–30B) spill over and slow down.Reddit | |
| Intel Arc B570 (similar to B580 benchmarks): | |
| Significantly faster: often 25–40+ t/s on mid-sized models (e.g., 8B–27B quantized) with SYCL or optimized Vulkan. | |
| Better scaling for larger models due to dedicated VRAM and bandwidth. Recent optimizations (e.g., Q8_0 kernels) boost it further. Strong prompt processing.YouTube | |
| Mac Mini M4 / M4 Pro: | |
| Base M4 (16–32GB): ~18–40+ t/s on 7B–8B models (llama.cpp Metal or better with MLX/Ollama). Great for lighter use. | |
| M4 Pro (48–64GB): 11–18 t/s on 32B models; handles 70B quantized reasonably. MLX backend often 2–3x faster than older llama.cpp Metal for decode (e.g., 100+ t/s on smaller models).Like2byte | |
| Unified memory shines for large models without "VRAM" walls. Prompt processing can be very fast. | |
| Overall Winner for Raw Speed: Arc B570 generally leads in sustained llama.cpp performance for most model sizes, especially with dedicated hardware. Mac Mini (especially Pro with MLX) is very competitive or better for Apple-optimized workflows and efficiency. 890M is the weakest of the three but respectable for an iGPU.Medium | |
| Key Factors | |
| Model Size & Memory: | |
| Mac Mini excels with unified memory — easiest for big models (e.g., 70B on 48–64GB Pro). | |
| Arc B570 handles larger models well with dedicated VRAM. | |
| 890M limited by shared RAM allocation and bandwidth sharing with CPU. | |
| Software & Ease: | |
| Mac: MLX/Ollama is plug-and-play and often fastest. llama.cpp Metal is solid. | |
| Arc B570: Excellent SYCL support in llama.cpp + Vulkan. Mature for Intel GPUs. | |
| 890M: Vulkan works reliably; ROCm is patchy/unofficial (needs workarounds). Linux preferred for best results. | |
| Power & Form Factor: | |
| Mac Mini: Quietest, most efficient, desktop-friendly. | |
| 890M: Best for laptops/mini-PCs (very low power). | |
| B570: Higher performance but needs a discrete GPU setup (more power/heat). | |
| Other: Mac has great ecosystem integration. Discrete GPUs like B570 win for raw throughput and future-proofing with bigger models. 890M is great value in an APU but not competitive with the others for heavy inference. | |
| Recommendation: | |
| Best pure performance: Intel Arc B570 (dedicated GPU advantage). | |
| Best convenience/efficiency: Mac Mini M4 Pro (especially with MLX). | |
| Budget/portable iGPU: 890M if you're already on Ryzen AI hardware. | |
| Test with your specific models via llama-bench or Ollama/LM Studio. Results depend heavily on config! Let me know model sizes or use cases for more tailored advice.</div> | |
| </div> | |
| <div class="message user"> | |
| <div class="role">User</div> | |
| <div class="content">compare (table) with m5/intel/amdgpu price/performance</div> | |
| </div> | |
| <div class="message assistant"> | |
| <div class="role">Grok AI</div> | |
| <div class="content">Thought for 15sGFX1150 (Radeon 890M) vs. Intel Arc B570 vs. Mac Mini M5 (and M5 Pro) for llama.cpp — Price/Performance Comparison (as of mid-2026) | |
| AspectAMD GFX1150 / Radeon 890M (Ryzen AI 9 HX 370)Intel Arc B570 (Discrete)Apple Mac Mini M5 / M5 ProTypeIntegrated GPU (iGPU in APU)Discrete GPU (Battlemage Xe2)Unified Memory Architecture (SoC)Approx. PriceMini PC: $700–$1,200 (32–64GB RAM config) | |
| Laptop: $1,100–$1,500~$200–$260 (10GB GDDR6)Base M5: ~$699+ | |
| M5 Pro: ~$1,300–$1,600+ (higher RAM configs)MemoryShared LPDDR5X/DDR5 (up to 64–128GB in mini PCs)Dedicated 10GB GDDR6Unified 16–64GB+ (higher on Pro)Memory Bandwidth~80–120+ GB/s (shared)~300+ GB/s~270–600+ GB/s (M5 Pro/Max class)llama.cpp Perf (approx. tokens/sec)7B–13B Q4/Q5: 15–30 t/s | |
| 20B–32B: 8–18 t/s (Vulkan/ROCm)7B–27B: 25–45+ t/s (SYCL/Vulkan, strong optimizations)M5: Excellent on small-mid (40–70+ t/s MLX/llama.cpp on 8B) | |
| M5 Pro: 15–40+ t/s on 32B–70B Q4 (MLX often fastest)Best ForPortable/low-power, small-medium modelsRaw speed, larger models on budget discreteConvenience, large models (unified mem), efficiency, ecosystemPower DrawVery low (15–54W total APU)~75–120WExtremely efficient (quiet, low TDP)Model Size HandlingGood up to ~20–30B (limited by shared RAM)Strong for 27B+ quantizedExcellent for 70B+ on higher RAM configsSoftwareVulkan reliable; ROCm patchyExcellent SYCL + VulkanBest-in-class MLX / Metal / OllamaPrice/Perf ValueBest for integrated/portable setupsStrongest raw value for discrete perfBest convenience & efficiency; higher upfront for big RAM | |
| Key Takeaways (Price/Performance) | |
| Best Raw Performance per Dollar: Intel Arc B570 — At ~$220–260, it delivers significantly higher sustained t/s than the 890M iGPU and competes well with higher-end options for mid-sized models. Excellent for a dedicated LLM PC build.YouTube | |
| Best Efficiency & Convenience: Mac Mini M5 / M5 Pro — Unified memory + MLX makes it seamless for large contexts/models. M5 Pro offers great scaling and is often faster in optimized workflows than the others for equivalent power. Quiet and plug-and-play.Promptquorum | |
| Best Portable/Integrated: GFX1150 (890M) — Solid in Ryzen AI mini PCs or laptops, but trails in absolute speed due to shared memory and backend limitations. Great if you need mobility or ultra-low power.Amd | |
| Notes: | |
| Performance varies by exact model (e.g., 7B vs 70B), quantization, context length, and backend (SYCL shines on Intel, MLX on Apple, Vulkan everywhere). | |
| Prices are approximate street prices; Mac configs with 48–128GB unified RAM cost more. | |
| For heavy use, factor in full system cost (e.g., + RAM/CPU for discrete B570 setup). | |
| The Arc B570 wins pure price/performance for most llama.cpp users wanting speed without Apple ecosystem lock-in. Mac Mini M5 Pro is the premium pick for ease and large models. 890M is respectable but not competitive in raw throughput. | |
| Test with your models using llama-bench! Let me know specific models/RAM budgets for more details.</div> | |
| </div> | |
| <div class="message user"> | |
| <div class="role">User</div> | |
| <div class="content">Is it possible to setup agent, which work comparable with cursor agent in terms of speed, quality? compare as table</div> | |
| </div> | |
| <div class="message assistant"> | |
| <div class="role">Grok AI</div> | |
| <div class="content">Thought for 7sYes, it is possible to set up a local or hybrid agent that approaches Cursor’s agent (Composer mode) in quality and speed, but with trade-offs. Cursor excels due to its polished IDE integration, fast cloud frontier models (Claude Opus/Sonnet, GPT-5.x, Composer 2.5), and seamless agentic workflow. Local setups can get close (70–95% quality on coding tasks) with strong hardware + optimized tools.Morphllm | |
| Key Local/OSS Alternatives | |
| Continue.dev (best overall local): VS Code/JetBrains extension, excellent Ollama/llama.cpp support. | |
| OpenCode / Cline / Roo Code: Strong agentic capabilities, MCP/tool use, terminal + editor. | |
| Void Editor: Open-source Cursor-like fork. | |
| Aider: Excellent CLI agent (git-native). | |
| Hybrid: Use Cursor/Void with local models via OpenAI-compatible server (llama.cpp/Ollama). | |
| Comparison Table: Cursor Agent vs. Local Setups on Your Hardware Options | |
| AspectCursor (Cloud Agent)Local on Mac Mini M5/Pro (MLX/Ollama)Local on Intel Arc B570 (SYCL/llama.cpp)Local on GFX1150 / 890M (Vulkan/ROCm)Speed (Response / Tokens)Very Fast (cloud frontier models, low latency)Excellent (30–70+ t/s small models; 15–40 t/s 32B+ with MLX)Strong (25–50+ t/s mid models; good scaling)Decent (15–35 t/s small-mid; slower on large)Quality (Reasoning/Code Accuracy)Highest (Claude/GPT-5 tuned for agentic)Very Close (Qwen3-Coder, DeepSeek, GLM-4.7)Close (strong on optimized models)Good but lower on complex tasksAgentic CapabilitiesBest-in-class (multi-file, Composer loop)Strong with Continue/OpenCode/ClineStrong with Continue + SYCLAdequate (Vulkan limits some features)Context & Large CodebasesExcellent (cloud)Excellent (unified memory, 48–64GB+)Very Good (dedicated VRAM)Limited (shared RAM)Setup ComplexityPlug-and-play ($20/mo Pro)Medium (Ollama + Continue.dev)Medium (llama.cpp + Continue)Higher (ROCm workarounds)Privacy / OfflineLimited (sends code)FullFullFullCost$20–200+/mo (usage)Hardware only (one-time)~$220 GPU + PCIncluded in Ryzen AI mini-PC (~$700–1200)Best ForPolished daily driver, complex projectsEfficiency, large models, Apple ecosystemRaw speed/value on Windows/LinuxPortable/low-power setupsOverall Comparable?ReferenceClosest match (esp. M5 Pro)Excellent price/perfRespectable but trails | |
| Key Takeaways (Mid-2026) | |
| Mac Mini M5 Pro is the strongest for a Cursor-like local experience thanks to unified memory + MLX framework. It often feels snappiest for agentic coding with 32B–70B quantized models.Medium | |
| Arc B570 offers the best raw price/performance for discrete GPU acceleration — great with SYCL in llama.cpp and tools like Continue.dev. | |
| 890M works well for lighter workflows but struggles with very large contexts or heavy agent loops compared to the others. | |
| Quality Gap: Top local models (Qwen3-Coder, DeepSeek-Coder-V2, GLM-4.7) are within 5–15% of Claude/GPT on coding benchmarks when properly prompted. Agent tools like OpenCode/Cline bridge the workflow gap significantly.Mightybot | |
| Recommended Setup for Comparable Experience: | |
| Install Ollama or llama.cpp (faster). | |
| Use Continue.dev or Void Editor + OpenCode/Cline. | |
| Pair with strong coder models (e.g., Qwen3 32B or equivalent). | |
| Hybrid option: Run Cursor + local OpenAI-compatible endpoint for best of both. | |
| Local agents are now very viable — many developers prefer them for privacy, cost, and control. With the hardware you were comparing, a Mac Mini M5 Pro or B570 build + Continue.dev will get you closest to Cursor’s speed/quality. | |
| Let me know your preferred OS, model sizes, or specific workflow (e.g., web dev, large repo) for more detailed setup steps!</div> | |
| </div> | |
| <div class="message user"> | |
| <div class="role">User</div> | |
| <div class="content">What if using 3x Intel Arc at time? Which CLI agent is best for the agents tasks? Parrallelism? Latency?</div> | |
| </div> | |
| <div class="message assistant"> | |
| <div class="role">Grok AI</div> | |
| <div class="content">Thought for 8s3x Intel Arc B570 (or similar Battlemage) setup for local agents is a strong, cost-effective multi-GPU configuration for agentic LLM tasks (coding, research, multi-step workflows). It provides substantial VRAM (~24–30GB+ total, depending on exact B570 variant) and compute for larger models or parallel inference.Phoronix | |
| Multi-GPU Performance with 3x Arc B570 | |
| llama.cpp SYCL Backend (best for Intel Arc): Supports multi-GPU. | |
| Layer splitting or tensor parallelism works, but scaling is best up to ~2 GPUs for many workloads. With 3+ cards, SYCL often still beats single-GPU (better than Vulkan, which can degrade beyond 2). | |
| Expect good gains for large models (e.g., 32B–70B+ quantized) that don't fit on one card, or higher throughput via batching/multiple instances. | |
| Real-world: 2x cards is often the "sweet spot"; 3x adds more capacity/parallelism but with diminishing returns due to PCIe bandwidth limits (no NVLink equivalent). Use ONEAPI_DEVICE_SELECTOR or --tensor-split flags for control.Phoronix | |
| Benefits for Agents: | |
| Run larger context models (better reasoning/quality). | |
| Parallel serving: Multiple model instances or concurrent requests (e.g., via llama-server or vLLM-like setups with SYCL/IPEX-LLM). | |
| Throughput: Excellent for multi-agent systems where sub-agents query the model simultaneously. | |
| Limitations: PCIe interconnect is the bottleneck (consumer motherboards). Power/heat/PCIe lanes matter — need a strong PSU and suitable motherboard. SYCL optimizations continue improving for Battlemage.Wangjuntestdoc.readthedocs | |
| Overall: Transforms a single B570 into a mini workstation capable of Cursor-like (or better in privacy/speed for local) agent performance on complex tasks. Pair with high-RAM CPU (e.g., Arrow Lake or Ryzen) for best results. | |
| Best CLI Agent for Agentic Tasks | |
| For CLI-focused agentic tasks (coding, refactoring, multi-step automation) on local hardware: | |
| Top Recommendation: OpenCode (or Aider as strong alternative).Morphllm | |
| CLI AgentStrengths for Agentic TasksMulti-Agent / Parallelism SupportLatency & Hardware Fit (3x Arc)Local Model QualityOverall for Your SetupOpenCodeProvider-agnostic, strong terminal UI, subagents, extensibleExcellent — native subagents, parallel execution, multi-sessionLow latency with local SYCL; scales with multi-GPU servingHigh (any local model)Best overall — flexible & parallel-readyAiderGit-native pair programming, fast edits, reliableGood via multiple instances / git worktreesVery low (CLI lightweight)HighExcellent for git-heavy codingClineVS Code integration + CLI, step-by-step approvalStrong subagent/parallel modesGoodHighBest if mixing IDE + CLIGooseBroader automation beyond code, MCP toolsSolid multi-taskGoodHighGood for general agentsOthers (Roo Code, etc.)VariesVariableGoodHighNiche | |
| Why OpenCode stands out: Most-starred open-source option, excellent local support (Ollama/llama.cpp), built-in multi-agent workflows (subagents for parallel tasks like research + coding). Runs fully offline with your 3x Arc setup.Opencode | |
| Parallelism Tips: Use git worktrees for isolation + multiple agent instances. Many tools (including OpenCode) support spawning subagents. For inference parallelism, run a multi-GPU llama.cpp server and point agents to it (or multiple servers on different GPUs).Medium | |
| Latency: | |
| Single request: Low with optimized SYCL (fast decode on Arc). Multi-GPU helps more with prefill/large context than single-token latency. | |
| Agent loops: Parallel subagents reduce wall-clock time significantly (e.g., analyze multiple files/codebases at once). Expect Cursor-comparable or better experienced speed with strong local models (Qwen3-Coder, DeepSeek, etc.). | |
| Recommended Full Setup | |
| Hardware: 3x Arc B570 + sufficient PCIe lanes/RAM (64GB+). | |
| Inference: llama.cpp with SYCL (or IPEX-LLM) in server mode for multi-GPU. | |
| Agent: OpenCode or Aider + local coder model. | |
| Workflow: Combine with Continue.dev (IDE) for hybrid Cursor-like experience. | |
| This setup can match or exceed Cursor agent speed/quality for many tasks (especially privacy-focused or large-context) while being fully local and cheaper long-term. | |
| Trade-offs vs. Cursor: Slightly more setup, but superior control, no usage fees, and massive parallelism potential with 3 GPUs. Let me know your OS, preferred models, or specific tasks for exact commands!</div> | |
| </div> | |
| <div class="message user"> | |
| <div class="role">User</div> | |
| <div class="content">what is best agentic code model for the setup?</div> | |
| </div> | |
| <div class="message assistant"> | |
| <div class="role">Grok AI</div> | |
| <div class="content">Thought for 18sBest agentic coding model for your 3x Intel Arc B570 + llama.cpp (SYCL) setup (mid-2026) | |
| Top Recommendation: Qwen3-Coder series (especially Qwen3-Coder-Next / 27B–35B or MoE variants) | |
| Why it's the best fit: | |
| Excellent performance on agentic benchmarks (SWE-Bench, Terminal-Bench, tool-calling, long-horizon reasoning, multi-file edits). | |
| Strong native tool-use and recovery from execution failures — ideal for OpenCode/Aider/Cline workflows. | |
| GGUF support is mature; runs very well on llama.cpp SYCL (Intel-native backend). | |
| With 3x B570 (~24–30GB+ total VRAM), you can comfortably run larger quantized versions (Q4/Q5_K_M or MoE sparse) or use tensor parallelism / layer splitting for bigger contexts (up to 128K–262K tokens) and faster inference.Bibek-poudel.medium | |
| High decode speeds on Arc hardware with recent optimizations; great balance of quality and speed for agent loops.Discuss.huggingface | |
| Strong Alternatives (ranked for your multi-GPU setup): | |
| ModelSize / TypeAgentic StrengthsVRAM Fit on 3x B570Speed (est. t/s)Best ForNotesQwen3-Coder-Next27B–35B+ MoETop tool-calling, SWE-Bench, multi-step codingExcellent (full offload + large ctx)25–45+Overall best for agentsStrongest local coding-agent pick right nowGLM-4.7 / GLM-5.1~30B–40B active (MoE)Long-horizon stability, terminal agents, UI/code qualityVery Good20–40Complex refactoring & tool loopsExcellent judgment; strong on Terminal-BenchDevstral Small24BPurpose-built for agentic workflows (multi-file, debugging)Excellent (lighter)30–50+Git-heavy / repo explorationMistral's agentic specialistDeepSeek V4 / R130B+ active (MoE)Algorithmic coding, reasoningGood25–45Pure code generationGreat but slightly behind on pure agenticKimi K2.6Large MoESub-agent swarmsPossible with splittingVariableMulti-agent parallelismHeavier; better if you run multiple instances | |
| Why These Over Others? | |
| Chinese models (Qwen, GLM, DeepSeek) currently dominate open coding/agentic leaderboards. | |
| Qwen3-Coder consistently ranks highest for practical agentic coding with local tools like OpenCode/Aider.Securityboulevard | |
| Your 3x Arc setup gives enough headroom for parallelism: Run the main agent on 2 GPUs + a smaller specialist model on the third, or serve multiple sub-agents concurrently via llama-server. | |
| Setup Tips for Maximum Performance | |
| Use llama.cpp with SYCL (best for Arc) + --tensor-split or multi-GPU flags. | |
| Quant: Start with Q5_K_M or Q4_K_M for balance. | |
| Context: Leverage 128K+ with your VRAM headroom. | |
| Agent: OpenCode (recommended earlier) pairs exceptionally well with these models. | |
| Test with llama-bench and your specific agent tasks. | |
| Bottom line: Start with Qwen3-Coder-Next (27B–35B class) — it delivers the closest-to-Cursor quality for agentic coding on your hardware while being fully local and highly parallelizable. GLM-4.7/5.1 is a close second if you prioritize long multi-step stability. | |
| Let me know your target model size, context length, or specific tasks (e.g., web dev, large repo) for exact GGUF links or run commands!</div> | |
| </div> | |
| <div class="message user"> | |
| <div class="role">User</div> | |
| <div class="content">vllm setup performance, based on result from indi hackers like https://www.roger.lol/?</div> | |
| </div> | |
| <div class="message assistant"> | |
| <div class="role">Grok AI</div> | |
| <div class="content"><div class="relative group flex flex-col justify-center w-full max-w-[var(--content-max-width)] pb-0.5 items-start" id="response-0fa8ceb5-6915-4067-af5b-177f2fab86f7" style="scroll-margin-top: var(--scroll-margin-top, 0px);"><div dir="auto" data-testid="assistant-message" class="message-bubble relative rounded-3xl text-primary min-h-7 prose dark:prose-invert break-words prose-p:opacity-100 prose-strong:opacity-100 prose-li:opacity-100 prose-ul:opacity-100 prose-ol:opacity-100 prose-ul:my-1 prose-ol:my-1 prose-li:my-2 last:prose-li:mb-3 prose-li:ps-1 prose-li:ms-1 w-full max-w-none"><div class="thinking-container mb-3"><div class="flex flex-col"></div></div><div class="relative"><div class="relative response-content-markdown markdown [&_a:not(.not-prose)]:text-current [&_a:not(.not-prose):hover]:text-primary [&_a:not(.not-prose):hover]:decoration-primary [&_a:not(.not-prose)]:underline [&_a:not(.not-prose)]:decoration-primary/30 [&_a:not(.not-prose)]:underline-offset-2 [&>:first-child:not(.not-prose)]:mt-0 [&>:last-child:not(.not-prose)]:mb-0"><p dir="auto" class="break-words last:mb-0 max-md:leading-[155%] max-md:mb-4 max-md:last:mb-0" style="white-space: pre-wrap;"><strong class="font-semibold">vLLM setup performance on 3x Intel Arc B570 (Battlemage / Xe2) for agentic coding</strong> — based on indie hackers like <strong class="font-semibold">Roger Ngo (roger.lol)</strong> and community reports (mid-2026).<a href="https://www.roger.lol/blog/accessible-ai-vllm-on-intel-arc" target="_blank" rel="noopener noreferrer nofollow" class="no-copy ms-1 inline text-nowrap print:hidden py-[2.5px] ps-2 pe-2 rounded-full align-[0.5px] bg-surface-l1 text-[13px] leading-none font-medium !text-fg-secondary hover:!text-fg-primary visited:text-fg-secondary focus:outline-none !no-underline hover:!no-underline citation" data-state="closed">Roger</a></p> | |
| <h3 dir="auto" class="font-semibold text-xl max-md:font-display max-md:text-[19px] max-md:leading-[155%] max-md:tracking-[0px] max-md:mb-3 max-md:mt-3">Overview</h3> | |
| <p dir="auto" class="break-words last:mb-0 max-md:leading-[155%] max-md:mb-4 max-md:last:mb-0" style="white-space: pre-wrap;"><strong class="font-semibold">vLLM</strong> (with Intel XPU backend) is an excellent production-oriented serving engine for your multi-GPU setup. It excels at <strong class="font-semibold">concurrent requests</strong>, <strong class="font-semibold">paged attention/KV cache</strong>, and <strong class="font-semibold">high throughput</strong> — ideal for agentic workflows (multiple sub-agents, tool-calling loops, parallel tasks in OpenCode/Aider).</p> | |
| <p dir="auto" class="break-words last:mb-0 max-md:leading-[155%] max-md:mb-4 max-md:last:mb-0" style="white-space: pre-wrap;">It often outperforms <strong class="font-semibold">llama.cpp SYCL</strong> in <strong class="font-semibold">serving scenarios</strong> (concurrent users/agents) and scales decently with tensor parallelism (TP), though 2x GPUs is usually the sweet spot; 3x adds capacity for larger models or more parallelism.<a href="https://www.youtube.com/watch?v=iqG7WdZPf_Q" target="_blank" rel="noopener noreferrer nofollow" class="no-copy ms-1 inline text-nowrap print:hidden py-[2.5px] ps-2 pe-2 rounded-full align-[0.5px] bg-surface-l1 text-[13px] leading-none font-medium !text-fg-secondary hover:!text-fg-primary visited:text-fg-secondary focus:outline-none !no-underline hover:!no-underline citation" data-state="closed">YouTube</a></p> | |
| <p dir="auto" class="break-words last:mb-0 max-md:leading-[155%] max-md:mb-4 max-md:last:mb-0" style="white-space: pre-wrap;"><strong class="font-semibold">Roger.lol</strong> (key indie reference) shows <strong class="font-semibold">vLLM XPU on dual B580</strong> delivering <strong class="font-semibold">very fast</strong> results — e.g., ~80–83+ t/s on GPT-OSS-20B at full 128K context (much faster than llama.cpp in their tests). Your 3x B570 (slightly slower per card than B580, ~10–15% less) should scale similarly or better with extra headroom.<a href="https://www.youtube.com/watch?v=iqG7WdZPf_Q" target="_blank" rel="noopener noreferrer nofollow" class="no-copy ms-1 inline text-nowrap print:hidden py-[2.5px] ps-2 pe-2 rounded-full align-[0.5px] bg-surface-l1 text-[13px] leading-none font-medium !text-fg-secondary hover:!text-fg-primary visited:text-fg-secondary focus:outline-none !no-underline hover:!no-underline citation" data-state="closed">YouTube</a></p> | |
| <h3 dir="auto" class="font-semibold text-xl max-md:font-display max-md:text-[19px] max-md:leading-[155%] max-md:tracking-[0px] max-md:mb-3 max-md:mt-3">Performance Expectations (3x B570, ~24–30GB total VRAM)</h3> | |
| <ul dir="auto" class="marker:text-secondary"> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Single model, TP=2 or 3</strong>: Strong scaling for 20B–70B+ quantized models (Q4/Q5/FP8/MXFP4). High decode throughput due to optimized kernels.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Agentic use</strong>: Excellent for <strong class="font-semibold">parallel sub-agents</strong> or concurrent tool calls — vLLM handles batching and high QPS better than llama.cpp.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Rough Benchmarks</strong> (extrapolated from Roger + similar B580/B70 reports):<ul dir="auto" class="marker:text-secondary"> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">20B class</strong> (e.g., GPT-OSS-20B, Qwen3-Coder ~27B): 60–85+ t/s decode (128K ctx) on 2–3 GPUs.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">30B–40B MoE/active</strong>: 30–55 t/s with good quantization.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Concurrent agents</strong>: Handles multiple streams far better than single-instance llama.cpp.</li> | |
| </ul></li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">vs llama.cpp SYCL</strong>: vLLM wins on <strong class="font-semibold">throughput/serving</strong> and KV cache efficiency; llama.cpp often better for single-user low-latency decode on smaller models.</li> | |
| </ul> | |
| <p dir="auto" class="break-words last:mb-0 max-md:leading-[155%] max-md:mb-4 max-md:last:mb-0" style="white-space: pre-wrap;"><strong class="font-semibold">Multi-GPU Notes</strong>: Use <span class="text-sm px-1 rounded-sm !font-mono bg-orange-400/10 text-orange-500 dark:bg-orange-300/10 dark:text-orange-300">--tensor-parallel-size 2</span> or 3. PCIe limits full linear scaling (no NVLink), but P2P works well on Battlemage. Intel's LLM-Scaler (vLLM fork) helps with Arc Pro/consumer tuning.<a href="https://vllm.ai/blog/2025-11-11-intel-arc-pro-b" target="_blank" rel="noopener noreferrer nofollow" class="no-copy ms-1 inline text-nowrap print:hidden py-[2.5px] ps-2 pe-2 rounded-full align-[0.5px] bg-surface-l1 text-[13px] leading-none font-medium !text-fg-secondary hover:!text-fg-primary visited:text-fg-secondary focus:outline-none !no-underline hover:!no-underline citation" data-state="closed">Vllm</a></p> | |
| <h3 dir="auto" class="font-semibold text-xl max-md:font-display max-md:text-[19px] max-md:leading-[155%] max-md:tracking-[0px] max-md:mb-3 max-md:mt-3">Setup Summary (Based on Roger.lol + Official Intel)</h3> | |
| <ol dir="auto" class="marker:text-secondary"> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">OS</strong>: Ubuntu 24.04.3+ LTS (best driver/kernel support for Xe2).</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Drivers/oneAPI</strong>: Install Intel Xe drivers, Level Zero, oneAPI base toolkit.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Python</strong>: 3.12 (critical for XPU).</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Install</strong>:<ul dir="auto" class="marker:text-secondary"> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal">Torch XPU wheels.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal">Build vLLM from source with <span class="text-sm px-1 rounded-sm !font-mono bg-orange-400/10 text-orange-500 dark:bg-orange-300/10 dark:text-orange-300">VLLM_TARGET_DEVICE=xpu</span>.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal">Use Intel's <span class="text-sm px-1 rounded-sm !font-mono bg-orange-400/10 text-orange-500 dark:bg-orange-300/10 dark:text-orange-300">intel/vllm:xpu</span> Docker images for easier multi-GPU (LLM-Scaler recommended).</li> | |
| </ul></li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Run Example</strong> (adapted for 3x):<div dir="auto" class="not-prose"><div class="relative not-prose @container/code-block [&_div+div]:!mt-0 mb-3 mt-3 @md:-mx-4 @md:-mr-4" data-testid="code-block"><div class="border border-warm-gray-100 dark:border-neutral-900 rounded-xl"><div class="flex flex-row px-4 py-2 h-10 items-center rounded-t-xl bg-white"><span class="font-mono text-xs text-secondary select-none">text</span><div class="ml-auto flex flex-row gap-0.5 print:hidden"><div class="flex flex-row gap-0.5" style="opacity: 1;"></div></div></div><div class="shiki not-prose relative [&_pre]:overflow-auto [&_pre]:rounded-b-lg [&_pre]:px-4 [&_pre]:py-4 !p-0" style="border-radius: 0px 0px 12px 12px; border-top-color: currentcolor; border-top-style: none; border-top-width: medium; border-right-color: ; border-right-style: ; border-right-width: ; border-bottom-color: ; border-bottom-style: ; border-bottom-width: ; border-left-color: ; border-left-style: ; border-left-width: ; border-image-source: ; border-image-slice: ; border-image-width: ; border-image-outset: ; border-image-repeat: ; padding: 0px 16px 16px; margin-top: 0px; font-size: 0.9em; font-family: var(--font-ibm-plex-mono); line-height: 1.5em; background-color: white; color: var(--fg-primary); display: block; overflow: auto;"><pre class="shiki slack-ochin" tabindex="0" style="background-color: rgb(255, 255, 255); color: rgb(0, 35, 57);"><code><span class="line"><span>vllm serve <model> --device xpu --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --max-model-len 131072</span></span></code></pre></div><div></div></div><div class="false flex flex-col gap-2 h-full"></div></div></div>(Adjust TP; use scripts from roger.lol for env vars/LD_LIBRARY_PATH).<a href="https://www.roger.lol/blog/accessible-ai-vllm-on-intel-arc" target="_blank" rel="noopener noreferrer nofollow" class="no-copy ms-1 inline text-nowrap print:hidden py-[2.5px] ps-2 pe-2 rounded-full align-[0.5px] bg-surface-l1 text-[13px] leading-none font-medium !text-fg-secondary hover:!text-fg-primary visited:text-fg-secondary focus:outline-none !no-underline hover:!no-underline citation" data-state="closed">Roger</a></li> | |
| </ol> | |
| <h3 dir="auto" class="font-semibold text-xl max-md:font-display max-md:text-[19px] max-md:leading-[155%] max-md:tracking-[0px] max-md:mb-3 max-md:mt-3">Pros/Cons for Your Agent Setup</h3> | |
| <ul dir="auto" class="marker:text-secondary"> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Pros</strong>: High throughput, concurrent agents, mature OpenAI-compatible API → plug into OpenCode/Continue.dev. Great with Qwen3-Coder.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Cons</strong>: Higher VRAM overhead than llama.cpp; setup more involved (esp. bare-metal); occasional stability tweaks needed for multi-GPU.</li> | |
| <li class="break-words whitespace-pre-wrap [&>ul]:whitespace-normal [&>ol]:whitespace-normal"><strong class="font-semibold">Latency</strong>: Good single-stream (comparable or better than SYCL for batched); excels in agent loops with parallelism.</li> | |
| </ul> | |
| <p dir="auto" class="break-words last:mb-0 max-md:leading-[155%] max-md:mb-4 max-md:last:mb-0" style="white-space: pre-wrap;"><strong class="font-semibold">Recommendation</strong>: Use <strong class="font-semibold">vLLM XPU</strong> as your primary serving backend for agentic tasks on 3x B570 — pair it with <strong class="font-semibold">OpenCode</strong> pointing to the local OpenAI endpoint. It gets you closest to Cursor-like speed/quality in a multi-agent setup while being fully local. For single-threaded low-latency, fallback to llama.cpp SYCL.<a href="https://www.reddit.com/r/LocalLLaMA/comments/1gymtfp/best_inference_engine_for_intel_arc/" target="_blank" rel="noopener noreferrer nofollow" class="no-copy ms-1 inline text-nowrap print:hidden py-[2.5px] ps-2 pe-2 rounded-full align-[0.5px] bg-surface-l1 text-[13px] leading-none font-medium !text-fg-secondary hover:!text-fg-primary visited:text-fg-secondary focus:outline-none !no-underline hover:!no-underline citation" data-state="closed">Reddit</a></p> | |
| <p dir="auto" class="break-words last:mb-0 max-md:leading-[155%] max-md:mb-4 max-md:last:mb-0" style="white-space: pre-wrap;">Check <strong class="font-semibold">roger.lol/blog/accessible-ai-vllm-on-intel-arc</strong> for detailed dual-B580 guide (easily adaptable to 3x B570). Test with your Qwen3-Coder model! Let me know if you need exact commands or troubleshooting.</p></div></div><section class="inline-media-container flex flex-col gap-1 clear-both"></section></div><div class="order-first sticky hidden @md/mainview:block top-11 h-0 opacity-0 group-focus-within:opacity-100 group-hover:opacity-100 -ml-[36px] @lg/mainview:-ml-[40px] -left-[26px]"><div class="min-h-7 py-4 flex flex-col gap-0.5"></div></div><div class="action-buttons h-8 mt-0.5 mb-2 flex flex-row flex-wrap w-full justify-between last-response print:hidden"><div class="flex items-center gap-[2px] w-max rounded-lg text-xs px-2 start-0 @md/chat:start-3 -ml-4" style="bottom: 87.9844px;"><div class="opacity-0 group-focus-within:opacity-100 group-hover:opacity-100 [.last-response_&]:opacity-100 disabled:opacity-0 group-focus-within:disabled:opacity-60 group-hover:disabled:opacity-60 [.last-response_&]:disabled:opacity-60"><div class="inline-flex flex-row gap-1 items-center w-fit py-1 ps-1 pe-2 rounded-full cursor-pointer bg-surface-l1 border border-border-l1 hover:bg-surface-l1-hover" role="button" aria-label="57 sources" data-state="closed"><div class="flex items-center justify-center overflow-hidden rounded-full bg-surface border border-border-l1 size-5" style="z-index: 3;"><img class="size-4 shrink-0 m-0 rounded-full" alt="" role="presentation" src="https://www.google.com/s2/favicons?domain=reddit.com&sz=256"></div><div class="flex items-center justify-center overflow-hidden rounded-full bg-surface border border-border-l1 size-5" style="z-index: 2; margin-inline-start: -12px; mask: radial-gradient(12px at -3px 50%, transparent 99%, rgb(255, 255, 255) 100%);"><img class="size-4 shrink-0 m-0 rounded-full" alt="" role="presentation" src="https://www.google.com/s2/favicons?domain=linkedin.com&sz=256"></div><div class="flex items-center justify-center overflow-hidden rounded-full bg-surface border border-border-l1 size-5" style="z-index: 1; margin-inline-start: -12px; mask: radial-gradient(12px at -3px 50%, transparent 99%, rgb(255, 255, 255) 100%);"><img class="size-4 shrink-0 m-0 rounded-full" alt="" role="presentation" src="https://www.google.com/s2/favicons?domain=youtube.com&sz=256"></div><div class="truncate text-sm">57 sources</div></div></div></div></div><div aria-hidden="true" class="absolute bottom-0 h-px"></div><div class="flex flex-col gap-1 mt-2 items-start w-full" style="opacity: 1; transform: none;"></div></div></div> | |
| </div> | |
| </body> | |
| </html> | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment