Anemll Anemll

ANE INT8 W8A8 Benchmark: ~1.7-1.9x FP16 Throughput on Apple Silicon

Demonstrates that Apple Neural Engine (ANE) achieves significantly higher throughput with INT8 W8A8 quantization vs FP16, consistent with native INT8 datapath support.

Results (M5, h17g, single ANE cluster)

Summary

Method	FP16	INT8 W8A8	Ratio

Pi Coding Agent + Apple Foundation Models

This gist shows a working local Pi provider setup for Apple's fm serve Chat Completions endpoint.

It supports both Apple Foundation Models exposed by the fm CLI:

fm/system: on-device Apple Foundation Model, configured as 4K context
fm/pcc: Private Cloud Compute model, configured as 32K context

Pi Coding Agent + Apple Foundation Models

This gist shows a working local Pi provider setup for Apple's fm serve Chat Completions endpoint.

It supports both Apple Foundation Models exposed by the fm CLI:

fm/system: on-device Apple Foundation Model, configured as 4K context
fm/pcc: Private Cloud Compute model, configured as 32K context

Kimi Delta Attention (KDA): Algorithm and vLLM PR #27654

This note explains the Kimi Delta Attention algorithm and maps it to the merged implementation in vllm-project/vllm#27654. The analysis is based on the PR's final head commit f48a1cd, the Kimi Linear paper, and the reference Flash Linear Attention implementation.

1. Executive summary

KernelDMA 1 MiB prefetch notch — portable profile

Post: Getting 50 GB/s Back Out of the ANE (Eileen Yoon, 2026-08-10). This repo’s Bonsai 27B export is not on the notch. Use this note plus the profiler to check other chips (M1 Max, M3 Ultra, M4, M5, …) and to confirm ANEC has no 1 MiB KernelDMA-split flag.

Gist (same scripts): https://gist.github.com/Anemll/39f657dc48b402747bdd96458edd415f

M5 W8A8 FFN expansion: splits and prefill vectors

The complete 2048 -> 8192 projection becomes about 2.1–2.5x faster when split into 2/4/8 output-channel branches on this M5. TP2 is a strong default for this tested shape; more splits do not consistently improve it. These are sequential branches in one ANE graph, not distributed tensor parallelism.

All measurements below include the full output: one native evaluation of all branches and concatenation. Compile/load time and external application overhead are excluded. This is an expansion projection, not a full gated FFN (no gate, activation, or down projection).

Shared weight scale

Vectors	TP1 µs	TP2 µs	TP4 µs	TP8 µs	TP2 speedup	TP2 µs/vector

Live decode TPS, TTFT & prefill throughput for Pi (v2)

A model-neutral throughput footer for Pi, validated against @earendil-works/pi-coding-agent 0.85.1.

It observes Pi's standard assistant-stream events, so it works with local and hosted models. For real server-side prefill throughput it can optionally read Prometheus metrics (e.g. vLLM /metrics); prefix caching is handled correctly, which client-side usage.input cannot.

Pi image-history filter for DeepSeek on Spark

Work around this vLLM error without restarting the model server:

At most 4 image(s) may be provided in one prompt. (parameter=image)

Pi sends images from retained conversation history, not just the latest turn. This extension keeps the newest four images in each outgoing request and

	import FoundationModels
	import Playgrounds
	import Foundation

	let session = LanguageModelSession()
	let start = Date()
	let response = try await session.respond(to: "What is Apple Neural Engine and how to use it?")
	let responseText = response.content // Replace 'value' with the actual property name from LanguageModelSession.Response<String> that holds the string payload.
	print(responseText)
	let end = Date()