M5 ANE INT8/W8A8 1 MiB coefficient-window slowdown: shared vs per-channel weight scales; standalone native eval_us reproducer

README.md

M5 ANE INT8/W8A8: slowdown at a 1 MiB weight window

Standalone reproducer for an 8-bit-integer ANE convolution slowdown on Apple M5. Both weights and activations are quantized (W8A8); this is not weight-only INT8 with FP16 convolution. Model I/O is FP16, with quantize/dequantize operations around the convolution.

Related FP16 reproducer: https://gist.github.com/Anemll/39f657dc48b402747bdd96458edd415f

Run

Download all four source/document files into one directory. Requires Apple Silicon macOS, Xcode command-line tools (clang), Python 3.9+, NumPy, and coremltools. Tested with Python 3.9.25, coremltools 9.0, NumPy 2.0.2 on Apple M5 (Mac17,2), macOS 27.0 (26A5425a).

python3.9 -m pip install 'coremltools==9.0' 'numpy==2.0.2'

# Default: compare shared vs per-channel WEIGHT scales
python3.9 profile_int8_dma.py --out run_both

# Shared weight scale only: exact 1 MiB on-boundary
python3.9 profile_int8_dma.py --out run_tensor --scales tensor

# Check reverse measurement order
python3.9 profile_int8_dma.py --out run_reverse --scales both --reverse

# Same byte count, different shape: 4096 x Cin=4032/4096
python3.9 profile_int8_dma.py --out run_cout4096 --cout 4096 --scales both

Every run needs a fresh output directory. The script writes models, MIL, all timing samples, medians, payload GB/s, and ratios to report.json. Default: 10 warmups + 64 timed evaluations per shape. Avoid competing ANE workloads. If aned changes PID, the run is marked invalid. This uses a private ANE API and may need adaptation on other OS releases.

This is the single-cluster, 16-core lattice. Do not use these dimensions to infer Ultra behavior. A flat result on another chip is a valid outcome, not evidence that the script failed.

Keeping the weight window at 1 MiB

INT8 payload per core:

(Cout / 16) * Cin * 1 byte

Cout	Cin	Weight bytes/core	MiB/core
8192	2016	1,032,192	0.984375
8192	2048	1,048,576	1.0
4096	4032	1,032,192	0.984375
4096	4096	1,048,576	1.0

The shared-weight-scale variant uses one scale for all weights. Separate HWX inspection on this M5 verified sixteen coefficient descriptors of exactly 0x100000 bytes for each on-boundary shape. HWX tools are not required or included in this reproducer; descriptor sizes are prior observations, not dynamically verified by this script.

Per-channel scales add coefficient bytes. For 8192 x 2048, the inspected size was 0x100400 per core: 1 MiB weights + 1 KiB scales. The extra 1 KiB is 512 output-channel scales x 2 bytes, interleaved as sixteen [64-byte scales + 64-KiB weights] blocks. It is not activation data. The HWX programs one combined coefficient range per core, with no separately programmed scale-fetch task; it cannot establish the physical bus transaction count.

Both variants use a shared activation scale (0.001) at input and output. “Per-channel” here refers to weight scales, not activation scales.

Measured M5 results

Native call medians, 10 warmups + 64 samples:

Cout	Scale layout	Off Cin / µs	On Cin / µs	On/off
8192	Shared	2016 / 342.25	2048 / 878.52	2.57x
8192	Per-channel	2016 / 341.12	2048 / 882.27	2.59x
4096	Shared	4032 / 354.12	4096 / 851.60	2.40x
4096	Per-channel	4032 / 359.62	4096 / 826.96	2.30x

The standalone packaged script was rerun successfully: Cout=8192 gave 2.60x shared / 2.52x per-channel; Cout=4096 gave 2.38x / 2.33x.

The slowdown persists with per-channel scales; the extra bytes do not avoid it. Exact total descriptor-size equality alone is therefore too narrow a description of the trigger. These results do not prove a prefetch-wrap mechanism or separate internal DMA streams.

What is timed and verified

• The host measures elapsed time around _ANEClient evaluateWithModel, not Core ML predict(), compilation, or model loading. It includes native call overhead; it is not a hardware-only DMA counter.
• GB/s is nominal INT8 weight payload divided by that elapsed time, excluding scales and activation traffic.
• The generator checks that INT8 weight storage and activation quantization survive model conversion. The convolution weight is constant 127 with FP16 scale approximately 0.001/127, matching the earlier FP16 test's approximately 0.001 weight. This is a synthetic timing test, not a numerical-accuracy benchmark.
• Separate M5 HWX inspection confirmed an 8-bit integer convolution between format-conversion tasks. The reverse-engineered decoder labels its operand format code 1 as UINT8, despite signed INT8 in MIL; do not cite this as proof of a signed hardware encoding.
• Running the script alone does not inspect compiler-selected instruction formats on a new chip/OS.

Files

• profile_int8_dma.py: W8A8 model generation and CLI.
• native_utils.py: native build/timing and chip identification helpers.
• kernel_dma_mib_host.m: original native timing host from the FP16 gist.
• README.md: instructions and measured results.

FFN splits and prefill vectors (M5 extension)

See FFN_SPLIT_PREFILL.md for TP1/2/4/8-style output-channel splits and 1/4/16/32-vector measurements. The complete 2048 -> 8192 projection, including all branches and concatenation, was about 2.1–2.5x faster when split. Shared and per-channel weight-scale results are included.

python3.9 profile_int8_tp_prefill.py --out ./tp_prefill_run

This extension also needs tp_host.m and the existing native_utils.py. It does not require HWX tools. These are single-M5 graph splits, not distributed tensor parallelism or a complete gated FFN.

FFN_SPLIT_PREFILL.md

M5 W8A8 FFN expansion: splits and prefill vectors

The complete 2048 -> 8192 projection becomes about 2.1–2.5x faster when split into 2/4/8 output-channel branches on this M5. TP2 is a strong default for this tested shape; more splits do not consistently improve it. These are sequential branches in one ANE graph, not distributed tensor parallelism.

All measurements below include the full output: one native evaluation of all branches and concatenation. Compile/load time and external application overhead are excluded. This is an expansion projection, not a full gated FFN (no gate, activation, or down projection).

Shared weight scale

Vectors	TP1 µs	TP2 µs	TP4 µs	TP8 µs	TP2 speedup	TP2 µs/vector
1	843.29	365.52	377.90	370.08	2.31x	365.52
4	835.94	377.50	371.04	381.88	2.21x	94.38
16	892.83	362.56	375.27	388.83	2.46x	22.66
32	909.40	358.69	376.67	393.75	2.54x	11.21

Per-channel weight scales

Vectors	TP1 µs	TP2 µs	TP4 µs	TP8 µs	TP2 speedup	TP2 µs/vector
1	853.73	372.71	393.96	374.88	2.29x	372.71
4	847.02	385.12	375.73	402.75	2.20x	96.28
16	872.31	378.65	386.23	395.94	2.30x	23.67
32	877.65	391.17	381.92	395.60	2.24x	12.22

Method and limits

• Apple M5, Mac17,2, macOS 27.0; Python 3.9.25, coremltools 9.0.
• Input layout [1, 2048, 1, vectors]; vectors = 1, 4, 16, 32. Each vector is processed in the same evaluation, not separate calls.
• TP1/2/4/8 output widths: 8192/4096/2048/1024 per branch. Total weight count and full output shape stay fixed.
• Nominal weight bytes/core/branch: 1 MiB, 0.5 MiB, 0.25 MiB, 0.125 MiB. These smaller branches are evaluated together; this is not an independent isolated 0.5 MiB kernel timing.
• Signed INT8 weight storage and activation quantization in MIL. Both variants have shared activation scales; “per-channel” refers to weights.
• One fixed deterministic dense random weight matrix is sliced across branches. Unlike identical constant branches, distinct weights prevent common-subexpression elimination from removing work. The converted MIL retains exactly TP convolution and TP compressed-weight operations.
• Ten warmup evaluations plus 64 timed samples per configuration per round, two rounds in forward/reverse order. Tables use medians of all 128 samples. Full per-round samples are in results.json.
• Output checks against an integer-derived quantized reference and the unsplit graph passed exactly for every case (maximum absolute difference 0). Native output extraction happens after timing. The test input is constant 0.01; this is a performance/correctness smoke test, not model accuracy validation.
• The ANE daemon PID stayed unchanged. Small differences between TP2/TP4/TP8 vary by round; do not treat a few percent as a stable ranking.
• The benchmark does not inspect HWX. Sizes above are logical weight payloads; compiler scheduling, buffer sizes, and physical traffic may differ. No claim is made that the compiler physically runs the graph branches concurrently.
• All timings are native evaluateWithModel elapsed call times. Logical effective TOPS includes the multiply-add count and is not a hardware counter.

Reproduce

Download profile_int8_tp_prefill.py, tp_host.m, and native_utils.py alongside the original gist files. Requires Python 3.9+, NumPy, coremltools 9.0 and clang.

python3.9 profile_int8_tp_prefill.py --out ./tp_prefill_run

Use a fresh output directory. The script compiles 32 configurations, then runs both timing orders and verifies outputs. Results are saved in report.json. No HWX tools are required.

kernel_dma_mib_host.m

// Compact ANE eval_us host for the KernelDMA 1 MiB notch profiler.
// Ordinary compiler MIL only. No custom HWX, no vault, no 0-4 dump.
// Usage: kernel_dma_mib_host <model.mlmodelc> <warm> <timed>
#import <CoreVideo/CoreVideo.h>
#import <Foundation/Foundation.h>
#import <IOSurface/IOSurfaceObjc.h>
#import <objc/runtime.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

@interface _ANEClient : NSObject
+ (id)sharedConnection;
- (bool)compileModel:(id)model options:(id)options qos:(unsigned int)qos error:(id *)error;
- (bool)loadModel:(id)model options:(id)options qos:(unsigned int)qos error:(id *)error;
- (_Bool)evaluateWithModel:(id)model options:(id)options request:(id)request
                       qos:(unsigned int)qos error:(id *)error;
@end

@interface _ANEModel : NSObject
@property(retain, nonatomic) NSDictionary *modelAttributes;
+ (id)modelAtURL:(id)url key:(id)key;
@end

@interface _ANEIOSurfaceObject : NSObject
+ (id)objectWithIOSurface:(IOSurface *)iosurface;
@end

@interface _ANERequest : NSObject
+ (id)requestWithInputs:(id)inputs inputIndices:(id)indices outputs:(id)outputs
          outputIndices:(id)indices2 perfStats:(id)stats procedureIndex:(id)index;
@end

static NSDictionary *opts(void) {
  const char *ident = getenv("ANE_IDENTITY");
  NSString *identity = (ident && ident[0]) ? [NSString stringWithUTF8String:ident]
                                          : @"kernel_dma_mib_probe";
  return @{@"kANEFModelIdentityStrKey" : identity, @"kANEFModelType" : @"kANEFModelMIL"};
}

static IOSurface *surface_from_dict(NSDictionary *dict, NSString *name) {
  NSDictionary *props = @{
    IOSurfacePropertyKeyWidth : dict[@"Width"] ?: @1,
    IOSurfacePropertyKeyHeight : dict[@"Height"] ?: @1,
    IOSurfacePropertyKeyBytesPerElement : @2,
    IOSurfacePropertyKeyPixelFormat : @(kCVPixelFormatType_OneComponent16Half),
    IOSurfacePropertyKeyName : name,
  };
  if (dict[@"Batches"]) {
    NSNumber *tensor_size = @([dict[@"Batches"] intValue] * [dict[@"BatchStride"] intValue]);
    props = @{
      IOSurfacePropertyKeyWidth : tensor_size,
      IOSurfacePropertyKeyHeight : @1,
      IOSurfacePropertyKeyName : name,
      IOSurfacePropertyKeyPixelFormat : @(kCVPixelFormatType_OneComponent16Half),
    };
  }
  return [[IOSurface alloc] initWithProperties:props];
}

static void fill_constant(IOSurface *ios, NSDictionary *d, float value) {
  NSUInteger b = MAX(1, [d[@"Batches"] integerValue]);
  NSUInteger c = MAX(1, [d[@"Channels"] integerValue]);
  NSUInteger h = MAX(1, [d[@"Height"] integerValue]);
  NSUInteger w = MAX(1, [d[@"Width"] integerValue]);
  NSUInteger rs = [d[@"RowStride"] unsignedIntegerValue] ?: w * 2;
  NSUInteger ps = [d[@"PlaneStride"] unsignedIntegerValue] ?: h * rs;
  NSUInteger bs = [d[@"BatchStride"] unsignedIntegerValue] ?: c * ps;
  IOSurfaceRef ref = (__bridge IOSurfaceRef)ios;
  IOSurfaceLock(ref, 0, NULL);
  uint8_t *base = IOSurfaceGetBaseAddress(ref);
  memset(base, 0, IOSurfaceGetAllocSize(ref));
  __fp16 half = (__fp16)value;
  for (NSUInteger nb = 0; nb < b; nb++)
    for (NSUInteger nc = 0; nc < c; nc++)
      for (NSUInteger nh = 0; nh < h; nh++)
        for (NSUInteger nw = 0; nw < w; nw++)
          memcpy(base + nb * bs + nc * ps + nh * rs + nw * 2, &half, 2);
  IOSurfaceUnlock(ref, 0, NULL);
}

int main(int argc, char **argv) {
  if (argc != 4) {
    fprintf(stderr, "usage: %s model.mlmodelc warm timed\n", argv[0]);
    return 2;
  }
  int warm = atoi(argv[2]), timed = atoi(argv[3]);
  if (warm < 0 || timed < 1 || timed > 128) return 2;
  @autoreleasepool {
    NSError *err = nil;
    NSString *path = [NSString stringWithUTF8String:argv[1]];
    id client = [_ANEClient sharedConnection];
    id model = [_ANEModel modelAtURL:[NSURL fileURLWithPath:path] key:@"ANE_model"];
    if (![client compileModel:model options:opts() qos:21 error:&err] || err) {
      NSLog(@"compile failed %@", err);
      return 3;
    }
    err = nil;
    if (![client loadModel:model options:opts() qos:21 error:&err] || err) {
      NSLog(@"load failed %@", err);
      return 4;
    }
    NSDictionary *attr = [model modelAttributes];
    NSArray *procedures = attr[@"ANEFModelDescription"][@"ANEFModelProcedures"];
    NSArray *networks = attr[@"NetworkStatusList"];
    if (procedures.count != 1 || networks.count != 1) return 5;
    NSArray *inputs = networks[0][@"LiveInputList"];
    NSArray *outputs = networks[0][@"LiveOutputList"];
    if (inputs.count != 1 || outputs.count != 1) return 5;
    IOSurface *in = surface_from_dict(inputs[0], inputs[0][@"Name"]);
    IOSurface *out = surface_from_dict(outputs[0], outputs[0][@"Name"]);
    if (!in || !out) return 6;
    fill_constant(in, inputs[0], 0.01f);
    id request = [_ANERequest
        requestWithInputs:@[ [_ANEIOSurfaceObject objectWithIOSurface:in] ]
             inputIndices:procedures[0][@"ANEFModelInputSymbolIndexArray"]
                  outputs:@[ [_ANEIOSurfaceObject objectWithIOSurface:out] ]
            outputIndices:procedures[0][@"ANEFModelOutputSymbolIndexArray"]
                perfStats:@[]
          procedureIndex:@0];
    NSMutableArray *samples = [NSMutableArray array];
    int total = warm + timed;
    for (int i = 0; i < total; i++) {
      err = nil;
      uint64_t start = clock_gettime_nsec_np(CLOCK_UPTIME_RAW);
      BOOL ok = [client evaluateWithModel:model
                                  options:@{@"kANEFDisableIOFencesUseSharedEventsKey" : @0}
                                  request:request
                                      qos:0x21
                                    error:&err];
      double us = (clock_gettime_nsec_np(CLOCK_UPTIME_RAW) - start) / 1000.0;
      if (!ok || err) {
        NSLog(@"evaluate failed %@", err);
        return 8;
      }
      if (i >= warm) [samples addObject:@(us)];
    }
    NSMutableString *json = [NSMutableString stringWithString:@"{\"eval_us\":["];
    for (NSUInteger i = 0; i < samples.count; i++) {
      if (i) [json appendString:@","];
      [json appendFormat:@"%.3f", [samples[i] doubleValue]];
    }
    [json appendString:@"]}"];
    printf("DMA_JSON %s\n", json.UTF8String);
    return 0;
  }
}

native_utils.py

"""Native ANE host utilities from the original KernelDMA gist."""
import os, sys, platform, subprocess, json, statistics
from pathlib import Path
HOST_M = Path(__file__).with_name('kernel_dma_mib_host.m')

def sysctl(name):
    run = subprocess.run(['sysctl', '-n', name], capture_output=True, text=True, timeout=5)
    return run.stdout.strip() if run.returncode == 0 else ''

def guess_chip(hw_model, brand):
    brand_l = (brand or '').lower()
    model = hw_model or ''
    text = f'{brand_l} {model.lower()}'
    for name in ('m5', 'm4', 'm3', 'm2', 'm1'):
        if name in text:
            extra = ''
            for suf in ('ultra', 'max', 'pro'):
                if suf in brand_l:
                    extra = ' ' + suf.title()
                    break
            return ('Apple ' + name.upper() + extra).strip()
    if hw_model.startswith('Mac17'):
        return 'Apple M5-class (hw.model)'
    if hw_model.startswith('Mac16'):
        return 'Apple M4-class (hw.model)'
    if hw_model.startswith('Mac15'):
        return 'Apple M3-class (hw.model)'
    if hw_model.startswith('Mac14'):
        return 'Apple M2-class (hw.model)'
    if hw_model.startswith('Mac13') or hw_model.startswith('MacBookPro18'):
        return 'Apple M1-class (hw.model)'
    return 'unknown'

def guess_ane_cores(chip_name):
    text = (chip_name or '').lower()
    if 'ultra' in text:
        return 32
    return 16

def chip_info(label=None):
    brand = sysctl('machdep.cpu.brand_string')
    model = sysctl('hw.model')
    guessed = guess_chip(model, brand)
    cores = guess_ane_cores(guessed)
    return dict(
        hw_model=model,
        brand=brand,
        platform=platform.platform(),
        guessed_chip=guessed,
        guessed_ane_cores=cores,
        dual_cluster=cores >= 32,
        label=label or guessed,
        python=sys.version.split()[0],
    )

def build_host(dest):
    dest = Path(dest)
    run = subprocess.run(
        ['clang', '-O2', '-fobjc-arc', str(HOST_M), '-o', str(dest),
         '-framework', 'Foundation', '-framework', 'CoreVideo',
         '-framework', 'IOSurface',
         '-F/System/Library/PrivateFrameworks', '-framework', 'AppleNeuralEngine'],
        capture_output=True, text=True, timeout=30)
    if run.returncode:
        raise RuntimeError(run.stderr[-4000:])
    return dest

def time_mlmodelc(host, mlc, warm, timed):
    env = dict(os.environ, ANE_IDENTITY='kernel_dma_mib_probe')
    for name in ('ANE_INMEM_HWX', 'ANE_INPUT_SPLIT'):
        env.pop(name, None)
    run = subprocess.run(
        [str(host), str(mlc), str(warm), str(timed)],
        capture_output=True, text=True, timeout=120, env=env)
    if run.returncode:
        raise RuntimeError((run.stderr or run.stdout)[-4000:])
    line = [row for row in run.stdout.splitlines() if row.startswith('DMA_JSON ')]
    if not line:
        raise RuntimeError('native host printed no DMA_JSON')
    payload = json.loads(line[-1][9:])
    samples = [float(x) for x in payload['eval_us']]
    return dict(eval_us_samples=samples, median_eval_us=statistics.median(samples))

profile_int8_dma.py

"""W8A8 ANE KernelDMA boundary profiler. Python 3.9+, coremltools 9.0."""
import argparse, json, pathlib, subprocess, sys
import numpy as np
import coremltools as ct
from coremltools.converters.mil import Builder as mb
from coremltools.converters.mil.mil import types
from coremltools.models.utils import compile_model
import native_utils as base

def build(cin,cout,mode,dest, per_tensor=False):
    dest.mkdir(parents=True)
    # Same constant weight as original, represented explicitly as signed INT8.
    w=np.full((cout,cin,1,1),127,dtype=np.int8)
    scale=np.full((1 if per_tensor else cout,1,1,1),np.float16(0.001/127),dtype=np.float16)
    @mb.program(input_specs=[mb.TensorSpec(shape=(1,cin,1,1),dtype=types.fp16)],opset_version=ct.target.iOS18)
    def prog(x):
        weight=mb.constexpr_blockwise_shift_scale(data=w,scale=scale,name='int8_weights')
        if mode=='w8a8':
            x=mb.quantize(input=x,scale=np.float16(0.001),output_dtype='int8')
            x=mb.dequantize(input=x,scale=np.float16(0.001))
        y=mb.conv(x=x,weight=weight,pad_type='valid',strides=[1,1])
        if mode=='w8a8':
            y=mb.quantize(input=y,scale=np.float16(0.001),output_dtype='int8')
            y=mb.dequantize(input=y,scale=np.float16(0.001))
        return mb.identity(x=y,name='y')
    model=ct.convert(prog,convert_to='mlprogram',compute_precision=ct.precision.FLOAT16,minimum_deployment_target=ct.target.iOS18,skip_model_load=True)
    ops=list(model._mil_program.functions['main'].operations)
    compressed=[o for o in ops if o.op_type=='constexpr_blockwise_shift_scale']
    assert len(compressed)==1
    data=compressed[0].data.val
    assert data.dtype==np.int8 and data.nbytes==cin*cout
    audit={'weight_dtype':str(data.dtype),'weight_shape':list(data.shape),'weight_bytes':data.nbytes,'weight_bytes_per_core_nominal':data.nbytes//16,'scale_bytes':scale.nbytes,'ops':[o.op_type for o in ops]}
    if mode=='w8a8':
        assert 'quantize' in audit['ops'] and 'dequantize' in audit['ops']
    (dest/'mil.txt').write_text(str(model._mil_program))
    model.save(str(dest/'model.mlpackage'))
    compile_model(str(dest/'model.mlpackage'),destination_path=str(dest/'model.mlmodelc'))
    audit['compiled_files']={str(p.relative_to(dest/'model.mlmodelc')):p.stat().st_size for p in (dest/'model.mlmodelc').rglob('*') if p.is_file()}
    return audit

def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument('--out', type=pathlib.Path, required=True, help='Fresh output directory')
    ap.add_argument('--scales', choices=['tensor','channel','both'], default='both')
    ap.add_argument('--cout', type=int, choices=[4096,8192], default=8192)
    ap.add_argument('--warm', type=int, default=10)
    ap.add_argument('--timed', type=int, default=64)
    ap.add_argument('--reverse', action='store_true')
    args=ap.parse_args()
    if args.warm < 0 or not 1 <= args.timed <= 128: ap.error('warm >= 0; timed between 1 and 128')
    chip=base.chip_info()
    if chip['guessed_ane_cores'] != 16: ap.error('This reproducer uses the single-cluster 16-core lattice; Ultra needs other dimensions.')
    out=args.out.resolve()
    out.mkdir(parents=True,exist_ok=False)
    host=base.build_host(out/'kernel_dma_mib_host')
    cins=[2016,2048] if args.cout==8192 else [4032,4096]
    if args.reverse: cins.reverse()
    layouts=['tensor','channel'] if args.scales=='both' else [args.scales]
    report={'chip':chip,'coremltools':ct.__version__,'numpy':np.__version__,'metric':'native_evaluate_call_us','warm':args.warm,'timed':args.timed,'aned_before':subprocess.getoutput('pgrep -x aned'),'points':[],'summaries':[],'status':'running'}
    def save(): (out/'report.json').write_text(json.dumps(report,indent=2)+'\n')
    try:
        for layout in layouts:
            rows=[]
            for cin in cins:
                dest=out/f'{layout}_{args.cout}x{cin}'
                print('COMPILE',layout,args.cout,cin,flush=True)
                audit=build(cin,args.cout,'w8a8',dest,per_tensor=layout=='tensor')
                row={'scales':layout,'cin':cin,'cout':args.cout,'mil_audit':audit}
                row.update(base.time_mlmodelc(host,dest/'model.mlmodelc',args.warm,args.timed))
                row['gbps_weight_payload']=args.cout*cin/row['median_eval_us']/1000
                rows.append(row);report['points'].append(row);save()
                print('TIME',layout,cin,row['median_eval_us'],row['gbps_weight_payload'],flush=True)
            off,on=sorted(rows,key=lambda r:r['cin'])
            ratio=on['median_eval_us']/off['median_eval_us']
            report['summaries'].append({'scales':layout,'ratio_on_over_off':ratio,'class': 'host_dominated_invalid' if on['median_eval_us']>=5000 else 'slowdown' if ratio>=1.3 else 'no_slowdown_on_this_tile'})
        report['aned_after']=subprocess.getoutput('pgrep -x aned')
        report['daemon_pid_unchanged']=report['aned_before']==report['aned_after']
        report['status']='complete' if report['daemon_pid_unchanged'] else 'invalid_daemon_changed'
        save();print(json.dumps(report['summaries'],indent=2))
        return 0 if report['status']=='complete' else 1
    except Exception as exc:
        report.update(status='failed',error=str(exc));save();raise

if __name__=='__main__':sys.exit(main())

profile_int8_tp_prefill.py

"""Compare complete W8A8 2048->8192 projections with output-channel splits."""
import argparse,json,os,pathlib,subprocess,sys,statistics,time
import numpy as np
import coremltools as ct
from coremltools.converters.mil import Builder as mb
from coremltools.converters.mil.mil import types
from coremltools.models.utils import compile_model
HERE=pathlib.Path(__file__).resolve().parent
import native_utils as native

CIN,COUT=2048,8192
WS=np.float16(0.001/127)
AS=np.float16(0.001)

def build(dest,weights,vectors,tp,layout):
    dest.mkdir(parents=True)
    chunks=np.split(weights,tp,axis=0)
    @mb.program(input_specs=[mb.TensorSpec(shape=(1,CIN,1,vectors),dtype=types.fp16)],opset_version=ct.target.iOS18)
    def prog(x):
        q=mb.quantize(input=x,scale=AS,output_dtype='int8')
        a=mb.dequantize(input=q,scale=AS)
        outputs=[]
        for i,w in enumerate(chunks):
            scale=np.full((1 if layout=='tensor' else COUT//tp,1,1,1),WS,dtype=np.float16)
            weight=mb.constexpr_blockwise_shift_scale(data=w,scale=scale,name=f'w{i}')
            y=mb.conv(x=a,weight=weight,pad_type='valid',strides=[1,1],name=f'conv{i}')
            y=mb.quantize(input=y,scale=AS,output_dtype='int8')
            y=mb.dequantize(input=y,scale=AS)
            outputs.append(y)
        y=outputs[0] if tp==1 else mb.concat(values=outputs,axis=1)
        return mb.identity(x=y,name='y')
    model=ct.convert(prog,convert_to='mlprogram',compute_precision=ct.precision.FLOAT16,minimum_deployment_target=ct.target.iOS18,skip_model_load=True)
    ops=list(model._mil_program.functions['main'].operations)
    assert sum(o.op_type=='conv' for o in ops)==tp
    assert sum(o.op_type=='constexpr_blockwise_shift_scale' for o in ops)==tp
    (dest/'mil.txt').write_text(str(model._mil_program))
    model.save(str(dest/'model.mlpackage'))
    compile_model(str(dest/'model.mlpackage'),destination_path=str(dest/'model.mlmodelc'))
    return {'conv_count':tp,'weight_bytes':weights.nbytes,'nominal_weight_bytes_per_core_per_branch':CIN*(COUT//tp)//16}

def evaluate(host,dest,warm,timed):
    env=dict(os.environ,ANE_OUTPUT_PATH=str(dest/'output.fp16'),ANE_IDENTITY='int8_tp_prefill')
    run=subprocess.run([str(host),str(dest/'model.mlmodelc'),str(warm),str(timed)],capture_output=True,text=True,env=env,timeout=120)
    (dest/'native.log').write_text(run.stdout+run.stderr)
    if run.returncode:raise RuntimeError('native return %d: %s'%(run.returncode,run.stderr[-1500:]))
    data=json.loads(next(l[9:] for l in run.stdout.splitlines() if l.startswith('DMA_JSON ')))
    return data['eval_us'],np.fromfile(dest/'output.fp16',dtype=np.float16)

def main():
    ap=argparse.ArgumentParser();ap.add_argument('--out',type=pathlib.Path,required=True)
    ap.add_argument('--warm',type=int,default=10);ap.add_argument('--timed',type=int,default=64)
    ap.add_argument('--rounds',type=int,default=2)
    args=ap.parse_args()
    if args.warm<0 or not 1<=args.timed<=128 or args.rounds<1: ap.error('warm >= 0, timed 1..128, rounds >= 1 required')
    if native.chip_info()['guessed_ane_cores']!=16: ap.error('This shape uses the 16-core lattice; Ultra is not supported')
    out=args.out.resolve();out.mkdir(parents=True,exist_ok=False)
    native.HOST_M=HERE/'tp_host.m';host=native.build_host(out/'tp_host')
    # Identical full weight matrix for all layouts/splits. Distinct branches prevent CSE.
    weights=np.random.default_rng(20260912).integers(-127,128,size=(COUT,CIN,1,1),dtype=np.int16).astype(np.int8)
    expected=np.clip(np.rint(weights.astype(np.int32).sum(axis=(1,2,3))*10*float(WS)),-128,127).astype(np.float32)*float(AS)
    report={'chip':native.chip_info(),'coremltools':ct.__version__,'cin':CIN,'cout':COUT,'vectors':[1,4,16,32],'splits':[1,2,4,8],'seed':20260912,'warm':args.warm,'timed':args.timed,'rounds':args.rounds,'status':'running','aned_before':subprocess.getoutput('pgrep -x aned'),'points':[]}
    def save(): (out/'report.json').write_text(json.dumps(report,indent=2)+'\n')
    save();cases=[];references={}
    try:
        for layout in ['tensor','channel']:
            for vectors in [1,4,16,32]:
                for tp in [1,2,4,8]:
                    dest=out/f'{layout}_v{vectors}_tp{tp}'
                    print('BUILD',dest.name,flush=True)
                    audit=build(dest,weights,vectors,tp,layout)
                    row={'layout':layout,'vectors':vectors,'tp':tp,'audit':audit,'round_samples_us':[]}
                    cases.append((row,dest));report['points'].append(row);save()
        for rnd in range(args.rounds):
            ordered=cases if rnd%2==0 else list(reversed(cases))
            for row,dest in ordered:
                samples,output=evaluate(host,dest,args.warm,args.timed)
                assert output.size==COUT*row['vectors'] and np.isfinite(output).all()
                values=output.astype(np.float32).reshape(COUT,row['vectors'])
                err=float(np.max(np.abs(values-expected[:,None])))
                if err>float(AS)*1.1:raise RuntimeError(f'Output reference mismatch {dest.name}: {err}')
                key=(row['layout'],row['vectors'])
                if row['tp']==1:references[key]=values
                if key in references:
                    delta=float(np.max(np.abs(values-references[key])))
                    if delta>float(AS)*1.1:raise RuntimeError(f'Split mismatch {dest.name}: {delta}')
                    row['max_abs_vs_unsplit']=delta
                row['max_abs_vs_reference']=err
                row['round_samples_us'].append(samples)
                row['round_medians_us']=[statistics.median(s) for s in row['round_samples_us']]
                row['median_us']=statistics.median(sum(row['round_samples_us'],[]))
                row['us_per_vector']=row['median_us']/row['vectors']
                row['effective_tops']=2*CIN*COUT*row['vectors']/row['median_us']/1e6
                save();print('TIME',rnd,dest.name,row['median_us'],'error',err,flush=True)
        for row,_ in cases:
            baseline=next(r['median_us'] for r,_ in cases if r['tp']==1 and r['vectors']==row['vectors'] and r['layout']==row['layout'])
            row['speedup_vs_tp1']=baseline/row['median_us']
        report['aned_after']=subprocess.getoutput('pgrep -x aned')
        assert report['aned_before']==report['aned_after'],'ANE daemon changed'
        report['status']='complete';save()
    except Exception as e:
        report.update(status='failed',error=str(e));save();raise
if __name__=='__main__':main()

tp_host.m

// Compact ANE eval_us host for the KernelDMA 1 MiB notch profiler.
// Ordinary compiler MIL only. No custom HWX, no vault, no 0-4 dump.
// Usage: kernel_dma_mib_host <model.mlmodelc> <warm> <timed>
#import <CoreVideo/CoreVideo.h>
#import <Foundation/Foundation.h>
#import <IOSurface/IOSurfaceObjc.h>
#import <objc/runtime.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

@interface _ANEClient : NSObject
+ (id)sharedConnection;
- (bool)compileModel:(id)model options:(id)options qos:(unsigned int)qos error:(id *)error;
- (bool)loadModel:(id)model options:(id)options qos:(unsigned int)qos error:(id *)error;
- (_Bool)evaluateWithModel:(id)model options:(id)options request:(id)request
                       qos:(unsigned int)qos error:(id *)error;
@end

@interface _ANEModel : NSObject
@property(retain, nonatomic) NSDictionary *modelAttributes;
+ (id)modelAtURL:(id)url key:(id)key;
@end

@interface _ANEIOSurfaceObject : NSObject
+ (id)objectWithIOSurface:(IOSurface *)iosurface;
@end

@interface _ANERequest : NSObject
+ (id)requestWithInputs:(id)inputs inputIndices:(id)indices outputs:(id)outputs
          outputIndices:(id)indices2 perfStats:(id)stats procedureIndex:(id)index;
@end

static NSDictionary *opts(void) {
  const char *ident = getenv("ANE_IDENTITY");
  NSString *identity = (ident && ident[0]) ? [NSString stringWithUTF8String:ident]
                                          : @"kernel_dma_mib_probe";
  return @{@"kANEFModelIdentityStrKey" : identity, @"kANEFModelType" : @"kANEFModelMIL"};
}

static IOSurface *surface_from_dict(NSDictionary *dict, NSString *name) {
  NSDictionary *props = @{
    IOSurfacePropertyKeyWidth : dict[@"Width"] ?: @1,
    IOSurfacePropertyKeyHeight : dict[@"Height"] ?: @1,
    IOSurfacePropertyKeyBytesPerElement : @2,
    IOSurfacePropertyKeyPixelFormat : @(kCVPixelFormatType_OneComponent16Half),
    IOSurfacePropertyKeyName : name,
  };
  if (dict[@"Batches"]) {
    NSNumber *tensor_size = @([dict[@"Batches"] intValue] * [dict[@"BatchStride"] intValue]);
    props = @{
      IOSurfacePropertyKeyWidth : tensor_size,
      IOSurfacePropertyKeyHeight : @1,
      IOSurfacePropertyKeyName : name,
      IOSurfacePropertyKeyPixelFormat : @(kCVPixelFormatType_OneComponent16Half),
    };
  }
  return [[IOSurface alloc] initWithProperties:props];
}

static void fill_constant(IOSurface *ios, NSDictionary *d, float value) {
  NSUInteger b = MAX(1, [d[@"Batches"] integerValue]);
  NSUInteger c = MAX(1, [d[@"Channels"] integerValue]);
  NSUInteger h = MAX(1, [d[@"Height"] integerValue]);
  NSUInteger w = MAX(1, [d[@"Width"] integerValue]);
  NSUInteger rs = [d[@"RowStride"] unsignedIntegerValue] ?: w * 2;
  NSUInteger ps = [d[@"PlaneStride"] unsignedIntegerValue] ?: h * rs;
  NSUInteger bs = [d[@"BatchStride"] unsignedIntegerValue] ?: c * ps;
  IOSurfaceRef ref = (__bridge IOSurfaceRef)ios;
  IOSurfaceLock(ref, 0, NULL);
  uint8_t *base = IOSurfaceGetBaseAddress(ref);
  memset(base, 0, IOSurfaceGetAllocSize(ref));
  __fp16 half = (__fp16)value;
  for (NSUInteger nb = 0; nb < b; nb++)
    for (NSUInteger nc = 0; nc < c; nc++)
      for (NSUInteger nh = 0; nh < h; nh++)
        for (NSUInteger nw = 0; nw < w; nw++)
          memcpy(base + nb * bs + nc * ps + nh * rs + nw * 2, &half, 2);
  IOSurfaceUnlock(ref, 0, NULL);
}

int main(int argc, char **argv) {
  if (argc != 4) {
    fprintf(stderr, "usage: %s model.mlmodelc warm timed\n", argv[0]);
    return 2;
  }
  int warm = atoi(argv[2]), timed = atoi(argv[3]);
  if (warm < 0 || timed < 1 || timed > 128) return 2;
  @autoreleasepool {
    NSError *err = nil;
    NSString *path = [NSString stringWithUTF8String:argv[1]];
    id client = [_ANEClient sharedConnection];
    id model = [_ANEModel modelAtURL:[NSURL fileURLWithPath:path] key:@"ANE_model"];
    if (![client compileModel:model options:opts() qos:21 error:&err] || err) {
      NSLog(@"compile failed %@", err);
      return 3;
    }
    err = nil;
    if (![client loadModel:model options:opts() qos:21 error:&err] || err) {
      NSLog(@"load failed %@", err);
      return 4;
    }
    NSDictionary *attr = [model modelAttributes];
    NSArray *procedures = attr[@"ANEFModelDescription"][@"ANEFModelProcedures"];
    NSArray *networks = attr[@"NetworkStatusList"];
    if (procedures.count != 1 || networks.count != 1) return 5;
    NSArray *inputs = networks[0][@"LiveInputList"];
    NSArray *outputs = networks[0][@"LiveOutputList"];
    if (inputs.count != 1 || outputs.count != 1) return 5;
    IOSurface *in = surface_from_dict(inputs[0], inputs[0][@"Name"]);
    IOSurface *out = surface_from_dict(outputs[0], outputs[0][@"Name"]);
    if (!in || !out) return 6;
    fill_constant(in, inputs[0], 0.01f);
    id request = [_ANERequest
        requestWithInputs:@[ [_ANEIOSurfaceObject objectWithIOSurface:in] ]
             inputIndices:procedures[0][@"ANEFModelInputSymbolIndexArray"]
                  outputs:@[ [_ANEIOSurfaceObject objectWithIOSurface:out] ]
            outputIndices:procedures[0][@"ANEFModelOutputSymbolIndexArray"]
                perfStats:@[]
          procedureIndex:@0];
    NSMutableArray *samples = [NSMutableArray array];
    int total = warm + timed;
    for (int i = 0; i < total; i++) {
      err = nil;
      uint64_t start = clock_gettime_nsec_np(CLOCK_UPTIME_RAW);
      BOOL ok = [client evaluateWithModel:model
                                  options:@{@"kANEFDisableIOFencesUseSharedEventsKey" : @0}
                                  request:request
                                      qos:0x21
                                    error:&err];
      double us = (clock_gettime_nsec_np(CLOCK_UPTIME_RAW) - start) / 1000.0;
      if (!ok || err) {
        NSLog(@"evaluate failed %@", err);
        return 8;
      }
      if (i >= warm) [samples addObject:@(us)];
    }
    const char *outputPath = getenv("ANE_OUTPUT_PATH");
    if (outputPath) {
      NSDictionary *d = outputs[0];
      NSUInteger B = MAX(1, [d[@"Batches"] integerValue]);
      NSUInteger C = MAX(1, [d[@"Channels"] integerValue]);
      NSUInteger H = MAX(1, [d[@"Height"] integerValue]);
      NSUInteger W = MAX(1, [d[@"Width"] integerValue]);
      NSUInteger rs = [d[@"RowStride"] unsignedIntegerValue] ?: W * 2;
      NSUInteger ps = [d[@"PlaneStride"] unsignedIntegerValue] ?: H * rs;
      NSUInteger bs = [d[@"BatchStride"] unsignedIntegerValue] ?: C * ps;
      IOSurfaceRef ref = (__bridge IOSurfaceRef)out;
      IOSurfaceLock(ref, kIOSurfaceLockReadOnly, NULL);
      const uint8_t *ptr = IOSurfaceGetBaseAddress(ref);
      FILE *fp = fopen(outputPath, "wb");
      if (!fp) return 9;
      for (NSUInteger b=0; b<B; b++)
        for (NSUInteger c=0; c<C; c++)
          for (NSUInteger h=0; h<H; h++)
            for (NSUInteger w=0; w<W; w++)
              if (fwrite(ptr+b*bs+c*ps+h*rs+w*2, 2, 1, fp)!=1) return 9;
      fclose(fp);
      IOSurfaceUnlock(ref, kIOSurfaceLockReadOnly, NULL);
    }
    NSMutableString *json = [NSMutableString stringWithString:@"{\"eval_us\":["];
    for (NSUInteger i = 0; i < samples.count; i++) {
      if (i) [json appendString:@","];
      [json appendFormat:@"%.3f", [samples[i] doubleValue]];
    }
    [json appendString:@"]}"];
    printf("DMA_JSON %s\n", json.UTF8String);
    return 0;
  }
}

tp_prefill_results.csv