smellslikeml smellslikeml

Remyx contributions to `huggingface/peft`

Three parameter-efficient fine-tuning methods surfaced via Outrider, drafted through the smellslikeml/peft fork, and shepherded into huggingface/peft — the fine-tuning ecosystem's hub. This gist tracks what's landed, what's in review, what's pending on external coordination, and how the shape of these contributions compares against the year's other merged tuner PRs.

Why `huggingface/peft` is where the leverage lives

Per Appendix G Table 15 of Scaling DoRA (arXiv:2603.22276), five of six DoRA-supporting fine-tuning frameworks route their implementation through PEFT:

DoRA Factored Kernel — Algorithmic Analysis

Composite illustration for smellslikeml/peft#18 + remyxai/dora-factored-kernel.

Compares the dense DoRA norm-computation path used by HuggingFace PEFT + 5 other major frameworks (torchtune, Unsloth, SWIFT, LLaMA-Factory, Axolotl per arXiv:2603.22276 Appendix G Table 15) against the factored decomposition — 38× to 634× per-module memory reduction across d_in ∈ {1K…16K}, r=64, bf16.

From surveying gaps to shipping the anchor

An earlier writeup (Findings from running remyxai-cli explore across 6 production repos) walked through what happens when you dispatch recent arxiv papers as draft integrations against a set of production repos: each paper hits the ranker, preflight identifies the extension point the paper needs, and Outrider opens an Issue naming what's missing. The by-product is a per-repo gap analysis — a catalog of the extension points those repos lack, and the papers currently blocked on them.

Findings from running remyxai-cli explore across 6 production repos

Recent AI research lands in existing codebases through specific extension points — modules, callbacks, or data-structure fields where a new method can plug in. Which extension points a repo provides determines which methods can be tried against it without a rewrite. We ran an agentic method-search loop that dispatches recent arxiv papers as draft integrations against 6 production repos; the by-product across 36 cycles was a per-repo inventory of the specific extension points those repos are missing.

The dispatch mode

Packaged as a CLI subcommand in remyxai-cli #46:

remyxai outrider explore --repo owner/name \

GLM Tries, Opus Triages: Behavioral Differences in Research-to-Code Agents

A controlled comparison across 19 paired runs spanning 19 repository forks — 38 individual workflow executions total — running an identical paper-implementation pipeline (remyxai/outrider — Claude Code under the hood, with glm-5.2 routed at z.ai's Coding Plan endpoint vs default Opus). The pipeline ran in two modes that probe different parts of the workflow:

Selection-pass mode (n=9): no pin; each provider freely selects its own paper from the candidate pool. Exercises the full pipeline including selection + verification gates.
Pin-method mode (n=10): same paper pinned on each fork, both providers run their full chain on identical input. Isolates implementation-side behavior on a forced pick.

The aggregate verdict comes from the n=19 union; the two mode-specific breakdowns below show where the difference comes from.

	import torch
	from fastapi import FastAPI, Request
	from pydantic import BaseModel
	from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

	model_name = "name of model"
	prompt_template = """ ### Input: {} ### Response: {}"""
	stop_token_ids = [0]

	app = FastAPI()

	import asyncio
	import nats
	import uuid


	async def aggregate_inferences(nats_url, request_subject, data, timeout=10):
	nc = await nats.connect(nats_url)

	responses = []

	# Launch nats-server
	# wget https://huggingface.co/remyxai/stablelm-zephyr-3B_localmentor/resolve/main/ggml-model-q4_0.gguf -o stablelm-localmentor_2.gguf
	import nats
	import asyncio
	from llama_cpp import Llama


	async def llm_runner(nats_url, model_path, subject):
	nc = await nats.connect(nats_url)
	llm = Llama(model_path)

	#!/usr/bin/env python
	# coding=utf-8
	# Copyright 2023 The HuggingFace Inc. team. All rights reserved.
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#

	#!/bin/bash

	# Ensure a list of clips is provided
	if [ $# -eq 0 ]; then
	echo "Please provide a list of clips to process and concatenate."
	exit 1
	fi

	# Directory to store processed clips
	mkdir -p processed_clips