smellslikeml

From surveying gaps to shipping the anchor

An earlier writeup (Findings from running remyxai-cli explore across 6 production repos) walked through what happens when you dispatch recent arxiv papers as draft integrations against a set of production repos: each paper hits the ranker, preflight identifies the extension point the paper needs, and Outrider opens an Issue naming what's missing. The by-product is a per-repo gap analysis — a catalog of the extension points those repos lack, and the papers currently blocked on them.

smellslikeml

Findings from running remyxai-cli explore across 6 production repos

Recent AI research lands in existing codebases through specific extension points — modules, callbacks, or data-structure fields where a new method can plug in. Which extension points a repo provides determines which methods can be tried against it without a rewrite. We ran an agentic method-search loop that dispatches recent arxiv papers as draft integrations against 6 production repos; the by-product across 36 cycles was a per-repo inventory of the specific extension points those repos are missing.

The dispatch mode

Packaged as a CLI subcommand in remyxai-cli #46:

remyxai outrider explore --repo owner/name \

smellslikeml

GLM Tries, Opus Triages: Behavioral Differences in Research-to-Code Agents

A controlled comparison across 19 paired runs spanning 19 repository forks — 38 individual workflow executions total — running an identical paper-implementation pipeline (remyxai/outrider — Claude Code under the hood, with glm-5.2 routed at z.ai's Coding Plan endpoint vs default Opus). The pipeline ran in two modes that probe different parts of the workflow:

• Selection-pass mode (n=9): no pin; each provider freely selects its own paper from the candidate pool. Exercises the full pipeline including selection + verification gates.
• Pin-method mode (n=10): same paper pinned on each fork, both providers run their full chain on identical input. Isolates implementation-side behavior on a forced pick.

The aggregate verdict comes from the n=19 union; the two mode-specific breakdowns below show where the difference comes from.

Reproducing this

smellslikeml

import torch
from fastapi import FastAPI, Request
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "name of model"
prompt_template = """ ### Input: {}  ### Response: {}"""
stop_token_ids = [0]

app = FastAPI()

smellslikeml

import asyncio
import nats
import uuid


async def aggregate_inferences(nats_url, request_subject, data, timeout=10):
    nc = await nats.connect(nats_url)

    responses = []

smellslikeml

# Launch nats-server
# wget https://huggingface.co/remyxai/stablelm-zephyr-3B_localmentor/resolve/main/ggml-model-q4_0.gguf -o stablelm-localmentor_2.gguf
import nats
import asyncio
from llama_cpp import Llama


async def llm_runner(nats_url, model_path, subject):
    nc = await nats.connect(nats_url)
    llm = Llama(model_path)

smellslikeml

#!/usr/bin/env python
# coding=utf-8
# Copyright 2023 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#

smellslikeml

#!/bin/bash

# Ensure a list of clips is provided
if [ $# -eq 0 ]; then
    echo "Please provide a list of clips to process and concatenate."
    exit 1
fi

# Directory to store processed clips
mkdir -p processed_clips

smellslikeml

model:
  arch: video_llama
  model_type: pretrain_vicuna
  freeze_vit: True
  freeze_qformer: True
  max_txt_len: 160
  end_sym: "###"
  low_resource: True

  frozen_llama_proj: False

smellslikeml

import os
import time

import cv2
import hashlib
import numpy as np
from PIL import Image
from absl import logging
import tritonclient.http