HSILA

ChEBI-guided role-based retrieval

We sought a chemistry-interpretable view of the behaviour learned during ChEmbed training. Specifically, we asked whether a natural-language query about a chemical role would retrieve PubChem descriptions of compounds that ChEBI directly assigns to that role more effectively. PubChem supplied the descriptions, while ChEBI supplied structured role annotations independently of the embedding models.

Retaining one 20 to 500 word description per PubChem CID and removing exact duplicate texts yielded 44,887 retrieval candidates. We linked these PubChem records to ChEBI using explicit database links or exact full-InChI matches. If several usable PubChem CIDs matched the same ChEBI entity, we retained one CID to avoid duplication. This produced 42,068 ChEBI-to-PubChem mappings involving 40,656 PubChem CIDs. A role-based group contains the distinct mapped CIDs sharing the same direct ChEBI has_role annotation and does not imply structural relatedness.

Of 1,313 roles with at le

HSILA

ChEBI-guided chemical-role retrieval experiment

Aim

We wanted an intuitive way to examine what changed after ChEmbed training. The specific question was whether a natural-language query about a chemical role would retrieve PubChem descriptions of compounds that ChEBI assigns to that role more effectively.

Starting data: PubChem names and descriptions

We started from BASF-AI/PubChem-Raw. The fields relevant to this experiment were the PubChem CID, compound title, textual description, and chemical identity information used to connect PubChem records with ChEBI.

HSILA

Short report: Chemical-entity representation analysis

Objective

We investigated whether ChEmbed training produces a more chemically meaningful representation space than the original Nomic model. Our initial idea was that descriptions of chemically related entities, identified using ChEBI relationships, might become more clearly grouped after training. Such a result could provide an intuitive visual explanation of what the model learned.

HSILA

Why ChEmbed improves on ChemRxiv retrieval but not on encyclopedic benchmarks: an embedding-geometry analysis

1. The question

Our evaluation shows a pattern that a reviewer will immediately question:

Benchmark	Text type	Metric	Base model	ChEmbed	Change
ChemRxiv Retrieval (ours)	chemistry literature paragraphs	MRR@10	0.796	0.879	up
ChemTEB encyclopedic retrieval tasks	chemistry-topic Wikipedia-style passages	MRR@10	0.728	0.672	down

HSILA

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.10"
# dependencies = [
#   "datasets==4.8.5",
#   "huggingface-hub==1.17.0",
#   "mteb==2.12.30",
#   "pyarrow==24.0.0",
#   "sentence-transformers==5.5.1",
#   "torch==2.12.0",

HSILA

Report: Literature/Academic Retrieval Tasks

Rubric

Included:

• Scientific papers, academic corpora
• Citations, abstracts from scholarly literature
• Scholarly QA grounded in academic papers

Excluded:

HSILA

{
  "dataset_revision": "5377aa18f309ec440ff6325a4c2cd3362c2cb8d7",
  "task_name": "ChemRxivRetrieval",
  "mteb_version": "2.6.8",
  "scores": {
    "test": [
      {
        "ndcg_at_1": 0.625,
        "ndcg_at_3": 0.71766,
        "ndcg_at_5": 0.74022,

HSILA

\documentclass[tikz,border=2pt]{standalone}
\usepackage{amsmath,bm}
\usepackage{tikz}
% \usetikzlibrary{arrows.meta}  % if you need extra arrowheads
\begin{document}
\begin{tikzpicture}
    \node at (0,1.8) (h) {$\longleftarrow\,$Representation$\,\longrightarrow$};
    \node[draw, circle] at (0,-1) (x) {$\bm{x}$};
    \node[draw, circle] at (-2.5,0) (x1) {$\tilde{\bm{x}}_i$};
    \node[draw, circle] at (2.5,0) (x2) {$\tilde{\bm{x}}_j$};

HSILA

import os
import numpy as np
from typing import Any


import torch
import mteb


from transformers import AutoTokenizer

HSILA

import torch
from torch import nn
import onnx
import onnxruntime

a = torch.tensor([12, 0, 5, 154], dtype=torch.float32)

# a.to(torch.bool) = tensor([True, False, True, True])
# ~a.to(torch.bool) = tensor([False, True, False, False])