We wanted an intuitive way to examine what changed after ChEmbed training. The specific question was whether a natural-language query about a chemical role would retrieve PubChem descriptions of compounds that ChEBI assigns to that role more effectively.
We started from BASF-AI/PubChem-Raw. The fields relevant to this experiment were the PubChem CID, compound title, textual description, and chemical identity information used to connect PubChem records with ChEBI.
The description table contained 408,530 rows representing 332,075 unique PubChem CIDs. A description being present did not automatically make it useful for this analysis, because many records were only a few words long. We retained CIDs with at least one nonempty description between 20 and 500 words, selected one description per CID using a fixed rule, and removed exact duplicate descriptions. This left 44,887 compounds with distinct, usable descriptions.
The models embedded only the description text. Compound titles were retained for identification and for the figures.
PubChem provided the descriptions to embed, but the text alone did not give us a consistent reference for deciding which compounds share a chemical role. ChEBI provides explicit, structured links between chemical entities, chemical classes, and chemical roles. We therefore used ChEBI to define related sets independently of the embedding models.
The first ChEBI step was identity matching, not grouping. We connected a PubChem compound to a ChEBI entity when the PubChem data contained an explicit ChEBI link or when the full InChI matched exactly. After this step, 40,639 PubChem CIDs met all three practical requirements for the analysis:
- a PubChem CID;
- a usable PubChem description; and
- at least one retained match to a ChEBI entity.
A PubChem CID could match more than one ChEBI entity. This mapping step only established that records in the two databases referred to the same chemical entity. It did not yet say how that entity was related to other compounds.
We next used the structured ChEBI records to ask which of the 40,639 mapped compounds participated in a relationship with another mapped compound or belonged to a shared ChEBI family. We audited eight relationship categories:
| Relationship category | What one unit represents | Eligible units |
|---|---|---|
| Conjugate acid/base | One directly linked acid/base pair | 6,212 |
| Functional parent | One compound and its directly linked functional parent | 12,284 |
| Parent hydride | One compound and its directly linked parent hydride | 483 |
| Has part | One directly linked whole/part pair | 1,443 |
| Enantiomer | One directly linked enantiomer pair | 668 |
| Tautomer | One directly linked tautomer pair | 77 |
| Chemical-role family | Compounds linked to the same role by direct has_role records |
517 |
| Chemical-class family | Compounds linked to the same parent class by is_a records |
2,284 |
Here, a relationship unit is either one pair of compounds or one family of compounds, depending on the category. For a pair to be eligible, both compounds had to map back to usable PubChem descriptions. For a role or class family, the included compounds had to satisfy the same mapping and description requirements, and the family had to have between 4 and 30 available members. The ChEBI role or class concept itself was not embedded as a PubChem description.
The eight categories produced 23,968 relationship units covering 32,109 distinct compounds. The other 8,530 mapped compounds did not occur in one of these eligible units. This does not mean that all 32,109 compounds had a chemical-role annotation. It means that each appeared in at least one of the eight audited relationship categories.
The number 32,109 describes the relationship audit, not the final search catalog. Every question in the experiment still searched all 44,887 usable PubChem descriptions. The reported retrieval experiment used only the direct chemical-role families as reference sets. The other seven categories were not treated as correct answers to the role questions.
For the role experiment, a compound became a member of a family when ChEBI contained a direct statement of the form “this compound has this role.” We grouped together the mapped compounds pointing to the same role concept.
Before applying the family-size rule, 1,313 ChEBI role concepts had at least one mapped compound with a usable PubChem description. Of these, 598 roles had only 1 to 3 available compounds, 517 had between 4 and 30, and 198 had more than 30. The primary experiment used all 517 roles in the middle group. The lower limit avoided trying to evaluate a family from only one or two examples, while the upper limit kept very broad roles from dominating the main comparison. The larger families were examined separately as a sensitivity check.
Across the 517 retained role families, there were 5,769 compound-to-role links involving 4,485 unique compounds. These numbers differ because the same compound can have more than one ChEBI role.
For every retained role, we asked the same three forms of question:
- T1:
Which compounds act as [role]? - T2:
Find chemical entities with the role of [role]. - T3:
What chemicals can function as [role]?
This produced 1,551 questions, or three questions for each of the 517 roles. For each model, every question was compared with the complete set of 44,887 PubChem descriptions, which were ordered from the most to the least relevant according to that model.
ChEBI provided the reference list of compounds directly annotated with the queried role. The comparison asked whether those known members appeared earlier in the ranking after ChEmbed training than they did with the original Nomic model. A description without the relevant ChEBI annotation was left unlabeled, not declared chemically wrong, because ChEBI may not list every compound that can genuinely have a role.
The clearest result came from ChEmbed-progressive. Compared with Nomic, the descriptions of ChEBI-listed compounds appeared earlier in the results for 434 of the 517 role families, or about 84 percent. They appeared later for the remaining 83 families. The effect was therefore broad, but not universal. ChEmbed-vanilla showed a smaller improvement.
We summarized retrieval using two complementary measures. Known-member average precision measures how early the ChEBI-listed members appear throughout the complete ranking, with a higher value indicating that more members were placed near the top. Known-member recall at 10 measures the average fraction of a role family's listed members recovered within the first ten results.
| Model | Known-member average precision | Known-member recall at 10 |
|---|---|---|
| Nomic | 0.176 | 20.3% |
| ChEmbed-vanilla | 0.202 | 22.7% |
| ChEmbed-progressive | 0.315 | 33.4% |
The recall-at-10 result gives the most direct reading: the first ten Nomic results recovered about one fifth of the listed members of a typical role family, while the progressive model recovered about one third. The same overall direction appeared across all three question wordings and under alternative mapping and family filters.
In about 88 percent of the compound-to-role links, the PubChem description literally contained the queried ChEBI role name. We repeated the comparison after removing that exact phrase wherever it appeared in the descriptions of known family members. Most of the improvement became smaller. This indicates that the model often benefits from connecting the language of the question with explicit role terminology and its surrounding context. A reduced progressive advantage remained, but this control does not justify claiming that the model inferred chemical roles from molecular structure alone.
To make the aggregate result concrete, we use the ChEBI role EC 3.4.21.6 (coagulation factor Xa) inhibitor as a visual example. The exact T3 query used for ranking was:
What chemicals can function as EC 3.4.21.6 (coagulation factor Xa) inhibitor?
ChEBI linked eight PubChem entities to this role in the eligible corpus. With Nomic, 3 of those 8 entities appeared in the first ten results. With ChEmbed-progressive, 7 of 8 appeared in the first ten. The figure shows the molecular structures and the rank of every listed entity before and after training.
This is an unusually strong example, near the 99th percentile of family improvements, so it illustrates the behavior rather than representing the typical effect. The set also contains edoxaban, edoxaban tosylate, and edoxaban tosylate monohydrate as separate PubChem entities. Whether those forms should be counted separately is one of the points on which we would value chemistry feedback.
The supported conclusion is that ChEmbed training, especially progressive training, made many training-exposed PubChem descriptions easier to retrieve using explicit chemical-role questions.
The experiment does not show that ChEmbed learned a complete chemical ontology, a generally better molecular-similarity space, or chemical roles independently from textual evidence.
These are five concrete choices for which we would value chemical judgment. We are not asking the reader to validate the embedding calculations or give the whole experiment a simple yes-or-no approval.
- Using a ChEBI role as a family: If several compounds have the same direct ChEBI
has_roleannotation, is it chemically reasonable to treat them as members of one retrieval family for a general question about that role, even when their structures or mechanisms differ? - Role breadth and completeness: Are some types of ChEBI roles too broad, too narrow, or too chemically mixed for this use? Could incomplete role annotation make a family with 4 to 30 mapped members misleading?
- Factor Xa example: Do the eight ChEBI-linked PubChem entities in the accompanying figure form a coherent example, and is the question “What chemicals can function as EC 3.4.21.6 (coagulation factor Xa) inhibitor?” phrased appropriately?
- Different forms of the same compound: The factor Xa set includes edoxaban, edoxaban tosylate, and edoxaban tosylate monohydrate as separate database entities. Is it appropriate to count them separately here, or would that overstate the chemical diversity of the retrieved set?
- Claim boundary: Given that much of the improvement becomes smaller when the exact role wording is removed, is “better retrieval for explicit chemical-role questions” a fair description, provided that we do not claim structure-based chemical reasoning?