We sought a chemistry-interpretable view of the behaviour learned during ChEmbed training. Specifically, we asked whether a natural-language query about a chemical role would retrieve PubChem descriptions of compounds that ChEBI directly assigns to that role more effectively. PubChem supplied the descriptions, while ChEBI supplied structured role annotations independently of the embedding models.
Retaining one 20 to 500 word description per PubChem CID and removing exact duplicate texts yielded 44,887 retrieval candidates. We linked these PubChem records to ChEBI using explicit database links or exact full-InChI matches. If several usable PubChem CIDs matched the same ChEBI entity, we retained one CID to avoid duplication. This produced 42,068 ChEBI-to-PubChem mappings involving 40,656 PubChem CIDs. A role-based group contains the distinct mapped CIDs sharing the same direct ChEBI has_role annotation and does not imply structural relatedness.
Of 1,313 roles with at least one available mapped entity, 597 had 1 to 3 members, 518 had 4 to 30, and 198 had more than 30. The primary cohort used the 518 groups with 4 to 30 available entities, containing 5,783 compound-to-role assignments and 4,494 unique entities. We required at least four compounds so that each role could be evaluated as a group. Roles with more than 30 compounds were evaluated separately because they were much broader.
For each role, we used three fixed question forms:
- "Which compounds act as [role]?"
- "Find chemical entities with the role of [role]."
- "What chemicals can function as [role]?"
The resulting 1,554 queries each ranked all 44,887 descriptions using Nomic, ChEmbed-vanilla, and ChEmbed-progressive. The directly annotated CIDs formed the reference set for that role. Average precision summarizes how early and consistently these reference descriptions appear across the complete ranking, while recall at 10 is the proportion appearing among the first ten results.
| Model | Average precision | Recall at 10 |
|---|---|---|
| Nomic | 0.176 | 20.4% |
| ChEmbed-vanilla | 0.203 | 22.7% |
| ChEmbed-progressive | 0.315 | 33.4% |
ChEmbed-progressive improved 435 of the 518 primary role-based groups relative to Nomic. The improvement held across all three query templates. A separate sensitivity analysis restricted to one-to-one PubChem-to-ChEBI correspondences produced an essentially identical model advantage, while including groups above 30 preserved the same overall direction. Figure 1 summarizes the group-level comparison and the role-phrase masking control.
Figure 1. Chemical-role retrieval across the 518-group primary cohort. (A) Group-level average precision (AP) for ChEmbed-progressive versus Nomic; points above the diagonal improved after training. (B) Mean group-level AP for Nomic, ChEmbed-vanilla, and ChEmbed-progressive. (C) The same comparison before and after masking the exact ChEBI role phrase in the reference descriptions.
The factor Xa inhibitor role was retained as a continuity example before the full-corpus analysis. For the exact scored query, "What chemicals can function as EC 3.4.21.6 (coagulation factor Xa) inhibitor?", 3 of 8 directly annotated PubChem entities appeared among Nomic's first 10 results, compared with 7 of 8 for ChEmbed-progressive, as shown in Figure 2. This example illustrates the rank changes for individual compounds; the overall conclusion is based on all 518 role-based groups. The eight PubChem records correspond to six inhibitor identities because three records describe edoxaban, edoxaban tosylate, and edoxaban tosylate monohydrate.
Figure 2. Retrieval ranks for the eight PubChem records directly annotated by ChEBI as coagulation factor Xa inhibitors. For the displayed query, three records appeared among Nomic's first ten results and seven among ChEmbed-progressive's first ten. The eight records correspond to six inhibitor identities because edoxaban, edoxaban tosylate, and edoxaban tosylate monohydrate are separate records. Structures identify the records; retrieval used description text only.
To assess how much explicit role wording contributed to the result, we repeated the evaluation after removing the exact queried ChEBI role label from the corresponding reference descriptions. That label appeared in 5,110 of the 5,783 evaluated CID-to-role links, or 88%. After masking it, average precision fell from 0.176 to 0.052 for Nomic and from 0.315 to 0.091 for ChEmbed-progressive. Explicit role wording is therefore an important part of the retrieval signal. A smaller progressive advantage remained, but this control does not identify its mechanism. Because the role-member descriptions came from a PubChem source used during training, this analysis characterizes learned retrieval behaviour rather than held-out chemical knowledge. The supported interpretation is that ChEmbed, particularly the progressive variant, better aligns explicit chemical-role questions with relevant PubChem descriptions. It does not show inference from molecular structure or learning of a complete chemical ontology.