Checkpoints on Hugging Face (download access on request): retro-lm 0.6B · retro-lm 1.7B

One step (System 1)

• Top-1 means the first proposal matches the reactants recorded in the patent; top-10 means a match appears anywhere in the ten. Both sides are canonicalized with RDKit before comparing. A chemically valid alternative counts as a miss, so these are lower bounds.
• Every complete arm answered the same benchmark reactions. Hatched rows (Muse Spark Open, Kimi K3.1 Pro, DeepSeek V4 Pro, via OpenRouter at provider-default reasoning) ran out of API credit partway and are scored only on the reactions they answered, shown as n; compare them with care.
• The frontier LLMs get the same 5-shot prompt. The OpenAI and Anthropic APIs cannot switch reasoning off, so "low" is their minimum. Refusals count as misses. LLM time is median wall time per request, network included; LLM cost is list price for the tokens used (OpenRouter arms: the charge it reported).
• The 0.6B (beam search) has one row per serving engine, SGLang and vLLM, each with its own served accuracy, cost at its best batched throughput and time for one request at a time, on one H200 without network. The 1.7B (beam 10 on 20 spellings of each product) has offline accuracy and cost and its time served on SGLang. The Full table adds the offline 0.6B run for reference. Only the 0.6B runs live here.
• Live accuracy counts only benchmark reactions you ran in this browser session; with a handful it is noisy. A cached answer returns at once, so its speed bar shows the original run.
• USPTO-50K is public and years old, so memorisation by any model, retro-lm's base model included, cannot be ruled out.

Full routes (System 2)

• A target is solved when the search finds a route whose every leaf is in the purchasable catalogue, within the model-call budget. Both proposers ran under the same Retro* search, budget and stock, so the comparison isolates the one-step model.
• The sample is a sample, so the 95% interval is roughly wide. The published rows are on all . The published Retro* rows use a trained value function this search does not have; Retro*-0, like this search, does not, so it is the closer comparison. Read retro-lm as "in the range of the published rows", not as beating them.
• The points below the run's budget calls are upper bounds read from that run (first solution within N calls), not separate runs capped at N. Only the last point is a measured solve rate.
• Most targets have no overlap with training data (neither the target nor its reference route); retro-lm's solve rate on those is .

Step judge

• The judge question: is this reactant set a plausible precursor of this product? AUROC measures ranking, calibration error (ECE) whether a stated probability means what it says. The OpenAI Decisions API and TypeSafe JEV are decision-only models: they score steps but cannot propose reactants.
• retro-lm judges with its own log-likelihood, calibrated on a validation split. It is weakest on its own plausible-but-wrong proposals (AUROC lower there), the case a route search cares about most.

• It proposes reactants only. It says nothing about conditions, temperature, solvent, yield, side products or hazards.
• The benchmark scores one step back, the reactants for the final reaction. The full-route mode chains such steps with a search, but a found route is only as good as each proposed step; nothing checks that it works in a flask.
• Training data is patent chemistry, weighted toward medicinal chemistry. Other reaction classes are untested.
• A proposal is a starting point for a chemist, not a validated procedure.