
==== Front
J Chem Inf Model
J Chem Inf Model
ci
jcisd8
Journal of Chemical Information and Modeling
1549-9596
1549-960X
American Chemical Society

39154287
10.1021/acs.jcim.4c00554
Article
RLSynC: Offline–Online Reinforcement Learning for Synthon Completion
Baker Frazier N. †
Chen Ziqi †
https://orcid.org/0000-0001-9392-2431
Adu-Ampratwum Daniel ‡
https://orcid.org/0000-0002-6842-1165
Ning Xia *†¶§‡
† Department of Computer Science and Engineering, College of Engineering, The Ohio State University, Columbus, Ohio 43210, United States
‡ Division of Medicinal Chemistry and Pharmacognosy, College of Pharmacy, The Ohio State University, Columbus, Ohio 43210, United States
¶ Translational Data Analytics Institute, The Ohio State University, Columbus, Ohio 43210, United States
§ Department of Biomedical Informatics, College of Medicine, The Ohio State University, Columbus, Ohio 43210, United States
* Email: ning.104@osu.edu.
18 08 2024
09 09 2024
64 17 67236735
29 03 2024
15 07 2024
05 07 2024
© 2024 The Authors. Published by American Chemical Society
2024
The Authors
https://creativecommons.org/licenses/by-nc-nd/4.0/ Permits non-commercial access and re-use, provided that author attribution and integrity are maintained; but does not permit creation of adaptations or other derivative works (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Retrosynthesis is the process of determining the set of reactant molecules that can react to form a desired product. Semitemplate-based retrosynthesis methods, which imitate the reverse logic of synthesis reactions, first predict the reaction centers in the products and then complete the resulting synthons back into reactants. We develop a new offline–online reinforcement learning method RLSynC for synthon completion in semitemplate-based methods. RLSynC assigns one agent to each synthon, all of which complete the synthons by conducting actions step by step in a synchronized fashion. RLSynC learns the policy from both offline training episodes and online interactions, which allows RLSynC to explore new reaction spaces. RLSynC uses a standalone forward synthesis model to evaluate the likelihood of the predicted reactants in synthesizing a product and thus guides the action search. Our results demonstrate that RLSynC can outperform state-of-the-art synthon completion methods with improvements as high as 14.9%, highlighting its potential in synthesis planning.

U.S. National Library of Medicine 10.13039/100000092 1R01LM014385-01 Division of Information and Intelligent Systems 10.13039/100000145 IIS-2133650 document-id-old-9ci4c00554
document-id-new-14ci4c00554
ccc-price
==== Body
pmc1 Introduction

Retrosynthesis is the process of determining the set of reactant molecules that can react to form a desired product molecule. It is essential to drug discovery, where medicinal chemists seek to identify feasible synthesis reactions for desired molecules (i.e., synthesis planning1,2). The recent development on computational retrosynthesis methods using deep learning3−7 has enabled high-throughput and large-scale prediction for many products, facilitating medicinal chemists to conduct synthesis planning much more efficiently. Among the existing computational retrosynthesis methods, semitemplate-based retrosynthesis methods8−10 imitate the reverse logic of synthesis reactions: they first predict the reaction centers in the products, and then transform (complete) the resulting synthons–the molecular structures from splitting the products at the reaction centers, back into reactants. Semitemplate-based retrosynthesis methods enable necessary interpretability as to where the reactions happen among the reactants and how the products are synthesized, and thus have high practical utility to inform synthesis planning.

Existing retrosynthesis methods typically train predictive or generative models to transform from products to their reactants, under the supervision of training data with known reactions. The objective during model training is to reproduce the reactions in the training data, and the model performance is evaluated by comparing the predicted reactions of a product against its known reactions. While these models can accurately recover the known reactions for products, they do not have the capability of exploring and learning new reaction patterns not present in the training data. To address this issue, in this manuscript, we develop a new multiagent reinforcement learning method with offline learning and online data augmentation, denoted as RLSynC, for synthon completion in semitemplate-based methods. Figure 1 presents the overall idea of RLSynC. We focus on semitemplate-based methods due to their interpretability, practical utility and state-of-the-art performance.9,10 We particularly focus on their synthon completion step, because reaction centers can be predicted very accurately,10 but synthon completion is often more complicated.11

Figure 1 Overview of RLSynC.

Specifically, RLSynC assigns one agent to each synthon, all of which complete the synthons by conducting actions step by step in a synchronized fashion. All the agents share the same action selection policy and select the optimal actions with full observation of other agents’ states. RLSynC learns the policy from offline training episodes, and augmented training data generated through online interactions. The augmented data introduce new reaction patterns not included in training data, and thus allow RLSynC to explore new reaction spaces. RLSynC uses a reward function to evaluate the likelihood of the predicted reactants in synthesizing a product, and thus guides the action search. We compare RLSynC with state-of-the-art synthon completion methods. Our results demonstrate that RLSynC can outperform these methods with improvements as high as 14.9%, highlighting its potential in synthesis planning. To the best of our knowledge, RLSynC is the first reinforcement learning method for synthon completion. Our code is available at https://github.com/ninglab/RLSynC.

2 Related Work

2.1 Retrosynthesis

Deep-learning-based retrosynthesis methods can be categorized into three groups: template-based, template-free and semitemplate-based. Template-based methods3,12,13 use reaction templates extracted from known reactions to transform a product directly into reactants, and thus are limited to reactions covered by the templates. Template-free methods4−6,14 typically utilize the sequence representation of molecules (SMILES) and employ Transformer models to translate product SMILES strings into reactant SMILES strings, without using reaction templates. For example, RSMILES5 uses a Transformer to decode the reactant SMILES strings from the product SMILES strings. RetroFormer6 is a template-free retrosynthesis method. RetroFormer6 embeds both the SMILES strings and molecular graphs of products, and uses the embeddings to predict reaction center regions and generate reactant SMILES strings. However, these methods may generate SMILES strings that violate SMILES grammars or chemical rules.

Semitemplate-based methods7−10,15 have two steps: (1) they first identify the reaction centers and break the product into synthons using reaction centers; and then (2) they complete synthons into reactants. GraphRetro9 predicts reaction centers by learning from the molecular graphs of products, and then completes synthons by classifying the subgraphs based on whether they can realize the difference between synthons and reactants. G2Retro10 also predicts reaction centers from molecular graphs, and then completes synthons by sequentially adding rings or bonds. RetroPrime8 is a semitemplate based retrosynthesis method. RetroPrime8 employs two Transformers to first translate the SMILES strings of products to synthons, and then synthons to reactants.

Recent works have leveraged structural information and attention mechanisms to perform semitemplate-based and template-free retrosynthesis. G2GT16 leverages a graph-to-graph transformer to transform the product into predicted reactants directly, combining the richness of graph-based representation with the versatility of template-free methods. RetroExplainer17 is a semitemplate-based method, which leverages graph transformers, contrastive learning, and multitask learning to enhance its predictive capability. MARS18 uses graph attention to predict the sequence of edits and motifs required to transform the product into reactants. They differentiate their approach by the use of chemical motifs, which are larger than atoms but smaller than leaving groups.

Existing work primarily uses exact match accuracy@N to evaluate single-step retrosynthesis. However, exact match accuracy does not account for synthetically viable predictions not matching the ground truth. Schwaller et al.19 propose using forward synthesis models to evaluate single-step retrosynthesis techniques. Similarly, we use forward synthesis for evaluation with MAP@N, NDCG@N, and Diversity@N. However, RLSynC also uses forward synthesis in training, allowing RLSynC to explore new reaction patterns.

2.2 Reinforcement Learning

Deep reinforcement learning methods have been developed to design new small molecules. For example, GCPN20 uses a graph convolutional policy network to sequentially add new atoms and bonds to construct new molecules. MolDQN21 uses Morgan fingerprints22 to represent molecules, and learns a deep Q-network to guide the addition or change of atoms and bonds, modifying molecules to have desired properties. PGFS23 uses a policy gradient method to learn de novo molecule design by sequentially selecting commercially available reactants and simulating the forward synthesis path. Reinforcement learning has also been applied for biological sequence design. For example, DynaPPO24 uses a model-based variant of proximal-policy optimization to generate DNA and peptide sequences with desired properties. TCRPPO25 learns a mutation policy to mutate sequences of T-cell receptors to recognize specific peptides.

Reinforcement learning has been used for multistep retrosynthetic planning, which seeks to find an optimal sequence of multiple reactions to synthesize a product. Traditionally, tools such as AiZynthFinder2 implement a tree search algorithm (e.g., Monte Carlo tree search) to plan multistep synthesis routes. However, reinforcement learning can also be used to navigate the multistep retrosynthesis space. For example, Schreck et al.26 trains an agent to select reactions from a list to construct the sequence backward starting from the product, until all the reactants of the first reaction in the sequence are purchasable. GRASP27 trains a goal-driven actor-critic policy to select reactions from a tree of possible reactions until it reaches purchasable reactants in the leaf nodes of the tree. In these methods, reinforcement learning is used to select reactions rather than predicting reactions. However, there is very limited work applying reinforcement learning to retrosynthesis. RCSearcher28 applies a deep Q network to search a molecular graph for reaction centers. In contrast, RLSynC uses reinforcement learning for synthon completion.

Recent efforts29,30 in reinforcement learning leverage offline–online learning to accelerate or improve learning. AWAC29 is an extension of actor-critic learning that combines prior offline demonstration with online experience to accelerate the learning of robotics tasks. PEX30 constructs a policy set to combine offline and online policies. PEX freezes and retains the offline policy during online training and samples actions from the offline and online policies, allowing it to benefit from the offline experience without overwriting it. These works emphasize the usefulness of combining offline data with online experience. Inspired by this, RLSynC also leverages offline data to accelerate learning and online interactions to explore the retrosynthesis prediction space.

3 Definitions and Notations

A synthesis reaction involves a set of reactants {Ri} and a product molecule P that is synthesized from these reactants. Each reactant Ri has a corresponding synthon Mi, which represents the substructures of Ri that appear in P. The connection point of these synthons to form the product, typically a bond, is referred to as the reaction center. Figure 2 presents the retrosynthesis process. In retrosynthesis, a typical semitemplate-based method first identifies the reaction center and thus the corresponding synthons {Mi}, and then completes the synthons back to reactants {Ri}.

Figure 2 Semitemplate-based retrosynthesis process.

To complete Mi to Ri, atoms may be added to Mi one at a time, through establishing new bonds. In each step t, the intermediate molecular structure generated from Mi is denoted as Mit. With abuse of terms, such intermediate molecular structures are referred to as current synthons. In this manuscript, we focus on the reactions with only two reactants, because this is the most common case in synthesis reactions.11 In this manuscript, the two terms “pair of reactants” and “reaction” for a product are used interchangeably, when no ambiguity is raised; the term “prediction” refers to the prediction of the two reactants of a product. Table 1 presents the key definitions and notations.

Table 1 Key Notations

Notation	Meaning	
P	product molecule	
(R1, R2)	a pair of reactants	
(M1, M2)	a pair of synthons	
(M1t, M2t)	a pair of current synthons at step t	
A	an agent	
t/T	time step/time step limit	
MDP	Markov decision process:	
/st	State space/a state at step t	
/a1t/a2t	Action space/an action used to update M1t/M2t at step t	
Ait	the set of feasible actions for Ai at time step t	
	state transition function	
	reward function for terminal states	

RLSynC is employed under the assumption that reaction centers are predetermined or can be accurately predicted. This is because the potential reaction centers are typically limited, especially in the case of small molecules. According to Chen et al.,10 reaction center prediction can achieve as high as 97.2% accuracy. However, synthon completion is often more complicated and can be realized in a variety of ways.11 However, RLSynC can be generalized to one-reactant or multireactant cases by having one or multiple agents, given how RLSynC completes synthons (Section 4.2).

4 Methods

4.1 RLSynC Model

RLSynC assigns one agent, denoted as A, to each synthon and uses the agent to transform its synthon into a reactant through a sequence of actions, assuming the reaction center is known. This transformation is achieved through a Markov Decision Process (MDP), denoted as MDP = , including a state space , an action space , a transition function and a reward function .

4.1.1 State Space ()

RLSynC has a discrete state space describing the MDP status. Each state is represented as1

where t denotes the steps of actions (Section 4.1.2); M1 and M2 are the synthons from P that are assigned to agent A1 and A2, respectively; M1t and M2t are the current synthons generated from M1 and M2 after t (t = 0, ..., T) steps of actions by A1 and A2, respectively (M10 = M1, M20 = M2); P is the product molecule; and T is the step limit and thus sT is a terminal state. The current synthons (M1T, M2T) in sT are the predicted reactants. Inspired by previous work,21,31 we impose a step limit T to control the number of addition and provide a guarantee of termination. In RLSynC, T is set to 3 because 89.10% of the synthons in the benchmark USPTO-50K data set can be completed with the addition of up to 3 atoms. Increasing T would not significantly increase coverage of the ground truth reactants, (e.g., T = 6 will cover only an additional 2.96% of the synthons in the benchmark data set), but would increase the complexity of problem and the search algorithm (see Section 4.2.4). Please note that RLSynC supports setting T to any number, so this could be adjusted for other data sets. When no ambiguity is raised, st,P is represented as st with P dropped.

4.1.2 Action Space ()

RLSynC has two types of actions in its action space : (1) adding atoms via bonds, denoted as ADD, and (2) no operation (i.e., doing nothing), denoted as NOOP. Rather than directly mimic the complex electron cloud mechanisms of chemical reactions, RLSynC uses this discretized action space that fits well with our reinforcement learning framework. Using ADD actions to add atoms one by one, the agents are able to explore diverse synthon completions while maintaining a small action space, with the ultimate results of the actions being the reactants. This is inspired by existing MDP-based methods for molecule design.20,21 Using the NOOP action, the agent can skip a step when it is not necessary to add atoms. For ADD, RLSynC allows 12 types of atoms (B, C, N, O, F, Si, P, S, Cl, Se, Br, and I) via single, double or triple bonds, and thus 36 types of additions. These additions are sufficient to complete 98.42% of the synthons in those two-synthon cases in the benchmark data within 3 steps. The other 1.58% of the synthons require metal atoms (Cu, Mg, Sn, and Zn), but these are uncommon in organic chemistry reactants and insufficiently represented in our data set. Therefore, we exclude these from our action space. No other atom types or bond types were present in our data set. Adding more ADD actions offers very little additional coverage, but expands the action space. Please note, these restrictions apply only to the action space, not the state space. RLSynC can handle any atom or bond type in the original input synthon, but can only modify the synthons using the actions in .

Thus, the action space is denoted as follows:2

where each ADDi corresponds to a specific atom type and bond type combination. The atom additions have to satisfy the following constraints:a) The new atoms are only added to the reaction centers or atoms that are added through the previous actions;

b) The bonds connecting the new added atoms and the current synthons obey structural or valency rules;

c) The types of these new bonds exist in the training data.

At each step t (t = 0, ..., T), each agent Ai selects an action ait from , and applies the action to its current synthon Mit (i = 1, 2). The two agents act in a synchronized fashion and start the next step t + 1 only when both finish step t. Note that each agent at step t has perfect observations of its own action and current synthon, and also the other agent’s current synthon. This full observation allows the agents to share the same policy without exchanging information, and thus simplifies the policy learning (Section 4.2.3).

4.1.3 Transition Function ()

The transition function (st+1|st, {a1t, a2t}) in RLSynC calculates the probability of MDP transitioning to state st+1, given the state st and actions {a1t, a2t} at step t. In RLSynC, is deterministic, that is, (st+1|st, {a1t, a2t}) = 1.

4.1.4 Reward Function ()

RLSynC uses a final binary reward to guide its agents. At the terminal step T, if the predicted reactants M1T and M2T exactly match the reactants given for the product P in the training data, gives sT a reward of 1. If the predicted reactants do not exactly match the reactants given for the product, that does not necessarily mean they are incorrect predictions. They may still be synthetically viable. To account for this, a standalone forward synthesis prediction model is applied to predict the products that can be synthesized from M1T and M2T. If P is among the top-5 predictions by this model, gives sT a reward of 1; otherwise, a reward of 0. RLSynC uses Molecular Transformer32 as the forward synthesis prediction model, because it is the state of the art and achieves very high accuracy for forward synthesis prediction.33 Please note that Molecular Transformer is trained from a separate data set with no overlap with the training data for RLSynC, eliminating the possibility of data leakage and bias in calculating rewards. However, RLSynC is not bound to Molecular Transformer and can be easily adapted to any other forward synthesis prediction models for . Predicted reactants that receive positive rewards are referred to as correct predictions.

4.1.5 State-Action Representation

The state-action pairs will be used to learn a state-action Q-value function (discussed later in Section 4.2.1). RLSynC represents a state-action pair (st, {a1t, a2t}) as follows:3

where i = 1, 2 indexing the agent of interest, j = 3 – i indexing the other agent; m’s are the Morgan fingerprint vectors for the corresponding synthons M’s (Mit will be transformed to Mit+1 by ait, and Mit+1 is represented by mit+1); p is the Morgan fingerprint vector for the product P; and ⊕ is the concatenation operation. The use of Morgan fingerprints is inspired by Zhou et al.21 Morgan fingerprints22 capture molecular substructure information, are easy to construct and do not require representation learning.

4.2 RLSynC Training and Prediction

4.2.1 Offline Training

At a state st = {M1, M2, M1t, M2t, P, T – t}, RLSynC uses a state-action value function QΘ(st, {a1t, a2t}), parametrized by Θ, to estimate the future rewards of st if the actions a1t and a2t are applied by A1 and A2 on M1t and M22, respectively. QΘ(st, {a1t, a2t}) is modeled as a multilayer fully connected neural network, with (st, {a1t, a2t}) represented as hi,t (eq 3) as input to the neural network.

4.2.1.1 Offline Training Episode Generation

RLSynC uses offline data of precomputed episodes to learn QΘ. An episode refers to a trajectory from s0 to sT for a product P, that is, RLSynC computes all true episodes from training data that include known reactions to synthesize given products. For each product in the training set, RLSynC also computes 4 random episodes from a set of random reactions that are not included in the training data. These random reactions are generated by taking random actions on the synthons of the products in the training data, and their rewards are calculated using . While all the true episodes for known reactions have positive rewards, most episodes for random reactions have zero rewards. By training on both positive- and zero-reward episodes, the agents can learn actions to take as well as actions to avoid, thereby improving their overall performance.

4.2.1.2 Q-Value Function Learning

From the offline training episodes, RLSynC uses a SARSA34-like approach to approximate the Q-value Q̃(st, {a1t, a2t}) as follows:4

where 0 < γ < 1 is the discount factor and is the reward function. With the approximate Q-values, RLSynC learns QΘ(st, {a1t, a2t}) by minimizing the following loss function:5

where is the set of all {st, {a1t, a2t}} pairs from the offline training episodes and is its size; qt = Q̃(st, {a1t, a2t}) is calculated from eq 4; is the -norm regularizer over Θ; and α > 0 is a trade-off parameter.

4.2.2 Training Data Augmentation via Online Interactions

The offline training has the advantage of sourcing experience from known reactions, and thus exposing RLSynC to successful synthon completion patterns (i.e., the true episodes). However, it also eliminates the opportunity for RLSynC to explore and discover new synthon completion patterns that are not included in the training data. To mitigate this issue, inspired by Nair et al.,29 RLSynC deploys an innovative strategy to augment the offline training data via online interactions, denoted as RLSynC-aug.

4.2.2.1 Augmentation through Greedy Action Selection

RLSynC first learns QΘ using the known and random reactions as discussed above (Section 4.2.1). The optimized agents are then applied to the original unaugmented training data (only known reactions, excluding random reactions), and generate one new episode for each unique product in the training data from the actions they take and the rewards the actions receive (action selection discussed later in Section 4.2.3). These new episodes are added to and augment the training data. QΘ is then relearned from this new augmented training data and the new agents generate another set of new episodes for further training data augmentation. The above process is iterated until the performance of the agents on the separate validation set is not improved.

4.2.2.2 Augmentation through Top-N Prediction Search

Once the agents’ performance is stabilized, a more aggressive strategy is deployed to generate additional new episodes from the training products through top-N prediction search (discussed in Section 4.2.4) and tracking back the episodes leading to the top-N predictions, resulting in N episodes for each product. Multiple iterations of such top-N-prediction-based new episode generation are conducted and training data is augmented until validation set performance ceases to improve. A final QΘ will be learned over the final augmented training data.

Augmenting the training data with additional positive-reward episodes exposes RLSynC to new reactions that do not exist in the training data, allowing RLSynC to explore beyond the limits of the data. Augmenting the training data with additional zero-reward episodes helps RLSynC correct inaccurate Q-value estimations, enhancing its robustness and effectiveness. Algorithm 1 presents RLSynC-aug.

4.2.3 Action Selection Policy

In predicting the reactants of a new product P (i.e., in validation set or test set), the agents of RLSynC predict and select actions to complete synthons via up to T steps. To predict the action ait for agent Ai at time t, RLSynC first identifies the set of all chemically feasible atom additions, denoted as Ait (), for Mit satisfying the constraints as in Section 4.1.2. Ai samples the best action ait ∈ Ait (i = 1, 2) if the action has the maximum predicted Q-value, as follows:6

7

That is, each agent assumes NOOP from the other agent, but observes the other agent’s current synthon in st in order to select its next optimal action. Please note that the two agents share the same QΘ function and follow the same policy.

4.2.4 Top-N Prediction Search

RLSynC uses a novel greedy search algorithm, denoted as RLSynC-search, to identify the top-N predicted reactions with the highest Q-values. RLSynC-search is applied to the training data during online iterations to predict top-N pairs for training data augmentation RLSynC-search is also used after training is completed to predict the top-N sets of reactants for new products (e.g., the test set). This provides additional options for downstream tasks, such as multistep synthesis planning. Specifically, at step t = 0, instead of selecting only one action, each agent selects k actions with the top-k highest Q-values calculated based on eq 6 and 7 (i.e., instead of “max” in eq 6 and 7, use “top-k”). Thus, there will be k2 possible next states, resulting from all possible action combinations from the two agents. In each of the possible next states, each agent selects again k actions for its current synthon. Through T steps, this process will result in k2T predicted reactions at the terminal state. RLSynC sorts all these predicted reactions using their Q-values and selects the top-N predictions. While this process can be expensive, the actions on each of the possible next states can be done independently, and this process can be implemented in parallel. In our experiments, we used k = 3 and T = 3, and therefore the computational cost is low. The algorithm is presented in Algorithm 2.

5 Experimental Settings

5.1 Data

We use the benchmark USPTO-50K data set35 in our experiments, which contains 50,016 chemical reactions. We use the same training, validation, and testing division as in the literature,7 resulting in 40,008 reactions for training, 5,001 for validation, and 5,007 for testing. From each set, we use only the reactions that satisfy the following constraints to train/evaluate RLSynC:a) The reaction has exactly two reactants;

b) The synthons can be completed to the ground-truth reactants by adding no more than three atoms.

After applying the above filter, our training, validation, and test sets contain 25,225, 3,172 and 3,167 reactions, respectively. The baseline models are trained using all 40,008 training reactions and tested on the same test set as RLSynC.

Reactions that contain metal atoms (Cu, Mg, Sn, and Zn) are included in the training, validation, and testing data, despite the limitations of the action space (Section 4.1.2). RLSynC can still learn to complete these synthons using alternative leaving groups based on feedback from the forward synthesis reward function. During training, these reactions provide diverse opportunities for the agents to learn to predict such alternative leaving groups. During validation and testing, these reactions highlight the agents’ abilities to achieve a reward of 1 without matching the ground truth.

5.2 Baselines

We compare RLSynC against three state-of-the-art retrosynthesis methods, including GraphRetro,9 G2Retro,10 and RSMILES.5 In the experiments of synthon completion (Section 6.1), we compare RLSynC against the synthon completion components of G2Retro, GraphRetro, and the semitemplate-based version of RSMILES, referred to as RSMILES-p2s2r, because, as semitemplate-based methods, they have a synthon completion component that can be used separately. Each of these methods takes a different approach to retrosynthesis prediction:GraphRetro9 uses graph message passing networks to perform semitemplate-based retrosynthesis. GraphRetro treats reaction center prediction as a set of bond classification problems. For synthon completion, GraphRetro selects leaving groups to append from a limited set of options.

G2Retro10 also uses graph message passing networks to perform semitemplate-based retrosynthesis. G2Retro learns a set of atom and bond regression problems to score reaction centers. G2Retro completes synthons by performing a sequence of additions to the molecule.

RSMILES5 treats retrosynthesis as a sequence-to-sequence prediction task and uses a Transformer36 model trained on root-aligned SMILES. By aligning the training set SMILES, RSMILES minimizes the edit distance between the product and reactant strings, making it easier for the model to learn accurate predictions. RSMILES is also able to augment its training data with multiple SMILES aligned to different root atoms. RSMILES provides both template-free and semitemplate-based versions.RSMILES-p2r: The template-free version of RSMILES directly outputs the reactant SMILES based on the product SMILES input.

RSMILES-p2s2r: The semitemplate-based version of RSMILES uses two sequence-to-sequence models: the first produces synthons given a product, and the second produces reactants given the synthons.

5.3 Evaluation Metrics

We evaluate different methods in terms of the correctness, diversity, and validity of their predicted reactants. Consistent with common practice in reinforcement learning, we use the reward function in these metrics. Please note that the reward function’s forward synthesis model is standalone; it remains fixed while training RLSynC. During testing, RLSynC does not use the reward function to predict the reactants, but uses the learned QΘ to guide its prediction.

5.3.1 Correctness Metrics

Two predicted reactants of a product are considered correct if they receive a reward of 1 (Section 4.1.4). To measure the correctness of top-N predicted reactant pairs, we use mean average precision at top-N (MAP@N). We define MAP@N as follows:8

where is the test set, P is a product in , is the reward (Section 4.1.4), {M1T, M2T}P,k is the k-th ranked predicted reactants (i.e., at the terminal step T, see Section 4.1.1) for P. Higher MAP@N indicates better correctness among top-N predictions. Note that MAP@N is different from accuracy@N in retrosynthesis prediction,5,9,10 which compares the predictions only with the ground-truth reactions.

We also use normalized discounted cumulative gain at top-N (NDCG@N),37 which is a popular metric in evaluating ranking. In our experiments, NDCG@N uses the rewards as gains and captures both the rewards and the ranking positions of the predictions. Higher NDCG@N indicates that correct predictions tend to be ranked higher.

5.3.2 Diversity Metrics

A diverse set of correct predictions enables a broad range of viable options to synthesize a product, and thus is preferred in synthesis planning. We measure diversity using the dissimilarity among the top-N correct predictions, as follows:9

where rj is the j-the ranked predicted reactions for P (i.e., rj = {M1T, M2T}P,j); is the reward for rj (for correct predictions, ; otherwise 0); dsim(ri)P is the minimum dissimilarity of ri to higher ranked correct predicted reactions for product P as defined below:where sim(ri, rj) is the Tanimoto similarity between two predicted reactions. Following the definition from Chen et al.,10 we define similarity between predicted pairs of reactants as follows:10

where MkjT is the k-th predicted reactant (k = 1, 2) for the j-th ranked prediction for the product P; and simM is the Tanimoto similarity38 between two molecules. Note that in eq 10, we assume reactions have exactly two predicted reactants. Baseline methods can predict reactions with any number of reactants. Again, following Chen et al.,10 when either predicted reaction has fewer or more than two reactants, we define similarity as follows:where MiT denotes a disconnected, composite molecule consisting of all of the predicted reactants for ri.

5.3.3 Significance Testing

We compare the performance of RLSynC with the best performing baseline for each metric at N using a two-sample paired t test on the test set (Section 5.1), where the scores for each test example from RLSynC and the best baseline are used as paired samples. We consider results to be significantly different if the reported p-value is less than 0.05.

6 Results

We first compare RLSynC directly to the synthon completion components of the baselines by providing known reaction centers to all the methods. Please note that all the methods are tested only on reactions that have two reactants; RLSynC is trained on reactions with only two reactants (Section 5.1); all the baseline models are from their original publications and implementations, and were trained over USPTO-50K data set35 (i.e., including reactions of various numbers of reactants).

6.1 Evaluation on Synthon Completion

6.1.1 Correctness Evaluation

Table 2 and Table 3 present the performance of all the methods in terms of MAP@N and NDCG@N in completing given synthons. As Table 2 shows, in terms of MAP@N, RLSynC outperforms the baseline methods consistently for N ∈ [2, 10]. G2Retro is the best baseline, with the best MAP@N among all baselines at N ∈ [2, 10], and very close to the best MAP@1 achieved by RSMILES-p2s2r. RLSynC significantly outperforms G2Retro on 9 results, with the best improvement 14.9% at N = 10, average improvement 7.2% over N ∈ [2, 10], and 8 improvements statistically significant. As N increases, the performance improvement from RLSynC over other methods also increases, indicating that RLSynC generates more positive-reward reactions than other methods among top-N predictions. This capability of RLSynC could facilitate the design of multiple synthesis reactions in synthesis planning. Note that as N increases, top-N predictions tend to include more incorrect predictions, leading to a decrease in MAP@N.

Table 2 MAP@N for Synthon Completiona

N	1	2	3	4	5	6	7	8	9	10	
RSMILES-p2s2r	0.953	0.863	0.795	0.734	0.681	0.636	0.601	0.571	0.545	0.523	
GraphRetro	0.912	0.861	0.827	0.804	0.779	0.747	0.715	0.687	0.662	0.643	
G2Retro	0.950	0.893	0.855	0.823	0.789	0.753	0.720	0.691	0.665	0.645	
RLSynC	0.927	0.898	0.874	0.845	0.822	0.803	0.784	0.769	0.754	0.741	
imprv. (%)	–2.7*	0.6	2.2*	2.7*	4.2*	6.6*	8.9*	11.3*	13.4*	14.9*	
a The best performance for each N is in bold, and the best performance among the baseline method is underlined. The row “imprv. (%)” presents the percentage improvement of RLSynC over the best-performing baseline methods (underlined). The * indicates that the improvement is statistically significant at 95% confidence level.

Table 3 NDCG@N for Synthon Completiona

N	1	2	3	4	5	6	7	8	9	10	
RSMILES-p2s2r	0.953	0.883	0.831	0.784	0.742	0.707	0.678	0.652	0.630	0.611	
GraphRetro	0.912	0.872	0.846	0.827	0.807	0.784	0.760	0.739	0.719	0.703	
G2Retro	0.950	0.905	0.876	0.851	0.825	0.798	0.773	0.750	0.729	0.712	
RLSynC	0.927	0.905	0.886	0.865	0.847	0.832	0.817	0.805	0.793	0.782	
impv. (%)	–2.7*	0.0	1.1*	1.6*	2.7*	4.3*	5.7*	7.3*	8.8*	9.8*	
a The annotations in this table are the same as those in Table 2.

Similar trends can be observed in Table 3: In terms of NDCG@N, RLSynC consistently outperforms the best baseline method G2Retro at N ∈ [3, 10], all with statistically significant improvement; the best improvement is 9.8% at N = 10, and average improvement 5.2% over N ∈ [3, 10]. RLSynC achieves the same performance as G2Retro at N = 2. RSMILES-p2s2r achieves the best NDCG@1 performance among all the methods, but its performance dramatically decreases for larger N. GraphRetro’s performance is between G2Retro and RSMILES-p2s2r. The difference between MAP@N and NDCG@N is that NDCG@N discounts the impact of low-ranking correct predictions. The fact that RLSynC achieves both high MAP@N and NDCG@N indicates that RLSynC predicts more correct reactions at high ranks (e.g., top 2, top 3).

G2Retro completes synthons into reactants by sequentially attaching substructures (i.e., bonds or rings) starting from the reaction centers. Unlike G2Retro, which exclusively learns from known reactions, RLSynC benefits from random reactions and online iterations of augmented data, therefore, it can overcome the potential limitations and biases in the known reactions. In addition, RLSynC can adapt and improve from past mistakes by learning from online iterations of data augmentation, and thus correct inaccurate Q-value predictions in zero-reward episodes. These advantages enable RLSynC to finally outperform G2Retro.

RSMILES-p2s2r employs a Transformer to translate root-aligned synthon SMILES strings into reactant strings. It also augments the training and test synthon SMILES strings with varied atom orders to improve the performance further. However, RSMILES-p2s2r is trained to recover the unique SMILES strings of ground-truth reactants in the training data, disregarding other possible reactants to synthesize the same product. In contrast, by augmenting data with online interactions, particularly through top-N search, RLSynC focuses beyond just the top-1 prediction and aims to maximize the rewards for the overall top-N predictions, and achieves better MAP@N for N ∈ [2, 10] than RSMILES-p2s2r. GraphRetro formulates synthon completion as a classification problem over subgraphs. However, it ignores the impact of predicted subgraphs on the overall structures of resulting reactants, which may lead to incorrect predictions. Unlike GraphRetro, RLSynC predicts reactants by adding feasible bonds and atoms to current synthons under the guidance of rewards for resulting molecules. As a result, RLSynC outperforms GraphRetro on MAP@N at N ∈ [1, 10].

6.1.2 Diversity Evaluation

Table 4 presents the performance of different methods in terms of Diversity@N (eq 9) among their correctly predicted reactants. Similar trends to those for MAP@N and NDCG@N can be observed for Diversity@N. RLSynC outperforms the best baseline, GraphRetro, at N ∈ [2, 10], all with statistical significance, with the best improvement 6.8% at N = 10, and average improvement 6.8% over N ∈ [2, 10].

Table 4 Diversity@N for Correct Synthon Completiona

N	2	3	4	5	6	7	8	9	10	
RSMILES-p2s2r	0.129	0.114	0.104	0.096	0.089	0.084	0.080	0.077	0.072	
GraphRetro	0.156	0.150	0.146	0.142	0.137	0.131	0.126	0.121	0.117	
G2Retro	0.154	0.148	0.143	0.137	0.131	0.125	0.120	0.116	0.113	
RLSynC	0.164	0.156	0.154	0.147	0.142	0.136	0.131	0.128	0.125	
impv. (%)	5.1*	4.0*	5.5*	3.5*	3.6*	3.8*	4.0*	5.8*	6.8*	
a The annotations in this table are the same as those in Table 2. Diversity@1 is always 0 based on the definition (eq 9), so it is not presented in this table.

While all the baseline methods are limited by the synthon completion patterns within known reactions from the training data, RLSynC is able to discover patterns not present in the training data by learning from augmented data via online interactions. These newly discovered patterns could contribute to the better Diversity@N at N ∈ [2, 10] for RLSynC. Higher Diversity@N indicates a higher variety of correctly predicted reactants. Please note that diversity in predicted reactions is always desired, as it can enable the exploration of multiple synthetic options. This makes RLSynC a potentially preferable tool in synthetic design.

6.1.2.1 Diversity in Leaving Groups

We evaluate the ability of RLSynC to discover leaving groups–the difference between synthons and their completed reactants–that are not present in the training data. The ground-truth reactants in our training data set contain 35 distinct leaving groups. RLSynC uses 229 leaving groups in its correct top-10 predictions on the test set. Among the correct top-10 predictions on the test set, 46.8% use novel leaving groups. This highlights the ability of RLSynC to explore novel reaction patterns, which can facilitate new synthesis routes in synthetic planning.

6.2 Validity Evaluation

We also evaluate the validity of the predicted reactions returned by retrosynthesis methods. We consider a predicted reaction to be valid if all the SMILES strings for the predicted reactants are valid. Valid SMILES strings should obey standard valency rules. Validity at top-N is calculated as the percentage of valid predictions among all the top-N predictions.

We also evaluate the validity of the completed synthons by the different methods. RLSynC and G2Retro always achieve 100% validity among its top-N predictions (N ∈ [1, 10]). This is because they enforce validity (Section 4.1.2), and complete synthons by adding atoms and bonds obeying valency rules. RSMILES-p2s2r can achieve on average 99.3% validity among top-N (N ∈ [1, 10]) predictions, and GraphRetro can achieve 98.4% validity. RSMILES-p2s2r formulates synthon completion as a sequence-to-sequence translation problem, so it cannot guarantee valency or the validity of the output strings. GraphRetro leverages a graph-based model of molecules, but uses a more relaxed set of valency rules when editing the molecular graphs.

6.3 Evaluation on Data Augmentation

6.3.1 Correctness Evaluation

Figure 3a and Figure 3b present the performance in terms of MAP@N and NDCG@N from RLSynC over different online iterations of data augmentation, respectively. Note that online data augmentation is determined by the continuity of performance improvement on the validation set (line 16 in Algorithm 1), and therefore, the performance on the test set may not be strictly improving over iterations. Even though, as Figure 3a shows, with 8 online iterations of data augmentation, RLSynC is able to improve its MAP@N performance over all N values. For example, RLSynC improves its MAP@1 performance from 0.763 at iteration 0 (i.e., the initial model using only known and random reactions) to 0.927 after 8 iterations, that is, 21.5% improvement. Note that at iteration 6, the online data augmentation is switched from using only 1 new episode (line 10 in Algorithm 1) to the top-5 new episodes for each product, which significantly boosts performance. Similar trends are observed for NDCG@N.

Figure 3 RLSynC performance from data augmentation iterations: a, MAP@N; b, NDCG@N; c, Diversity@N. The scale for each heatmap is set based on its range to highlight the trend across training iterations. The cell values have been rounded to two decimal places for readability.

Figure 4a and Figure 4b present the performance improvement in terms of MAP@N and NDCG@N from iterations 6, 7, and 8 over the initial model, respectively. From these figures, we see higher improvement from later iterations over the initial model. For example, in Figure 4a, the 8-th iteration improves MAP@10 by 40.1%, whereas the 6-th iteration improves MAP@10 by 32.5%. Similar trends can be observed for NDCG@N. Note that the episodes generated for data augmentation in later iterations are derived from agents that have been trained on data from previous iterations. These episodes are more likely to contain correct reactions, and subsequent agents will benefit from training on these correct reactions, leading to better performance in later iterations.

Figure 4 Improvement from Top-5 prediction search: a, MAP@N; b, NDCG@N; c, Diversity@N. The line colors and markers for b and c are the same as those given in the legend for a.

6.3.2 Diversity Evaluation

Figure 3c and Figure 4c present the Diversity@N for RLSynC across different data augmentation iterations. The best diversities result from iteration 8, with Diversity@10 as 0.125 as an example. Iterations 6, 7, and 8 provide large improvements to diversity. These iterations augment the data set with 5 unique episodes per product, chosen by agents that have been trained for multiple iterations. In contrast, the first five iterations, which augment the data set with only one episode per product, do not always improve Diversity@N but improve MAP@N and NDCG@N.

6.4 Case Study

RLSynC can predict multiple reactions from the same reaction centers. This can provide chemists with multiple options during synthetic planning. In order to illustrate this, we highlight an example product (Figure 5a) from our test set, the ground-truth reactants (Figure 5b), and the corresponding top-8 predictions from RLSynC (Figure 5c–5j). This example is an amide coupling reaction with a diamine and a carboxylic acid as the ground truth reactants (Figure 5b). RLSynC correctly predicts N,N-dimethylethane-1,2-diamine and aryl carboxylic acid as the reactants in its top-2 prediction (Figure 5d). This prediction matches the ground-truth reactants, but is not the only synthetically useful prediction. RLSynC also predicts N,N-dimethylethane-1,2-diamine/acyl chloride (Figure 5c) and N,N-dimethylethane-1,2-diamine/acyl bromide (Figure 5e) as reactant pairs. These are also practically useful amide coupling reactions for this product. Interestingly, RLSynC predicts another common reaction type in its top-7 and top-8 predictions (Figure 5i and 5j). Here, RLSynC predicts amidation reactions of esters with N,N-dimethylethane-1,2-diamine. These are also synthetically useful results, although high temperatures are typically required for this reaction to occur.

Figure 5 Predicted reactions by RLSynC for case study: a, product; b, the ground-truth reactants in USPTO-50K; c–j, top predicted reactants.

However, the top-4, top-5, and top-6 predictions (Figure 5f–5h) cannot be considered synthetically useful because of the unusual groups present on the acyl donor, such as the boron derivatives in the Figure 5h. Note that the top-4 and top-6 predictions still receive a reward of 1, indicating that the forward synthesis model may reward predictions which lack practical utility. Despite this, RLSynC is able to provide multiple synthetically useful predictions of different reaction types from the same reaction center. This makes RLSynC a desirable tool for synthetic planning.

7 Discussion and Conclusions

We developed RLSynC, a novel multiagent reinforcement learning method with offline training and online data augmentation for synthon completion. RLSynC has two agents to complete the two synthons of a product into reactants. The two agents share the same action selection policy learned from known reactions, random reactions, and reactions generated and deliberately selected during the online data augmentation iterations. Using a forward synthesis prediction model as the reward function, RLSynC achieves superior performance on synthon completion compared to the state-of-the-art methods. RLSynC provides a new and versatile framework that can enable a more comprehensive evaluation paradigm. Even more importantly, RLSynC enables the exploration of reaction patterns not seen in training, making it especially suitable for new reaction discovery purposes.

By using a reward function composed of multiple evaluation functions, it allows reaction evaluation with respect to the corresponding evaluation metrics, particularly when the reactions do not match the known reactions. Ideally, if high-throughput synthesis reactions can be conducted in laboratories over the predicted reactants, the reaction outcomes (e.g., yield) can be used as the reward. Even more importantly, RLSynC enables the exploration of new reactions that are not included in the ground truth, but are still feasible based on the reaction evaluation (i.e., the reward function), through online iterations of data augmentation. This feature makes RLSynC especially suitable for new reaction discovery purposes, and provide reactions with respect to specific evaluation metrics.

RLSynC uses a standalone forward synthesis prediction model in its reward function. This reward function guides the training and exploration process, and has a profound impact on the final model. In our experiments, we see examples where the forward synthesis model rewards reactions which are not practically useful to chemists. A more precise forward synthesis model is necessary to improve our model; however, this is outside the scope of the current work. Fortunately, the RLSynC framework is not dependent on a particular reward function. When a better forward synthesis model becomes available, RLSynC can leverage it with minimal modification.

We exclude metal atoms (Cu, Mg, Sn, or Zn) from the action space because the data sets used to train the agents and forward synthesis model do not have strong support for metal–organic reactants. The exploration capabilities of RLSynC require an accurate reward function to guide the agents to learn realistic and useful synthon completion patterns. Agents may learn unrealistic patterns if metal atoms are included in the action space without a reward function that can reliably evaluate metal–organic reactions. Metal–organic reactants are uncommon in the USPTO reaction data sets,39,40 which include both our training set and the training set for the forward synthesis model. Specifically, our training set has metal atoms in only 1.4% of reactions and the forward synthesis model has metal atoms in only 4.2% of reactions.32 This underrepresentation limits our ability to accurately reward predictions containing metals using the forward synthesis model. The RLSynC framework could incorporate metals in the future if additional metal–organic reaction data becomes available.

RLSynC uses Morgan fingerprints to represent molecules, which have been widely used in previous work.1,21,41 However, there are other fingerprint representations42,43 that could be used in place of Morgan fingerprints within the RLSynC framework. Additionally, molecular graph representation learning has been shown to be effective in retrosynthesis prediction.9,10 A thorough comparison of these representations is beyond the scope of this work, but could be a fruitful area for future research.

In retrosynthesis prediction, how to evaluate predicted reactions automatically at scale is under-studied.44 Existing methods compare the predictions with known reactions (i.e., ground-truth) of products, and consider only the predictions that exactly match the ground-truth to be correct. However, as demonstrated in Chen et al.,10 many “incorrect” predictions can still be chemically possible, and may even represent more viable options. Thus, only comparing to the ground-truth may underestimate the performance. Moreover, making the recovery of known reactions the only optimization objective may result in retrosynthesis prediction models that lack the ability to discover novel reactions.

In future work, we will generalize RLSynC for products with up to three reactants (i.e., up to three synthons; 100.00% of reactions in benchmark data set). This can be done by allowing for three agents and empty synthons if there are fewer reactants. In these cases, the agents with empty synthons will only be able to choose NOOP. We will also explore extending the model beyond the T = 3 step limit so that RLSynC can learn synthon completion patterns that require more than 3 atom additions. Because of the limited data available for such patterns, this will likely require additional data augmentation or few-shot learning techniques. In addition, we will explore more complex ways of controlling the termination of each episode, such as having the agents learn to coordinate their actions to terminate the episode when the synthons are complete. This could allow us to remove the fixed step limit altogether, but may complicate the learning task for the agents.

Data Availability Statement

Our data and source code are publicly available online at https://github.com/ninglab/RLSynC.

Supporting Information Available

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acs.jcim.4c00554.Implementation details (PDF)

Supplementary Material

ci4c00554_si_001.pdf

The authors declare no competing financial interest.

Acknowledgments

This project was made possible, in part, by support from the National Science Foundation grant no. IIS-2133650 (X.N.) and the National Library of Medicine grant no. 1R01LM014385-01 (X.N., D.A.). Any opinions, findings and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agency.
==== Refs
References

Chen B. ; Li C. ; Dai H. ; Song L. Retro*: Learning Retrosynthetic Planning with Neural Guided A* Search. Proceedings of the 37th International Conference on Machine Learning 2020, 1608–1616.
Genheden S. ; Thakkar A. ; Chadimová V. ; Reymond J.-L. ; Engkvist O. ; Bjerrum E. AiZynthFinder: a fast, robust and flexible open-source software for retrosynthetic planning. J. Cheminf. 2020, 12 , 70 10.1186/s13321-020-00472-1.
Seidl P. ; Renz P. ; Dyubankova N. ; Neves P. ; Verhoeven J. ; Wegner J. K. ; Segler M. ; Hochreiter S. ; Klambauer G. Improving Few- and Zero-Shot Reaction Template Prediction Using Modern Hopfield Networks. J. Chem. Inf. Model. 2022, 62 , 2111–2120. 10.1021/acs.jcim.1c01065.35034452
Tu Z. ; Coley C. W. Permutation Invariant Graph-to-Sequence Model for Template-Free Retrosynthesis and Reaction Prediction. J. Chem. Inf. Model. 2022, 62 , 3503–3513. 10.1021/acs.jcim.2c00321.35881916
Zhong Z. ; Song J. ; Feng Z. ; Liu T. ; Jia L. ; Yao S. ; Wu M. ; Hou T. ; Song M. Root-aligned SMILES: a tight representation for chemical reaction prediction. Chemical Science 2022, 13 , 9023–9034. 10.1039/D2SC02763A.36091202
Wan Y. ; Hsieh C.-Y. ; Liao B. ; Zhang S. Retroformer: Pushing the Limits of End-to-end Retrosynthesis Transformer. Proceedings of the 39th International Conference on Machine Learning 2022, 22475–22490.
Yan C. ; Ding Q. ; Zhao P. ; Zheng S. ; YANG J. ; Yu Y. ; Huang J. RetroXpert: Decompose Retrosynthesis Prediction Like A Chemist. Advances in Neural Information Processing Systems 2020, 11248–11258.
Wang X. ; Li Y. ; Qiu J. ; Chen G. ; Liu H. ; Liao B. ; Hsieh C.-Y. ; Yao X. RetroPrime: A Diverse, plausible and Transformer-based method for Single-Step retrosynthesis predictions. Chem. Eng. J. 2021, 420 , 129845 10.1016/j.cej.2021.129845.
Somnath V. R. ; Bunne C. ; Coley C. ; Krause A. ; Barzilay R. Learning Graph Models for Retrosynthesis Prediction. Advances in Neural Information Processing Systems 2021, 9405–9415.
Chen Z. ; Ayinde O. R. ; Fuchs J. R. ; Sun H. ; Ning X. G2Retro as a two-step graph generative models for retrosynthesis prediction. Commun. Chem. 2023, 6 , 1–19. 10.1038/s42004-023-00897-3.36698004
Smith M. B. Organic synthesis; Elsevier, 2017.
Dai H. ; Li C. ; Coley C. ; Dai B. ; Song L. Retrosynthesis Prediction with Conditional Graph Logic Network. Advances in Neural Information Processing Systems. 2019.
Sacha M. ; Sadowski M. ; Kozakowski P. ; van Workum R. ; Jastrzȩbski S. Molecule-Edit Templates for Efficient and Accurate Retrosynthesis Prediction arXiv 2023, http://arxiv.org/abs/2310.07313.
Seo S.-W. ; Song Y. Y. ; Yang J. Y. ; Bae S. ; Lee H. ; Shin J. ; Hwang S. J. ; Yang E. GTA: Graph Truncated Attention for Retrosynthesis. Proceedings of the AAAI Conference on Artificial Intelligence 2021, 35 , 531–539. 10.1609/aaai.v35i1.16131.
Shi C. ; Xu M. ; Guo H. ; Zhang M. ; Tang J. A Graph to Graphs Framework for Retrosynthesis Prediction. Proceedings of the 37th International Conference on Machine Learning 2020, 8818–8827.
Lin Z. ; Yin S. ; Shi L. ; Zhou W. ; Zhang Y. J. G2GT: Retrosynthesis Prediction with Graph-to-Graph Attention Neural Network and Self-Training. J. Chem. Inf. Model. 2023, 63 , 1894–1905. 10.1021/acs.jcim.2c01302.36946514
Wang Y. ; Pang C. ; Wang Y. ; Jin J. ; Zhang J. ; Zeng X. ; Su R. ; Zou Q. ; Wei L. Retrosynthesis prediction with an interpretable deep-learning framework based on molecular assembly tasks. Nat. Commun. 2023, 14 , 6155 10.1038/s41467-023-41698-5.37788995
Liu J. ; Yan C. ; Yu Y. ; Lu C. ; Huang J. ; Ou-Yang L. ; Zhao P. MARS: A Motif-based Autoregressive Model for Retrosynthesis Prediction. arXiv 2022, http://arxiv.org/abs/2209.13178.
Schwaller P. ; Petraglia R. ; Zullo V. ; Nair V. H. ; Haeuselmann R. A. ; Pisoni R. ; Bekas C. ; Iuliano A. ; Laino T. Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy. Chemical Science 2020, 11 , 3316–3325. 10.1039/C9SC05704H.34122839
You J. ; Liu B. ; Ying R. ; Pande V. ; Leskovec J. Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation. Proceedings of the 32nd International Conference on Neural Information Processing Systems; Red Hook: NY, USA, 2018; pp 6412–6422.
Zhou Z. ; Kearnes S. ; Li L. ; Zare R. N. ; Riley P. Optimization of Molecules via Deep Reinforcement Learning. Sci. Rep. 2019, 9 , 10752 10.1038/s41598-019-47148-x.31341196
Morgan H. L. The Generation of a Unique Machine Description for Chemical Structures-A Technique Developed at Chemical Abstracts Service. J. Chem. Doc. 1965, 5 , 107–113. 10.1021/c160017a018.
Gottipati S. K. ; Sattarov B. ; Niu S. ; Pathak Y. ; Wei H. ; Liu S. ; Blackburn S. ; Thomas K. ; Coley C. ; Tang J. ; Chandar S. ; Bengio Y. Learning to Navigate The Synthetically Accessible Chemical Space Using Reinforcement Learning. Proceedings of the 37th International Conference on Machine Learning 2020, 3668–3679.
Angermueller C. ; Dohan D. ; Belanger D. ; Deshpande R. ; Murphy K. ; Colwell L. Model-based reinforcement learning for biological sequence design. International Conference on Learning Representations 2020.
Chen Z. ; Min M. R. ; Guo H. ; Cheng C. ; Clancy T. ; Ning X. T-Cell Receptor Optimization with Reinforcement Learning and Mutation Polices for Precision Immunotherapy. Research in Computational Molecular Biology. Cham, 2023, 13976 , 174–191. 10.1007/978-3-031-29119-7_11.
Schreck J. S. ; Coley C. W. ; Bishop K. J. M. Learning Retrosynthetic Planning through Simulated Experience. ACS Cent. Sci. 2019, 5 , 970–981. 10.1021/acscentsci.9b00055.31263756
Yu Y. ; Wei Y. ; Kuang K. ; Huang Z. ; Yao H. ; Wu F. GRASP: Navigating Retrosynthetic Planning with Goal-driven Policy. Advances in Neural Information Processing Systems 2022, 35 , 10257–10268.
Lan Z. ; Zeng Z. ; Hong B. ; Liu Z. ; Ma F. RCsearcher: Reaction Center Identification in Retrosynthesis via Deep Q-Learning. arXiv 2023, http://arxiv.org/abs/2301.12071.
Nair A. ; Gupta A. ; Dalal M. ; Levine S. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. arXiv 2021, http://arxiv.org/abs/2006.09359.
Zhang H. ; Xu W. ; Yu H. Policy Expansion for Bridging Offline-to-Online Reinforcement Learning. arXiv 2023, http://arxiv.org/abs/2302.00935.
Brockman G. ; Cheung V. ; Pettersson L. ; Schneider J. ; Schulman J. ; Tang J. ; Zaremba W. OpenAI Gym. arXiv 2016, http://arxiv.org/abs/1606.01540.
Schwaller P. ; Laino T. ; Gaudin T. ; Bolgar P. ; Hunter C. A. ; Bekas C. ; Lee A. A. Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction. ACS Cent. Sci. 2019, 5 , 1572–1583. 10.1021/acscentsci.9b00576.31572784
Jaume-Santero F. ; Bornet A. ; Valery A. ; Naderi N. ; Alvarez D. V. ; Proios D. ; Yazdani A. ; Bournez C. ; Fessard T. ; Teodoro D. Transformer Performance for Chemical Reactions: Analysis of Different Predictive and Evaluation Scenarios. J. Chem. Inf. Model. 2023, 63 , 1914–1924. 10.1021/acs.jcim.2c01407.36952584
Sutton R. S. ; Barto A. G. Reinforcement learning: An Introduction, 2nd ed.; MIT Press, 2018.
Schneider N. ; Stiefl N. ; Landrum G. A. What’s What: The (Nearly) Definitive Guide to Reaction Role Assignment. J. Chem. Inf. Model. 2016, 56 , 2336–2346. 10.1021/acs.jcim.6b00564.28024398
Vaswani A. ; Shazeer N. ; Parmar N. ; Uszkoreit J. ; Jones L. ; Gomez A. N. ; Kaiser L. ; Polosukhin I. Attention is All you Need. Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, Long Beach, CA, USA, 2017.
Järvelin K. ; Kekäläinen J. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 2002, 20 , 422–446. 10.1145/582415.582418.
Bajusz D. ; Rácz A. ; Héberger K. Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations. J. Cheminf. 2015, 10.1186/s13321-015-0069-3.
Lowe D. M. Extraction of chemical structures and reactions from the literature. Ph.D. thesis, Apollo - University of Cambridge Repository, 2012.
Lowe D. Chemical reactions from US patents (1976-Sep2016). https://figshare.com/articles/dataset/Chemical_reactions_from_US_patents_1976-Sep2016_/5104873/1, 2017.
Coley C. W. ; Rogers L. ; Green W. H. ; Jensen K. F. Computer-Assisted Retrosynthesis Based on Molecular Similarity. ACS Cent. Sci. 2017, 3 , 1237–1245. 10.1021/acscentsci.7b00355.29296663
Yin J. ; Zhang H. ; Sun X. ; You N. ; Mou M. ; Lu M. ; Pan Z. ; Li F. ; Li H. ; Zeng S. ; Zhu F. Decoding Drug Response with Structurized Gridding Map-based Cell Representation. IEEE Journal of Biomedical and Health Informatics 2023, 1–12. 10.1109/JBHI.2023.3342280.
Rogers D. ; Hahn M. Extended-Connectivity Fingerprints. J. Chem. Inf. Model. 2010, 50 , 742–754. 10.1021/ci100050t.20426451
Schwaller P. ; Petraglia R. ; Nair V. H. ; Laino T. Evaluation Metrics for Single-Step Retrosynthetic Models. Second Workshop on Machine Learning and the Physical Sciences; NeurIPS 2019: Vancouver, Canada, 2019.
