1. Introduction {#sec1-molecules-24-02183}
===============

Covalent inhibitors have a reactive functional group, sometimes known as a warhead, which can form a chemical bond with a target. In the past decade, there has been a growing interest in the design of drugs that can form a covalent bond with drug targets \[[@B1-molecules-24-02183]\]. Nearly 30% of the marketed drugs targeting enzymes are known to act by covalent addition, and targeted covalent inhibitors (TCIs) are becoming important \[[@B2-molecules-24-02183]\]. Drugs such as Boceprevir \[[@B3-molecules-24-02183]\], which operates by nucleophilic addition, Osimertinib \[[@B4-molecules-24-02183]\] (Michael addition), and Clopidogrel \[[@B5-molecules-24-02183]\] (disulfide formation) have been approved by the Food and Drug Administration (FDA). The pharmacological advantages of covalent inhibitors are being extensively studied, and it has been shown that covalent inhibitors can achieve longer drug residence times than non-covalent inhibitors \[[@B6-molecules-24-02183]\] and improve target selectivity \[[@B7-molecules-24-02183],[@B8-molecules-24-02183]\]. New methods for TCI design have been reported \[[@B9-molecules-24-02183],[@B10-molecules-24-02183],[@B11-molecules-24-02183],[@B12-molecules-24-02183],[@B13-molecules-24-02183],[@B14-molecules-24-02183]\] and there is an increasing demand for virtual screening of TCIs \[[@B15-molecules-24-02183],[@B16-molecules-24-02183],[@B17-molecules-24-02183],[@B18-molecules-24-02183]\]. Currently, many covalent docking tools are available, and an online comparative evaluation of covalent docking tools is available. Keserű and colleagues have highlighted the key factors influencing the docking performance of the tools that have been investigated and they give guidelines for selecting the optimal combination of warheads, ligands, and tools for different systems \[[@B19-molecules-24-02183]\].

It is worth mentioning a special form of covalent bonds: metal coordination bonds. Most metalloligands not only interact with biomolecules by binding to the therapeutic targets but also influence transportation in the blood or cell in which they are engaged. Medicinal inorganic chemistry has been studied for years \[[@B20-molecules-24-02183]\] and the computational challenge for predicting possible metal-mediated binding with protein-ligand docking methods is how to cope with the formation of one or more coordination bonds \[[@B21-molecules-24-02183]\]. There are some explorative works to tackle the limitations of the protein-ligand docking methods \[[@B22-molecules-24-02183],[@B23-molecules-24-02183]\], and they did well in simulating the covalent bonds between a protein and a metal species; our work focuses on common covalent ligands.

In this paper, we examine studies of the protocols and criteria used to select a covalent docking tool. Compared with non-covalent docking, the covalent docking process is sophisticated, and to evaluate these docking tools, we focus on the following issues: (1)classification of warhead chemotypes and the targets in receptors,(2)provision of a standard unbiased benchmark data set to test covalent docking tools objectively, and(3)elaboration of criteria used to measure the performance of covalent docking tools used in a warhead-receptor complex.

Our goals are (a) to provide a general protocol to evaluate existing covalent docking tools and evaluate new tools in the future; (b) to provide benchmark data for covalent docking evaluation (BCDE); and (c) to elaborate criteria enabling measurement of the performance of covalent docking tools.

This research involved 330 covalent complexes, original covalent ligand files, and the reaction files, which show the formation of covalent bonds between warhead atoms and reactive residues. The protocols have been tested with commonly used software packages (MOE, GOLD, CovDock, and ICM-Pro). More and more covalent docking tools are becoming available, such as DOCKTITE \[[@B24-molecules-24-02183]\], AutoDock \[[@B25-molecules-24-02183]\], Dockovalent \[[@B26-molecules-24-02183]\], and CovalentDock Cloud \[[@B27-molecules-24-02183]\]. Instead of regularly evaluating the docking tools \[[@B28-molecules-24-02183],[@B29-molecules-24-02183],[@B30-molecules-24-02183]\], we developed this protocol that may be useful for selecting a covalent docking tool for a specific target. The challenges for covalent docking tools have been previously summarized \[[@B31-molecules-24-02183],[@B32-molecules-24-02183]\].

2. Results {#sec2-molecules-24-02183}
==========

2.1. Covalent Binding Reaction Classification {#sec2dot1-molecules-24-02183}
---------------------------------------------

Based on the combinations of covalent binding reaction types (nucleophilic-addition/substitution/ring-opening/disulfide formation) and binding anchor types (cysteine/serine), covalent binding reactions can be divided into seven classes, as shown in [Table 1](#molecules-24-02183-t001){ref-type="table"}. The warheads are divided into 15 classes belonging to corresponding reaction types. Each warhead has one or more cores, which are used to covalently connect ligands and receptors.

The full list of Protein Data Bank (PDB, <http://www.rcsb.org/>) accession codes of the BCDE data set and additional docking information are available in the [Supplementary Information, Table S1](#app1-molecules-24-02183){ref-type="app"}.

2.2. Evaluation of Covalent Docking Tools {#sec2dot2-molecules-24-02183}
-----------------------------------------

The top-ranked binding mode (Best Scored Pose) and the conformation with minimum root-mean-square deviation (RMSD) values with crystal pose among all docked ligand poses (Best Sampled Pose) were used to evaluate the docking tools. The Best Sampled Pose might be a metric with little difference between tested tools because the sampling method is more developed than with the scoring function. This work was conducted based on the fact that covalent bonds are predefined and most scoring functions typically neglect the entire contribution from covalent bond formation. Specifically, we are interested in the maximal deviated pose for a comprehensive inspection of the evaluation. This measure has been verified to confirm the quality of the produced docking predictions \[[@B33-molecules-24-02183],[@B34-molecules-24-02183]\]. For each of the 330 binding modes retained in the benchmark data set, the RMSD of three poses from the reference binding mode was determined ([Figure 1](#molecules-24-02183-f001){ref-type="fig"}), which reveals that all docking tools were able to find a near-native pose (RMSD at most 2.0 Å) for 144\~187 of the 330 best scored poses and 203\~267 of the 330 best sampled poses. In terms of the maximal deviated pose, ICM-Pro has more frequencies among the large region (RMSD \> 2.0 Å) compared with other tools. The median and maximum RMSD distributions with RMSDs of 5.38 Å and 14.20 Å, respectively, show a distinct difference with the two poses. All simulations demonstrate that the employed docking tools generated qualified data to provide a reasonable test scenario for our work.

All docking RMSD values are listed in [Table S1 in the Supplementary Information](#app1-molecules-24-02183){ref-type="app"}, and computational costs in terms of docking time as well as disk space are listed in [Figure S1](#app1-molecules-24-02183){ref-type="app"}. The median RMSD calculations for every protein are listed in [Figures S2 and S3](#app1-molecules-24-02183){ref-type="app"}. Three criteria (precision, generality, and robustness) were introduced to evaluate a covalent docking tool for a given target. More discussions on "precision, generality, and robustness" can also be found in [Section 4.2](#sec4dot2-molecules-24-02183){ref-type="sec"}. Precision. *P*(*i*,*x*), the precision for the *i*th docking tool (*i* ∈ MOE, GOLD, CovDock, ICM-Pro) using the *x*th RMSD measurement *x* ∈ (Best Scored Pose, Best Sampled Pose), measured for each covalent docking tool demonstrates different docking performances for different reaction classes and warheads. *P*(*i*,*x*) can be calculated with formula (1) as follows:$$P\left( {i,x} \right) = Median_{j = 1}^{m}\left( {Median_{k}^{n}\left( {RMSD_{i,x,j,k}} \right)} \right)$$ where m is the total number of warhead types or the total number of receptor types, and n is the total number of co-crystal structures for a given warhead type, or a given receptor type. RMSD *i*,*x*,*j*,*k* is the RMSD value between the crystallographic ligand and the docked ligand pose produced by the *i*th docking tool with *x*th measurement for the *k*th co-crystal structure of the *j*th warhead type (the *j*th receptor type). Thus, a lower *P*(*i*,*x*) is better.

As shown in [Table 2](#molecules-24-02183-t002){ref-type="table"}, the heat maps (red = worst and green = best) show the RMSD medians of the top-ranked poses. With nitrile (Cys), for example, GOLD seems better than other tools with both precision criteria. Using the medians as overall docking precision measures that cross the reaction classes (the last line in [Table 2](#molecules-24-02183-t002){ref-type="table"}), MOE is better than other tools in terms of the Best Scored Pose (Median RMSD = 1.94 Å), and CovDock seems better than other tools in terms of the Best Sampled Pose (Median RMSD = 1.33 Å). We recognize that some warheads, such as alkyne, guanyl, nitrile (Ser), phosphonyl, and lactone appear no more than three times in the current RCSB PDB database. The evaluation protocol reported here can be repeated in the future when more instances are reported in the RCSB PDB database, or more covalent docking tools must be evaluated.

[Table 3](#molecules-24-02183-t003){ref-type="table"} demonstrates the relationships between the covalent tools and receptor types. With hydrolase, for example, CovDock seems better than other tools with both precision criteria. Using the medians as overall docking precision measures that cross the reaction classes (the last line at [Table 3](#molecules-24-02183-t003){ref-type="table"}), CovDock is better than other tools in terms of the Best Scored Pose (Median RMSD = 1.71 Å); and GOLD seems better than other tools in terms of the Best Sampled Pose (Median RMSD = 1.17 Å).

Generality. Generality is measured for the overall precision across all types of receptors or warheads for a given tool. The G (*i*,*x*), generality of the *i*th docking tool with the *x*th RMSD measurement can be calculated using the following formulas:$$G\left( {i,x} \right)_{receptor~type} = \sum\limits_{j = 1}^{m_{1}}\left( \frac{\sum_{l = 1}^{n}\left( k_{i,j,l,x} \right)_{~}}{N_{i,j}} \right)_{i} \times 100$$ $$G\left( {i,x} \right)_{warhead~type} = \sum\limits_{j = 1}^{m_{2}}\left( \frac{\sum_{l = 1}^{n}\left( k_{i,j,l,x} \right)_{~}}{N_{i,j}} \right)_{i} \times 100$$ where *G*(*i*,*x*) is the generality score for the *i*th docking tool using the *x*th RMSD measurement *x* ∈ (Best Scored Pose, Best Sampled Pose); *i* is for the *i*th docking tool (in this study, *i* ∈ MOE, GOLD, CovDock, ICM-Pro); *n* is the number of uniportIDs or warhead-cores; *m*~1~ is the number of receptor types tested; *m*~2~ is the number of the warhead types tested; *N~j~* is the number of total co-crystal structures for a given receptor type or a given warhead type used to evaluate the *i*th docking tool; and *k*~*i*,*j*,*l*,*x*~ is the number of the best median RMSD values among all the docking tools for the *i*th tool, the *j*th receptor type (or the *j*th warhead type), and the *l*th uniportID (or the *l*th warhead core-structure), measured with the *x*th method. The detailed calculations are listed in [Figures S1 and S2 and Tables S2--S9](#app1-molecules-24-02183){ref-type="app"}. Thus, a higher G (*i*,*x*) is better. The G (*i*,*x*) for the four docking tools based on receptor types are shown in [Figure 2](#molecules-24-02183-f002){ref-type="fig"}.

In terms of the overall generalities across all types of receptors for covalent docking tools, MOE appears to deliver a superior performance. However, the generality analyses based on warhead types show that Schrödinger's CovDock has a significantly better performance ([Figure 3](#molecules-24-02183-f003){ref-type="fig"}).

Mann-Whitney U-test results show that the above analyses are statistically significant (*p* \< 0.05). The detailed U-test results are listed in [SI Tables S10 and S11](#app1-molecules-24-02183){ref-type="app"}. As shown in [Table S8](#app1-molecules-24-02183){ref-type="app"}, the results of the analyses using Best Sampled Pose are not statistically significant. Therefore, we suggest using the Best Scored Pose method to evaluate docking tools.

Combining the results in [Figure 2](#molecules-24-02183-f002){ref-type="fig"} and [Figure 3](#molecules-24-02183-f003){ref-type="fig"}, we observed that MOE consistently produces superior docking results. However, if the generalities are based on warhead types, Schrödinger's CovDock is superior to other docking tools. **Robustness**. The robustness of a covalent docking tool is measured by the standard deviation of all RMSDs between native ligand poses and the docked poses covering all tested covalent binding complexes in the BCDE data set. The robustness analysis results for the four docking tools are depicted in [Figure 4](#molecules-24-02183-f004){ref-type="fig"}.

Based on [Figure 4](#molecules-24-02183-f004){ref-type="fig"}, we observed that ICM-Pro has the largest RMSD standard deviation, i.e., the lowest robustness, and MOE has the narrowest RMSD standard deviation, or the most robust character.

This study found that the overall performances of CovDock and MOE are similar, and the accuracy of docking is slightly better than obtained with GLOD and ICM-Pro, the other two tools with similar performances. The precise values vary depending on whether the subject is a receptor or a ligand. We found that different docking tools have different abilities when dealing with different targets and different warheads. In the best scored pose and the best sampled pose, the generality of CovDock is 13, that is, when processing 13 kinds of receptors, it is possible to obtain a lower RMSD than the other three tools, and these 13 kinds account for 41% of the entire effective target set. GOLD differs greatly for different targets, and gives better performance than other software on the three targets of Cathepsin S (CATS), Replicase polyprotein lab (REP), and Penicillin-binding protein A (PBPA) even though it performs slightly more poorly on the complete target set.

2.3. Profiling Covalent Docking Tools {#sec2dot3-molecules-24-02183}
-------------------------------------

In order to comprehensively estimate the capacity to produce a correct covalent binding pose for a docking tool, we investigated the following parameters: **Pose deviation and success and failure counts**: For a co-crystal structure, a docking tool can generate two RMSD values, Best Scored Pose, and Best Sampled Pose, denoted S~1~ and S~2~, respectively. Also, (S~1~--S~2~) is denoted as P-deviation, representing the difference between the two measurements of docking accuracies. For a given docking tool, if (S~1~ ≤ 2.0 Å) then it is considered as a success count (S-count); If (S~1~ \> 2.0 Å) and (S~2~ \> 2.0 Å) then it is considered as a failure count (F-count). The P-deviations, S-counts, and F-counts for the four evaluated docking tools have been computed and are listed in [Table S13](#app1-molecules-24-02183){ref-type="app"}.**P-deviation distributions**. These distributions for all docking tools can be used to measure docking error ranges of the tools as shown in [Figure 5](#molecules-24-02183-f005){ref-type="fig"} (the data are listed in [Table S14](#app1-molecules-24-02183){ref-type="app"}). MOE (green) and CovDock (red) are closer to Gaussian distribution. GOLD (blue) and ICM-Pro (black) have two significant waves around 2Å and 4.5 Å suggesting that these tools have extraordinary errors around the 2Å and 4.5 Å ranges.**Parameters with which to profile a docking tool**. While analyzing the docking results, we revealed four parameters that can characterize a docking tool: (a) S-Only count---the number of cases in which only one docking tool predicted the successful poses, (b) F-Only count---the number of cases in which only one docking tool predicted the failing poses, (c) S \> 2---the number of cases in which at least two docking tools predicted the successful poses, and (d) F \> 2---the number of cases in which at least two docking tools predicted the failing poses. These parameters were derived from [Table S13](#app1-molecules-24-02183){ref-type="app"}, and are listed in [Table S15](#app1-molecules-24-02183){ref-type="app"}. The profiles for the four docking tools are depicted in [Figure 6](#molecules-24-02183-f006){ref-type="fig"}.

In summary, there is a greater possibility for MOE to consistently produce precise and robust covalent docking results for many targets. However, CovDock can be better than MOE if the generality is for various warhead types, although its robustness is not as good as that of MOE. Based on the profiling analytic results in [Figure 6](#molecules-24-02183-f006){ref-type="fig"}, it can be concluded that MOE can predict the most correct binding poses while other tools fail, and can predict the least number of failing poses while other tools can correctly predict more failing poses. Both CovDock and MOE can predict the most correct binding poses while at least one other tool can do just as well (the S \> 2 column in [Figure 6](#molecules-24-02183-f006){ref-type="fig"}), although CovDock is significantly better than MOE. MOE, again, predicts the least failure poses while other tools perform equally as poorly (the F \> 2 column in [Figure 6](#molecules-24-02183-f006){ref-type="fig"}).

These studies are based on a test set of existing data, assessed on 20 December 2017, and current versions of the docking tools. We believe the observations may change over time along with the technology improvements and the increasing size of the BCDE data set. We will update the BCDE database when possible.

3. Discussion {#sec3-molecules-24-02183}
=============

Keserű and colleagues' paper is for comparatively evaluating covalent docking tools. Our work, however, focuses on proposing a protocol to select a suitable covalent docking tool for a specific target. Our intention is not to generally rank existing covalent docking tools, because each docking tool can be more favorable to a set of specific targets. It is commonly accepted that there is no docking tool that is superior to the other tools for all protein targets.

The parameters for performance measures in [Table 2](#molecules-24-02183-t002){ref-type="table"} and [Table 3](#molecules-24-02183-t003){ref-type="table"} and [Figure 4](#molecules-24-02183-f004){ref-type="fig"}, [Figure 5](#molecules-24-02183-f005){ref-type="fig"} and [Figure 6](#molecules-24-02183-f006){ref-type="fig"}, and [Figure 8](#molecules-24-02183-f008){ref-type="fig"} were only to demonstrate the process for selecting a covalent docking tool with a specific data set, not for generally ranking the docking tools.

3.1. Case Studies of Abnormal Docking Results {#sec3dot1-molecules-24-02183}
---------------------------------------------

Usually, the methods of Best Scored Pose and Bets Sampled Pose produce consistent results, however, they can produce extraordinarily different results. Here, we discuss two case studies.

### 3.1.1. Case Study 1 {#sec3dot1dot1-molecules-24-02183}

A hepatitis C virus NS3/4A protease co-crystal structure (PDB code: 1RTL) \[[@B35-molecules-24-02183]\] has a ligand covalently bound at Ser139. The four docking tools produced poor docking results, with the exception of CovDock with its Best Sampled Pose method (1.32Å). We found that only CovDock generated the ligand conformation that was closest to the crystallographic ligand pose. This pose helped two urea NHs of the ligand to form two hydrogen bonds at Ala157, which is critical for the ligand's pyrrolidine-5, 5-trans-lactam to open for covalent binding with Ser139. As shown in [Figure 7](#molecules-24-02183-f007){ref-type="fig"}, only CovDock recognized this near-native conformation in Best Scored Pose. As indicated in [Figure 7](#molecules-24-02183-f007){ref-type="fig"}, only CovDock generated a Best Scored Pose, in which the urea NHs are close to Ala157, and maintained the correct pose to bind covalently to Ser139. Other poses generated by MOE, GOLD, and ICM-Pro failed to keep the urea group close to Ala157 and form the critical hydrogen bonds. This case study confirms that one should consider both *Best Score Pose* and Best Sampled Pose measurements while evaluating the docking tools. In this case, even the *Best Score Pose* measurement failed, but the correct answer can still be reached by checking the Best Sampled Pose measurement.

### 3.1.2. Case Study 2 {#sec3dot1dot2-molecules-24-02183}

As shown in [Figure 8](#molecules-24-02183-f008){ref-type="fig"}, GOLD, CovDock, and ICM-Pro have significant deviations between Best Scored Pose and Best Sampled Pose measurements when docking GH4, the ligand of Cathepsin L, into the co-crystal structure (PDB code: 5MQY) \[[@B36-molecules-24-02183]\]. They all produce correct binding poses using the Best Sampled Pose measurement but failed with the Best Scored Pose measurement. MOE, however, was able to generate correct poses with both measurements. Further analysis indicated that the failed poses were due to the two rings (rings A and B) at the tertiary of the fluoropyrimidine nitrile that exchange the sub-binding pockets. The corrected docking poses should be ring-A for S3, and ring-B for S2. Only MOE docked the rings into the correct sub-binding pockets with both measurements. Although the favorable interactions of the ring-B in the aromatic cleft of the S2 pocket were identified in the docking position, the pose was not selected as the highest scoring conformation due to the potential hydrophobic collapse. Earlier results in the Cambridge Structural Database (CSD) showed that the ring-B attached through the unsubstituted propyl linker accounted for 27% of the collapsed conformation in all the crystal structures with this motif. As the ligand needs to bind to hCatL in an extended conformation, an energy penalty is involved in disrupting the collapsed conformation.

3.2. Binding Pocket Features and Docking Performance {#sec3dot2-molecules-24-02183}
----------------------------------------------------

Specific structural features of a binding site, such as the location of an anchor residue, binding pocket size, and the flexibility of the binding pocket can be important for covalent docking. For example, there are 61 covalent kinase inhibitors (CKIs) in the BCDE data set (see [Table S16](#app1-molecules-24-02183){ref-type="app"}).

Cysteine is an anchor residue for kinases. This covalent anchor can be found in as many as eight domains: Remote-Cys, Beta4-4, Catalytic-3, Hinge, DFG-3, Extended front pocket, Front pocket, and P-loop. The possibility of the anchor residing at one particular domain, however, can vary. Based on our BCDE data set, the distribution of cysteine across kinase domains is depicted in [Figure 9](#molecules-24-02183-f009){ref-type="fig"}. The top three possible domains in which the anchor resides are Front pocket (31), DFG-3 (14), and P-loop (9). These domains offer greater possibilities for a cysteine to be used for covalent inhibitor design \[[@B37-molecules-24-02183]\].

The front pocket is exposed to a polar environment suitable for the reaction and provides sufficient open space for the movement of the nucleophilic group. In contrast, the hinge region is not easily accessed by an electrophilic group of a ligand due to steric effects. The ligand has difficulty in accessing the thiol group of the cysteine, and it is hard for a covalent reaction to take place. Our results demonstrate that a docking tool can have better performance in the front pocket than in the hinge area ([Figure 10](#molecules-24-02183-f010){ref-type="fig"}). This observation is consistent with the work reported by Zhao et al. \[[@B38-molecules-24-02183]\].

The cysteine near the DFG-3 domain is close to the polar area, while in the P-loop, it is the β carbon of the side chain and the thiol of cysteine that mainly contribute to the hydrophobic interaction and then help with the covalent attachment to a ligand. All docking tools have poorer performances in DFG, P-loop, and hinge domains. All tools have unstable performances at other domains, and this may be due to the lack of enough co-crystal structures for the evaluation.

The flexibility of the domain hosting an anchor residue, in this case, cysteine or serine, affects the docking performance. The flexibility can be measured by the solvent accessible surface area (SASA) and the dihedral angle of N--CA--CB--SG/OG \[[@B39-molecules-24-02183]\]. The relations of the docking accuracy (measured in Best Scored Pose) and SASA and the dihedral angle are depicted in [Figure 11](#molecules-24-02183-f011){ref-type="fig"}a,b.

From [Figure 11](#molecules-24-02183-f011){ref-type="fig"}a, we can observe that all docking tools can produce the ligand poses within acceptable accuracies (RMSD ranging from 2.5 to 1.5 Å) when SASA is less than 10 Å^2^, meaning the anchor residue is less flexible. However, the performances of docking tools will behave differently when the flexibility of the anchor residue increases. The performances of CovDock and GOLD fluctuate significantly, but MOE and ICM-Pro are not significantly disturbed by the SASA increase.

In contrast, the dihedral angle can change the performance of all the docking tools except MOE, as shown in [Figure 11](#molecules-24-02183-f011){ref-type="fig"}b. Interestingly, both MOE and CovDock perform well at −180°--150°. All tools have poor results at −120°--90° and 120°--150°, which may be due to the absence of evaluation data.

4. Materials and Methods {#sec4-molecules-24-02183}
========================

4.1. Benchmark Data for Covalent Docking Evaluation (BCDE) {#sec4dot1-molecules-24-02183}
----------------------------------------------------------

The data curation process used for BCDE was as follows: (1) 191 UniportIDs for covalent binding targets were derived from the open literature and the cBindingDB database \[[@B42-molecules-24-02183]\], (2) the UniportIDs were used to query the RCSB PDB database, resulting in 6,584 PDB entries on 20 December 2017; (3) these PDB entries were filtered, and only co-crystal complexes with resolutions \<3.0 Å were retained; (4) these co-crystal complexes were then examined for ligands non-covalently bound to their protein targets, and buffers, components, cofactors, metals, and saccharides were excluded. As a result, the number of remaining co-crystal structures was 2,235, and these were further processed using the following filters:DNA complexes excludedproteins that have covalent bindings with ligands at cysteine and serine residues includedligand and receptor connected with a single bond includedco-factors within 8 Å to the ligand excludedligands with at least 5 non-hydrogen atoms and less than 30 rotatable bonds were included

The BCDE data set obtained according to these criteria consisted of 330 qualified protein--ligand complexes, which contain diverse warhead species and binding geometries. The 330 structurally unique ligands covalently targeted 104 proteins ([Figure 12](#molecules-24-02183-f012){ref-type="fig"}).

To ensure the BCDE data quality, we inspected every co-crystal structure complex with Discovery Studio 4.5. For non-monomer complexes, we retained only those with one chain with covalent linkages. Water molecules, metals, co-factors, and crystallization agents were removed from target structures. In order to prevent any bias from the use of coordinates present in the protein--ligand X-ray crystal structures, all the ligands were derived from the complex and restored to the structure before the covalent bond was formed (this is a re-docking process). The ligands were saved in different files in MOL/MOL2/SDF formats. To make a covalent docking, this tool objectively docks the ligands back to the targets, and the original coordinates of the ligands are zeroed. The entire data set is available at <http://www.rcdd.org.cn/cbinderdb/>.

The programs with functionalities that could support covalent docking are restricted to sampling ligand conformations under the constraint provided by a predefined bond between a ligand and a protein. Since the warhead chemistry associated with the covalent ligands involves different reaction energetics and product geometries, the chemical moieties of the ligands in the BCDE data were individually inspected and corrected when necessary.

4.2. Criteria for Evaluating Covalent Binding Tools {#sec4dot2-molecules-24-02183}
---------------------------------------------------

Criteria were established to evaluate the docking performance of covalent docking tools, and three aspects of each docking tool, "precision", "generality", and "robustness" were considered. Each metric here corresponds to a typical practical application of docking tools, e.g., reproducing co-crystallized conformation with precision, or covering targets and reactions as much as possible. The concept of each metric is explained below. **Precision**. This refers to the capacity of a covalent docking tool to reproduce the experimental binding mode, which is represented by the RMSD value of a calculated binding conformation and experimental data. The prediction accuracy was expressed in terms of the RMSD values of the ligand heavy atoms from the conformation co-crystallized in the PDB entry.**Generality**. This refers to the capacity of a covalent docking tool to encompass different covalent binding systems. It is a measure of the overall precision across protein targets and reactions of all kinds. We marked the minimum RMSD among these tools for a single object then counted the number of marks in each tool. The goal was to find a covalent docking tool that is sufficiently general to be able to cope with most covalent systems.**Robustness**. This refers to the consistency of a covalent docking tool with the same docking process. Notwithstanding different structures of the same target or different targets of the same family or different families, there should be no unreasonable errors in the docking results. We calculated the average and deviation of the RMSD given by each docking tool in the whole data set.

4.3. Protocol for Selecting Tools {#sec4dot3-molecules-24-02183}
---------------------------------

There should be an instructive protocol to select the most appropriate covalent docking tool. We created a flow chart ([Figure 13](#molecules-24-02183-f013){ref-type="fig"}) that helps researchers choose tools. First, it was necessary to generate our own standard data set. To do this, we had to collect all PDB files for our own covalent system, dispose of proteins and ligands with specialized software and, if necessary, prepare a defined reaction file according to the warhead reaction. Second, we examined the docking with candidate covalent docking tools. Finally, we compared and evaluated the results using measurement criteria and selected the most appropriate tool for subsequent research. This is a guide to selection of covalent docking tools.

4.4. Docking Programs and Scoring Functions {#sec4dot4-molecules-24-02183}
-------------------------------------------

In this study, we utilized the docking programs MOE, GOLD, Schrodinger, and ICM-Pro for an evaluation of their ability to reproduce the co-crystallized covalent binding mode. In addition to GOLD's implemented scoring functions (GoldScore), it has been optimized for the prediction of binding positions, which is consistent with our purpose; the other systems use a default scoring function.

MOE version 2016.08 was licensed from Chemical Computing Group ULC. The DOCK module of MOE (Molecular Operating Environment) \[[@B43-molecules-24-02183]\] achieves conformational sampling by Placement methodology. The Placement phase of Dock generates poses from ligand conformations. Specifically, poses are generated by the reaction or transformation methodology of the Combinatorial Library. We customized reactions for those not included in the predefined reaction library using MarvinSketch \[[@B44-molecules-24-02183]\]. The complex PDB file, which represents the receptor, was prepared using default parameters. The SDF file can help users to repair the pre-reaction structure of the ligand. The GBVI/WSA ΔG is a force field-based scoring function \[[@B45-molecules-24-02183]\] that estimates the free energy of binding of the ligand from a given pose. Ligand poses were ranked according their binding free energy.

GOLD (Genetic Optimization for Ligand Docking) version 1.8.2 was licensed from Cambridge Crystallographic Data Centre, Cambridge, UK. It is a genetic algorithm for docking flexible ligands into protein binding sites. It is one of the first and most widely applied programs that uses docking with a preformed bond \[[@B46-molecules-24-02183],[@B47-molecules-24-02183]\]. The program assumes that there is just one atom linking the ligand to the protein, which in this project was either S or O. Protein and ligand files were set up with the link atom identified, and appear in both the protein or ligand input files. The complex PDB file and Mol2 file, which present receptor and ligand, respectively, were prepared with default parameters. Then, the link atom in the ligand was forced to attach to the link atom in the protein. The correct geometry of the bond between ligand and protein was ensured with an angle-bending potential from the Tripos Force Field \[[@B48-molecules-24-02183]\] added to the fitness function. On evaluating the score for the docked ligand, the angle-bending energy for the link atom was included in the calculation of the fitness score, which is the negative of the sum of the component energy terms. As a consequence, larger fitness scores are better.

CovDock was is included in Maestro version 2018.2, which was licensed from Schrodinger. LLC. CovDock is a workflow developed by Schrödinger for pose prediction and scoring of covalently bound ligands. It was built upon a foundation of the well-tested Glide \[[@B49-molecules-24-02183]\] docking algorithm and Prime \[[@B50-molecules-24-02183]\] structure refinement methodology for accurate prediction of non-covalently docked poses. There are two protocols for covalent docking. "Pose Prediction" mode \[[@B44-molecules-24-02183]\], although time-consuming, is suitable for lead-optimization projects. Due largely to reduced sampling, the "CovDock-VS" mode \[[@B51-molecules-24-02183]\] is 10--40 times faster than the "Pose Prediction" mode. According to our research purpose, the "Pose Prediction" mode was selected. The complex PDB file and SDF file that, respectively, present the receptor and the ligand, were prepared using default parameters. The ligand functional group was represented by a SMARTS pattern \[[@B51-molecules-24-02183]\] whose atomic index identifies the atom that forms a bond with the acceptor. When the ligands were docked, all modes were tested to match the ligand. A unique match on a single ligand was considered a reasonable match and a covalent bond was formed. Affinity score was calculated as the average between the GlideScore for the pre-reaction noncovalent docking pose and the in-place score for the post-reaction pose. All docking projects were completed on the Tianhe Starlight platform of the NSCC-GZ (National Supercomputer Center in Guangzhou).

ICM-Pro version 3.8 was licensed from MolSoft LLC. Molsoft LLC developed an efficient methodology, referred to as ICM \[[@B52-molecules-24-02183]\], which employs a Monte Carlo minimization algorithm and is based on a ligand description in internal coordinates. At first, according to the reaction type (RXN files in our data set), the warhead of the ligand was linked with the side-chain of cysteine or serine. During the conformational sampling process, the method can make a random step in a Monte Carlo procedure when armed with knowledge of the side-chain torsion angles. This biased probability Monte Carlo (BPMC) procedure \[[@B53-molecules-24-02183]\] randomly selects the subspace first, then takes a step to a new random position independent of the previous position. All the stereoisomers in the chemical table are docked and evaluated by the ICM scoring function \[[@B54-molecules-24-02183]\], and docking solutions are ranked by the score, which is usually a function of different energy terms based on a force-field. The lower the ICM score, the higher the chance the ligand can bind successfully.

In general, for the same protein, the calculation time increases as the number of ligand stereoisomers increases, so the number of stereoisomers was arbitrarily limited to 10. We choose the final 10 docking poses for further analysis.

5. Conclusions {#sec5-molecules-24-02183}
==============

In this study, we report a standard test data set BCDE, which fills a gap in the collection of benchmark data sets for assessing the performance of covalent docking tools. We also propose three quantitative measures (precision, generality, and robustness) for validating a covalent docking tool for protein---ligand covalent binding studies. Four profiling parameters were suggested for cross-validating covalent docking tools. Parameters such as the size and flexibility of the protein pocket, the electrophilic warhead, the chemical reactions, and the physical and chemical properties of a molecule impacting on covalent docking performance have been investigated. The performance of an ideal covalent docking tool should be independent of receptor class, warhead type, and reaction mechanisms. Our recommendations for selecting a covalent binding tool are based on the precision, generality, and robustness measures from validating the docking suites with a standard data set.

This work was based on an unevenly distributed data set in terms of receptor classifications, warhead types, and reaction mechanisms. For example, cysteine-related covalent binding data are abundantly reported but other types of covalent bindings are inadequately reported. The BCDE data should be updated from time to time to be more useful in covalent binding studies.

We would like to thank Shimin Su, Yuyao Yang, Liangyue Wang, and Lizhao Hu for participating in the preparation and evaluation of data and docking studies for this project.

The following are available online, Table S1: PDB codes, uniprotIDs, information and RMSD values for complexes included in the Cov, Figure S1: Some computional costs in this work, Figure S2: Median RMSD value of Best Scored Pose on each target, Table S2: The number of best RMSD value cross all the docking tools for a given tool based on the receptor type for Best Scored Pose, Table S3: The generalities calculated by the generality(G(i,x)) formula for four docking tool with Best Scored Pose measurement based on the receptor type, Figure S3: Median RMSD value of Best Sampled Pose on each target, Table S4: The number of best RMSD value cross all the docking tools for a given tool based on the receptor type for Best Sampled Pose, Table S5: The generalities calculated by the generality(G(i,x)) formula for four docking tool with Best Sampled Pose measurement based on the receptor type, Table S6: The number of best RMSD value cross all the docking tools for a given tool based on the warhead type for Best Scored Pose, Table S7: The generalities calculated by the generality(G(i,x)) formula for four docking tool with Best Scored Pose measurement based on the warhead type, Table S8: The number of best RMSD value cross all the docking tools for a given tool based on the warhead type for Best Sampled Pose, Table S9: The generalities calculated by the generality(G(i,x)) formula for four docking tool with Best Sampled Pose measurement based on the warhead type, Table S10: Mann-Whitney U-test results of the difference between the Best Scored Pose and Best Sampled Pose RMSD sets on each of four docking tools, Table S11: Mann-Whitney U-test results of the difference between RMSD sets of the docking tools on Best Scored Pose and Best Sampled Pose on measurement, Table S12: The robustness analysis results for the four docking tools, Table S13: The P-deviations, S-counts, and F-counts for docking tools, Table S14: The P-deviations distributions for all docking tools, Table S15: Parameters to profile a docking tool, Table S16: Kinase domain distribution in BCDE and their PDB id.

###### 

Click here for additional data file.

Conceptualization, J.X., C.W., J.D. and H.Z.; methodology, C.W.; software/computing resource, Y.L.; validation, C.W., X.Y. and J.X.; formal analysis, C.W.; investigation, C.W.; resources, D.W. and Y.L.; data curation, J.D.; writing---original draft preparation, C.W.; writing---review and editing, J.X.; visualization, J.D.; supervision, J.X.; project administration, X.Y., Q.G. and H.Z.; funding acquisition, J.X., D.W. and Y.L.

This work has funded in part by the National Key R&D Program of China (2017YFB02034043), the science & technology program of Guangzhou (201604020109), GD Frontier & Key Techn. Innovation Program (2015B010109004), GD-NSF (2016A030310228), and Guangdong Provincial Key Lab. Of Construction Foundation (2011A060901014).

The authors declare no conflict of interest.

![Stacked bar chart of docking performance over all 330 reference binding modes. The frequency of all RMSDs from the crystal ligand among best scored pose (**a**), best sampled pose (**b**), and maximal deviated pose (**c**).](molecules-24-02183-g001){#molecules-24-02183-f001}

![The generalities for the four docking tools based on receptor types.](molecules-24-02183-g002){#molecules-24-02183-f002}

![The generalities for the four docking tools based on warhead types.](molecules-24-02183-g003){#molecules-24-02183-f003}

![The robustness analysis results for the four docking tools. The details are listed in [Table S12](#app1-molecules-24-02183){ref-type="app"}.](molecules-24-02183-g004){#molecules-24-02183-f004}

![The P-deviation distributions for all docking tools.](molecules-24-02183-g005){#molecules-24-02183-f005}

![The profiles for the four docking tools.](molecules-24-02183-g006){#molecules-24-02183-f006}

![Ligand CPX covalently docked into NS3/4A protease (1RTL) with the four docking tools. Yellow: Best Scored Pose; Cyan: Best Sampled Pose; Green: Native Ligand Pose. S~1~ is the RMSD between the native pose and the Best Score Pose; S~2~ is the RMSD between the native pose and the Best Sampled Pose.](molecules-24-02183-g007){#molecules-24-02183-f007}

![GH4 covalently docked into Cathepsin L (5MQY) with the four docking tools. Yellow: Best Scored Pose; Cyan: Best Sampled Pose; Green: Native Ligand Pose. S~1~ is the RMSD between the native pose and the Best Score Pose; S~2~ is the RMSD between the native pose and the Best Sampled Pose.](molecules-24-02183-g008){#molecules-24-02183-f008}

![The distribution of kinase domains in the data set.](molecules-24-02183-g009){#molecules-24-02183-f009}

![Impact of the position of reactive cysteine residue on the docking performance.](molecules-24-02183-g010){#molecules-24-02183-f010}

![The relations of the docking accuracy and the solvent accessible surface area (SASA) or the dihedral angle. (**a**) Relation of the SASA of reactive residue and docking accuracy. SASA was calculated with GETAREA \[[@B40-molecules-24-02183]\] (<http://curie.utmb.edu/getarea.html>). (**b**) The relation of the dihedral angle and the docking accuracy. The dihedral angle of the reactive residue was calculated with R package Bio3D \[[@B41-molecules-24-02183]\].](molecules-24-02183-g011){#molecules-24-02183-f011}

![The distribution of protein types in the BCDE data set. The proteins are grouped by EC (Enzyme Commission) classification codes. All the uniprotIDs for the 330 proteins are listed in [Table S1](#app1-molecules-24-02183){ref-type="app"}.](molecules-24-02183-g012){#molecules-24-02183-f012}

![Flowchart for selecting covalent docking tools.](molecules-24-02183-g013){#molecules-24-02183-f013}

molecules-24-02183-t001_Table 1

###### 

Covalent binding reaction classification and warhead classes.

  Reaction Class                     Reaction Type                      Warhead Class                      Warhead Core                       Example                            Frequency \*
  ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- --------------
  Nucleophilic Addition              Nucleophilic Addition by Cys       Nitrile(cys)                       ![](molecules-24-02183-i001.jpg)   ![](molecules-24-02183-i002.jpg)   42
  Alkene(cys)                        ![](molecules-24-02183-i003.jpg)   ![](molecules-24-02183-i004.jpg)   84                                                                    
  ![](molecules-24-02183-i005.jpg)   ![](molecules-24-02183-i006.jpg)   18                                                                                                       
  ![](molecules-24-02183-i007.jpg)   ![](molecules-24-02183-i008.jpg)   15                                                                                                       
  Carbonyl(cys)                      ![](molecules-24-02183-i009.jpg)   ![](molecules-24-02183-i010.jpg)   13                                                                    
  ![](molecules-24-02183-i011.jpg)   ![](molecules-24-02183-i012.jpg)   9                                                                                                        
  ![](molecules-24-02183-i013.jpg)   ![](molecules-24-02183-i014.jpg)   3                                                                                                        
  Alkyne                             ![](molecules-24-02183-i015.jpg)   ![](molecules-24-02183-i016.jpg)   1                                                                     
  Guanyl                             ![](molecules-24-02183-i017.jpg)   ![](molecules-24-02183-i018.jpg)   2                                                                     
  Nucleophilic Addition by Ser       Nitrile (ser)                      ![](molecules-24-02183-i019.jpg)   ![](molecules-24-02183-i020.jpg)   3                                  
  Carbonyl (ser)                     ![](molecules-24-02183-i021.jpg)   ![](molecules-24-02183-i022.jpg)   3                                                                     
  ![](molecules-24-02183-i023.jpg)   ![](molecules-24-02183-i024.jpg)   26                                                                                                       
  Boric acid                         ![](molecules-24-02183-i025.jpg)   ![](molecules-24-02183-i026.jpg)   8                                                                     
  Nucleophilic Substitution          Nucleophilic Substitution (cys)    Halide                             ![](molecules-24-02183-i027.jpg)   ![](molecules-24-02183-i028.jpg)   13
  ![](molecules-24-02183-i029.jpg)   ![](molecules-24-02183-i030.jpg)   2                                                                                                        
  Others                             ![](molecules-24-02183-i031.jpg)   ![](molecules-24-02183-i032.jpg)   13                                                                    
  ![](molecules-24-02183-i033.jpg)   ![](molecules-24-02183-i034.jpg)   3                                                                                                        
  Nucleophilic Substitution by Ser   Phosphonyl                         ![](molecules-24-02183-i035.jpg)   ![](molecules-24-02183-i036.jpg)   3                                  
  Ring Opening                       Ring Opening by Cys                Heterocyclic                       ![](molecules-24-02183-i037.jpg)   ![](molecules-24-02183-i038.jpg)   11
  ![](molecules-24-02183-i039.jpg)   ![](molecules-24-02183-i040.jpg)   6                                                                                                        
  Ring Opening by Ser                Lactam                             ![](molecules-24-02183-i041.jpg)   ![](molecules-24-02183-i042.jpg)   39                                 
  ![](molecules-24-02183-i043.jpg)   ![](molecules-24-02183-i044.jpg)   1                                                                                                        
  Lactone                            ![](molecules-24-02183-i045.jpg)   ![](molecules-24-02183-i046.jpg)   2                                                                     
  Disulfide Formation                Disulfide Formation                Sulfydryl                          ![](molecules-24-02183-i047.jpg)   ![](molecules-24-02183-i048.jpg)   10

\* Number of co-crystal structures collected in the benchmark data for covalent docking evaluation (BCDE) data set.

molecules-24-02183-t002_Table 2

###### 

Covalent docking performance measured by both Best Scored Pose and Best Sampled Pose based on reaction and warhead classifications. The RMSD data in the table are the backbone RMSD median values for a given warhead and a given tool. The RMSD data points are depicted in heat maps. The best RMSD poses (\<1 Å) are encoded in deep green, and the worst RMSD poses (\>5 Å) are encoded in deep red. *n* represents the total number of complexes for a given warhead.

  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Reaction Class                         Warhead        *n*        Best Scored Pose (RMSD)   Best Sampled Pose (RMSD)                                                           
  -------------------------------------- -------------- ---------- ------------------------- -------------------------- --------- --------- ---------- ---------- ------ ------ -----------------------------------
  **Nucleophilic Addition by Cys**       Nitrile        42         2.26                      1.1                        1.27      3.74      1.6        0.91       1.24   1.56   5 Å\
                                                                                                                                                                                ![](molecules-24-02183-i049.jpg)\
                                                                                                                                                                                1 Å

  Alkene                                 117            1.99       2.47                      1.47                       1.68      1.4       1.59       1.18       1.41          

  Carbonyl                               25             2.27       2.12                      2.05                       3.72      1.5       1.77       1.33       1.52          

  Alkyne                                 1              2.01       0.67                      0.42                       0.39      1.9       0.67       0.35       0.39          

  Guanyl                                 2              1.78       1.46                      0.81                       1.37      1.4       0.98       0.81       1.37          

  **Nucleophilic Substitution by Cys**   Halide         15         1.67                      2.31                       4.37      1.06      1.3        2.17       4.37   0.88   

  Others                                 16             2.1        2.19                      2.12                       2.09      1.6       1.47       1.3        1.77          

  **Ring Opening by Cys**                Heterocyclic   17         1.71                      1.74                       1.87      4.88      1.4        1.48       1.33   4.72   

  **Disulfide Formation**                Sulfydryl      10         1.94                      2.94                       2.52      2.92      1.3        2.47       2.52   1.95   

  **Nucleophilic Addition by Ser**       Nitrile        3          1.38                      2.54                       0.73      2.66      1.3        1.77       0.52   2.66   

  Carbonyl                               29             1.61       3.54                      3.1                        2.03      1.3       1.58       1.4        1.36          

  Boronic Acid                           8              1.94       2.57                      2.94                       2.44      1.5       1.88       2.74       2.44          

  **Nucleophilic Substitution by Ser**   Phosphonyl     3          2.32                      2.33                       5.81      1.68      1.5        2.08       3.25   1.68   

  **Ring Opening by Ser**                Lactam         40         1.73                      2.95                       1.7       2.53      1.5        1.62       1.7    1.48   

  Lactone                                2              2.09       2.56                      2.52                       5.66      1.5       1.6        1.6        4.98          

  ***P*(*i*,*x*)**                       **1.94**       **2.33**   **2.05**                  **2.53**                   **1.5**   **1.6**   **1.33**   **1.56**                 
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

molecules-24-02183-t003_Table 3

###### 

Covalent docking performance as measured by both Best Scored Pose and Best Sampled Pose based on receptor types. The RMSD data in the table are the backbone RMSD median values for a given warhead and a given tool. The RMSD data points are depicted in heat maps. The best RMSD poses (\<1 Å) are encoded in deep green, and the worst RMSD poses (\>5 Å) are encoded in deep red. n represents the total number of complexes for a given warhead.

  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Receptor Type               *n*       Best Scored Pose (RMSD)   Best Sampled Pose (RMSD)                                                                
  --------------------------- --------- ------------------------- -------------------------- ---------- ---------- ---------- --------- ---------- ------ -----------------------------------
  **Hydrolase**               204       2.07                      2.33                       1.71       2.76       1.54       1.51      1.39       1.61   5 Å\
                                                                                                                                                          ![](molecules-24-02183-i049.jpg)\
                                                                                                                                                          1 Å

  **Transferase**             83        1.8                       2.24                       1.3        1.34       1.34       1.46      1.06       1.24   

  **Ligase**                  3         2.9                       2.63                       1.08       1.41       1.25       1.04      0.84       1.41   

  **Lyase**                   1         0.96                      1.23                       1.24       1.34       0.9        1.17      1.19       1.34   

  **Oxidoreductase**          6         1.7                       1.44                       2.37       1.13       1.18       1.06      2.37       1.13   

  **Isomerase**               6         1.49                      1.2                        1.02       2.25       1.33       0.78      1.01       2.25   

  **Transcription**           18        2.38                      6.15                       2.15       3.42       1.6        3.03      1.71       2.1    

  **Viral Protein**           4         3.91                      0.81                       4.29       4.93       2.97       0.81      1.3        4.93   

  **Metal binding protein**   5         1.72                      5.54                       5.43       2.37       1.04       4.54      3          1.67   

  ***P*(*i*,*x*)**            **1.8**   **2.24**                  **1.71**                   **2.25**   **1.33**   **1.17**   **1.3**   **1.61**          
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
