==== Front BMC Bioinformatics BMC Bioinformatics BMC Bioinformatics 1471-2105 BioMed Central London 3523 10.1186/s12859-020-3523-9 Research Decoy selection for protein structure prediction via extreme gradient boosting and ranking Akhter Nasrin nakhter3@gmu.edu 1 http://orcid.org/0000-0002-6223-8570Chennupati Gopinath gchennupati@lanl.gov 2 Djidjev Hristo djidjev@lanl.gov 2 Shehu Amarda amarda@gmu.edu 134 1 grid.22448.380000 0004 1936 8032Department of Computer Science, George Mason University, Fairfax, 22030 VA USA 2 grid.148313.c0000 0004 0428 3079Information Sciences (CCS-3) Group, Los Alamos National Laboratory, Bikini At al Rd., Los Alamos, 87545 USA 3 grid.22448.380000 0004 1936 8032Department of Bioengineering, George Mason University, Fairfax, 22030 VA USA 4 grid.22448.380000 0004 1936 8032School of Systems Biology, George Mason University, Manassas, 20110 VA USA 9 12 2020 9 12 2020 2020 21 Suppl 1 Publication of this supplement has not been supported by sponsorship. Information about the source of funding for publication charges can be found in the individual articles. The articles have undergone the journal's standard peer review process for supplements. IM was co-author on two papers in the supplement but was not involved with the peer review process of these two papers. No other competing interests were declared.18915 4 2020 29 4 2020 © The Author(s) 2020Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.Background Identifying one or more biologically-active/native decoys from millions of non-native decoys is one of the major challenges in computational structural biology. The extreme lack of balance in positive and negative samples (native and non-native decoys) in a decoy set makes the problem even more complicated. Consensus methods show varied success in handling the challenge of decoy selection despite some issues associated with clustering large decoy sets and decoy sets that do not show much structural similarity. Recent investigations into energy landscape-based decoy selection approaches show promises. However, lack of generalization over varied test cases remains a bottleneck for these methods. Results We propose a novel decoy selection method, ML-Select, a machine learning framework that exploits the energy landscape associated with the structure space probed through a template-free decoy generation. The proposed method outperforms both clustering and energy ranking-based methods, all the while consistently offering better performance on varied test-cases. Moreover, ML-Select shows promising results even for the decoy sets consisting of mostly low-quality decoys. Conclusions ML-Select is a useful method for decoy selection. This work suggests further research in finding more effective ways to adopt machine learning frameworks in achieving robust performance for decoy selection in template-free protein structure prediction. Keywords Decoy selectionMachine learningEnergy landscapeBasinsPurityXGBoostRankingLANL Laboratory Directed Research and Development (LDRD) program20160317ERhttp://dx.doi.org/10.13039/501100008982National Science Foundation19000618th IEEE International Conference on Computational Advances in Bio and medical Sciences (ICCABS 2018) Las Vegas, NV, USA 18-20 October 2018 issue-copyright-statement© The Author(s) 2020 ==== Body Background Protein molecules play a vital role in controlling the biological activities of a cell. There are a number of attempts in wet laboratories to determine biologically-active/native tertiary structures as a route to decoding protein function [1]. Technological advances have now made it possible to generate hundreds of thousands of tertiary structures for a given amino-acid sequence, known as decoys, in a few CPU hours [2]. The multiplicity of decoys necessitates recognizing high-quality, near-native decoys among hundreds of thousand of decoys in an ensemble. Identifying these near-native decoys is a challenging problem in computational structural biology, and is known as decoy selection. Template-free methods, which generate low-energy tertiary structures in the absence of one or more structural templates from homogeneous sequences, have now become prominent. The most popular ones include Rosetta [3] and Quark [4]. To compute the low-energy structures, these methods employ stochastic optimization to find local minimum of a selected energy/scoring function. A well known fact is that energy bias often does not lead to tertiary structures that are close to the native. Therefore, identifying near-natives from a large ensemble of decoys remains an open problem [5]. Consequently, other decoy selection strategies gained momentum due to the weak role of energy in recognizing near-native conformations, which is reflected in Critical Assessment of protein Structure Prediction (CASP) [5] series of community wide experiments. Clustering-based methods dominate the model quality assessment (MQA) performed in CASP. Clustering-based decoy selection methods work on the notion that decoys are randomly distributed around the native structure which a consensus method ought to reveal. The clustering-based decoy selection performs better when the ensemble consists of mostly good quality decoys. However, if the sampling of decoys in the decoy generation stage is sparse, resulting in many dissimilar decoys in an ensemble, consensus methods fail to recognize exceptionally good decoys [6]. Moreover, the time complexity incurred in clustering a large decoy ensemble creates another bottleneck. In addressing the above challenges in decoy selection, we propose an alternative approach that takes advantage of the consensus methods and a machine learning technique. As described in [7], protein energy landscape reveals important statistical information regarding the conformational organization and pathway. In this paper, we leverage the quantitative knowledge garnered from the energy landscape of a protein molecule in a machine learning framework to address the challenges in decoy selection. Supervised machine learning methods are gaining prominence in computational biology applications. These methods generate predictive models that learn subtle patterns from the data without making any prior assumptions [8]. One of the biggest challenges for these predictive models is to succeed even when the dataset is extremely imbalanced. Data imbalance is a common problem in computational biology and bioinformatics [9]. For instance, one of the benchmark proteins in our experiments contains only 0.005% of positive instances (near-natives) among 58,491 decoys. Even in such a sparse decoy set, the proposed method successfully identifies the near-natives. Our method works as follows: first, the method extracts local structures from the energy landscape probed through a template-free protein structure prediction method; next, a machine learning-based decoy selection method uses these local structures to finally select groups of good quality decoys. The method outperforms state-of-the-art decoy selection strategies in [10]. Related work The diverse collection of decoy selection strategies can be categorized into single-model, multi-model, quasi-single, and machine learning (ML) methods. Single-model methods predict quality on a per-decoy basis [11], these are physics-based and/or knowledge-based. Physics-based methods employ different atomic interactions such as electrostatic, Van Der Waals interactions, hydrogen bonding [12–14], whereas the knowledge-based scoring functions employ statistical analysis of known native structures [15–17]. Between these two methods, knowledge-based methods are known to be more successful in predicting high quality decoys [18, 19]. Cluster-based methods work on the premise that the decoys are randomly distributed around the ’true’ answer [20, 21], which is not entirely valid due to the inherent bias associated with the template-free protein structure prediction methods used to generate the decoys. Apart from the huge time-complexity incurred by clustering a large decoy ensemble, the cluster-based methods often fail to identify good quality decoys (near-natives) for hard targets, which are more sparsely sampled [6]. Despite the bottlenecks, cluster-based decoy selection strategies have been the most popular methods in the decoy selection literature. Quasi-single models combine the single-model and consensus methods. First, some high quality reference structures are selected, then the remaining decoys in the ensemble are compared with the reference structures [22]. These methods are shown to perform better [5, 23, 24]. Recent investigations are employing machine learning (ML) methods for decoy selection [25–27]. For instance, work in [28] uses Support Vector Machine (SVM) and uses a statistical scoring function GOAP [29] to distinguish native decoys from the non-native ones. Decoy selection through machine learning are mostly single-model methods. These methods leverage structural features of proteins to assess decoy quality. Work in [30] employs non-negative matrix factorization for selecting the best cluster of decoys and the the best decoy in the decoy set, which can be further extended to large scale using the the distributed implementations [31] of NMF. Deep learning has also become a popular approach to address ML problems in bioinformatics [32]. Along with a variety of applications, such as DNA sequencing [33], enzyme function prediction [34], de-novo prediction of membrane proteins [35], protein contact map prediction [36], and protein secondary structure prediction [37], deep learning has been successfully utilized for protein decoy selection as well. For instance, a deep belief network-based protein quality estimation (decoy selection) model DeepQA outperforms SVM-based methods and achieves state-of-the-art performance on the CASP dataset [38]. Convolutional neural network-based models have also observed success in protein decoy selection [38–40]. In this paper, we prefer to investigate shallow models, which, unlike deep architectures, do not place such high demands on the size of the training dataset in relation to the number of parameters. As our ability to expediently generate or obtain structure data grows, deep learning will surely provide an interesting way forward that we plan to pursue in tandem with strategies to reduce the dimensionality of the loss function. In this paper, we employ an ML technique to a multi-model method that exploits local structures extracted from an energy landscape [41]. The proposed ML-based multi-model method offers promising results in terms of higher true positives and lower false positives. Methods First, we elaborate on the concept of energy landscape that forms the basis of our decoy selection method. Energy landscapes to basins The energy landscape is an instance of a more general fitness landscape that comprises a set of points X, a neighborhood N(X) defined on X, a distance metric on X, and a fitness function f:X→ℝ≥0 that assigns a fitness to every point in X. Moreover, the points in X secure neighbors via the neighborhood function. In the context of decoy selection, the points x∈X represent decoy structures, and the fitness function often designates an energy function. Effectively, the energy landscape of decoy structures characterizes the mapping of structures to their internal energy and provides important quantitative information about the structure space. A protein energy landscape features an ensemble of structural states near or far from the native state and an extensive collection of intermediate states that shape the multi-modal and multi-dimensional nature of the landscape [41]. The concept of a basin is connected to a local/focal minimum. A focal minimum in a landscape is surrounded by a basin of attraction, which is the set of points on the landscape from which steepest descent/ascent converges to that focal optimum. Barriers separate basins and regulate transitions of a system between different structural states corresponding to basins in the landscape. Under the energy landscape treatment, the biologically-active/native state(s) can be determined by identifying corresponding basins, which requires one to extract the underlying organization of decoys to identify basins in the landscape. One approach to achieve this objective is to embed the decoys in a connectivity data structure and utilize energies to identify basins. Consider an Ω set of decoys. The Ω can be embedded in a nearest-neighbor graph (nn-graph) G=(V,E) [42]. The vertex set V is populated with the decoys, and the edge set E is populated by inferring the neighborhood structure of the landscape. The distance between two structures is measured via root-mean-squared-deviation (RMSD) after each of the structures is superimposed over some reference structures (arbitrarily, chosen to be the first in the ensemble); the superimposition minimizes differences due to rigid-body motions. Each vertex u∈V is connected to vertices v∈V if d(u,v)≤ε, where ε is a user-defined parameter. If the landscape has been sampled sparsely and in a non-uniform way, there is a possibility of creating a disconnected graph from a small ε value. One way to prevent such scenario is to increase the ε while controlling the density of the resulting nn-graph via the number of nearest neighbors of u. The local minima of the landscape can be detected by analyzing the nn-graph. A vertex u∈V is a local minimum if ∀v∈V f(u)≤f(v), where v∈N(u) (N(u) denotes the neighborhood of u). The remaining vertices are then assigned to basins as follows. Each vertex u is associated a negative gradient estimated by selecting the edge (u,v) that maximizes the ratio [f(u)−f(v)]/d(u,v). From each vertex u that is not a local minimum, the negative gradient is followed (via the edge that maximizes the above ratio) until a local minimum is reached. Vertices that reach the same local minimum are assigned to the basin associated with that minimum. Basin selection via basin ranking The basins, extracted from the energy landscape, can be useful in decoy selection. Work in [10] shows that simple, ranking-based basin selection strategies outperform a standard clustering-based decoy selection method in terms of purity (percentage of true positives, penalizes the selected basin by the extent of false positives found in that basin). Basins can be ranked as a combination of basin characteristics. For instance, basins can be ranked merely as size (S), as a combination of size and the energy (S+E) of the focal minimum of that basin. The size of basin is computed by the number decoys that belong to a basin. On the other hand, size and energy are used as conflicting objectives in a multi-objective, Pareto-based selection strategy. In a multi-objective optimization, solution A dominates solution B, if A is better than or equal to B for all optimization objectives, and for at least one objective, A is strictly better than B. In the context of basins, Pareto Rank (PR) of Basin A is the number of basins that dominate A. The Pareto Count (PC) of basin A is the number of basins that A dominates. Specifically, basins can be ranked with their PR, or with PR and PC (PR+PC). Empirical studies conducted in [10] demonstrate the superiority of the Pareto-based basin selection strategies over both cluster-based, size and energy-based decoy selection methods. Despite good performance, ranking-based decoy selection strategies are unable to perform consistently well over all test cases regardless of their difficulty levels. Neither S+E nor PR+PC can provide fair performance (less false positives and more true positives in the selected clusters/basins) over all or most of the test cases. One would prefer a decoy selection method that is able to provide reasonably good performance for all or most of the test cases regardless of difficulty level or heterogeneity in structural characteristics. This is the premise of the work presented in this paper. Decoy selection via ML and ranking Shortcomings of ranking-based basin selection strategies necessitate a new basin selection strategy. On that premise, we present a novel basin-based decoy selection method, referred to as ML-Select, that employs machine learning techniques. The method operates in two phases: the first phase captures n pure basins; while the second phase purifies the selected n basins and offers top k purified basins as output. Both the phases involve fitting a regression model and a selection approach (ranking) based on the regression results. To generalize across all possible difficulty levels of proteins, we randomly select two proteins per difficulty level (easy, medium, hard) to train the models. Therefore, the performance of our models is independent of a test case and difficulty levels. We now describe the two phases of ML-Select in further detail. Phase 1 In this phase, ML-Select predicts the purity of basins and ranks them based on the predicted values. We use two kinds of attributes: Pareto and graph-based attributes as features to build the regression model. The Pareto-based features are PR and PC, computed from treating basin size and focal energy as two conflicting optimization objectives [10]. We assign the ranks to each basin that are calculated based on the PR and PC values associated with the given basin. Specifically, each basin is assigned two ranks based on their PR and PC values, which serve as two different features. The graph-based feature, number of connected components, characterizes a spatial attribute of the graphical representation of basins. The extracted basins from the nn-graph (of all the decoys in the dataset) using the Structural Bioinformatics Library (SBL) [42] are essentially bags of decoys. Estimating the spatial structure of these decoys in a specific basin is hard. Therefore, we consider the number of connected components as one of the features for ML-Select. In order to easily recover the relative spatial organization of the decoys comprising a basin, we construct m different nearest-neighbor graphs using the decoys populating m different basins. We use pdist+1Å for the distance threshold to create the nearest-neighbor graphs, where pdist refers to the average pairwise distance between the decoys of the basins. Depending on the distance between the decoys in a basin, the corresponding graph may consist of one or more connected components, which signify the structural attribute of a basin. Figure 1 shows an example graphical representation of the components in a basin. We rank the basins based on the predicted purity and pass the top n basins to the second phase for further purification. Fig. 1 Three components in one of the basin-graphs of 1dtja Phase 2 In the second phase, we predict the root mean-squared-deviation (rmsd) of a decoy from the true native. The training set of this phase uses the same proteins as in the first phase. However, the features in the second phase are different from that of the previous phase. We use twenty features of which three are knowledge-based potentials and the remaining are energy scores from Rosetta suite [43]. The three knowledge-based features are: RW, RWplus [44] and dDFIRE [45]. RW is distance-dependent atomic potential and RWplus is side-chain orientation dependent potential; the third feature is dDFIRE, which improves the DFIRE statistical potential by adding an orientation dependency. The remaining 17 features are energy terms in the REF2015 scoring function [46] in the Rosetta suite of scoring functions. The 17 Rosetta REF2015 energy terms are the Lennard-Jones attractive and repulsive terms that capture interactions between atoms in different residues, the Lazaridis-Karplus solvation energy, the intra-residue Lazaridis-Karplus solvation energy term, the asymmetric solvation energy term, the Lennard-Jones repulsive term that captures interactions between atoms in the same residue, the Coulombic electrostatic potential with a distance-dependent dielectric, the Proline ring closure energy and energy of the psi angle of preceding residue, the backbone-backbone hydrogen-bonding energy term between atoms close and distant in the primary sequence, the sidechain-backbone and sidechain-sidechain hydrogen-bonding energy term, the Ramachandran preferences term, the (backbone) omega dihedral term, the probability of amino acid given torsion values for the phi and psi backbonee angles, the internal energy of sidechain rotamers term (as derived from Dunbrack’s statistics), and a special torsional potential term to keep the tyrosine hydroxyl in the plane of the aromatic ring. The top n pure basins from the first phase are treated as test cases. That is, we build n regression models for n basins that are passed to the second phase from the first phase. Each of these basins are further purified as follows. In a given basin from phase 1, if the predicted rmsd of a decoy falls short of pre-defined threshold (dist_thresh, explained later in the implementation details), we remove that decoy from a test case basin. Effectively, the decoys that are further away from the true native are removed from the selected basins. As a result, the purity of the selected basin improves. We rank the basins based on the resulting purity after the non-native decoy elimination and offer the top k basins as a result at the end of second phase. The purification process in this phase poses a threat of eliminating a good decoy (ones near the native). We mitigate this effect with a shift in the pre-defined distance threshold, dist_thresh ± τ, where τ ∈ {10%, 20%, 25%} of the pre-defined threshold. The effect of the threshold variation on purity is discussed later in the results. Evaluation metrics We evaluate the performance of our approach using two metrics: percentage of true positives (n) and purity (p). At a given distance threshold dist_thresh (explained in the implementation details), n is the ratio of number of true near-natives in the selected basin B1−x, where x∈{1,2,3}, to the total number of true near-natives in that decoy ensemble. This metric resembles the Sensitivity (recall or true positive rate) measure. However, even significantly high n might become less effective if the number of false positives in the selected basin is high, where, a random draw from the selected basin would result in a lower probability of offering a true near-native. The metric p compensates this scenario by penalizing a large basin (or a group of selected basins) containing a large number of true and false positives to the extent of the false positive population present in that basin. p is computed as a ratio of the number of true positives to the size of a basin (or a group of basins). Therefore, a basin with a large number of false positives results in a low purity regardless of the number of true positives in that basin. In essence, purity metric resembles the precision of our method. Specifically, we discuss the performance of ML-Select and four other competing methods in terms of purity metric due to its balanced treatment towards false and true positives. For evaluation, we select these metrics that focus more on true and false positives rather than on true and false negatives because here we are more concerned with increasing the probability of selecting a true positive from the selected basins in a random draw, which can be achieved by minimizing the false positives and maximizing the true positives. Implementation details We use a distance threshold of 1Å for creating the nn-graph of a decoy ensemble via SBL [42]. Since Rosetta decoy generation protocol may produce sparse samples, a low threshold may result in a disconnected graph. To address this problem, we increase the initial threshold until the graph is connected. Minimum distance from a decoy in an ensemble to the true native is referred to as min_dist. For a protein with a known native structure, all decoys under the threshold dist_thresh are deemed as near-natives. As there are three different categories of test cases, we set the dist_thresh parameter to determine the near-natives on a per-case basis. More specifically, dist_thresh is set to 2Å for the easy cases (min_dist<1Å). For the medium cases (1Å≤min_dist<2Å), dist_thresh is either 2.5Å or 3Å. For the hard cases (3Å3.0.CO;2-U 16 McConkey BJ Sobolev V Edelman M Discrimination of native protein structures using atom–atom contact scoring Proc Natl Acad Sci 2003 100 6 3215 20 10.1073/pnas.0535768100 12631702 17 Simons KT Ruczinski I Kooperberg C Fox BA Bystroff C Baker D Improved recognition of native-like protein structures using a combination of sequence-dependent and sequence-independent features of proteins Proteins Struct Funct Bioinforma 1999 34 1 82 95 10.1002/(SICI)1097-0134(19990101)34:1<82::AID-PROT7>3.0.CO;2-A 18 Park B Levitt M Energy functions that discriminate x-ray and near-native folds from well-constructed decoys J Mol Biol 1996 258 2 367 92 10.1006/jmbi.1996.0256 8627632 19 Felts AK Gallicchio E Wallqvist A Levy RM Distinguishing native conformations of proteins from decoys with an effective free energy estimator based on the opls all-atom force field and the surface generalized born solvent model Proteins Struct Funct Bioinforma 2002 48 2 404 22 10.1002/prot.10171 20 Lorenzen S Zhang Y Identification of near-native structures by clustering protein docking conformations Proteins Struct Funct Bioinforma 2007 68 1 187 94 10.1002/prot.21442 21 Estrada T, Armen R, Taufer M. Automatic selection of near-native protein-ligand conformations using a hierarchical clustering and volunteer computing. In: Proceedings of the First ACM International Conference on Bioinformatics and Computational Biology. ACM: 2010. p. 204–13. 10.1145/1854776.1854807. 22 Jing X Wang K Lu R Dong Q Sorting protein decoys by machine-learning-to-rank Sci Rep 2016 6 31571 10.1038/srep31571 27530967 23 He Z Alazmi M Zhang J Xu D Protein structural model selection by combining consensus and single scoring methods PloS ONE 2013 8 9 74006 10.1371/journal.pone.0074006 24 Pawlowski M Kozlowski L Kloczkowski A Mqapsingle: A quasi single-model approach for estimation of the quality of individual protein structure models Proteins Struct Funct Bioinforma 2016 84 8 1021 8 10.1002/prot.24787 25 Manavalan B Lee J Lee J Random forest-based protein model quality assessment (rfmqa) using structural features and potential energy terms PloS ONE 2014 9 9 106542 10.1371/journal.pone.0106542 26 Nguyen SP, Shang Y, Xu D. Dl-pro: A novel deep learning method for protein model quality assessment. In: Neural Networks (IJCNN), 2014 International Joint Conference On. IEEE: 2014. p. 2071–8. 10.1109/ijcnn.2014.6889891. 27 Hurtado DM, Uziela K, Elofsson A. Deep transfer learning in the assessment of the quality of protein models. arXiv preprint. 2018. arXiv:1804.06281. 28 Mirzaei S, Sidi T, Keasar C, Crivelli S. Purely structural protein scoring functions using support vector machine and ensemble learning. IEEE/ACM Trans Comput Biol Bioinforma. 2016. 10.1109/tcbb.2016.2602269. 29 Zhou H Skolnick J Goap: a generalized orientation-dependent, all-atom statistical potential for protein structure prediction Biophys J 2011 101 8 2043 52 10.1016/j.bpj.2011.09.012 22004759 30 Akhter N, Vangara R, Chennupati G, Alexandrov BS, Djidjev H, Shehu A, Non-Negative Matrix Factorization for Selection of Near-Native Protein Tertiary Structures. In: IEEE Int Conf Bioinforma Biomed (BIBM). IEEE: 2019. p. 70–73. 31 Chennupati G, Vangara R, Skau E, Djidjev H, Alexandrov B. Distributed non-negative matrix factorization with determination of the number of latent features. Journal Supercomput. 2020:1–31. 32 Li Y, Huang C, Ding L, Li Z, Pan Y, Gao X. Deep learning in bioinformatics: Introduction, application, and perspective in the big data era. Methods. 2019. 10.1101/563601. 33 Li Y Han R Bi C Li M Wang S Gao X Deepsimulator: a deep simulator for nanopore sequencing Bioinformatics 2018 34 17 2899 908 10.1093/bioinformatics/bty223 29659695 34 Li Y Wang S Umarov R Xie B Fan M Li L Gao X Deepre: sequence-based enzyme ec number prediction by deep learning Bioinformatics 2017 34 5 760 9 10.1093/bioinformatics/btx680 35 Wang S Fei S Wang Z Li Y Xu J Zhao F Gao X Predmp: a web server for de novo prediction and visualization of membrane proteins Bioinformatics 2018 35 4 691 3 10.1093/bioinformatics/bty684 36 Wang S Sun S Li Z Zhang R Xu J Accurate de novo prediction of protein contact map by ultra-deep learning model PLoS Comput Biol 2017 13 1 1005324 10.1371/journal.pcbi.1005324 37 Wang S Peng J Ma J Xu J Protein secondary structure prediction using deep convolutional neural fields Sci Rep 2016 6 18962 10.1038/srep18962 26752681 38 Cao R Bhattacharya D Hou J Cheng J Deepqa: improving the estimation of single protein model quality with deep belief networks BMC Bioinformatics 2016 17 1 495 10.1186/s12859-016-1405-y 27919220 39 Sato R Ishida T Protein model accuracy estimation based on local structure quality assessment using 3d convolutional neural network PloS ONE 2019 14 9 0221347 10.1371/journal.pone.0221347 40 Hou J, Wu T, Cao R, Cheng J. Protein tertiary structure modeling driven by deep learning and contact distance prediction in casp13. Proteins Struct Funct Bioinforma. 2019. 10.1002/prot.25697. 41 Nussinov R Wolynes PG A second molecular biology revolution? the energy landscapes of biomolecular function Phys Chem Chem Phys 2014 16 14 6321 2 10.1039/c4cp90027h 24608340 42 Cazals F Dreyfus T The structural bioinformatics library: modeling in biomolecular science and beyond Bioinformatics 2017 33 7 997 1004 28062450 43 Burman SSR, Mulligan VK. Scoring Tutorial. https://www.rosettacommons.org/demos/latest/tutorials/scoring/scoring. Accessed 20 June 2018. 44 Zhou H Zhou Y Distance-scaled, finite ideal-gas reference state improves structure-derived potentials of mean force for structure selection and stability prediction Protein Sci 2002 11 11 2714 26 10.1110/ps.0217002 12381853 45 Yang Y Zhou Y Specific interactions for ab initio folding of protein terminal regions with secondary structures Proteins Struct Funct Bioinforma 2008 72 2 793 803 10.1002/prot.21968 46 Alford RF Leaver-Fay A Jeliazkov JR O’Meara MJ DiMaio FP Park H Shapovalov MV Renfrew PD Mulligan VK Kappel K The rosetta all-atom energy function for macromolecular modeling and design J Chem Theory Comput 2017 13 6 3031 48 10.1021/acs.jctc.7b00125 28430426 47 Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Stat. 2001:1189–232. 48 Chen T, Guestrin C. Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining. ACM: 2016. p. 785–94. 10.1145/2939672.2939785. 49 RW Potential. https://zhanglab.ccmb.med.umich.edu/RW/. Accessed 5 Jul 2018. 50 dDFIRE/DFIRE2 Energy Calculation. http://sparks-lab.org/yueyang/DFIRE/dDFIRE-service.php/. Accessed 8 Jul 2018. 51 Meiler J Baker D Coupled prediction of protein secondary and tertiary structure Proc Natl Acad Sci U S A 2003 100 21 12105 10 10.1073/pnas.1831973100 14528006 52 DeBartolo J, Hocky G, Wilde M, Xu J, Freed KF, Sosnick TR. Protein structure prediction enhanced with evolutionary diversity: SPEED. 2010; 19(3):520–34. 10.1002/pro.330. 53 Olson B, Shehu A. Multi-objective stochastic search for sampling local minima in the protein energy surface. In: ACM Conf on Bioinf and Comp Biol (BCB). Washington, D. C.: 2013. p. 430–9. 10.1145/2506583.2506590. 54 Molloy K Saleh S Shehu A Probabilistic search and energy guidance for biased decoy sampling in ab-initio protein structure prediction IEEE/ACM Trans Comput Biol and Bioinf 2013 10 5 1162 75 10.1109/TCBB.2013.29 55 Zhang GJ Zhou GX Yu XF Hao H Yu L Enhancing protein conformational space sampling using distance profile-guided differential evolution IEEE/ACM Trans Comput Biol and Bioinf 2017 14 6 1288 301 10.1109/TCBB.2016.2566617 56 Zhang G, Ma L, Wang X, Zhou X. Secondary structure and contact guided differential evolution for protein structure prediction. IEEE/ACM Trans Comput Biol and Bioinf. 2018. 10.1109/TCBB.2018.2873691. preprint. 57 Zhang J Xu D Fast algorithm for population-based protein structural model analysis Proteomics 2013 13 2 221 9 10.1002/pmic.201200334 23184517 58 Garcia S Herrera F An extension on "statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons J Mach Learn Res 2008 9 2677 94 59 Demšar J Statistical comparisons of classifiers over multiple data sets J Mach Learn Res 2006 7 Jan 1 30 60 Chennupati G, Azad RMA, Ryan C. Performance optimization of multi-core grammatical evolution generated parallel recursive programs. In: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation. ACM: 2015. p. 1007–14. 10.1145/2739480.2754746.