
==== Front
PLoS Comput Biol
PLoS Comput Biol
plos
PLOS Computational Biology
1553-734X
1553-7358
Public Library of Science San Francisco, CA USA

39255309
10.1371/journal.pcbi.1012423
PCOMPBIOL-D-23-00581
Research Article
Research and Analysis Methods
Animal Studies
Experimental Organism Systems
Model Organisms
Zebrafish
Research and Analysis Methods
Model Organisms
Zebrafish
Research and Analysis Methods
Animal Studies
Experimental Organism Systems
Animal Models
Zebrafish
Biology and Life Sciences
Organisms
Eukaryota
Animals
Vertebrates
Fish
Osteichthyes
Zebrafish
Biology and Life Sciences
Zoology
Animals
Vertebrates
Fish
Osteichthyes
Zebrafish
Biology and Life Sciences
Developmental Biology
Life Cycles
Larvae
Biology and Life Sciences
Psychology
Behavior
Social Sciences
Psychology
Behavior
Biology and Life Sciences
Toxicology
Toxicity
Medicine and Health Sciences
Pathology and Laboratory Medicine
Toxicology
Toxicity
Physical Sciences
Mathematics
Statistics
Statistical Data
Biology and Life Sciences
Toxicology
Medicine and Health Sciences
Pathology and Laboratory Medicine
Toxicology
Biology and Life Sciences
Physiology
Biological Locomotion
Swimming
Biology and Life Sciences
Evolutionary Biology
Population Genetics
Biology and Life Sciences
Genetics
Population Genetics
Biology and Life Sciences
Population Biology
Population Genetics
Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies
Autoencoder-based behavioral pattern recognition outperforms standard statistical methods
https://orcid.org/0000-0001-9429-1838
Green Adrian J. Conceptualization Formal analysis Methodology Validation Writing – original draft Writing – review & editing 1 2 *
Truong Lisa Data curation Investigation Validation Writing – review & editing 3
Thunga Preethi Formal analysis Writing – review & editing 1
Leong Connor Investigation Validation Writing – review & editing 3
Hancock Melody Formal analysis Writing – review & editing 1
Tanguay Robyn L. Funding acquisition Resources Supervision Writing – review & editing 3
https://orcid.org/0000-0001-7815-6767
Reif David M. Conceptualization Funding acquisition Project administration Resources Supervision Writing – review & editing 1 4
1 Bioinformatics Research Center, Department of Biological Sciences, NC State University, Raleigh, North Carolina, United States of America
2 Sciome LLC, Research Triangle Park, North Carolina, United States of America
3 Department of Environmental and Molecular Toxicology, Oregon State University, Corvallis, Oregon, United States of America
4 Predictive Toxicology Branch, Division of Translational Toxicology, National Institute of Environmental Health Sciences, Durham, North Carolina, United States of America
Scarpino Samuel V. Editor
Northeastern University, UNITED STATES OF AMERICA
The authors have declared that no competing interests exist.

* E-mail: ajgreen4@ncsu.edu
10 9 2024
9 2024
20 9 e101242312 4 2023
15 8 2024
https://creativecommons.org/publicdomain/zero/1.0/ This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

Zebrafish have become an essential model organism in screening for developmental neurotoxic chemicals and their molecular targets. The success of zebrafish as a screening model is partially due to their physical characteristics including their relatively simple nervous system, rapid development, experimental tractability, and genetic diversity combined with technical advantages that allow for the generation of large amounts of high-dimensional behavioral data. These data are complex and require advanced machine learning and statistical techniques to comprehensively analyze and capture spatiotemporal responses. To accomplish this goal, we have trained semi-supervised deep autoencoders using behavior data from unexposed larval zebrafish to extract quintessential “normal” behavior. Following training, our network was evaluated using data from larvae shown to have significant changes in behavior (using a traditional statistical framework) following exposure to toxicants that include nanomaterials, aromatics, per- and polyfluoroalkyl substances (PFAS), and other environmental contaminants. Further, our model identified new chemicals (Perfluoro-n-octadecanoic acid, 8-Chloroperfluorooctylphosphonic acid, and Nonafluoropentanamide) as capable of inducing abnormal behavior at multiple chemical-concentrations pairs not captured using distance moved alone. Leveraging this deep learning model will allow for better characterization of the different exposure-induced behavioral phenotypes, facilitate improved genetic and neurobehavioral analysis in mechanistic determination studies and provide a robust framework for analyzing complex behaviors found in higher-order model systems.

Author summary

We demonstrate that a deep autoencoder using raw behavioral tracking data from zebrafish toxicity screens outperforms conventional statistical methods, resulting in a comprehensive evaluation of behavioral data. Our models can accurately distinguish between normal and abnormal behavior with near-complete overlap with existing statistical approaches, with many chemicals detectable at lower concentrations than with conventional statistical tests; this is a crucial finding for the protection of public health as exposure can lead to a range of neurodevelopmental disorders, including cognitive and other behavioral deficits. Our deep learning models enable the identification of new substances capable of inducing aberrant behavior, and we generated new data to demonstrate the reproducibility of these results. Thus, neurodevelopmentally active chemicals identified by our deep autoencoder models may represent previously undetectable signals of subtle individual response differences. Our method elegantly accounts for the high degree of behavioral variability associated with the genetic diversity found in a highly outbred population, as is typical for zebrafish research, thereby making it applicable to multiple laboratories generating similar data. Utilizing the vast quantities of control data generated during high-throughput screening is one of the most innovative aspects of this study and to our knowledge is the first study to explicitly develop a deep autoencoder model for anomaly detection in large-scale toxicological behavior studies.

http://dx.doi.org/10.13039/100000066 National Institute of Environmental Health Sciences ES030287 Tanguay Robyn L. http://dx.doi.org/10.13039/100000066 National Institute of Environmental Health Sciences ES030007 https://orcid.org/0000-0001-7815-6767
Reif David M. http://dx.doi.org/10.13039/100000066 National Institute of Environmental Health Sciences ES025128 https://orcid.org/0000-0001-7815-6767
Reif David M. http://dx.doi.org/10.13039/100000066 National Institute of Environmental Health Sciences ES033243 https://orcid.org/0000-0001-7815-6767
Reif David M. http://dx.doi.org/10.13039/100000054 National Cancer Institute CA161608 https://orcid.org/0000-0001-7815-6767
Reif David M. the Intramural Research Program of the NIH ZIAES103385 https://orcid.org/0000-0001-7815-6767
Reif David M. This research was supported by the National Institutes of Health (NIH) grant awards ES030287 (RLT, LT), ES030007 (AJG, DMR), ES025128 (DMR), ES033243 (DMR), and CA161608 (AJG, DMR). This research was supported [in part] by the Intramural Research Program of the NIH, ZIAES103385 (DMR). The funders played no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. PLOS Publication Stagevor-update-to-uncorrected-proof
Publication Update2024-09-20
Data AvailabilityThe code and data required to replicate findings reported in the article are available at https://github.com/Tanguay-Lab/Manuscripts/tree/main/Green_et_al_(2024)_Manuscript.
Data Availability

The code and data required to replicate findings reported in the article are available at https://github.com/Tanguay-Lab/Manuscripts/tree/main/Green_et_al_(2024)_Manuscript.
==== Body
pmcIntroduction

Significant progress continues to be made in our understanding of neurodevelopmental disorders such as autism spectrum disorder, attention deficit hyperactivity disorder (ADHD), developmental delay, learning disabilities, and other neurodevelopmental problems. As incidences continue to rise globally and affect 10–15% of all births, more work must be done to improve our understanding of these disorders [1–3]. Meta-analyses suggest strong and consistent epidemiological evidence that the developing nervous system is particularly vulnerable to low-level exposure to widespread environmental contaminants, as the anatomical and functional architecture of the human brain is mainly determined by developmental transcriptional processes during the prenatal period [3–7]. Therefore, identifying associations between developmental exposures and neurological effects is a core objective to improve public health by informing disease and disability prevention [1,8].

As the number of environmental contaminants grows to nearly one million, comprehensive data on the neurodevelopmental toxicity of these contaminants remain sparse or nonexistent [3,9–11]. In response, high-throughput screening (HTS) assays have been developed to expedite chemical toxicity testing using in vitro and in vivo systems [12–14]. However, in vitro cell and cell-free assays cannot fully capture systemic organismal responses in terms of anatomy, physiology, or behavior [15]. Zebrafish (Danio rerio) have emerged as an ideal model for studying low-level chemical exposure because of their high fecundity, rapid development, genetic tractability, and amenability to high-throughput data generation [12,16,17]. The zebrafish brain’s structural organization, cellular morphology, and neurotransmitter systems are very similar to other vertebrates, including chickens, rats, and humans [18–21]. Furthermore, zebrafish have behavioral patterns highly similar to mammals, and genetic homologs for 70% of human genes and 82% of human disease genes, making them a powerful model organism for revealing the neuronal developmental pathways underlying behavior [22–24].

Zebrafish larvae show swimming patterns essential for their survival following swim bladder development at four to five days post-fertilization (dpf), including exploration, foraging, and escape response which can be assessed using various locomotor behavioral assays [25,26] while more advanced continuous swimming, schooling, and reproductive behavior is still developing. One of these assays, the larval photomotor response (LPR), utilizes a sudden transition from light to dark, eliciting a stereotyped large-angle O-bend, followed by several minutes of increased movement, which gradually reduces [27,28]. Exposure to toxicants has been shown to alter this stereotypical behavioral response [24,29]. Current HTS for behavioral neurotoxicity focuses heavily on analyzing locomotor behavior using distance moved and population-based statistical methods [24,30]. However, while the behavior repertoire of larval zebrafish is less sophisticated when compared to that of adult zebrafish and other higher-order vertebrates, they are capable of numerous distinct behaviors [24,31,32]. These behaviors, such as thigmotaxis, and light avoidance cannot always be captured when using distance moved as a sole indicator of neurobehavioral toxicity in analyses of this data. Moreover, as most laboratory zebrafish populations feature significant genetic heterogeneity, individual responses to exotic toxicants cannot be expected to be homogeneous for simplistic measures such as distance moved [33].

Improved accessibility to computing resources and application interfaces, together with recent advances in deep-learning makes it possible to analyze complex behavioral data in novel ways and predict neurodevelopmental toxicity [34–36]. The volume and diversity of data generated during HTS experiments, combined with the variety in toxicological response within populations, present an opportunity that is well-suited for machine learning (ML). In particular, analysis of zebrafish HTS data from five dpf larvae exposed to 1,060 unique chemicals reveals that only 8% of chemical-concentration pairs (a unique combination of chemical and concentration, e.g. 6.4 μM Nicotine) exhibit changes in distance moved [30], which is low given the known toxicity profiles of the chemical set. The traditional methods for analyzing zebrafish behavior data are primarily based on measurement of distance moved and instances of variations in the movement patterns, velocity changes and spatial preference is lost due to the sheer volume of data and complexity. Additionally, the traditional analysis methods is unable to identify meaningful patterns due to the noise and variability. This challenge provides an opportunity to apply methods developed for anomaly detection from areas such as financial fraud [37], medical application faults [38], security systems intrusion [39], system faults [40], and others [41,42]. Such ML techniques would allow for a more holistic evaluation of zebrafish behavior by learning complex features such as movement patterns, velocity changes and spatial preferences associated with “normal” behavior and flagging subtle deviations. These intricate nuances could be indicative of chemical toxicity and can often be missed by traditional assays relying solely on measuring distance moved as a metric. In anomaly detection, we learn the pattern of a normal process, and anything that does not follow this pattern is classified as an anomaly. This learning model is particularly applicable, as many HTS data sets have large amounts of control data to analyze [30]. One intriguing approach to achieving this is by applying an autoencoder [43–48]. An autoencoder is a neural network of two modules, an encoder and a decoder [47,49]. The encoder learns the underlying features of a process, and these features are typically in a reduced dimension. The decoder then uses this reduced dimension to recreate the original data from these underlying features.

In the present study, we trained deep autoencoder models to recognize the pattern of quintessential larval zebrafish behavior and identify abnormal behavior following developmental chemical exposure. The performance of our deep autoencoders was compared against a two sample Kolmogorov–Smirnov test (K-S test), a standard for behavioral assessment. In addition to model development, we assessed the features driving performance through feature permutation and generated new confirmatory data to assess model reproducibility and confirm novel findings.

Results

Statistical classification of behavior

A two sample Kolmogorov–Smirnov test (K-S test) was used to compared treated vs control distance moved and angular velocity in light/dark cycling in zebrafish larvae at five dpf. We identified 40 chemical-concentration combinations from nine chemicals and 28 chemical-concentration combinations from nine chemicals capable of inducing a significantly different (p < 0.05) behavioral response using both distance moved and angular velocity, respectively (S2 Table). While 10 chemicals were identified using both methods, nine chemicals were similar, with distance moved finding a significant difference in multi-walled carbon nanotubes at 75 and 100 μM and angular velocity finding a difference in sodium 2-(N-ethylperfluorooctane-1-sulfonamido)ethyl phosphate at 0.25 μM. Considering that distance moved revealed more chemical-concentration combinations in this screening application, we used this metric to identify abnormal larvae to ensure a sufficient number for training the autoencoder models. Using the 30th and 70th percentiles, we defined 227 individual larvae as abnormal (Fig 1A). These 227 larvae formed the validation set used to test the performance of our models.

10.1371/journal.pcbi.1012423.g001 Fig 1 Assessment autoencoder performance.

(A) Schematic representation of the differences in statistical and autoencoder based classification of behavioral response in larval zebrafish. (B) Venn diagram showing overlap between statistical and autoencoder classified abnormal zebrafish. (C) Evaluating the change in model performance when the values of a single feature are randomly shuffled. Kappa–Cohen’s Kappa statistic, AUROC—area under the receiver operating characteristic. Figure depicts means ± SEM. (D) Coefficients of variation for each of the main numerical features.

Training performance

Autoencoder models were trained using only control data for each of the activity states (hypoactive, normal, and hyperactive) per phase of the second light cycle. This resulted in six trained models (S1 Fig the training loss plots for the models). Table 1 shows the results for the six deep autoencoder models trained using control data and validated using data from zebrafish defined as abnormal using the K-S test. All the models performed well with values ranging from 0.615–0.867 and 0.740–0.922 for the Kappa and AUROC, respectively. As expected, the models consistently produced high specificity (SP) levels as this value indicated how well the models classify control data. There was greater variability in the sensitivity (SE) with the dark phase models matching or outperforming the light phase models for each activity state. Further, we observed a noteworthy trend across all models producing high positive predictive value (PPV). Overall, these results show that deep autoencoders trained using control data is capable of distinguishing between normal and abnormal larval zebrafish behavior with a high degree of accuracy.

10.1371/journal.pcbi.1012423.t001 Table 1 Deep autoencoder model performance in behavioral classification.

Table showing performance of model trained using different activity states of the control data in both light and dark phases.

Model	Performance Metrics	
Baseline Control Activity Level	Light Phase	SE	SP	PPV	Kappa	AUROC	
Hypoactive	Light	78.5	100	99.7	0.867	0.892	
Dark	78.3	98.0	88.4	0.800	0.882	
Normal	Light	48.3	99.7	93.1	0.615	0.740	
Dark	73.3	94.8	77.6	0.695	0.840	
Hyperactive	Light	79.2	97.5	85.5	0.790	0.883	
Dark	86.9	97.5	90.2	0.855	0.922	

Evaluation of unknowns

Using the six trained models, we evaluated the 2,719 treated zebrafish larvae (Fig 1). The autoencoders correctly classified 156 of the 227 larvae that fell below or above the 30th and 70th percentiles, respectively. In addition, our deep autoencoders identified 463 larvae as abnormal from the 2,492 larvae defined as normal using the K-S test (Fig 1B). The majority (422) of these 619 larvae were from one of 66 chemical-concentration combinations from 13 chemicals (Table 2). The deep autoencoders successfully identified nine of the ten statistically abnormal chemicals and identified these chemicals at or below the lowest concentration shown to be statistically significant. While the deep autoencoders did not identify Perfluorodecylphosphonic acid as capable of inducing abnormal behavior, but they did identify 3-Perfluoropentyl propanoic acid (5:3), Perfluoro-n-octadecanoic acid, 8-Chloroperfluorooctylphosphonic acid, and Nonafluoropentanamide, which were missed in the statistical testing framework. These results, summarized in Fig 2, show that deep autoencoders can match the performance of the K-S test and are more sensitive at detecting abnormal behavior.

10.1371/journal.pcbi.1012423.g002 Fig 2 Summary of behavioral analysis pipeline and results.

Utilizing our analysis pipeline produced six deep autoencoder models (three for the light phase and three for the dark phase) capable of classifying larval zebrafish behavior with high Kappa and AUROC values. The trained models were then used to classify the non-significant exposed larvae and identified Nonafluoropentanamide, Perfluorohexanesulfonic acid, (Heptafluoropropyl)trimethylsilane, 2-Methylphenanthrene, 8-Chloroperfluorooctylphosphonic acid, Perfluoro-n-octadecanoic acid, and others as capable of inducing abnormal behavior.

10.1371/journal.pcbi.1012423.t002 Table 2 Autoencoders identified chemicals.

Table showing chemicals and concentrations flagged for displaying abnormal behavioral effects when evaluated using Autoencoder. Compounds that were picked up by Autoencoder, but not KS test are highlighted in red.

CASRN	Chemical Name	Concentration (μM)	
71751-41-2	Abamectin	0.1, 0.2, 0.4, 0.6	
308068-56-6	Multi-Walled Carbon Nanotube	10, 23.2, 50, 75, 100	
2531-84-2	2-Methylphenanthrene	1, 2.54, 6.45, 16.4, 35, 74.8, 100	
832-69-9	1-Methylphenanthrene	1, 2.54, 6.45, 16.4, 35, 74.8, 100	
914637-49-3	3-Perfluoropentyl propanoic acid (5:3)	0.25	
192-51-8	Dibenzo[e-l]pyrene	0.01, 0.025, 0.065, 0.164, 0.35, 0.75, 1, 2.54, 16.4, 35, 100	
16517-11-6	Perfluoro-n-octadecanoic acid	0.25	
355-46-4	Perfluorohexanesulfonic acid	0.015, 0.14, 0.41, 3.7, 11.1, 33.3, 66.5, 100	
3834-42-2	(Heptafluoropropyl)trimethylsilane	0.015, 0.046, 0.41, 1.23, 11.1, 33.3	
	8-Chloroperfluorooctylphosphonic acid	0.167	
31253-34-6	2-Aminohexafluoropropan-2-ol	0.015, 0.046, 0.41, 1.23, 3.7, 11.1, 33.3, 66.5, 100	
13485-61-5	Nonafluoropentanamide	0.41, 3.7, 11.1	
439-14-5	Diazepam	1, 3, 5, 8, 12	

Features driving improved autoencoder performance

To determine the features in the model that were most important in driving classification performance, we employed permutation feature importance. This technique is a model agnostic inspection technique used for any fitted estimator to determine the importance of each feature in the model. Larger the decrease in model performance (Kappa or AUROC) when a single feature value is randomly shuffled, the more important the feature. Our results, shown in Fig 1C, indicate that phase, trial time, x position, and y position are the largest drivers of model performance, while distance moved and velocity contribute very little. Coefficients of variation show greater variability in the x and y positional data between control and exposed groups compared to either velocity or distance moved (Fig 1D). This trend is consistent irrespective of the larval activity state (hypoactive, normal activity, or hyperactive) relative to their respective controls (Fig 3).

10.1371/journal.pcbi.1012423.g003 Fig 3 Coefficients of variation per larval activity state.

Coefficients of variation (CVs) for each of the main numerical features (A–C) in the light (D–F) and in the dark. Columns show CVs of larval zebrafish significantly (p < 0.05) (A, D) hypoactive, (B, E) normal activity, or (C, F) hyperactive relative to their respective controls.

Experimental confirmation of autoencoder findings

To provide an unbiased evaluation of the final model fits, we generated new data using 2-Methylphenanthrene, and Nonafluoropentanamide. The data collected confirmed that our models accurately classified all controls as normal while detecting similar levels of abnormal behavior response across the concentration range (Fig 4) (p > 0.15). These results show that the trained model is capable of producing similar results across experimental replicates.

10.1371/journal.pcbi.1012423.g004 Fig 4 Experimental model evaluation.

Comparison of the performance of deep autoencoder models between the training set and two chemicals identified by the models to elicit abnormal larval zebrafish behavior. Percent of larval zebrafish classified as abnormal based on their behavioral response to developmental exposure to (A) 2-Methylphenanthrene and (B) Nonafluoropentanamide.

Discussion

Statistical analysis identified 39 chemical-concentration combinations from ten chemicals capable of inducing a significantly different (p < 0.05) behavioral response. Utilizing the 227 abnormal individuals identified by the statistical test as our validation set, we trained six deep autoencoder models using control data for each of the activity states (hypoactive, normal, and hyperactive). All of the resulting models performed well with values ranging from 0.615–0.867 and 0.740–0.922 for the Kappa and AUROC, respectively. All models achieved SP values above 94.8% and PPV values above 77.6% while SE values for all dark phase models outperformed the light phase models for each activity state (Table 1). Assessment of permutation feature importance indicates that phase, trial time, x-position, and y-position are the largest drivers of model performance (Fig 1C). The calculated coefficients of variation shed some light on this surprising finding (Fig 1D). They show that variation in the x and y positional data is greater than observed for velocity or distance moved between control and exposed groups. These differences in variation likely make it easier for the models to distinguish between treated and exposed groups.

When we examined exposed larvae defined as normal using the K-S test (Fig 1), our deep autoencoders identified 66 chemical-concentration combinations from 12 chemicals (Table 2) with Perfluoro-n-octadecanoic acid, 8-Chloroperfluorooctylphosphonic acid, and Nonafluoropentanamide only identified by our autoencoders. These results show that a deep autoencoder-based model can classify larval zebrafish behavior as normal or abnormal with very good efficacy and often identified abnormal behaviors at lower concentrations than current statistical methods. Further, the models identified three novel chemicals, Perfluoro-n-octadecanoic acid, 8-Chloroperfluorooctylphosphonic acid, and Nonafluoropentanamide as capable of inducing abnormal behavior (Fig 3). While making a definitive claim will require further experimentation, it does appear that the autoencoder method is particularly sensitive at detecting changes due to PFAS exposure. PFAS are associated with increased glutamate levels in the hippocampus and catecholamine levels in the hypothalamus, decreased dopamine in the whole brain after PFAS exposure, and increased extracellular glutamate has been observed in the hippocampus epileptic rats [50,51]. Thus, it is reasonable to infer that these neurochemical changes are capable of altering autoencoder-detectable patterns without changing locomotor magnitude or direction.

Recognition and categorization of swimming patterns in larvae is a challenging task and a number of approaches have been used. These can range from subjective analysis based on experienced observations [31,52] or through the application of unsupervised ML [27,32,53–57]. These studies have focused on the analysis and categorization of behavioral patterns in wild-type strains [27,57], mutant strains [32,53], or larvae exposed to neuroactive chemicals [32] but do not classify behavior as normal or abnormal. In addition, these unsupervised approaches have utilized highspeed camera systems which are medium to low throughput and have limited potential in the screening of tens of thousands of chemicals for behavioral effects. As introduced above, classification of behavior is one of the primary goals of toxicological screening and tends to result in highly imbalanced datasets and lend themselves to anomaly detection methodologies. While these methods are common in manufacturing [41–43,58], information systems [38,40], security systems [39,45], and financial fraud [37] they have only very recently been applied to biological data [44,59,60]. To the best of our knowledge, this is the first study to explicitly develop a deep autoencoder model for anomaly detection in toxicological behavior studies.

Overall, our results show that a deep autoencoder utilizing raw behavioral tracking data from five dpf zebrafish larvae can accurately distinguish between normal and abnormal behavior. We show that these results are reproducible and allow for the identification of new compounds capable of eliciting abnormal behavior. Further, our models were able to identify abnormal behavior following chemical exposure at lower concentrations than with traditional statistical tests such as the two sample Kolmogorov–Smirnov test (K-S test). Our approach accounts for the high degree of behavioral variability associated with the genetic diversity found within a highly outbred population typical of zebrafish studies, thereby making it extensible to use across labs. Our deep autoencoders only needed seven hundred controls and a three-minute light and three-minute dark cycle to identify differences. The majority of zebrafish labs have historical or the ability to generate similar data that can be used to train their own deep autoencoder models. Looking to the future, neurodevelopmentally active chemicals identified using our deep autoencoder models may represent heretofore undetectable signals of subtle differences in individual responses, suggesting chemicals that should be investigated further as eliciting differential population responses (i.e. interindividual susceptibility differences).

These findings will facilitate the application of behavioral characterization methods discussed above, such as ZebraZoom [32], using highspeed cameras to identify the behavioral traits most perturbed by the chemical exposure and allow for more mechanistic discovery. One of the key innovations presented in this study is leveraging vast amounts of control data generated as part of any high-throughput screening (HTS)–setting the stage for predictive toxicological applications and safety assessments for the enormous backlog of as-yet untested chemicals.

Materials and methods

This section describes the autoencoder models utilizing a semi-supervised ML algorithm and logistic regression (LR) to discriminate between normal and abnormal behavior in chemically exposed five dpf zebrafish. An overview of our approach is shown in Fig 2. Briefly, we created and trained six autoencoder models for each phase of the assay; namely, hyperactive, normal, and hypoactive depending on the control movement in the light or dark phases of the assay. Finally, treated plates were tested on one of these, depending on which category, its controls fell under. We used experimental data collected on a large and diverse compound set of 30 chemicals including an insecticide, nanomaterial, perfluorinated chemicals, and aromatic pollutants at a range of concentrations (133 chemical-concentration pairs) to assess the neurotoxic effects of these chemicals following developmental exposure (S1 Table).

Ethics statement

This study was conducted in accordance with the guidelines and regulations set forth by the Institutional Animal Care and Use Committee (IACUC) at Oregon State University. The protocol was reviewed and approved by the IACUC under the approval number 2021–0227. All procedures involving animals were performed in compliance with the ethical standards of the institution and adhered to the principles of humane animal treatment.

Zebrafish husbandry

Tropical 5D wild-type zebrafish were housed at Oregon State University’s Sinnhuber Aquatic Research Laboratory (SARL, Corvallis, OR) in densities of 1000 fish per 100-gallon tank according to the Oregon State University Animal Use Care and Protocol: 2021–0227 [61]. Fish were maintained at 28°C on a 14:10 h light/dark cycle in recirculating filtered water, supplemented with Instant Ocean salts. Adult, larval and juvenile fish were fed with size-appropriate GEMMA Micro food 2–3 times a day (Skretting). Spawning funnels were placed in the tanks the night prior, and the following morning, embryos were collected and staged [62,63]. Embryos were maintained in embryo medium (EM) in an incubator at 28°C until further processing. EM consisted of 15 mM NaCl, 0.5 mM KCl, 1 mM MgSO4, 0.15 mM KH2PO4, 0.05 mM Na2HPO4, and 0.7 mM NaHCO3 [63].

Developmental chemical exposure

The empirical data used to develop our model were gathered as described in Truong et al. and Noyes et al. [12,64,65]. The experimental design consisted of the 30 unique chemicals tested (S1 Table) with at least 7 replicates (an individual embryo in singular wells of a 96-well plate) at each concentration for each chemical. The concentrations evaluated were based on preliminary studies within the authors’ lab to span lethal and sub-lethal concentration range were possible based on physical chemicals properties including solubility.

Developmental toxicity assessments

Mortality and morphology

At 24 hours post-fertilization (hpf), embryos were screened for mortality, developmental delay, and spontaneous movement [12]. At 120 hpf, mortality, craniofacial abnormalities (eye, snout and jaw), body axis abnormalities, edema (yolk sac and pericardial edema), upright body abnormalities (swim bladder, somite and circulation), touch response brain abnormalities (brain, otic vesicle and pectoral fin), pigment, notochord, and trunk abnormalities (trunk and caudal fin) [12,66,67]. The incidence of abnormality across all morphology endpoints were evaluated as binary outcomes. Any individuals identified with a physical abnormality were excluded from any behavioral analysis as these abnormalities might confound the results.

Photomotor responses

The larval photomotor response (LPR) assay was conducted at 120 hpf when the 96-round well plates of larvae were placed into a Zebrabox (Viewpoint LifeSciences) and larval movement was recorded. The recorded videos were then tracked with Ethovision XT v.11 analysis software for 24 min across 3 cycles of 3 min light: 3 min dark with an initial 6 minute dark acclimation period. The trial time(s), x-position, y-position, distance moved (μm), and velocity (mm/s) by each larva in the 2nd light/dark cycle were the features used for behavioral assessment (S2 Fig). The 2nd light/dark cycle was chosen as it exhibited less noise than the 1st cycle and was less influenced by any learning that might have occurred in the 3rd cycle. For all assessments, data were collected from embryos exposed to nominal concentrations of chemical and uploaded under a unique well-plate identifier into a custom LIMS (Zebrafish Acquisition and Analysis Program [ZAAP])–a MySQL database and analyzed using custom R scripts that were executed in the LIMS background [29].

Data preprocessing and statistical analysis pipeline

Preprocessing

All data processing, statistical analysis and ML were implemented in Python using the open source libraries Tensorflow [68], Keras [69], Scikit-learn [70], Pandas [71], and Numpy [72] within a purpose build Singularity container environment [73]. The x-position and y-position data was standardized relative to the center of each well and forward filled if datapoints were missing. Outliers were normalized to the maximum likely distance a zebrafish larva could move in 1/25th of a second. Considering that the average length of a 5 dpf larval zebrafish is 3.9 mm and can move about 2.5 times it’s body length during a startle response (120 frames at 1000 frames/second) the threshold for distance moved in our system was set at 3.25 mm per frame [53,74]. This resulted in 5,445 of the 30,825,000 frames being normalized.

Statistical analysis

A two sample Kolmogorov–Smirnov test (K-S test), a non-parametric two-sided test with no adjustments for normality or multiple comparisons, was used to compare each chemical-concentration combination with their respective same plate controls (p < 0.05). Interexperimental zebrafish larval response to light/dark cycling is highly variable (S2 Fig). Therefore, it was essential to group the unexposed controls based on the mean from individual 96-well plates compared to mean movement for unexposed controls across all plates. Controls from individual plates with statistically significant (p < 0.01) differences in movement compared to the average of all controls were grouped together as hyperactive, normal, or hypoactive. Following grouping the K-S test was used to compare Individuals in the 30th and 70th percentiles of each chemical-concentration combination were defined as abnormal.

Autoencoder architecture

Deep autoencoders were developed using zebrafish control data to distinguish between normal and abnormal zebrafish behavior. The model was trained on a Dell R740 containing two Intel Xeon processors with 18 cores per processor, 512 GB RAM, and a Tesla-V100-PCIE (31.7 GB). The autoencoders consisted of an input and output layer of fixed-size based on the size of a single phase (25 frames per 180s) of the second light cycle (4500 frames by 5 features). The encoder network was composed of eight fully connected hidden layers using a normal kernel initialization, tanh activation, a dropout value of 0.2, L1 and L2 regularization values of 1e-05, and an adadelta optimizer. The size of each hidden layer was reduced by increasing multiples of 15 and resulted in a compressed representation (bottleneck) size of 250. The decoder network was composed of six fully connected hidden layers using tanh activation, and a dropout value of 0.2. All hidden layers used an adadelta optimizer (learning_rate = 0.001, rho = 0.95, and epsilon = 1e-07) and mean squared error for the loss function [75–77]. For each model, we optimized the hyperparameters (i.e., the number of hidden layers, the number of nodes in the layers, loss functions, optimizers, regularization rates, and dropout rates) by grid search technique trained on all control data over 500 epochs using Cohens Kappa statistic as the objective metric. The final encoder models were trained over the course of 125000 epochs. The resulting compressed representation was used as input into a logistic regression layer trained using a 100 fold cross-validation with each fold consisting of 4000 epochs using a limited-memory BFGS solver. The code and dataset are available at GitHub [https://github.com/Tanguay-Lab/Manuscripts/tree/main/Green_et_al_(2024)_Manuscript].

Network performance and evaluation

The data showed strong normal vs abnormal class imbalance (Fig 1). Classifiers may be biased towards the major class (normal) and therefore, show poor performance accuracy for the minor class (abnormal) [78]. Normal vs abnormal classification accuracy was evaluated using a confusion matrix, Cohen’s Kappa statistic, and area under the receiver operating characteristic (AUROC) as Kappa and AUROC measure model accuracy, while compensating for simple chance [79]. The primary metrics we used from the confusion matrix included sensitivity (SE), specificity (SP), and positive predictive value (PPV) as these parameters give us the true positive rate, true negative rate, and the proportion of true positives amongst all positive calls [80–82]. Chemical-concentration combinations were defined as abnormal if the autoencoders identified more individual as abnormal in the exposed than their respective controls and at least 25% of the individuals were abnormal. Permutation feature importance was used to evaluate which features are the most important for model performance. In brief, one feature (variable) is shuffled randomly and all features are fed into the model the resulting Kappa and AUROC values are calculated. This is repeated 1000 times per feature and average Kappa and AUROC are calculated across each shuffle [83]. To determine why one feature might be more important than another a coefficient of variation was calculated for each of the features in the control and exposed groups (variation() in the SciPy package).

Experimental confirmation of autoencoder findings

Following model development two chemicals were identified for follow-up laboratory testing. We generated new data using 2-Methylphenanthrene, and Nonafluoropentanamide. 2-Methylphenanthrene was chosen as the autoencoder identified it was different from controls at a much lower concentration than a K-S test of distance moved and angular velocity while Nonafluoropentanamide was selected as it was not identified using either a K-S test of distance moved and angular velocity. Similarity between the results was determined by comparing fourth order polynomial curve fits with and a significance threshold of p < 0.05.

Supporting information

S1 Table Study chemicals and their common use.

(XLSX)

S2 Table Statistical results for behavioral response analysis.

(XLSX)

S1 Fig Loss function results during training.

Changes of loss functions during the training of (A) light-hypoactive controls, (B) light-normal controls, (C) light-hyperactive controls, (D) dark-hypoactive controls, (E) dark-normal controls, (F) dark-hyperactive controls. Blue line–training data (controls-only), orange line–test data (abnormal-only).

(TIF)

S2 Fig Interexperimental behavioral response to light/dark cycling in control larval zebrafish.

Zebrafish larvae were statically exposed to a chemical from six hpf until five dpf. At five dpf, behavior was measured under environmental conditions of continuous light for three minutes (0–180) followed by three minutes of dark (180–360). This plot shows representative control behavior data (n = 7 per line) classified as hyperactive (blue line), normal (green line) or hypoactive (purple line). The insert shows an example of larval behavioral tracks produced by Ethovision XT software. Figure depicts means ± SEM.

(TIF)

We would like to thank the staff at Sinnhuber Aquatic Research Laboratory, and John Lam for his contribution to reprocessing videos.

10.1371/journal.pcbi.1012423.r001
Decision Letter 0
Komarova Natalia L. Section Editor
Scarpino Samuel V. Academic Editor
© 2024 Komarova, Scarpino
2024
Komarova, Scarpino
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Submission Version0
15 Dec 2023

Dear Dr. Reif,

Thank you very much for submitting your manuscript "Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Samuel V. Scarpino

Academic Editor

PLOS Computational Biology

Natalia Komarova

Section Editor

PLOS Computational Biology

***********************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Overall the authors did a could job at highlighting why one would want to opt to use deep autoencoders when analyzing behavioral data. A few sections were hard to follow. For example, the authors refer "traditional statistical methods" throughout the manuscript and vaguely comment on some of the caveats of the such methods for analyzing toxicological behavioral data. But the latter is done only at a superficial level. The authors don't put forward enough the intricacies of the toxicological behavioral data. What is particular about those data? Moreover, what are the specific caveats of the traditional statistical methods? Is one of the main findings of this study that autoencoders helped to discriminate between normal and abnormal behaviors for even very low concentration of toxins?

Below I provide further details of the aforementioned sections that could be rephrase to both improve flow and clarity of the manuscript -- especially for readers who might not be familiar to such data or methods.

L19: I would suggest to change to "model organism" rather than tool.

L75: How is this finding specifically crucial for public health? Be more specific

L81-82: "...making it applicable to multiple laboratories" Do the authors mean that due the high flexibility of autoencoders, other people working on similar type of highly variable data would be able to apply similar network architecture to their data?

L138: what does "dpf" stands for?

L138-142: It would be great if the authors could be more specific by elaborating how anomaly detection could potentially solve that issue mentioned in the example rather than vaguely mentioning that this issue could be tackled by machine learning. Perhaps the authors can specify the exact issues, from an analytical point of view, how does the traditional methods not account for them? How does anomaly detection provide an alternative path to solve this issue?

L158: "traditional statistical methodologies" - this is very vague, which methods are the authors specifically referring to?

L231: Do the authors refer to K-S test as the traditional statistical analysis?

L354 -- L363: In general I had a hard time understanding how the K-S test were performed? Did the authors compare the empirical cumulative distributions of individuals from different groups (hypo-, normal, hyperactive) to their controls to classify whether they were normal vs abnormal? What is the threshold of "normal" v/s "abnormal" behaviors?

Reviewer #2: The authors present a timely study concept of the implementation of autoencoder-based pattern recognition that is described as an improvement upon standard statistical methods of classification of zebrafish exposed to various neurotoxic chemical treatments. Despite a useful discussion of the need for more sophisticated analysis methods to be applied to behavioral classification as these techniques are increasingly implemented in various fields, the authors have not established a correct standard in the field for the comparison base that they use for their main analysis design.

A major drawback of the design of the study is that the authors frame all improvements with their autoencoder method against basic statistical methods that are applied to classification based upon a metric of distance traveled, velocity, x-position, and y-position. Since each of these 4 measures are discussed, it is unclear to the reader whether all 4 of these measures were used for the statistical model or if only distance traveled is used. Please clarify this in the revision.

This measure of distance traveled is not the typical standard in the field for classification of subjects that are treated with toxins. Rather, distance traveled is often utilized as a locomotor control, which often is not capable of detecting variation between subjects treated with a toxin.

If locomotor impairment in the form of reduced distance traveled is found, this standardly indicates that subjects are not able to demonstrate more complex effects of toxin exposure. Rather reduced distance traveled would show that subjects display gross locomotor impairment, which could be attributed to multiple effects. This gross locomotor impairment could be due to inhibition of motor control or behavioral impacts and cannot be attributed to either effect as they are confounded with one another when dose of toxin is high enough to induce locomotor impairment. For the reasons discussed above, other metrics of behavioral consequences of toxin exposure including angular velocity and turn angle are utilized by many of toxicological studies utilizing the zebrafish model as they are sensitive to toxin effects, which may otherwise not be measured using metrics of distance traveled and velocity alone.

This standard of comparison of the autoencoder method against the distance traveled / velocity metric is flawed for these reasons. As such, it would be a great improvement upon the basic points of this study if a more standard metric such as angular velocity, or turn angle is used as the comparison base for classification using standard statistical methods. Otherwise, it is already expected that the distance traveled metric should not vary across subjects if the subjects are receiving a dose of toxin that is known to cause a behavioral impact without gross locomotor impairment (since distance traveled is a locomotor control and is not suggested as a metric for differentiating subjects using standard methods).

The basic point of the deep autoencoder improving upon standard methods would be greatly strengthened if the authors perform an analysis based on metrics that are more established for assessing subtle toxic effects (that are shown even when distance traveled does not change, such as changed angular velocity or changed turn angle, compared to controls.) Please address some form of an analysis of metrics other than distance traveled and velocity as the basis of comparison against the autoencoder-based strategy this in revision of this manuscript.

Is there any improvement of the autoencoder strategy compared to metrics such as angular velocity and turn angle? If there is no additional improvement from the autoencoder strategy when using these more typical measurements of subtle toxic effects, then the authors should declare this in the manuscript for full transparency and for generalizability of their findings to the field as a whole, in which these measures are commonly utilized.

I have discussed this point in detail since it is essential to address to improve the theoretical basis of the study design. Aside from this point, there are some additional points below. It is uncertain how the doses of toxins were chosen in this study. Were they based on the literature from other researchers or arrived at due to preliminary / unpublished studies within the authors’ lab or previously published work within the authors' lab. In either case, it should be specifically cited how these doses were arrived upon and whether the authors assessed standard metrics for establishing dose such as LD50 for each toxin. Please address this in revision of this manuscript.

It is unclear to the reader how the authors established the subjects as hyperactive, normal, or hypoactive. Was this based on categorical splitting of the data into equally numbered groups as is done with a traditional median split? Or were these classes based on specific cut-off points of the subjects based on other studies in the field which have standard cut-offs for these metrics? Please address this in revision of this manuscript.

The discussion section would benefit from some explanation of the purported reasons why certain classes of chemicals are better differentiated by the autoencoder-based method vs. standard statistical methods. Does this vary according to chemical class / structure?

It is unclear what type of analysis was performed in Figure 4 to determine that the trained model and the test dataset’s model produce similar results or if this is simply a qualitative observation that the training and test data models seems similar to the authors. It would be valid to claim similar results on the basis of a non-significant statistical test, rather than just a visual inspection.

Some editorial changes: On the referenced GitHub page from lines 384-385, the Autoencoder Model Design and the ten Jupyter Notebook file links appear to be broken; In line 29 should be “abnormal behavioral effects” instead of “abnormal behavioral”

Based on the need to sufficiently address these points described above, I would recommend resubmitting the manuscript with major revision.

Reviewer #3: This manuscript submitted by Green and colleagues reports a method based on machine learning to improve identification of abnormal behaviour of zebrafish larvae. Considering that behaviour of zebrafish is a frequently used methods to evaluate (neuro)toxicity of chemicals and the large amount of chemicals not tested yet, any way to improve sensitivity or accuracy is of utmost importance.

The Ms is well organised and easy to read (this reviewer not being able to evaluate the computational part focused on the behavioural and fish parts) and globally acceptable for publication provided below comments are addressed. One point would deserve more in-depth discussion because some results are puzzling. Indeed, in some cases, a higher sensitivity can explain differences between statistical and autoencoder methods (Table 2), but in other ones (Multi-Walled Carbon Nanotube; 1-Methylphenanthrene, Dibenzo[e-l]pyrene, Aminohexafluoropropan-2-ol) differences are in the middle of the concentrations range. How the Authors can ensure they are not false positive?

Given that the procedure appears to require high-level computational expertise, how can this help as many labs as possible benefit from this new approach?

Specific comments

L116 The Authors write "Zebrafish larvae show mature swimming patterns … at four to five days post-fertilization (dpf), which can be assessed using various locomotor behavioral assays…"

The sense of "mature swimming pattern" is not clear; it does not correspond to a specific situation. In addition 1) behaviour is very variable between 4 and 6 dpf and 2) response to chemicals varies significantly with often no response at 4 dpf while defect can be detected at 5 dpf. The sens of mature should be precisely defined (or the sentence rewritten) and 4 to 5 dpf should be distinguished in terms of maturity of responses.

L256 "more recently" is not correct since most studies are older

L297 reference is made to Fig. 3 while ti should be Fig. 2.

L318 type of 96 well plates should be indicated since round well plates have proven to produce high occurrence of swim bladder inflation defects. According to figures round well plates were used, so percentage of malformed larvae (see next comment) should be provided. Medium renewal or its absence should be clearly indicated.

L324 morphological defects scored at 24 and 120 hpf should be indicated

L331-332 the recording protocol is not correctly described, 3 cycles of 3+3 phases are equivalent to 18 minutes and not 24. Perhaps because acclimation precedes the first cycle. This should be corrected, and for acclimation if any, was it in the light or in the dark?

L335-336 so what is the purpose of this third cycle?

L544 ref format not correct

Even though Tanguay's laboratory has produced a large amount of data dealing with the behavior of zebrafish, a fairer representation of the literature should be presented!

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No: Two links on Github appear to be broken upon attempt to access: On the referenced GitHub page from lines 384-385, the Autoencoder Model Design and the ten Jupyter Notebook file links appear to be broken

It is stated that the data is available upon request instead of being uploaded to a public repository.

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

10.1371/journal.pcbi.1012423.r002
Author response to Decision Letter 0
Submission Version1
10 Apr 2024

Attachment Submitted filename: PCOMPBIOL-D-23-00581_R1_Author response to Reviewers.docx

10.1371/journal.pcbi.1012423.r003
Decision Letter 1
Komarova Natalia L. Section Editor
Scarpino Samuel V. Academic Editor
© 2024 Komarova, Scarpino
2024
Komarova, Scarpino
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Submission Version1
6 May 2024

Dear Dr. Reif,

Thank you very much for submitting your manuscript "Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Samuel V. Scarpino

Academic Editor

PLOS Computational Biology

Natalia Komarova

Section Editor

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #2: Thank you for including thorough responses to the majority of the issues that I addressed in my initial review. The few items remaining to address are included below:

Supplementary figures or table information should be included to display the directionality of the effects described in Supplementary Table 2 and Supplementary Table 3. As is, it is unknown whether the chemicals cause an increase or decrease in each of the distance traveled and angular velocity measures for each chemical. Please include figures showing this or information in a supplementary table showing the mean and standard deviation (or standard error) and number of samples for each chemical and control levels for both distance traveled and angular velocity.

Please address the data availability issue described in the data availability section. I have included this here for reference: The full set of data and code are not provided as part of the manuscript or its supporting information or deposited to a public repository. Although part of the data are provided the manuscript states that you need to contact the authors to access the full dataset and code. This does not yet meet the PLOS Data policy description.

Reviewer #3: Thanks for having addressed most of the comments raised. Could you elaborate a bit about the way other laboratories will be able to adopt this approach on the one hand and on the other hand the flexibility of the behavioral protocol (since there is no standard protocol). Second point please provide information about lighting during acclimation.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: No: The full set of data and code are not provided as part of the manuscript or its supporting information or deposited to a public repository. Although part of the data are provided the manuscript states that you need to contact the authors to access the full dataset and code. This does not yet meet the PLOS Data policy description.

Reviewer #3: None

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #3: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

References:

Review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

10.1371/journal.pcbi.1012423.r004
Author response to Decision Letter 1
Submission Version2
1 Aug 2024

Attachment Submitted filename: Author response to Reviewers_July2024.docx

10.1371/journal.pcbi.1012423.r005
Decision Letter 2
Komarova Natalia L. Section Editor
Scarpino Samuel V. Academic Editor
© 2024 Komarova, Scarpino
2024
Komarova, Scarpino
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Submission Version2
15 Aug 2024

Dear Dr. Reif,

We are pleased to inform you that your manuscript 'Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Samuel V. Scarpino

Academic Editor

PLOS Computational Biology

Natalia Komarova

Section Editor

PLOS Computational Biology

***********************************************************

10.1371/journal.pcbi.1012423.r006
Acceptance letter
Komarova Natalia L. Section Editor
Scarpino Samuel V. Academic Editor
© 2024 Komarova, Scarpino
2024
Komarova, Scarpino
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
31 Aug 2024

PCOMPBIOL-D-23-00581R2

Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies

Dear Dr Reif,

I am pleased to inform you that your manuscript has been formally accepted for publication in PLOS Computational Biology. Your manuscript is now with our production department and you will be notified of the publication date in due course.

The corresponding author will soon be receiving a typeset proof for review, to ensure errors have not been introduced during production. Please review the PDF proof of your manuscript carefully, as this is the last chance to correct any errors. Please note that major changes, or those which affect the scientific understanding of the work, will likely cause delays to the publication date of your manuscript.

Soon after your final files are uploaded, unless you have opted out, the early version of your manuscript will be published online. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers.

Thank you again for supporting PLOS Computational Biology and open-access publishing. We are looking forward to publishing your work!

With kind regards,

Zsofia Freund

PLOS Computational Biology | Carlyle House, Carlyle Road, Cambridge CB4 3DN | United Kingdom ploscompbiol@plos.org | Phone +44 (0) 1223-442824 | ploscompbiol.org | @PLOSCompBiol
==== Refs
References

1 Neurodevelopmental Diseases. In: National Institute of Environmental Health Sciences [Internet]. 12 Jan 2021 [cited 12 Jan 2021]. Available: https://www.niehs.nih.gov/research/supported/health/neurodevelopmental/index.cfm.
2 Boyle CA , Boulet S , Schieve LA , Cohen RA , Blumberg SJ , Yeargin-Allsopp M , et al . Trends in the prevalence of developmental disabilities in US children, 1997–2008. Pediatrics. 2011;127 : 1034–1042. doi: 10.1542/peds.2010-2989 21606152
3 US EPA. Health: Neurodevelopmental Disorders–Report Contents. In: Health: Neurodevelopmental Disorders–Report Contents [Internet]. 10 Jun 2015 [cited 12 Jan 2021]. Available: https://www.epa.gov/americaschildrenenvironment/health-neurodevelopmental-disorders-report-contents/.
4 Grandjean P , Landrigan PJ . Neurobehavioural effects of developmental toxicity. The Lancet Neurology. 2014;13 : 330–338. doi: 10.1016/S1474-4422(13)70278-3 24556010
5 Rock KD , Patisaul HB . Environmental Mechanisms of Neurodevelopmental Toxicity. Curr Environ Health Rep. 2018;5 : 145–157. doi: 10.1007/s40572-018-0185-0 29536388
6 Green AJ , Planchart A . The neurological toxicity of heavy metals: A fish perspective. Comp Biochem Physiol C Toxicol Pharmacol. 2018;208 : 12–19. doi: 10.1016/j.cbpc.2017.11.008 29199130
7 Miller JA , Ding S-L , Sunkin SM , Smith KA , Ng L , Szafer A , et al . Transcriptional landscape of the prenatal human brain. Nature. 2014;508 : 199–206. doi: 10.1038/nature13185 24695229
8 A Blueprint for Brain Development. In: NIH Director’s Blog [Internet]. 8 Apr 2014 [cited 12 Jan 2021]. Available: https://directorsblog.nih.gov/2014/04/08/a-blueprint-for-brain-development/.
9 US EPA. About the TSCA Chemical Substance Inventory. In: About the TSCA Chemical Substance Inventory [Internet]. 2 Mar 2015 [cited 23 Aug 2019]. Available: https://www.epa.gov/tsca-inventory/about-tsca-chemical-substance-inventory.
10 Krewski D , Andersen ME , Tyshenko MG , Krishnan K , Hartung T , Boekelheide K , et al . Toxicity testing in the 21st century: progress in the past decade and future perspectives. Arch Toxicol. 2020;94 : 1–58. doi: 10.1007/s00204-019-02613-4 31848664
11 Wambaugh JF , Setzer RW , Reif DM , Gangwal S , Mitchell-Blackwood J , Arnot JA , et al . High-Throughput Models for Exposure-Based Chemical Prioritization in the ExpoCast Project. Environ Sci Technol. 2013; 130711145716006. doi: 10.1021/es400482g 23758710
12 Truong L , Reif DM , St Mary L , Geier MC , Truong HD , Tanguay RL . Multidimensional In Vivo Hazard Assessment Using Zebrafish. Toxicol Sci. 2014;137 : 212–233. doi: 10.1093/toxsci/kft235 24136191
13 Richard AM , Judson RS , Houck KA , Grulke CM , Volarath P , Thillainadarajah I , et al . ToxCast Chemical Landscape: Paving the Road to 21st Century Toxicology. Chem Res Toxicol. 2016;29 : 1225–1251. doi: 10.1021/acs.chemrestox.6b00135 27367298
14 Judson RS , Houck KA , Kavlock RJ , Knudsen TB , Martin MT , Mortensen HM , et al . In vitro screening of environmental chemicals for targeted testing prioritization: the ToxCast project. Environ Health Perspect. 2010;118 : 485–492. doi: 10.1289/ehp.0901392 20368123
15 Thomas RS , Black MB , Li L , Healy E , Chu T-M , Bao W , et al . A Comprehensive Statistical Analysis of Predicting In Vivo Hazard Using High-Throughput In Vitro Screening. Toxicol Sci. 2012;128 : 398–417. doi: 10.1093/toxsci/kfs159 22543276
16 Bugel SM , Tanguay RL , Planchart A . Zebrafish: A Marvel of High-Throughput Biology for 21st Century Toxicology. Curr Envir Health Rpt. 2014;1 : 341–352. doi: 10.1007/s40572-014-0029-5 25678986
17 Planchart A , Green AJ , Hoyo C , Mattingly CJ . Heavy Metal Exposure and Metabolic Syndrome: Evidence from Human and Model System Studies. Curr Environ Health Rep. 2018;5 : 110–124. doi: 10.1007/s40572-018-0182-3 29460222
18 Kalueff AV , Stewart AM , Gerlai R . Zebrafish as an emerging model for studying complex brain disorders. Trends in Pharmacological Sciences. 2014;35 : 63–75. doi: 10.1016/j.tips.2013.12.002 24412421
19 Lowery LA , Sive H . Strategies of vertebrate neurulation and a re-evaluation of teleost neural tube formation. Mechanisms of Development. 2004;121 : 1189–1197. doi: 10.1016/j.mod.2004.04.022 15327780
20 Tropepe V , Sive HL . Can zebrafish be used as a model to study the neurodevelopmental causes of autism? Genes, Brain and Behavior. 2003;2 : 268–281. doi: 10.1034/j.1601-183x.2003.00038.x 14606692
21 Horzmann KA , Freeman JL . Zebrafish Get Connected: Investigating Neurotransmission Targets and Alterations in Chemical Toxicity. Toxics. 2016;4 : 19. doi: 10.3390/toxics4030019 28730152
22 Howe K , Clark MD , Torroja CF , Torrance J , Berthelot C , Muffato M , et al . The zebrafish reference genome sequence and its relationship to the human genome. Nature. 2013;496 : 498–503. doi: 10.1038/nature12111 23594743
23 Postlethwait JH , Yan YL , Gates MA , Horne S , Amores A , Brownlie A , et al . Vertebrate genome evolution and the zebrafish gene map. Nat Genet. 1998;18 : 345–349. doi: 10.1038/ng0498-345 9537416
24 Basnet RM , Zizioli D , Taweedet S , Finazzi D , Memo M . Zebrafish Larvae as a Behavioral Model in Neuropharmacology. Biomedicines. 2019;7 : 23. doi: 10.3390/biomedicines7010023 30917585
25 Hernandez RE , Galitan L , Cameron J , Goodwin N , Ramakrishnan L . Delay of Initial Feeding of Zebrafish Larvae Until 8 Days Postfertilization Has No Impact on Survival or Growth Through the Juvenile Stage. Zebrafish. 2018;15 : 515–518. doi: 10.1089/zeb.2018.1579 30089231
26 Tegelenbosch RAJ , Noldus LPJJ, Richardson MK, Ahmad F. Zebrafish embryos and larvae in behavioural assays. Behaviour. 2012;149 : 1241–1281. doi: 10.1163/1568539X-00003020
27 Burgess HA , Granato M . Modulation of locomotor activity in larval zebrafish during light adaptation. Journal of Experimental Biology. 2007;210 : 2526–2539. doi: 10.1242/jeb.003939 17601957
28 Emran F , Rihel J , Dowling JE . A behavioral assay to measure responsiveness of zebrafish to changes in light intensities. J Vis Exp. 2008. doi: 10.3791/923 19078942
29 Truong L , Bugel SM , Chlebowski A , Usenko CY , Simonich MT , Simonich SLM , et al . Optimizing multi-dimensional high throughput screening using zebrafish. Reproductive Toxicology. 2016;65 : 139–147. doi: 10.1016/j.reprotox.2016.05.015 27453428
30 Zhang G , Truong L , Tanguay RL , Reif DM . A New Statistical Approach to Characterize Chemical-Elicited Behavioral Effects in High-Throughput Studies Using Zebrafish. PLoS ONE. 2017;12 : e0169408. doi: 10.1371/journal.pone.0169408 28099482
31 Kalueff AV , Gebhardt M , Stewart AM , Cachat JM , Brimmer M , Chawla JS , et al . Towards a Comprehensive Catalog of Zebrafish Behavior 1.0 and Beyond. Zebrafish. 2013;10 : 70–86. doi: 10.1089/zeb.2012.0861 23590400
32 Mirat O , Sternberg JR , Severi KE , Wyart C . ZebraZoom: an automated program for high-throughput behavioral analysis and categorization. Front Neural Circuits. 2013;7. doi: 10.3389/fncir.2013.00107 23386812
33 Balik-Meisner M , Truong L , Scholl EH , La Du JK , Tanguay RL , Reif DM . Elucidating Gene-by-Environment Interactions Associated with Differential Susceptibility to Chemical Exposure. Environmental Health Perspectives. 2018;126. doi: 10.1289/EHP2662 29968567
34 Arifoglu D , Bouchachia A . Activity Recognition and Abnormal Behaviour Detection with Recurrent Neural Networks. Procedia Computer Science. 2017;110 : 86–93. doi: 10.1016/j.procs.2017.06.121
35 Xia C , Fu L , Liu Z , Liu H , Chen L , Liu Y . Aquatic Toxic Analysis by Monitoring Fish Behavior Using Computer Vision: A Recent Progress. Journal of Toxicology. 2018;2018 : e2591924. doi: 10.1155/2018/2591924 29849612
36 Pereira TD , Shaevitz JW , Murthy M . Quantifying behavior to understand the brain. Nat Neurosci. 2020;23 : 1537–1549. doi: 10.1038/s41593-020-00734-z 33169033
37 Awoyemi JO, Adetunmbi AO, Oluwadare SA. Credit card fraud detection using machine learning techniques: A comparative analysis. 2017 International Conference on Computing Networking and Informatics (ICCNI). 2017. pp. 1–9. doi: 10.1109/ICCNI.2017.8123782
38 Pachauri G , Sharma S . Anomaly Detection in Medical Wireless Sensor Networks using Machine Learning Algorithms. Procedia Computer Science. 2015;70 : 325–333. doi: 10.1016/j.procs.2015.10.026
39 Sargolzaei A , Crane CD , Abbaspour A , Noei S . A Machine Learning Approach for Fault Detection in Vehicular Cyber-Physical Systems. 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA). 2016. pp. 636–640. doi: 10.1109/ICMLA.2016.0112
40 Warriach EU , Tei K . Fault Detection in Wireless Sensor Networks: A Machine Learning Approach. 2013 IEEE 16th International Conference on Computational Science and Engineering. 2013. pp. 758–765. doi: 10.1109/CSE.2013.116
41 Fazai R , Abodayeh K , Mansouri M , Trabelsi M , Nounou H , Nounou M , et al . Machine learning-based statistical testing hypothesis for fault detection in photovoltaic systems. Solar Energy. 2019;190 : 405–413. doi: 10.1016/j.solener.2019.08.032
42 Jaiswal V , Ruskin A . Mooring Line Failure Detection Using Machine Learning. OnePetro; 2019. doi: 10.4043/29511-MS
43 Nicholaus IT , Park JR , Jung K , Lee JS , Kang D-K . Anomaly Detection of Water Level Using Deep Autoencoder. Sensors (Basel). 2021;21 : 6679. doi: 10.3390/s21196679 34640997
44 Frassek M , Arjun A , Bolhuis PG . An extended autoencoder model for reaction coordinate discovery in rare event molecular dynamics datasets. J Chem Phys. 2021;155 : 064103. doi: 10.1063/5.0058639 34391359
45 Feng J , Liang Y , Li L . Anomaly Detection in Videos Using Two-Stream Autoencoder with Post Hoc Interpretability. Comput Intell Neurosci. 2021;2021 : 7367870. doi: 10.1155/2021/7367870 34354745
46 Ranjan C, Reddy M, Mustonen M, Paynabar K, Pourak K Dataset: Rare Event Classification in Multivariate Time Series. arXiv:180910717 [cs, stat]. 2019 [cited 4 Jan 2022]. Available: http://arxiv.org/abs/1809.10717.
47 Goodfellow I , Bengio Y , Courville A . Chapter 14—Autoencoders. Deep Learning. MIT Press; 2016. pp. 499–523.
48 Le Borgne Y-A , Siblini W , Lebichot B , Bontempi G . Autoencoders and anomaly detection—Reproducible Machine Learning for Credit Card Fraud detection—Practical handbook. Reproducible Machine Learning for Credit Card Fraud Detection—Practical Handbook. Université Libre de Bruxelles; 2022. Available: https://github.com/Fraud-Detection-Handbook/fraud-detection-handbook.
49 Gupta A , Singh S . ML | Classifying Data using an Auto-encoder. In: GeeksforGeeks [Internet]. 25 Jun 2019 [cited 10 Feb 2020]. Available: https://www.geeksforgeeks.org/ml-classifying-data-using-an-auto-encoder/.
50 Brown-Leung JM , Cannon JR . Neurotransmission Targets of Per- and Polyfluoroalkyl Substance Neurotoxicity: Mechanisms and Potential Implications for Adverse Neurological Outcomes. Chem Res Toxicol. 2022;35 : 1312–1333. doi: 10.1021/acs.chemrestox.2c00072 35921496
51 Soukupova M , Binaschi A , Falcicchia C , Palma E , Roncon P , Zucchini S , et al . Increased extracellular levels of glutamate in the hippocampus of chronically epileptic rats. Neuroscience. 2015;301 : 246–253. doi: 10.1016/j.neuroscience.2015.06.013 26073699
52 Fero K , Yokogawa T , Burgess HA . The Behavioral Repertoire of Larval Zebrafish. In: Kalueff AV , Cachat JM , editors. Zebrafish Models in Neurobehavioral Research. Totowa, NJ: Humana Press; 2011. pp. 249–291. doi: 10.1007/978-1-60761-922-2_12
53 Burgess HA , Granato M . Sensorimotor Gating in Larval Zebrafish. J Neurosci. 2007;27 : 4984–4994. doi: 10.1523/JNEUROSCI.0615-07.2007 17475807
54 Kimmel CB , Patterson J , Kimmel RO . The development and behavioral characteristics of the startle response in the zebra fish. Developmental Psychobiology. 1974;7 : 47–60. doi: 10.1002/dev.420070109 4812270
55 Budick SA , O’Malley DM . Locomotor repertoire of the larval zebrafish: swimming, turning and prey capture. Journal of Experimental Biology. 2000;203 : 2565–2579. doi: 10.1242/jeb.203.17.2565 10934000
56 Burgess HA , Granato M . Flote v2.1: Biological Tracking Software. 2007.
57 Zhang H , Lenaghan SC , Connolly MH , Parker LE . Zebrafish Larva Locomotor Activity Analysis Using Machine Learning Techniques. 2013 12th International Conference on Machine Learning and Applications. 2013. pp. 161–166. doi: 10.1109/ICMLA.2013.35
58 Fan C , Xiao F , Zhao Y , Wang J . Analytical investigation of autoencoder-based methods for unsupervised anomaly detection in building energy data. Applied Energy. 2018;211 : 1123–1135. doi: 10.1016/j.apenergy.2017.12.005
59 Homayouni H , Ray I , Ghosh S , Gondalia S , Kahn MG . Anomaly Detection in COVID-19 Time-Series Data. SN Comput Sci. 2021;2 : 279. doi: 10.1007/s42979-021-00658-w 34027432
60 Nwokedi EI , Bains R , Bidaut L , Wells S , Ye X , Brown JM . Unsupervised detection of mouse behavioural anomalies using two-stream convolutional autoencoders. ArXiv. 2021.
61 Barton CL , Johnson EW , Tanguay RL . Facility Design and Health Management Program at the Sinnhuber Aquatic Research Laboratory. Zebrafish. 2016;13 : S-39-S-43. doi: 10.1089/zeb.2015.1232 26981844
62 Kimmel CB , Ballard WW , Kimmel SR , Ullmann B , Schilling TF . Stages of embryonic development of the zebrafish. Dev Dyn. 1995;203 : 253–310. doi: 10.1002/aja.1002030302 8589427
63 Westerfield M. The zebrafish book: a guide for the laboratory use of zebrafish (Danio rerio). Eugene, OR: Eugene, OR: Univ. of Oregon Press, 2007.; 2007. Available: https://catalog.lib.ncsu.edu/catalog/NCSU2481113.
64 Noyes PD , Haggard DE , Gonnerman GD , Tanguay RL . Advanced Morphological—Behavioral Test Platform Reveals Neurodevelopmental Defects in Embryonic Zebrafish Exposed to Comprehensive Suite of Halogenated and Organophosphate Flame Retardants. Toxicol Sci. 2015;145 : 177–195. doi: 10.1093/toxsci/kfv044 25711236
65 Truong L , Rericha Y , Thunga P , Marvel S , Wallis D , Simonich MT , et al . Systematic developmental toxicity assessment of a structurally diverse library of PFAS in zebrafish. Journal of Hazardous Materials. 2022;431 : 128615. doi: 10.1016/j.jhazmat.2022.128615 35263707
66 Thunga P , Truong L , Tanguay RL , Reif DM . Concurrent Evaluation of Mortality and Behavioral Responses: A Fast and Efficient Testing Approach for High-Throughput Chemical Hazard Identification. Frontiers in Toxicology. 2021;3 . Available: https://www.frontiersin.org/articles/10.3389/ftox.2021.670496. doi: 10.3389/ftox.2021.670496 35295121
67 Zhang G , Marvel S , Truong L , Tanguay RL , Reif DM . Aggregate entropy scoring for quantifying activity across endpoints with irregular correlation structure. Reprod Toxicol. 2016;62 : 92–99. doi: 10.1016/j.reprotox.2016.04.012 27132190
68 Abadi Martín , Agarwal Ashish , Barham Paul , Brevdo Eugene , Chen Zhifeng , Citro Craig , et al . TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015. Available: https://www.tensorflow.org/.
69 U.S. Environmental Protection Agency. Comptox Chemicals Dashboard: Master List of PFAS Substances (Version2). 10 Aug 2021 [cited 19 May 2022]. Available: https://comptox.epa.gov/dashboard/chemical-lists/pfasmaster.
70 Pedregosa F , Varoquaux G , Gramfort A , Michel V , Thirion B , Grisel O , et al . Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research. 2011;12 : 2825–2830.
71 McKinney W. Data Structures for Statistical Computing in Python. Austin, Texas; 2010. pp. 56–61. doi: 10.25080/Majora-92bf1922-00a
72 Harris CR , Millman KJ , Walt SJ van der , Gommers R , Virtanen P , Cournapeau D , et al . Array programming with NumPy. Nature. 2020;585 : 357–362. doi: 10.1038/s41586-020-2649-2 32939066
73 Sylabs.io. Singularity. Sylabs.io; 2019. Available: https://sylabs.io/singularity/.
74 ZFIN Zebrafish Developmental Stages. [cited 5 Apr 2022]. Available: https://zfin.org/zf_info/zfbook/stages/index.html.
75 Ramachandran P, Zoph B, Le QV. Searching for Activation Functions. arXiv:171005941 [cs]. 2017 [cited 4 Sep 2020]. Available: http://arxiv.org/abs/1710.05941.
76 Osl M , Netzer M , Dreiseitl S , Baumgartner C . Applied Data Mining: From Biomarker Discovery to Decision Support Systems. In: Trajanoski Z , editor. Computational Medicine. Vienna: Springer Vienna; 2012. pp. 173–184. doi: 10.1007/978-3-7091-0947-2_10
77 He K, Zhang X, Ren S, Sun J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. 2015 IEEE International Conference on Computer Vision (ICCV). 2015. pp. 1026–1034. doi: 10.1109/ICCV.2015.123
78 Lemaître G , Nogueira F , Aridas CK . Imbalanced -learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning. Journal of Machine Learning Research. 2017;18 : 1–5.
79 Ben-David A. About the relationship between ROC curves and Cohen’s kappa. Engineering Applications of Artificial Intelligence. 2008;21 : 874–882. doi: 10.1016/j.engappai.2007.09.009
80 Pearson K. On the theory of contingency and its relation to association and normal correlation. Drapers Company Research Memoirs. Dulau and Co.; 1904. Available: https://archive.org/details/cu31924003064833/page/n1/mode/2up.
81 Townsend JT . Theoretical analysis of an alphabetic confusion matrix. Perception & Psychophysics. 1971;9 : 40–50. doi: 10.3758/BF03213026
82 Parikh R , Mathai A , Parikh S , Chandra Sekhar G , Thomas R . Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol. 2008;56 : 45–50. doi: 10.4103/0301-4738.37595 18158403
83 Breiman L. Random Forests. Machine Learning. 2001;45 : 5–32. doi: 10.1023/A:1010933404324
