==== Front F1000Res F1000Res F1000Research 2046-1402 F1000 Research Limited London, UK 10.12688/f1000research.27158.3 Software Tool Article Articles DeepCLEM: automated registration for correlative light and electron microscopy using deep learning [version 3; peer review: 1 approved Seifert Rick Data Curation Formal Analysis Investigation Methodology Software Validation Writing – Original Draft Preparation Writing – Review & Editing 12 Markert Sebastian M. Data Curation Investigation Resources Validation Writing – Review & Editing https://orcid.org/0000-0001-9069-156X 2 Britz Sebastian Data Curation Investigation Resources Validation Writing – Review & Editing https://orcid.org/0000-0002-7566-005X 2 Perschin Veronika Data Curation Investigation Resources Validation Writing – Review & Editing 2 Erbacher Christoph Data Curation Investigation Resources Validation Writing – Review & Editing 3 Stigloher Christian Conceptualization Data Curation Funding Acquisition Investigation Methodology Resources Supervision Writing – Review & Editing 2 Kollmannsberger Philip Conceptualization Data Curation Formal Analysis Investigation Methodology Software Supervision Validation Visualization Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0002-8049-6186 a1 1 Center for Computational and Theoretical Biology, University of Würzburg, Würzburg, 97074, Germany 2 Imaging Core Facility, Biocenter, University of Würzburg, Würzburg, 97074, Germany 3 Department of Neurology, University of Würzburg, Würzburg, 97074, Germany a philip.kollmannsberger@uni-wuerzburg.de No competing interests were disclosed. 28 12 2023 2020 9 127527 12 2023 Copyright: © 2023 Seifert R et al. 2023 https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. In correlative light and electron microscopy (CLEM), the fluorescent images must be registered to the EM images with high precision. Due to the different contrast of EM and fluorescence images, automated correlation-based alignment is not directly possible, and registration is often done by hand using a fluorescent stain, or semi-automatically with fiducial markers. We introduce “DeepCLEM”, a fully automated CLEM registration workflow. A convolutional neural network predicts the fluorescent signal from the EM images, which is then automatically registered to the experimentally measured chromatin signal from the sample using correlation-based alignment. The complete workflow is available as a Fiji plugin and could in principle be adapted for other imaging modalities as well as for 3D stacks. Correlative Microscopy Image Registration In-silico labeling Deep Learning Studienstiftung des Deutschen VolkesDeutsche ForschungsgemeinschaftGRK2581 STI700/1-1 327497565 SMM was supported by a PhD grant from the Studienstiftung des Deutschen Volkes (German Academic Scholarship Foundation). VP and CS are funded by Deutsche Forschungsgemeinschaft GRK2581 SPHINGOINF and CS by STI 700/1-1. Computational work was performed using the High-Performance Computing Cloud of Würzburg University (DFG project 327497565).This publication was supported by COST Action NEUBIAS (CA15124), funded by COST (European Cooperation in Science and Technology). The JEOL JSM-7500F field emission scanning electron microscope (SEM) was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 218894895 (INST 93/761-1 FUGG) and the structured illumination microscope (SIM) was funded by DFG - 261184502 (INST 93/823-1 FUGG). VP and CS were funded by DFG within RTG 2581 SphingoINF (project 6).Revised Amendments from Version 2 We added information on the training parameters, the hardware and the training time required to the methods section of the manuscript. The software availability section now contains details on how to contact the authors for support and questions. The documentation of the software was expanded to include more details especially for novice users, with additional information about when and how the method should be adapted to novel data by training a custom network model. The installation of the python environment was updated and tested on different systems. ==== Body pmcIntroduction Correlative Light and Electron Microscopy (CLEM) combines the high resolution of electron microscopy (EM) with the molecular specificity of fluorescence microscopy. In super-resolution array tomography (srAT) for example, serial sections are imaged first under the fluorescence microscope using super-resolution techniques such as structured illumination microscopy (SIM), and then in the electron microscope 1 . With this technique, it is possible to identify and assign molecular identities to subcellular structures such as electrical synapses 1, 2 or microdomains in bacterial membranes 3 that cannot be resolved by EM due to insufficient contrast. To visualize and interpret the results of CLEM, the fluorescent images must be registered to the EM images with high accuracy and precision. Due to the different contrasts of EM and fluorescence images, automated correlation-based image alignment, as used e.g. for aligning EM serial sections 4 , is not directly possible. Registration is often done by hand using a fluorescent chromatin stain 2 , or semi-automatically with fiducial markers using tools such as eC-CLEM 5 . Further improvement and automation of the registration process is of great interest to make CLEM scalable to larger datasets. Deep Learning using convolutional neural networks (CNNs) has become a powerful tool for various tasks in microscopy, including denoising and deconvolution as well as classification and segmentation, reviewed in 6 and 7. One interesting application of CNNs is the prediction of fluorescent labels from transmitted light images of cells, also called “in silico labeling” 8, 9 . We show here that this approach can be used to predict the fluorescent chromatin stain in electron microscopy images of cell nuclei. The predicted “ in silico” chromatin images are sufficiently similar to real experimental chromatin images acquired with SIM to use them for automated correlation-based registration of CLEM images. Based on this observation, we developed “DeepCLEM”, a fully automated CLEM registration workflow implemented in Fiji 10 and based on CNNs. Methods Data acquisition We used previously acquired imaging data of Caenorhabditis elegans and of human skin samples from healthy subjects. Sample preparation as well as the acquisition of the imaging data has been previously described in detail 1, 2, 11 . Briefly, C. elegans worms were cryo-immobilized via high-pressure freezing and subsequently processed by freeze substitution. All samples were embedded in methacrylate resin and sectioned at 100 nm. Ribbons of consecutive sections were attached to glass slides and labeled with fluorophores. Live Hoechst 33342 was used to stain chromatin and immunolabeling was used to visualize molecular identities. The sections were then imaged with SIM super-resolution microscopy. Next, they were processed for electron microscopy by heavy metal contrasting and carbon coating. The regions of interest previously imaged with SIM were then imaged again on the same sections with scanning electron microscopy, resulting in pairs of images that needed to be correlated. Manual registration To prepare ground truth for network training, we manually registered the chromatin channel to the EM images as described in 2. We selected 30 subimages and super-imposed them in the software Inkscape. By reducing the opacity of the chromatin images, they could be manually resized, rotated and dragged until the Hoechst signal coincided with the electron-dense heterochromatin puncta in the underlying EM images. To generate own training data, reproducible methods retaining a record of all transforms are recommended. Implementation We implemented DeepCLEM as a Fiji 10 plugin, using CSBDeep 12 for network prediction. Preprocessing of the images as well as network training were performed in Python using scikit-image 13 and TensorFlow 14 . First, a neural network trained on manually registered image pairs predicts the fluorescent chromatin signal from previously unseen EM images ( Figure 1A). This "virtual" fluorescent chromatin image is then automatically registered to the experimentally measured chromatin signal from the sample using the “similarity” transform of the “Register Virtual Stack Slices” plugin in Fiji ( Figure 1B). The transformation parameters from this automated alignment are finally used to register the other SIM images that contain the signals of interest to the EM image ( Figure 1C). Figure 1. Schematic of the "DeepCLEM" workflow. From the EM image ( A), a CNN predicts the chromatin channel ( B), to which the SIM image ( C) is registered ( D). The same transform is applied to the channel of interest ( E) to obtain a CLEM overlay ( F). We trained the ProjectionCARE 2D UNet architecture from CSBDeep 12 with 3 layers and a batch size of 8 for 150 epochs, using the default parameters of CSBDeep unless stated otherwise. Training on a GPU took between one (RTX 2080 Ti, cuda 11.3, Windows 10) and three hours (Tesla P100, cuda 11.7, Ubuntu 20.04) and around 10 hours on a CPU (Intel i7-8700K at 3.70 GHz, Windows 10). The Jupyter notebook for training on custom datasets as well as documentation how to use it are available online (see software availability statement). Operation DeepCLEM requires Fiji 10 with CSBDeep 12 to run. The paths to the images and model file are entered in a user dialog ( Figure 2). After running DeepCLEM, the correlated images and a .XML file containing the transform parameters are written to the output directory. The workflow is summarized in Figure 1; instructions for installing and running DeepCLEM are included in the repository. The network included in DeepCLEM was trained on in-house data and may work on images with similar contrast, but in most cases, re-training will be necessary – details on the workflow and training parameters are given in a Jupyter notebook in the repository. Running this notebook on a directory with 30–40 aligned ground truth image pairs will yield a model file that can be loaded in the DeepCLEM Fiji plugin. Figure 2. GUI and input parameters for "DeepCLEM". Results Comparison of network architectures We trained DeepCLEM on correlative EM and SIM images of C. elegans and on human skin tissue and compared prediction and registration results for different network architectures and preprocessing routines. A generative adversarial network (pix2pix) showed promising results in some images from the skin dataset, but overall performance was best using the ProjectionCARE network from CSBDeep 12 . Optimization of preprocessing EM images had large differences in contrast even when acquired in the same laboratory. We compared different preprocessing routines, including normalization and histogram equalization, and found that standard histogram equalization in Fiji resulted in the best performance on our data. The best combination of preprocessing steps for optimizing contrast may however depend on the data. Quantitative evaluation We performed a quantification of the quality of the registration on four manually aligned images from an independent experiment, and applying a known shift or rotation. In 75% of cases the registration worked and had a very small error, while in 25% it was completely off by several 100 nm ( Table 1). If two of the test images were included in the training set, the error was much lower, so DeepCLEM works best if a small number of images of each experiment are manually aligned and added to the training data. The remaining images are then reliably aligned. We also varied the number of images in the training set and found that 30–40 ground truth images are sufficient to obtain good alignment on the test set. Table 1. Quantitative evaluation. When applying DeepCLEM on images from a different experiment not represented in the training data, registration failed in 25% of cases (top part, image 1–4). If two manually aligned images were included in the training set, all other test images were successfully registered (bottom part, images 1–2). Absolute error [nm] aligned shift in X (3125 nm) rotation 90° rotation 180° X Y X Y X Y X Y No images from same experiment included in training: image 1 65,2 54,0 287,9 215,7 12,2 2,0 102,9 84,8 image 2 69,9 31,0 9234,7 3440,5 276,8 72,0 271,4 1601,1 image 3 713,1 288,0 136,3 65,9 321,3 3,2 77,3 47,5 image 4 103,6 86,8 160,2 158,0 5771,9 4168,6 8861,2 1776,9 Two images from same experiment included in training: image 1 33,5 181,0 61,9 100,1 101,7 113,8 115,9 155,1 image 2 16,3 48,9 27,7 20,2 59,9 20,2 84,2 63,9 Discussion We developed “DeepCLEM”, a fully automated CLEM registration workflow implemented in Fiji 10 based on prediction of the chromatin stain from EM images using CNNs. Our registration workflow can easily be included in existing CLEM routines or adapted for imaging methods other than srAT where corresponding 2D slices need to be registered. If direct prediction of one modality from the other does not work, an alternative is to predict a common representation of both modalities, as described in Ref 15. While we found that "DeepCLEM" performs well under various conditions, it has some limitations: using chromatin staining for correlation requires the presence of at least three heterochromatin patches in the field of view. This limitation could be overcome by using e.g. propidium iodide to label the overall structure of the tissue. Widefield microscopy could be used where SIM is not available, but alignment quality is bounded by the lower-resolution channel. The popular CLEM registration tool eC-CLEM 5 has an “autofinder” function that detects corresponding features using spot finding or segmented regions. We did not perform a direct comparison, but results should be similar if suitable spots are found. If not, then image-to-image translation with DeepCLEM followed by point-based registration in eC-CLEM could be a promising alternative. Acknowledgements This publication was supported by COST Action NEUBIAS (CA15124), funded by COST (European Cooperation in Science and Technology). The JEOL JSM-7500F field emission scanning electron microscope (SEM) was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 218894895 (INST 93/761-1 FUGG) and the structured illumination microscope (SIM) was funded by DFG - 261184502 (INST 93/823-1 FUGG). VP and CS were funded by DFG within RTG 2581 SphingoINF (project 6). Data availability Source code, pretrained networks and example data as well as documentation and instructions how to train a model on custom data are available online at: https://github.com/CIA-CCTB/Deep_CLEM. For support, users are encouraged to start a new issue on github, or alternatively to contact the corresponding author via email. Software availability Source code available from: https://github.com/CIA-CCTB/Deep_CLEM. Archived source code at time of publication: https://doi.org/10.5281/zenodo.4095247 16 License: MIT License. 10.5256/f1000research.160461.r234014 Reviewer response for version 3 Jones Martin L. 1Referee https://orcid.org/0000-0003-0994-5652 1 Electron Microscopy Science Technology Platform, The Francis Crick Institute, London, UK 25 1 2024 Copyright: © 2024 Jones ML 2024 https://creativecommons.org/licenses/by/4.0/ This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Version 3recommendationapprove The addition of more novice-focused documentation is a good addition. Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Partly Is the rationale for developing the new software tool clearly explained? Yes Is the description of the software tool technically sound? Partly Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Partly Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes Reviewer Expertise: Image Analysis, CLEM, Machine Learning, Electron microscopy I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. 10.5256/f1000research.136822.r179427 Reviewer response for version 2 Schmied Christopher 1Referee https://orcid.org/0000-0003-2058-1124 1 AG Han Sun, Leibniz-Forschungsinstitut für Molekulare Pharmakologie, Berlin, Berlin, Germany 29 6 2023 Copyright: © 2023 Schmied C 2023 https://creativecommons.org/licenses/by/4.0/ This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Version 2recommendationapprove-with-reservations The authors present an image registration workflow to register light microscopy (LM) images into electron microscopy (EM) images for correlative light-electron microscopy (CLEM). To solve the multimodal registration problem, DeepCLEM predicts a LM image from the EM image using ProjectionCARE. The predicted LM image is then used to compute the necessary transformation of the LM image to the EM image. Apart from the pre-trained model, which is limited to data it has been trained on, the authors also describe how to train the model on their own data.  Overall, a very nice approach that has been well implemented. In my review I verified if the provided software package and its documentation allows the successful reproduction of the described workflow. I also assessed if the provided article sufficiently describes and discusses the proposed approach (i.e. scientific context and limitations).  Application of the pre-trained model: I was able to easily install the software as per documentation. The provided test images allow to easily verify the correct operation of the Fiji workflow. The output data is clearly and sufficiently described. Positive is that the transformation file is provided that allows to apply the transformation on other data. I found some points problematic in the documentation: "The EM image should look like the testing EM.png." This description is too unspecific for novice users. Important would be here to emphasize that this is in context to the application of the pre-trained model on new data.  Retraining: Great that the conda .yml is provided for easy installation of the dependencies. As a warning, I experienced that the creation of the conda environment took a long time on my machine. But could be specific to my setup. The possibilities of my review here are also limited since I do not have the possibility to generate my own training data from scratch and I can only test the provided data. Within these constraints, I was able to test the notebooks for their correct operation. Article: The article is overall well written and explains the context of the work. The authors also test and discuss the limitations of the approach. Further, providing avenues to overcome problems and limitations.  As a side note, not relevant to the review, in terms of the registration field in the biology focused sciences, I am always a bit surprised that software solutions such as elastix (https://elastix.lumc.nl/) are not more widely used and discussed. I would have expected that the solutions of multimodal image registration developed in the medical field would be at least somewhat relevant to CLEM. The article seems to also discuss another established registration tool (eC-CLEM) so there is sufficient other context for this publication. The methods section of the article is unfortunately a bit sparse. The actual network architecture is only mentioned in the results. No further detail is provided. It is fine if the default values of the out-of-the-box network are used. However, this should be clearly stated in the methods. I would also wish a better description of how the training has been performed. Including how long the training took and on which hardware this was performed. I really enjoyed that the authors provide all necessary code and example images on their GitHub repo. The only tiny thing missing is to clearly state their preferred way of users getting in touch for support. I would suggest clearly stating this in the software availability section.  I endorse the article under the following conditions: A more extensive methods section. Describing clearer what the architecture was and any important or changed hyperparameters. Also, the hardware and the training time should be specified.  Improve documentation from the perspective of a novice user.  State how users should get in touch for support.  Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Yes Is the rationale for developing the new software tool clearly explained? Yes Is the description of the software tool technically sound? Yes Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Partly Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes Reviewer Expertise: Data science, bioimage analysis, cell biology. I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. 10.5256/f1000research.136822.r146290 Reviewer response for version 2 Jones Martin L. 1Referee https://orcid.org/0000-0003-0994-5652 1 Electron Microscopy Science Technology Platform, The Francis Crick Institute, London, UK 30 8 2022 Copyright: © 2022 Jones ML 2022 https://creativecommons.org/licenses/by/4.0/ This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Version 2recommendationapprove No further comments, the authors have covered all of the comments very satisfactorily. Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Partly Is the rationale for developing the new software tool clearly explained? Yes Is the description of the software tool technically sound? Partly Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Partly Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes Reviewer Expertise: Image Analysis, CLEM, Machine Learning, Electron microscopy I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. 10.5256/f1000research.30002.r77401 Reviewer response for version 1 Rachel Reinhard 1Referee https://orcid.org/0000-0001-6367-1221 1 Centre for Electron Microscopy, Faculty of Biology and Preclinical Medicine, University of Regensburg, Regensburg, Germany 8 2 2021 Copyright: © 2021 Rachel R 2021 https://creativecommons.org/licenses/by/4.0/ This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Version 1recommendationapprove-with-reservations The manuscript by Seifert et al. (Univ. Würzburg, Germany) provides a convenient software tool (a macro name DeepCLEM) for linking fluorescent light microscopy images (here: structured illumination microscopy) to the information seen in electron micrographs (in this particular case: FE-SEM), obtained from serial sections on slides.  In a first glance, this short paper describes the development and application of a flexible and easy to handle software tool to a program which is open to many microscopy researchers, ImageJ / Fiji, in form of a macro, which can easily be added to any Fiji installation. Thus, it appears that this tool can be used by many groups aiming to analyze biological samples using correleative light and electron microscopy (CLEM).  A few questions arise when reading the manuscript - which do not modify the overall positive impression of this study.  The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere? What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed? - the authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow?  Histogram equalization of EM images - was this done with routines implemented in Fiji? Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this? With comments on my points listed above, and according improvements in the manuscript, this can be indexed - thus, approved with reservations. Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Partly Is the rationale for developing the new software tool clearly explained? No Is the description of the software tool technically sound? Partly Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Partly Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes Reviewer Expertise: TEM, STEM, tomography, CLEM; microbiology, cell biology I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Kollmannsberger Philip Julius-Maximilians-University Wurzburg, Germany 28 7 2022 R: The group uses Hoechst 33342 for staining and the biological objects are visualized by SIM - would it be sufficient to visualize the objects by 'standard' wide-field fluorescent microscopy, which is present in almost all biological laboratories, while SIM requires access to instruments which are not available everywhere? A: If the structures used for alignment (e.g., chromatin) can be resolved and the WF images can be manually aligned, it is also possible to obtain a prediction and do automated alignment, although the achievable quality of the registration is obviously limited by the resolution of the fluorescence channel. We now added this in the discussion of the scope and limitations of the tool. R: What kind of knowledge is necessary to perform the 'prediction of fluorescent signals from EM images', i.e. generating the "virtual" fluorescent images? Does this macro work as provided, i.e. 'fully automatically'? Is any user intervention needed? A: Using a trained network for prediction and alignment can be done fully automatically in Fiji, only the paths of the images and network must be provided in the user dialog. For retraining, some knowledge (e.g., setting up a python environment) is necessary. We now added a more detailed description of the training workflow. Currently, the training process should probably be performed by some more experienced person while using the trained network does not require any special knowledge. R: The authors trained their macro on their datasets. Is it likely that this macro "as trained in the author's lab" also works on images from other labs? or is it necessary to go through a training phase? if so: workflow? A: It may be possible to use the provided network trained on our data on images from other labs directly, however in most cases re-training is required. We now describe the required workflow in more detail and perform a quantitative comparison of registration quality where images from an experiment were either included in the training set or not. R: Histogram equalization of EM images - was this done with routines implemented in Fiji? A: We used the standard histogram equalization implemented in Fiji. We added this detail to the method description. R: Can the authors at least provide one set of parallel results obtained using DeepCLEM vs. ec-CLEM? or at least comment on this? A: We now comment on ec-CLEM and its “autofinder” functionality which is closest to our tool in terms of functionality. While ec-CLEM autofinder uses spot finding or centers of segmented regions as points of interest to perform registration on, we predict the image of the other modality and then use correlation-based alignment. 10.5256/f1000research.30002.r77801 Reviewer response for version 1 Jones Martin L. 1Referee https://orcid.org/0000-0003-0994-5652 1 Electron Microscopy Science Technology Platform, The Francis Crick Institute, London, UK 4 2 2021 Copyright: © 2021 Jones ML 2021 https://creativecommons.org/licenses/by/4.0/ This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Version 1recommendationapprove-with-reservations The authors present an automated method, deployed as a Fiji plugin, for registering 2D CLEM data. This is demonstrated on a "super resolution Array Tomography" (srAT) dataset, building on their previous method development work, where the sample is sliced into 100nm sections prior to imaging in SIM and SEM. The DeepCLEM plugin generates synthetic fluorescence data from the EM images that are aligned to the real fluorescence signal to perform the registration. This type of effort is crucial in harnessing the huge potential of correlative multimodal imaging as cross-modal registration is a significant bottleneck in many cases. The method of imaging both LM and EM on the processed and cut sections is perhaps not as widespread as other methods, such as, e.g. whole cell confocal imaging followed by resin-embedding, followed by EM imaging. The method used in this paper avoids the familiar CLEM registration problems of non-linear warping induced by the EM sample preparation and mismatched axial resolutions. This constrains the registration task to a "similarity" type transform rather than affine or non-rigid as is required in some other CLEM registration tasks. The open source plugin is easy to install and runs as described on the test data provided. A few points that I think should be addressed: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step. The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo? Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset. I believe the preferred capitalisation is "Fiji" and not "FIJI" ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible. For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training. In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model. If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists. The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives. An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial! The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well.  Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract. The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful Overall, the tool itself works as described on the data type provided, but I think there is a risk that users might be tempted to stretch beyond the current scope of the tool, so this scope should be made more clear in the text. Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Partly Is the rationale for developing the new software tool clearly explained? Yes Is the description of the software tool technically sound? Partly Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Partly Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes Reviewer Expertise: Image Analysis, CLEM, Machine Learning, Electron microscopy I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Kollmannsberger Philip Julius-Maximilians-University Wurzburg, Germany 28 7 2022 R: In the abstract and introduction, it is mentioned that "registration is often done by hand using a fluorescent chromatin stain" - this seems overly specific, in general CLEM alignment might make use of many different markers or features. Focusing on the specific stains is appropriate later when describing the specific use-case presented, but the authors might inadvertently deter potential users wanting to use different stains that could feasibly be trained for in the deep learning step. A: We removed the mention of a specific stain in the abstract to reflect the general applicability of the method. R: The abstract refers to a Fiji Macro, but the code appears to be a python/jython script that is to be installed as a plugin. Is there an additional *.ijm IJ1 macro available, or is this a typo? A: The code is a Jython script with graphical user interface to be run within Fiji. From the user's perspective, it is indeed a plugin. We changed “macro” to “plugin”, as suggested. R: Although it is common to call this type of method "automated", there is a front-loaded training step for the deep learning element. It isn't clear whether the trained model from the author's data would be sufficiently generalised to other datasets, or whether an amount of training will commonly be required for each different dataset. A: The trained network can be applied to different datasets if they have similar contrast, usually from the same microscope or sample type. Otherwise, (re-)training of the network is necessary. We now make this clearer in the manuscript and include a description of the training workflow. R: I believe the preferred capitalisation is "Fiji" and not "FIJI" A: Thank you, we corrected this. R: ec-CLEM is mentioned and the text implies that it is only used with fiducials, but it can also be used for aligning using the fluorescence and EM data directly via manual landmark placement on intrinsic structures and/or stains. ec-CLEM also has an "autofinder" functionality that seems like it should be mentioned as potentially the nearest functionality in an existing tool. It would be useful to see a comparison, or an explanation as to how DeepCLEM differs if a direct comparison isn't possible. A: We now refer to ec-CLEM autofinder which is similar in terms of functionality. It is based on finding corresponding features in both modalities, e.g., using wavelet-based spot detection or the centers of segmented regions, and then performing registration of these automatically detected landmarks. We did not perform a direct quantitative comparison. If suitable corresponding spots can be found (e.g. beads), the performance on the same images using the same registration algorithm should be comparable. In cases where spot finding or similar methods in ec-CLEM do not perform well because the images are too dissimilar or if there are no fiducial markers, then transferring the EM image to fluorescence using DeepCLEM could be used to generate input data for point-based registration in ec-CLEM. R: For the manual registration section, it would be better practice to use a more reproducible method, such as ec-CLEM or Fiji landmark-based registration (as opposed to the stated method using inkscape) since these methods retain a record of the steps and transforms. Although, in the context of the paper this is a means-to-an-end for acquiring training data for demonstration purposes, it would be preferable if the advice to potential users of the plugin were to err on the side of reproducible methods for their own training. A: We agree that these methods are more reproducible and added an additional note to the paragraph on manual registration to encourage users to use such reproducible methods. Since we already had created the training data, we did, however, not repeat the alignment. For assessing the performance (see below), we instead artificially displaced the two channels - in this case, the steps and transforms are also known exactly. R: In "Implementation" it would be useful to state a bit more detail about the training, for example training/validation split and suitable data quantities for users to train their own model. A: We now include more detail about the training workflow in the implementation section and compared the performance for different amounts of training images. R: If the goal is to enable non-computational researchers to perform the whole workflow, the GitHub README and Jupyter notebooks should have a bit more description, in particular with regards to training, a process that is quite alien with unfamiliar jargon for non-specialists. A: We extended both the description of the training parameters as well as the documentation of the entire workflow, including training in the Jupyter notebook. R: The text mentions "correlation based alignment in Fiji" - looking in the source code this is specifically the "Register Virtual Stack Slices" plugin, this should be mentioned in the text as it is a major computational step in the workflow. Also, the use of "similarity" as opposed to rigid, affine etc should be specified in the text since other CLEM practitioners may be more familiar with other alternatives. A: The text now explicitly mentions the use of the "Register Virtual Stack Slices" plugin and the "similarity" transform in our work, but in principle, our workflow should also work with other means of correlation-based registration. R: An important point that is missing is some kind of quantification of the quality of the registration compared to a gold-standard produced manually. Understanding this is likely to be very important for researchers who might be considering using this tool. If possible, to maximise usability, some sort of "confidence" metric would be very useful for the user, but I appreciate this is probably non-trivial! A: We now performed quantification of the quality of the registration using the manually aligned ground truth data as a reference where the best solution is already known. We observed that in most cases, the registration either works and has a very small error or is completely off by several 100 nm, which should be easy to spot. If a few images from an experiment are manually aligned and included in the training set, the registration of the remaining images improves significantly (Table 1). Instead of providing a confidence metric, which would indeed be non-trivial, we recommend visual inspection using the overlay. Possible ways to implement a confidence metric would be to output a confidence level for the neural network prediction, and to check the quality of the alignment using a correlation metric between fluorescent and predicted image. R: The authors propose extensions to their method for dealing with situations where nuclei are not present in the images. It would be useful if the authors could provide a brief focused description of the scope of the tool as it is currently configured, so that potential users can quickly assess whether it's suitable for their task or what they might need to do to extend it. For example, the GitHub repo suggests the image must have a minimum of 3 nucleoli - this should be mentioned in the body of the paper as well. A: We added more detail regarding the scope and limitations of this tool and its current requirements to the abstract, introduction, and discussion section, including the need for at least three heterochromatin patches if chromatin staining is used for registration. R: Related to the above, the title and abstract make no specific mention of the method being based on imaging both LM and EM in the cut sections as opposed to a perhaps more common workflow such as confocal on whole cells followed by EM data on sections or block face. To avoid potential confusion, this constraint should be made more clear in the abstract. The abstract and discussion mention adaptations to apply to other modalities and 3D stacks. Given the srAT-based application it is not obvious what these adaptations would be, so an elaboration on this point would be useful A: The way it is currently configured, DeepCLEM can in principle be used to register any CLEM data where corresponding 2D slices are available in both modalities, whether they were acquired on the same cut sections (srAT) or prior and post sectioning. The examples we provide however are restricted to srAT. We now state this in the text. Competing interests: No competing interests were disclosed. Competing interests: No competing interests were disclosed. Competing interests: No competing interests were disclosed. Competing interests: No competing interests were disclosed. Competing interests: No competing interests were disclosed. Competing interests: No competing interests were disclosed. Competing interests: No competing interests were disclosed. ==== Refs 1 Markert SM Britz S Proppert S : Filling the gap: adding super-resolution to array tomography for correlated ultrastructural and molecular identification of electrical synapses at the C. elegans connectome. Neurophotonics. 2016;3 (4 ): 041802. 10.1117/1.NPh.3.4.041802 27175373 2 Markert SM Bauer V Muenz TS : 3D subcellular localization with superresolution array tomography on ultrathin sections of various species. Methods Cell Biol. 2017;140 :21–47. 10.1016/bs.mcb.2017.03.004 28528634 3 García-Fernández E Koch G Wagner RM : Membrane Microdomain Disassembly Inhibits MRSA Antibiotic Resistance. Cell. 2017;171 (6 ):1354–1367. e20. 10.1016/j.cell.2017.10.012 29103614 4 Cardona A Saalfeld S Schindelin J : TrakEM2 Software for Neural Circuit Reconstruction. PLoS One. 2012;7 (6 ): e38011. 10.1371/journal.pone.0038011 22723842 5 Paul-Gilloteaux P Heiligenstein X Belle M : eC-CLEM: flexible multidimensional registration software for correlative microscopies. Nat Methods. 2017;14 (2 ):102–103. 10.1038/nmeth.4170 28139674 6 von Chamier L Laine RF Henriques R : Artificial intelligence for microscopy: what you should know. Biochem Soc Trans. 2019;47 (4 ):1029–1040. 10.1042/BST20180391 31366471 7 Belthangady C Royer LA : Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat Methods. 2019;16 (12 ):1215–1225. 10.1038/s41592-019-0458-z 31285623 8 Christiansen EM Yang SJ Ando DM : In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images. Cell. 2018;173 (3 ):792–803. e19. 10.1016/j.cell.2018.03.040 29656897 9 Guo SM Yeh LH Folkesson J : Revealing architectural order with quantitative label-free imaging and deep learning. eLife. 2020;9 :e55502. 10.7554/eLife.55502 32716843 10 Schindelin J Arganda-Carreras I Frise E : Fiji: an open-source platform for biological-image analysis. Nat Methods. 2012;9 (7 ):676–682. 10.1038/nmeth.2019 22743772 11 Uçeyler N Kafke W Riediger N : Elevated proinflammatory cytokine expression in affected skin in small fiber neuropathy. Neurology. 2010;74 (22 ):1806–1813. 10.1212/WNL.0b013e3181e0f7b3 20513817 12 Weigert M Schmidt U Boothe T : Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat Methods. 2018;15 (12 ):1090–1097. 10.1038/s41592-018-0216-7 30478326 13 Virtanen P Gommers R Oliphant TE : SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods. 2020;17 (3 ):261–272. 10.1038/s41592-019-0686-2 32015543 14 Abadi M Agarwal A Barham P : TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems.arXiv: 160304467 [cs].2016. 10.48550/arXiv.1603.04467 15 Pielawski N Wetzer E Öfverstedt J : CoMIR: Contrastive Multimodal Image Representation for Registration.In: Advances in Neural Information Processing Systems. Curran Associates, Inc.,2020;33 : 18433–18444. 10.48550/arXiv.2006.06325 16 Seifert R : CIA-CCTB/Deep_CLEM: First release of DeepCLEM (Version v1.0). Zenodo. 2020. 10.5281/zenodo.4095247