
==== Front
Nano Lett
Nano Lett
nl
nalefd
Nano Letters
1530-6984
1530-6992
American Chemical Society

39234957
10.1021/acs.nanolett.4c03069
Letter
Localized Plasmonic Structured Illumination Microscopy Using Hybrid Inverse Design
https://orcid.org/0000-0001-8060-2340
Wu Qianyi †⊥
Xu Yihao ‡⊥
Zhao Junxiang †
https://orcid.org/0000-0003-1084-6651
Liu Yongmin *‡§
https://orcid.org/0000-0002-5732-8109
Liu Zhaowei *†∥
† Department of Electrical and Computer Engineering, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, United States
‡ Department of Mechanical and Industrial Engineering, Northeastern University, Boston, Massachusetts 02115, United States
§ Department of Electrical and Computer Engineering, Northeastern University, Boston, Massachusetts 02115, United States
∥ Materials Science and Engineering Program, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, United States
* Yongmin Liu − Email: y.liu@northeastern.edu.
* Zhaowei Liu − Email: zhaowei@ucsd.edu.
05 09 2024
18 09 2024
24 37 1158111589
01 07 2024
28 08 2024
27 08 2024
© 2024 The Authors. Published by American Chemical Society
2024
The Authors
https://creativecommons.org/licenses/by-nc-nd/4.0/ Permits non-commercial access and re-use, provided that author attribution and integrity are maintained; but does not permit creation of adaptations or other derivative works (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Super-resolution fluorescence imaging has offered unprecedented insights and revolutionized our understanding of biology. In particular, localized plasmonic structured illumination microscopy (LPSIM) achieves video-rate super-resolution imaging with ∼50 nm spatial resolution by leveraging subdiffraction-limited nearfield patterns generated by plasmonic nanoantenna arrays. However, the conventional trial-and-error design process for LPSIM arrays is time-consuming and computationally intensive, limiting the exploration of optimal designs. Here, we propose a hybrid inverse design framework combining deep learning and genetic algorithms to refine LPSIM arrays. A population of designs is evaluated using a trained convolutional neural network, and a multiobjective optimization method optimizes them through iteration and evolution. Simulations demonstrate that the optimized LPSIM substrate surpasses traditional substrates, exhibiting higher reconstruction accuracy, robustness against noise, and increased tolerance for fewer measurements. This framework not only proves the efficacy of inverse design for tailoring LPSIM substrates but also opens avenues for exploring new plasmonic nanostructures in imaging applications.

Deep learning
Genetic algorithms
Photonics inverse design
Super-resolution microscopy
Plasmonics
Structured illumination microscopy
National Science Foundation 10.13039/100000001 CBET-1931777 Gordon and Betty Moore Foundation 10.13039/100000936 NA National Science Foundation 10.13039/100000001 MRI-2320437 National Science Foundation 10.13039/100000001 ECCS-2136168 National Science Foundation 10.13039/100000001 DMR-2202268 document-id-old-9nl4c03069
document-id-new-14nl4c03069
ccc-price
==== Body
pmcFluorescence imaging has revolutionized the visualization of biological structures and dynamics by offering remarkable image contrast and chemical specificity.1,2 The breakthroughs in super-resolution microscopy techniques3−5 further advanced our understanding of subcellular structures and processes at the nanoscale by overcoming the Abbe diffraction limit. Over the past two decades, numerous super-resolution microscopy methods, including single-molecule localization microscopy,6,7 stimulated emission depletion microscopy,8,9 structured illumination microscopy (SIM),10−12 and their derivatives have been developed.

Localized plasmonic structured illumination microscopy (LPSIM)13−16 is a super-resolution fluorescent imaging technique assisted by plasmonic nanoantenna arrays. These arrays, consisting of periodic plasmonic nanodiscs, generate near-field illumination patterns that incorporate the high-spatial-frequency details of the object into the detectable bandwidth of a microscope. Typically, at least nine subframes are collected to reconstruct each super-resolution frame by using a blind-SIM algorithm.17,18 LPSIM has experimentally achieved a 3-fold improvement in resolution beyond the diffraction limit and video speed for dynamic samples.19 Additionally, employing nanodisc arrays with a smaller pitch has further improved the resolution enhancement to five times.20 However, this comes at the cost of partially missing information between spatial frequency peaks, leading to potential deformations and artifacts in the reconstructed image.

The design of LPSIM arrays normally follows conventional trial-and-error procedures, which begin with an intuition-based initialization followed by iterative adjustments of geometric parameters. This process often involves parameter sweeping and necessitates a significant amount of time and computational resources. Moreover, for LPSIM, the simplicity and high symmetry of the commonly used unit cell offer a limited collection of tunable parameters, namely, the radius, height, and pitch of nanodiscs, thereby limiting the potential for finding the optimal design.21 Moreover, specific assumptions, such as uniformity in averaged illumination17 and low coherence among illuminations,22,23 are critical for reconstruction. However, systematically incorporating these constraints into the design process can be challenging. Although the accuracy of reconstructions primarily hinges on the illumination patterns generated by LPSIM substrates, a clear relationship between geometric parameters and imaging capabilities has been elusive.

To address the drawbacks of the conventional photonics design approach, researchers have turned to modern optimization algorithms, like genetic algorithms,24−26 topology optimization,27−29 and machine learning,30−35 to expedite the inverse design process. On the one hand, optimization methods can streamline the process by intelligently selecting subsequent design candidates, although they often rely on full-wave simulation for individual design characterization. On the other hand, machine learning—a data-driven approach—can entirely substitute full-wave simulation with a model trained on existing design-response data sets. Consequently, integrating these approaches into a hybrid model proves highly beneficial.36−43 It not only bypasses the time-intensive parameter sweeping and full-wave simulation but also enhances the exploration of diversiform photonics structures, allowing for greater flexibility in the design parameter choices.

In this study, we propose a computationally effective hybrid inverse design framework to optimize the silver nanoantenna array for LPSIM by combining deep learning and optimization methods. A convolutional neural network (CNN)-based predictor enables the rapid and accurate prediction of near-field electric fields produced by the LPSIM array. Utilizing the trained CNN, a genetic algorithm-based mutiobjective optimization (MOO) refines designs through selection, mutation, and reproduction. Simulated imaging results reveal the superiority of the optimized LPSIM substrate design over traditional designs in reconstructing fluorescent beads and biological structures. The optimized design exhibits a higher accuracy, robustness against noise, and increased tolerance for fewer measurements. The proposed framework not only shows the efficacy of inverse design techniques for tailoring LPSIM substrates with specific functionalities but also extends the possibilities for exploring new plasmonic nanostructures for applications in imaging and beyond.

To generate tunable structured illumination patterns with localized plasmonic fields, the nanoantenna array is illuminated with a laser at varying incident angles. Figure 1a shows an experimental illumination scheme. TM-polarized beams are used to illuminate the nanoantenna array because of their tunability when sweeping the illumination angles, which are controlled by a pair of galvo mirrors. The fluorescent signal from the sample is collected by an objective. For imaging purposes, an operational wavelength of 488 nm is selected. Silver is chosen as the nanoantenna material due to its strong plasmonic response at this specific wavelength.

Figure 1 Hybrid inverse design of nanoantenna arrays for LPSIM. (a) Illumination scheme of LPSIM. QWP, quarter wave plate; M, mirror; GM, galvo mirror; PP, polarizer plate; L, lens. (b) Schematic workflow of the CNN-assisted genetic algorithm for inverse design.

Figure 1b shows the schematic workflow of the CNN-assisted genetic algorithm. The evolution starts with a population of randomly generated unit cell designs represented by geometric parameters. Each design is fed into a trained CNN predictor, which provides rapid and accurate prediction of the periodic near-field electric fields generated by the LPSIM array. These predictions are used subsequently to compute several evaluation metrics that we aim to optimize. The designs with better metrics are stochastically selected, recombined, and mutated to form the new generation. This optimization process is repeated until the metrics converge or a certain number of generations is reached.

The initial preparation for training the CNN is the creation of a meticulously crafted training data set following a well-defined design framework. The unit cell of the LPSIM array (Figure 2a), called the meta-atom, exhibits a dimension of 200 × 200 nm and incorporates silver nanostructures patterned on a silica substrate. The silver structures featuring a thickness of 60 nm are embedded in an 80 nm poly(methyl methacrylate) (PMMA) layer. This type of metallic nanoantenna array can be mass-fabricated with nanoimprinting lithography as demonstrated in our previous work.14 Each meta-atom consists of one of the following three configurations: (i) a central nanostructure, (ii) four side nanostructures, or (iii) a central nanostructure surrounded by four side nanostructures, each chosen from a selection of four fundamental geometries: rectangle, ellipse, cross, and c-ring, as depicted in Figure 2b. The basic silver nanostructures are amenable to adjustments of their specific shapes on the x–y plane. To ensure simplicity, the meta-atom has mirror symmetry in both the x- and y-directions. Moreover, the size of the antennas and distances between antennas are constrained with a minimum of 30 nm to ensure the meta-atoms contain features within practical fabrication tolerance. Given the geometric configuration and parameters, a binary image representing the meta-atom is generated, with silver regions denoted by 1 and PMMA regions denoted by 0. The pixel size of the image is 1 nm.

Figure 2 Schematic representation of the CNN model and its prediction of the near-field electric fields generated by the LPSIM arrays. (a) Illustration of the unit cell, featuring silver nanostructures on a silica substrate and covered by a PMMA layer, with the dimension of 200 × 200 nm. (b) Illustration of the fundamental geometric shapes of the nanostructures, including rectangle, ellipse, cross, and c-ring. (c) Schematic of the CNN architecture employed for predicting the field distributions at various incident angles based on the input geometry. (d) Eight independent near-field distributions resulting from different incident angles. (e) Comparative analysis of field distributions. Field distributions for various unit cells show good agreement between the predictions from the CNN model and the ground-truth data obtained from the FDTD simulation. The MSE levels depend on geometric complexity but are always at the magnitude of 0.01 or smaller.

We use Lumerical FDTD to generate a sizable data set comprising designs and their corresponding responses (i.e., the field distributions at a plane 10 nm away from the surface of the LPSIM array). We simulate the meta-atom under illumination at 18 different incident angles. Due to inherent symmetry considerations, only 8 near-field distributions are independent (Supporting Information Section S1). A large data set comprising 10,000 structures and field pairs is generated, which we split into 9,000 and 1,000 for training and testing, respectively.

A CNN shown in Figure 2c (Supporting Information Section S2) is employed to facilitate rapid and precise field distribution predictions at every incident angle (Figure 2d). The trained CNN exhibits a remarkable ability to predict the near-field characteristics of novel meta-atoms unseen in the training data set. This is verified by achieving a mean-squared error (MSE) of approximately 0.01 between the predictions and the ground truth data obtained through FDTD simulations for the test data set (Figure 2e). It is worth highlighting that the MSE values are substantially influenced by the geometric complexity of the structures. Simpler shapes such as rectangles and ellipses exhibit lower errors, whereas more intricate structures, such as c-rings, tend to yield higher but still acceptable errors.

With the trained CNN model, we can readily predict the illumination patterns for arbitrary meta-atoms generated within our design framework in milliseconds. For photonic inverse design, many published works aiming at achieving high-dimensional target responses, such as the spectral response,44−47 often incorporate a generator like generative adversarial networks (GANs)48 and variational autoencoders (VAEs)49 that directly output the design of a structure for a given target response. In our case, instead of looking for specific target illumination patterns, we focus on optimizing a set of metrics serving as indicators of imaging performance in the context of SIM. These metrics with much smaller dimensions can be derived from analytical formulas given the field distributions. This innovative approach eliminates the need for a generator and reduces the dimensionality of the design problem.

For a certain meta-atom design, we evaluate the predicted illumination patterns with three metrics: (i) the uniformity of the total illumination, characterized by the standard deviation σ of the normalized average of all illumination patterns; (ii) the spatial cutoff frequency kc of the effective optical transfer function (OTF), defined as the frequency where the magnitude of OTF starts to fall below 1% of the DC component; (iii) similarity μt among the illuminations, represented by the t-averaged mutual coherence50 (Supporting Information Section S3).

The ideal illumination patterns have low σ, large kc, and low μt. Trade-offs need to be made when optimizing the three metrics. Thus, the design of LPSIM nanoantennas is optimized with well-established MOO algorithms. We utilize pymoo,51 an open-source MOO package in Python, so that the optimization can be seamlessly connected to the CNN model based on PyTorch. Pymoo offers a range of optimization algorithms, primarily based on genetic algorithms. We utilize the Non-Dominated Sorting Genetic Algorithm (NSGA-II).52Figure 3a illustrates the workflow, where we initially generate a set of random parameters to describe various meta-atoms and convert them into binary images. The corresponding near-field distributions predicted by CNN are then evaluated with the three metrics. The program selects the top-N best designs as parents for generating the next batch of design candidates, which involves the reproduction and mutation of the designs. After several iterations, a set of well-optimized final meta-atoms is obtained.

Figure 3 Workflow of multiobjective optimization and metric space representation. (a) Illustration of the workflow depicting the integration of a CNN with the NSGA-II MOO algorithm for the optimization of unit cell designs. (b) Evolution of the best values for the objective function and each metric per generation. The insets highlight the top design for each metric independently across the generations. (c) Correlation matrix of the three metrics for all randomly generated designs before optimization and the designs on the Pareto front. As the optimization proceeds, the metrics become more correlated, making it difficult to optimize them simultaneously. Therefore, they converge to the Pareto front. (d) Visualization of the metric space in both 3D and 2D projections. All generated data, the top 250 designs based on the customized objective function, and the Pareto front are marked in blue, yellow, and green, respectively. The purple and red stars represent the metrics from our previous LPSIM work (denoted as HEX) and the optimized design (denoted as OPT1) in Figure 4 for comparison.

We run the MOO for 100 generations, evaluating 500 meta-atoms in each generation using the CNN. The insets of Figure 3b show the evolution of metrics versus the number of generations. It should be noted that evaluating all 50,000 designs using full-wave simulation without CNN would be extremely time-consuming if not impossible. After completing the MOO, we obtain a Pareto front of the design problem that consists of a set of optimized meta-atoms, and each excels in at least one metric when compared to another design. Figure 3c compares the correlation among the three metrics for all of the designs before optimization and the designs on the Pareto front. The increase in the correlations among the metrics shows the effectiveness of optimization. To identify designs that simultaneously balance the three metrics, we look for the N-best designs which minimize a customized objective function1

where λ1, λ2, λ3 are positive weighted parameters and can be adjusted based on different requirements. Here, we selected the maximum value of these parameters to normalize the metrics, i.e., , where mi stands for σ, kc, and μt. We utilized the objective function to guide our selection process for the top 250 designs. Figure 3d shows the metric space in both 3D and 2D projections, with the top 250 data points highlighted in orange and the Pareto front in green. The purple star represents the metrics of a hexagonal design in our previous work14 for comparison, and we observe that the generated designs exhibit significantly improved metrics, which proves the effectiveness of our hybrid model in optimizing the meta-atoms for superior high-speed super-resolution performance.

We use randomly distributed fluorescent beads with a diameter of 40 nm (Figure 4a) as the object to numerically showcase the super-resolution imaging capability of the optimized LPSIM substrate design. We evaluate the performance of the hexagonal design in our previous work14 (Figure 4b), ten randomly selected designs from the training set (Figure 4c, Figure S1a, denoted as RAN1–RAN10), and one of the optimal designs (Figure 4d, denoted as OPT1). Additionally, several other designs from the Pareto front, which feature diverse geometries, are shown in Figure S2. The geometric parameters of these optimal designs are listed in Table S1. The evaluation metrics are listed in Table S2. We also examined the practical impact of fabrication deviations in Figure S3. These deviations are found to have minimal effect on the three metrics and, consequently, on the overall imaging performance.

Figure 4 Comparison of imaging results for different unit cell designs with simulated beads as the object. (a) Ground truth image of randomly distributed 40 nm fluorescent beads. Scale bar, 400 nm. (b–d) Unit cell of the hexagonal design, randomly picked design RAN10, and an optimal design OPT1, respectively. Scale bar, 40 nm. (e) OTF of the diffraction-limited imaging system. (f–h) Effective OTFs of the designs shown in (b–d), respectively. (i) Diffraction-limited image. Scale bar, 400 nm. (j–l) Blind-LPSIM reconstructions using 18 subimages for the designs shown in (b-d), respectively. Scale bar, 400 nm. (m-p) Zoom-in areas in the green dashed boxes in (i–l), respectively. Scale bar, 100 nm. (q–t) Zoom-in areas in the blue dashed boxes in (i–l), respectively. Scale bar, 100 nm.

The illumination patterns of each LPSIM substrate at 488 nm are obtained by using Lumerical FDTD. Figure 4e shows the OTF of the widefield imaging system, while Figure 4f-h and Figure S1b are the effective OTFs of the designs shown in Figure 4b-d and Figure S1a, respectively. The fluorescent images of beads at 520 nm collected by a 1.15 NA water immersion objective are simulated with Ii = (obj × illui)*PSF + Ni, where obj is the object, illui are the illumination patterns, * denotes the convolution operator, PSF is the point spread function of the imaging system, and Ni is Poisson noise considering a signal-to-noise ratio (SNR) of 20 dB. Figure 4i shows the simulated diffraction-limited images of the beads. Figure 4j-l and Figure S1c present the images reconstructed by using the blind-SIM algorithm. When illuminated with substrates consisting of randomly picked meta-atoms, most of the reconstructed images suffer from low resolution or contain a significant amount of artifacts. Specifically, most of them fail to resolve the quadruplet or resolve the four beads as two or three beads, shown in the zoomed-in reconstruction in Figure 4o and Figure S1d, and resolve multiple isolated beads as continuous lines in the scene with increased complexity (Figure 4s and Figure S1e). Despite the illuminations generated by the hexagonal design having a fairly high cutoff frequency and low similarity, the nonuniformity results in some beads appearing dimmer than others due to insufficient ignition by the sampling patterns during the measurements. This nonuniformity also leads to artifacts in the reconstructed images (Figure 4n,r). In contrast, the meta-atom optimized with the hybrid inverse design yields superior reconstruction with a two-point resolution of 80 nm and minimal artifacts (Figure 4l,p,t).

To demonstrate how LPSIM works for intricate biological objects and underscore the advantages of the proposed hybrid inverse design method, we simulate Blind-SIM reconstruction with LPSIM illumination generated by the hexagonal unit cell as well as the optimized meta-atom using microtubules (MTs) and endoplasmic reticulum (ER) as the object, as depicted in Figure 5a,b. The ground-truth (GT) images used here are from the data set BioSR.53

Figure 5 Demonstration of the improved LPSIM substrate design by using simulated imaging results for different types of biological objects: (a) microtubules, (b) endoplasmic reticulum. GT: ground truth. WF: widefield (diffraction limited). HEX18, HEX9: Blind-SIM reconstructions using 18 and 9 subimages for the hexagonal design shown in Figure 4b. OPT18, OPT9: Blind-SIM reconstructions using 18 and 9 subimages for the optimized design shown in Figure 4d. SNR is 20 dB for all measurements. Scale bar, 500 nm. Insets: corresponding Fourier transforms. (c) Comparison of the MS-SSIM versus the number of subimages for the hexagonal design and the optimized design. The SNR during measurement is 20 dB. (d) Comparison of the MS-SSIM versus the SNR during measurement for the hexagonal design and the optimized design using 18 subimages.

We evaluate the performance of different designs by analyzing image reconstructions produced with varying numbers of subimages (Figure 5a-c). Across both objects we observe that the reconstruction with measurements from the optimized design exhibits a superior match with the ground truth images in terms of the multiscale structural similarity index measure (MS-SSIM).54 The numbers of Blind-SIM iterations are selected for each reconstruction such that the MS-SSIM reaches its maximum, typically between 40 to 80. Although the reconstructions for the hexagonal design appear to be sharper and have higher peak signal-to-noise ratio (PSNR), they suffer from a conspicuous overemphasis on nodes (Figure S4c). This tendency to represent objects as interconnected dots leads to noticeable artifacts and excessively high contrast, rendering the representations less meaningful and authentic. In contrast, the reconstruction from the optimized design provided a more nuanced and accurate visual perception of the MT and ER. Figure 5c shows the MS-SSIM against the number of subimages utilized in reconstructions for both designs with 20 dB SNR measurements. When the number of subimages is reduced, the quality of both reconstructions degrades at a similar rate in terms of MS-SSIM.

We further test the imaging performance of the two LPSIM substrates across a range of SNRs from 25 down to 5 dB, as illustrated in Figure S4. Overall, the reconstructions for the hexagonal design consistently tend to overemphasize the nodes for various SNRs. This exacerbates as the SNR decreases down to 10 dB and below. It results in the reconstruction of continuous structures fragmenting into discrete pieces or even disappearing entirely. Conversely, the reconstruction from the optimized design demonstrates a superior alignment with the GT even at relatively low SNRs although dot-like noise artifacts also begin to emerge at an SNR of 5 dB. Figure 5d further illustrates the MS-SSIM versus the SNR for the hexagonal design and the optimized design utilizing 18 subimages. With degrading SNRs both reconstructions exhibited a similar degradation in quality in terms of MS-SSIM.

In conclusion, we present a hybrid inverse design framework, which integrates a CNN and a genetic algorithm-based MOO, to efficiently optimize silver nanoantenna arrays for LPSIM. It is important to highlight that despite its seemingly simple appearance with four elliptical shapes, the design of the OPT1 in Figure 4d is complex to be obtained through traditional design methods, which usually involve a two-step approach: initially identifying the optimal configuration of the four ellipses compared to other possible configurations and then fine-tuning the parameters of the chosen design. Our inverse design method streamlines this process by merging both steps and operates at a higher level of sophistication. Furthermore, it has the capability to discover various optimal designs along the Pareto front, catering to diverse user requirements.

The CNN model allows us to predict the illumination patterns rapidly and accurately for meta-atoms generated within the design framework. In our specific context, the meta-atoms comprise silver nanostructures belonging to the four designated antenna with fixed thickness. Only the planar geometries are subject to tuning. This mitigates the need to generate a complex and large data set, accelerates the convergence of the training process, and yields more accurate prediction. To augment the degree of freedom for potentially more optimized designs, a generalized predictor for near fields of free-form arbitrary nanostructures31,55 over a large region or array-level inverse design strategies56,57 may be used at a cost of increased computational demand. Exploring a broader range of materials and structured illumination scheme15 would further enhance the versatility of the hybrid model.

The proposed hybrid inverse design framework involves a complex data flow from structure to field and finally to metrics, making it more sophisticated and better suited for imaging tasks than existing frameworks that typically follow a simpler structure-to-response flow. Additionally, by predicting the field first, we can easily define and incorporate additional metrics to achieve a more holistic optimization, allowing the framework to be flexibly tailored to diverse applications. By training neural networks that model various forward processes and employing appropriate evaluation metrics, the proposed framework can be applied to diverse design scenarios, including but not limited to photonics, mechanics, and electric devices.

Supporting Information Available

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acs.nanolett.4c03069.Illumination scheme of LPSIM, details about the CNN and the enforcement of symmetric output, metrics for evaluation, additional simulated imaging results, geometric parameters of unit cell, metrices evaluation for different designs, parameter sensitivity analysis (PDF)

Supplementary Material

nl4c03069_si_001.pdf

Author Contributions

⊥ (Q.W. and Y.X.) These authors contributed equally.

The authors declare no competing financial interest.

Acknowledgments

Y.L. acknowledges the financial support of the National Science Foundation (CBET-1931777, ECCS-2136168, and DMR-2202268). Z.L. acknowledges the financial support from the Gordon and Betty Moore Foundation and the National Science Foundation (MRI-2320437).
==== Refs
References

Chalfie M. ; Tu Y. ; Euskirchen G. ; Ward W. W. ; Prasher D. C. Green Fluorescent Protein as a Marker for Gene Expression. Science 1994, 263 (5148 ), 802–805. 10.1126/science.8303295.8303295
Ntziachristos V. FLUORESCENCE MOLECULAR IMAGING. Annu. Rev. Biomed. Eng. 2006, 8 (1 ), 1–33. 10.1146/annurev.bioeng.8.061505.095831.16834550
Sahl S. J. ; Hell S. W. ; Jakobs S. Fluorescence nanoscopy in cell biology. Nat. Rev. Mol. Cell Biol. 2017, 18 (11 ), 685–701. 10.1038/nrm.2017.71.28875992
Schermelleh L. ; Ferrand A. ; Huser T. ; Eggeling C. ; Sauer M. ; Biehlmaier O. ; Drummen G. P. C. Super-resolution microscopy demystified. Nat. Cell Biol. 2019, 21 (1 ), 72–84. 10.1038/s41556-018-0251-8.30602772
Astratov V. N. ; Sahel Y. B. ; Eldar Y. C. ; Huang L. ; Ozcan A. ; Zheludev N. ; Zhao J. ; Burns Z. ; Liu Z. ; Narimanov E. ; et al. Roadmap on Label-Free Super-Resolution Imaging (Laser Photonics Rev. 17(12)/2023). Laser & Photonics Reviews 2023, 17 (12 ), 2370055 10.1002/lpor.202370055.
Rust M. J. ; Bates M. ; Zhuang X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 2006, 3 (10 ), 793–796. 10.1038/nmeth929.16896339
Betzig E. ; Patterson G. H. ; Sougrat R. ; Lindwasser O. W. ; Olenych S. ; Bonifacino J. S. ; Davidson M. W. ; Lippincott-Schwartz J. ; Hess H. F. Imaging Intracellular Fluorescent Proteins at Nanometer Resolution. Science 2006, 313 (5793 ), 1642–1645. 10.1126/science.1127344.16902090
Hell S. W. ; Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Opt. Lett. 1994, 19 (11 ), 780–782. 10.1364/OL.19.000780.19844443
Willig K. I. ; Harke B. ; Medda R. ; Hell S. W. STED microscopy with continuous wave beams. Nat. Methods 2007, 4 (11 ), 915–918. 10.1038/nmeth1108.17952088
Gustafsson M. G. L. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc. 2000, 198 (2 ), 82–87. 10.1046/j.1365-2818.2000.00710.x.10810003
Kner P. ; Chhun B. B. ; Griffis E. R. ; Winoto L. ; Gustafsson M. G. L. Super-resolution video microscopy of live cells by structured illumination. Nat. Methods 2009, 6 (5 ), 339–342. 10.1038/nmeth.1324.19404253
Lin R. ; Kipreos E. T. ; Zhu J. ; Khang C. H. ; Kner P. Subcellular three-dimensional imaging deep through multicellular thick samples by structured illumination microscopy and adaptive optics. Nat. Commun. 2021, 12 (1 ), 3148 10.1038/s41467-021-23449-6.34035309
Ponsetto J. L. ; Wei F. ; Liu Z. Localized plasmon assisted structured illumination microscopy for wide-field high-speed dispersion-independent super resolution imaging. Nanoscale 2014, 6 (11 ), 5807–5812. 10.1039/C4NR00443D.24740802
Ponsetto J. L. ; Bezryadina A. ; Wei F. ; Onishi K. ; Shen H. ; Huang E. ; Ferrari L. ; Ma Q. ; Zou Y. ; Liu Z. Experimental Demonstration of Localized Plasmonic Structured Illumination Microscopy. ACS Nano 2017, 11 (6 ), 5344–5350. 10.1021/acsnano.7b01158.28467053
Röhrich R. ; Koenderink A. F. Double moiré localized plasmon structured illumination microscopy. Nanophotonics 2021, 10 (3 ), 1107–1121. 10.1515/nanoph-2020-0521.
Yoo H. ; Lee H. ; Rhee W. J. ; Moon G. ; Lee C. ; Lee S. A. ; Shin J.-S. ; Kim D. Disordered Nanocomposite Islands for Nanospeckle Illumination Microscopy in Wide-Field Super-Resolution Imaging. Advanced Optical Materials 2021, 9 (15 ), 2100211 10.1002/adom.202100211.
Mudry E. ; Belkebir K. ; Girard J. ; Savatier J. ; Le Moal E. ; Nicoletti C. ; Allain M. ; Sentenac A. Structured illumination microscopy using unknown speckle patterns. Nat. Photonics 2012, 6 (5 ), 312–315. 10.1038/nphoton.2012.83.
Ayuk R. ; Giovannini H. ; Jost A. ; Mudry E. ; Girard J. ; Mangeat T. ; Sandeau N. ; Heintzmann R. ; Wicker K. ; Belkebir K. ; et al. Structured illumination fluorescence microscopy with distorted excitations using a filtered blind-SIMalgorithm. Opt. Lett. 2013, 38 (22 ), 4723–4726. 10.1364/OL.38.004723.24322116
Bezryadina A. ; Zhao J. ; Xia Y. ; Zhang X. ; Liu Z. High Spatiotemporal Resolution Imaging with Localized Plasmonic Structured Illumination Microscopy. ACS Nano 2018, 12 (8 ), 8248–8254. 10.1021/acsnano.8b03477.30044607
Bezryadina A. ; Zhao J. ; Xia Y. ; Lee Y. U. ; Zhang X. ; Liu Z. Localized plasmonic structured illumination microscopy with gaps in spatial frequencies. Opt. Lett. 2019, 44 (11 ), 2915–2918. 10.1364/OL.44.002915.
Molesky S. ; Lin Z. ; Piggott A. Y. ; Jin W. ; Vucković J. ; Rodriguez A. W. Inverse design in nanophotonics. Nat. Photonics 2018, 12 (11 ), 659–670. 10.1038/s41566-018-0246-9.
Oh J.-E. ; Cho Y.-W. ; Scarcelli G. ; Kim Y.-H. Sub-Rayleigh imaging via speckle illumination. Opt. Lett. 2013, 38 (5 ), 682–684. 10.1364/OL.38.000682.23455264
Yeh L.-H. ; Tian L. ; Waller L. Structured illumination microscopy with unknown patterns and a statistical prior. Biomed. Opt. Express 2017, 8 (2 ), 695–711. 10.1364/BOE.8.000695.28270977
Feichtner T. ; Selig O. ; Kiunke M. ; Hecht B. Evolutionary optimization of optical antennas. Phys. Rev. Lett. 2012, 109 (12 ), 127701 10.1103/PhysRevLett.109.127701.23005987
Huntington M. D. ; Lauhon L. J. ; Odom T. W. Subwavelength lattice optics by evolutionary design. Nano Lett. 2014, 14 (12 ), 7195–7200. 10.1021/nl5040573.25380062
Ren Y. ; Zhang L. ; Wang W. ; Wang X. ; Lei Y. ; Xue Y. ; Sun X. ; Zhang W. Genetic-algorithm-based deep neural networks for highly efficient photonic device design. Photonics Research 2021, 9 (6 ), B247 10.1364/PRJ.416294.
Phan T. ; Sell D. ; Wang E. W. ; Doshay S. ; Edee K. ; Yang J. ; Fan J. A. High-efficiency, large-area, topology-optimized metasurfaces. Light Sci. Appl. 2019, 8 , 48 10.1038/s41377-019-0159-5.31149333
Mansouree M. ; Kwon H. ; Arbabi E. ; McClung A. ; Faraon A. ; Arbabi A. Multifunctional 25D metastructures enabled by adjoint optimization. Optica 2020, 7 (1 ), 77 10.1364/OPTICA.374787.
Shi Z. ; Zhu A. Y. ; Li Z. ; Huang Y.-W. ; Chen W. T. ; Qiu C.-W. ; Capasso F. Continuous angle-tunable birefringence with freeform metasurfaces for arbitrary polarization conversion. Science Advances 2020, 6 (23 ), eaba3367 10.1126/sciadv.aba3367.32537506
Zhu D. ; Liu Z. ; Raju L. ; Kim A. S. ; Cai W. Building Multifunctional Metasystems via Algorithmic Construction. ACS Nano 2021, 15 (2 ), 2318–2326. 10.1021/acsnano.0c09424.33416319
Wiecha P. R. ; Muskens O. L. Deep Learning Meets Nanophotonics: A Generalized Accurate Predictor for Near Fields and Far Fields of Arbitrary 3D Nanostructures. Nano Lett. 2020, 20 (1 ), 329–338. 10.1021/acs.nanolett.9b03971.31825227
So S. ; Rho J. Designing nanophotonic structures using conditional deep convolutional generative adversarial networks. Nanophotonics 2019, 8 (7 ), 1255–1261. 10.1515/nanoph-2019-0117.
Liu Z. ; Zhu D. ; Lee K.-T. ; Kim A. S. ; Raju L. ; Cai W. Compounding Meta-Atoms into Metamolecules with Hybrid Artificial Intelligence Techniques. Adv. Mater. 2020, 32 (6 ), 1904790 10.1002/adma.201904790.
Ma W. ; Xu Y. ; Xiong B. ; Deng L. ; Peng R. W. ; Wang M. ; Liu Y. Pushing the Limits of Functionality-Multiplexing Capability in Metasurface Design Based on Statistical Machine Learning. Adv. Mater. 2022, 34 (16 ), e2110022 10.1002/adma.202110022.35167138
Yao K. ; Unni R. ; Zheng Y. Intelligent nanophotonics: merging photonics and artificial intelligence at the nanoscale. Nanophotonics 2019, 8 (3 ), 339–366. 10.1515/nanoph-2018-0183.34290952
Jiang J. ; Fan J. A. Global Optimization of Dielectric Metasurfaces Using a Physics-Driven Neural Network. Nano Lett. 2019, 19 (8 ), 5366–5372. 10.1021/acs.nanolett.9b01857.31294997
Kudyshev Z. A. ; Kildishev A. V. ; Shalaev V. M. ; Boltasseva A. Machine-learning-assisted metasurface design for high-efficiency thermal emitter optimization. Applied Physics Reviews 2020, 7 ( (2 ), ). 10.1063/1.5134792.
Xu Y. ; Zhang X. ; Fu Y. ; Liu Y. Interfacing photonics with artificial intelligence: an innovative design strategy for photonic structures and devices based on artificial neural networks. Photonics Research 2021, 9 (4 ), B135 10.1364/PRJ.417693.
Deng L. ; Xu Y. ; Liu Y. Hybrid inverse design of photonic structures by combining optimization methods with neural networks. Photonics and Nanostructures - Fundamentals and Applications 2022, 52 , 101073 10.1016/j.photonics.2022.101073.
Xu Y. ; Xiong B. ; Ma W. ; Liu Y. Software-defined nanophotonic devices and systems empowered by machine learning. Progress in Quantum Electronics 2023, 89 , 100469 10.1016/j.pquantelec.2023.100469.
Ma W. ; Liu Z. ; Kudyshev Z. A. ; Boltasseva A. ; Cai W. ; Liu Y. Deep learning for the design of photonic structures. Nat. Photonics 2021, 15 (2 ), 77–90. 10.1038/s41566-020-0685-y.
Jiang J. ; Chen M. ; Fan J. A. Deep neural networks for the evaluation and design of photonic devices. Nature Reviews Materials 2021, 6 (8 ), 679–700. 10.1038/s41578-020-00260-1.
Wiecha P. R. ; Arbouet A. ; Girard C. ; Muskens O. L. Deep learning in nano-photonics: inverse design and beyond. Photon. Res. 2021, 9 (5 ), B182–B200. 10.1364/PRJ.415960.
Ma W. ; Cheng F. ; Xu Y. ; Wen Q. ; Liu Y. Probabilistic Representation and Inverse Design of Metamaterials Based on a Deep Generative Model with Semi-Supervised Learning Strategy. Adv. Mater. 2019, 31 (35 ), e1901111 10.1002/adma.201901111.31259443
An S. ; Zheng B. ; Tang H. ; Shalaginov M. Y. ; Zhou L. ; Li H. ; Kang M. ; Richardson K. A. ; Gu T. ; Hu J. ; Multifunctional Metasurface Design with a Generative Adversarial Network. Adv. Opt. Mater. 2021, 9 ( (5 ), ). DOI: 10.1002/adom.202001433.
Sajedian I. ; Kim J. ; Rho J. Finding the optical properties of plasmonic structures by image processing using a combination of convolutional neural networks and recurrent neural networks. Microsyst Nanoeng 2019, 5 , 27 10.1038/s41378-019-0069-y.31240107
Unni R. ; Yao K. ; Zheng Y. Deep Convolutional Mixture Density Network for Inverse Design of Layered Photonic Structures. ACS Photonics 2020, 7 (10 ), 2703–2712. 10.1021/acsphotonics.0c00630.38031541
Liu Z. ; Zhu D. ; Rodrigues S. P. ; Lee K. T. ; Cai W. Generative Model for the Inverse Design of Metasurfaces. Nano Lett. 2018, 18 (10 ), 6570–6576. 10.1021/acs.nanolett.8b03171.30207735
Ma W. ; Liu Y. A data-efficient self-supervised deep learning model for design and characterization of nanophotonic structures. Science China Physics, Mechanics & Astronomy 2020, 63 ( (8 ), ). DOI: 10.1007/s11433-020-1575-2.
Elad M. Optimized Projections for Compressed Sensing. IEEE Transactions on Signal Processing 2007, 55 (12 ), 5695–5702. 10.1109/TSP.2007.900760.
Blank J. ; Deb K. Pymoo: Multi-Objective Optimization in Python. IEEE Access 2020, 8 , 89497–89509. 10.1109/ACCESS.2020.2990567.
Deb K. ; Pratap A. ; Agarwal S. ; Meyarivan T. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 2002, 6 (2 ), 182–197. 10.1109/4235.996017.
Qiao C. ; Li D. BioSR: a biological image dataset for super-resolution microscopy. figshare: 2022.10.6084/m9.figshare.13264793.v9
Wang Z. ; Simoncelli E. P. ; Bovik A. C. Multiscale structural similarity for image quality assessment. Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Nov 9-12, 2003 2003, 1392 , 1398–1402. 10.1109/ACSSC.2003.1292216.
Liu Z. ; Zhu D. ; Lee K. T. ; Kim A. S. ; Raju L. ; Cai W. Compounding meta-atoms into metamolecules with hybrid artificial intelligence techniques. Adv. Mater. 2020, 32 (6 ), 1904790 10.1002/adma.201904790.
Thureja P. ; Shirmanesh G. K. ; Fountaine K. T. ; Sokhoyan R. ; Grajower M. ; Atwater H. A. Array-Level Inverse Design of Beam Steering Active Metasurfaces. ACS Nano 2020, 14 (11 ), 15042–15055. 10.1021/acsnano.0c05026.33125844
Klopfer E. ; Dagli S. ; Barton D. III ; Lawrence M. ; Dionne J. A. High-Quality-Factor Silicon-on-Lithium Niobate Metasurfaces for Electro-optically Reconfigurable Wavefront Shaping. Nano Lett. 2022, 22 (4 ), 1703–1709. 10.1021/acs.nanolett.1c04723.35112873
