
==== Front
Nat Commun
Nat Commun
Nature Communications
2041-1723
Nature Publishing Group UK London

39266523
52331
10.1038/s41467-024-52331-4
Article
Behavioral biometric optical tactile sensor for instantaneous decoupling of dynamic touch signals in real time
Son Changil 1
http://orcid.org/0000-0001-9597-6863
Kim Jinyoung 1
Kang Dongwon 2
http://orcid.org/0000-0001-8889-1378
Park Seojoung 1
http://orcid.org/0009-0002-4702-7916
Ryu Chaeyeong 1
http://orcid.org/0000-0001-6856-6295
Baek Dahye 1
Jeong Geonyoung 1
Jeong Sanggyun 1
Ahn Seonghyeon 1
http://orcid.org/0000-0001-6903-0855
Lim Chanoong 1
Jeong Yundon 3
Eom Jeongin 2
http://orcid.org/0000-0001-5316-1690
Park Jung-Hoon 3
http://orcid.org/0000-0002-1572-9270
Lee Dong Woog 1
Kim Donghyuk dkim@unist.ac.kr

1
http://orcid.org/0000-0002-0376-9080
Kim Jungwook jungwkim@snu.ac.kr

4
http://orcid.org/0000-0003-2111-6101
Ko Hyunhyub hyunhko@unist.ac.kr

1
http://orcid.org/0000-0003-2968-2458
Lee Jiseok jiseok@unist.ac.kr

1
1 https://ror.org/017cjz748 grid.42687.3f 0000 0004 0381 814X School of Energy and Chemical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan Metropolitan City, 44919 Republic of Korea
2 https://ror.org/056tn4839 grid.263736.5 0000 0001 0286 5954 Department of Chemical and Biomolecular Engineering, Sogang University, Seoul, 04107 Republic of Korea
3 https://ror.org/017cjz748 grid.42687.3f 0000 0004 0381 814X Department of Biomedical Engineering, Ulsan National Institute of Science and Technology (UNIST), Ulsan Metropolitan City, 44919 Republic of Korea
4 https://ror.org/04h9pn542 grid.31501.36 0000 0004 0470 5905 Department of Chemical and Biological Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826 Republic of Korea
12 9 2024
12 9 2024
2024
15 800325 9 2023
28 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Decoupling dynamic touch signals in the optical tactile sensors is highly desired for behavioral tactile applications yet challenging because typical optical sensors mostly measure only static normal force and use imprecise multi-image averaging for dynamic force sensing. Here, we report a highly sensitive upconversion nanocrystals-based behavioral biometric optical tactile sensor that instantaneously and quantitatively decomposes dynamic touch signals into individual components of vertical normal and lateral shear force from a single image in real-time. By mimicking the sensory architecture of human skin, the unique luminescence signal obtained is axisymmetric for static normal forces and non-axisymmetric for dynamic shear forces. Our sensor demonstrates high spatio-temporal screening of small objects and recognizes fingerprints for authentication with high spatial-temporal resolution. Using a dynamic force discrimination machine learning framework, we realized a Braille-to-Speech translation system and a next-generation dynamic biometric recognition system for handwriting.

A sensitive upconversion nanocrystal-based biometric optical tactile sensor instantaneously and quantitatively decomposes dynamic touch signals into individual components of vertical normal and lateral shear force from a single image in real-time.

Subject terms

Characterization and analytical techniques
Sensors and biosensors
https://doi.org/10.13039/501100003725 National Research Foundation of Korea (NRF) 2021M3H4A3A01043764 2021M3H4A1A02056025 2022R1A2C2008120 Lee Jiseok Samsung Science and Technology Foundation (project number SRFC-TA1503-04)issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Behavioral biometric tactile systems are emerging technologies that identify people by analyzing the way they interact with their devices. For example, by measuring how fast the person writes or moves the pen and how much pressure they exert when interacting with the device, behavioral biometric tactile algorithms can authenticate the person signing a document1,2. Because personal habits are unique, behavioral biometric data is difficult to replicate, making these technologies better at preventing theft and fraud. Consequently, there is significant interest in using dynamic pressure sensing technology to identify and measure different types of forces that can be used for behavioral recognition3–14.

Optical tactile sensors are an attractive low-cost solution for interactive devices and tactile displays. While typical optical tactile sensors can measure normal force, information on in-plane motion, such as velocity, can only be obtained by positional analysis of two or more sequential images15–23. However, the resulting “time-averaged” velocity may not represent the true motion, especially when the directional change of motion is rapid with respect to the available rate of measurement, e.g., frames-per-second (fps) of a camera. (Supplementary Fig. 1). For more accurate information on in-plane motion, the optical tactile sensor should ideally also measure “instantaneous” velocity of the in-plane motion. Because instantaneous slip velocity is highly correlated to the shear force applied to the solid-solid interface in contact mechanics, having an optical tactile sensor capable of instantaneously measuring both normal and shear force would be desirable for studying behavioral biometrics. Optical polymers24,25, photonic crystals26, optical light guide27, quantum dots28, and organic dyes29,30 have been used to build optical tactile sensors. However, these systems cannot decompose dynamic elements of forces because they have slow signal response, high limits of detection, and low spatiotemporal resolution. To analyze fast movements such as handwriting in behavioral biometrics, we need optical tactile sensors that can quantitatively decompose applied force into vertical, normal, and lateral shear forces from a single image.

Here, we report a highly sensitive optical tactile sensor system that can instantaneously decouple dynamic touch signals in a single image for behavioral biometric analysis. Our tactile sensor contains a stress concentration layer that resembles the sensory architecture of the human skin and an optical tactile layer that contains an array of upconversion nanocrystals (UCNs). When force is applied to the sensor, the stress concentration layer amplifies and transfers the dynamic force to the tactile layer that interfaces with a total internal reflection (TIR) dove prism. Our UCNs-based optical tactile sensor generates unique axisymmetric or non-axisymmetric upconversion luminescence signals with spatiotemporal patterns that contain information on the motion of the object, such as direction, velocity, and magnitude of normal and shear force. Unlike conventional optical tactile sensors, our system can quantitatively decompose the applied force into individual components of vertical normal and lateral shear force from a single image in real time. We used our sensor system for surface profiling of objects, high-resolution fingerprint recognition, and anti-counterfeiting applications. Using a machine learning (ML) framework to analyze the luminescence tactile signals, we applied the sensor system in a small-scale Braille-to-Speech (BTS) translation system. Most importantly, our sensor accurately recognized behavioral biometric handwriting patterns. We expect such a simple sensor design to find ample applications in sophisticated behavioral biometric technologies.

Results

Design and principle of UCN-based optical tactile sensor for dynamic force decoupling

Our tactile sensor system includes a tactile sensor pad, a TIR dove prism and a charge-coupled device (CCD) camera (Fig. 1a, and Supplementary Fig. 2). The tactile sensor pad is composed of a stress concentration layer at the top, a thin (20 nm) anti-reflective platinum (Pt) layer in between, and a signal-generating tactile layer at the bottom (Fig. 1b). The stress concentration layer is designed to mimic the undulating interface of hard epidermis and soft dermis in human skin because such an interface is known to amplify applied forces by increasing interfacial area and localizing the applied force31,32. Further, mechanical mismatch between the rigid epidermis and soft dermis enhances force transmission from the skin to sensory mechanoreceptors33,34. To recreate this microarchitecture, we used stiff thermoplastic polyurethane (TPU, 52.5 MPa) and soft polydimethylsiloxane (PDMS, 1.76 MPa) with an undulated two-dimensional (2D) array of micro-hemispheres patterned at the interface (Fig. 1c, and Supplementary Fig. 3). The Pt layer in between serves to suppress diffuse reflection and improve the accuracy of the images (Supplementary Fig. 4). The bottom tactile layer contains an array of PDMS micro-hemispheres embedded with UCNs (Supplementary Fig. 5 and Supplementary Note 1). For our tactile sensor, we synthesized a hexagonal phase yellow luminescent lanthanide-doped UCNs (β-NaYF4:Yb3+/Er3+/Gd3+; 30/2/30 mol%) using a conventional hydrothermal method (Fig. 1d, e, and Methods for details). When homogenously embedded into the PDMS micro-hemispheres (Fig. 1f and Supplementary Fig. 6), their luminescence remained unchanged (Fig. 1g, and Methods for details). UCNs are advantageous for our system because they have a high spectral resolution (full width at half maximum of ~20 nm), low optical signal errors, and large anti-Stoke shifts. Further, UCNs do not require optical filters for luminescence color purification, have negligible background auto-fluorescence35 and are optically stable upon environmental perturbations36. Moreover, their luminescence colors can be easily tuned37 by simply adjusting the stoichiometry of lanthanide dopant ions (Supplementary Figs. 7 and 8, and Methods for details). UCNs are also amenable to large-scale synthesis38.Fig. 1 Upconversion nanocrystals (UCNs)-based optical tactile sensor resembling human skin.

a UCNs-based optical tactile sensor pad generating upconversion tactile signals via frustrated total internal reflection (TIR) of 980 nm near-infrared (NIR) illumination laser at the UM/prism interface. b Schematic of the optical tactile sensor pad consisting of a stress concentration layer made up of stiff thermoplastic urethane (TPU) interlocked with soft polydimethylsiloxane (PDMS) patterned with micro-hemispheres, a thin (20 nm) anti-reflective Pt layer, and a signal-generating tactile layer containing an array of micro-hemispheres (UMs). c Cross-sectional SEM images of the optical tactile sensor pad. (Scale bar, 100 µm). d TEM image of rod-like hexagonal phase lanthanide UCNs. (Scale bar, 100 nm). e Photograph of UCNs in cyclohexane excited by 980 nm NIR light display yellow luminescence. (Scale bar, 1 mm). f Luminescence image of UMs array upon 980 nm NIR excitation. (Scale bar, 200 µm) g, Emission spectra of yellow luminescence of UCNs and UCNs embedding in PDMS micro-hemispheres. h Principle of upconversion luminescence generation depending on the contact area changes between UM and TIR prism from the applied force. i, j Luminescence intensity profile under i static normal (axisymmetric luminescence) and j dynamic shear force (non-axisymmetric luminescence). k Decoupling of normal and shear forces from a single image frame using Machine Learning-based feature engineering. Source data are provided as a Source Data file.

In our system, signal generation depends on the contact between the UCNs-embedded micro-hemispheres (UMs) in the tactile layer and the TIR prism. Irradiating the prism with 980 nm near-infrared (NIR) laser generates evanescent waves in the thin (<100 nm) boundary layer close to the interface of the prism and UMs layer. When an object moves across our sensor pad, the stress concentration layer amplifies and transfers the applied dynamic forces to the UMs layer. As the contact area between the UM layer and TIR prism increases, the UMs produce a luminescence signal measurable by the CCD camera and luminescence spectrometer (Fig. 1h). This phenomenon can be ascribed to the transformation of the evanescence wave field into propagating far-fields at contact points between the UM and prism, where TIR is frustrated due to the decrease in the refractive index contrast according to the higher refractive index of PDMS (n = 1.4) compared to air (n = 1.0) (Supplementary Figs. 9 and 10 and Supplementary Note 2). At the contact points, the propagating far-field components can now penetrate the UM, effectively illuminating the UCNs within the UM volume. Consequently, locally compressed UMs with a more extensive contact area with the prism generate a stronger luminescence signal, while those without applied force do not yield measurable luminescence. It is noteworthy that the non-linear luminescence property of UCNs suppresses the initial luminescence caused by the evanescence wave field from the minimal contact between the UM layer and the TIR prism in the absence of applied force, consequently reducing background noise39. Locally pressed UMs with higher contact area with the prism produce higher luminescence signals, while those without applied force do not contact the prism and, therefore, do not produce luminescence (Supplementary Fig. 11a, b). When we applied a normal force to the top layer of the tactile sensor by vertically pressing a spherical indenter (5 mm diameter), the resulting luminescence signal was axisymmetric (Fig. 1i). The tactile sensor we fabricated exhibited uniform signal generation across the entire sensor pad or 10 different sensor pads and demonstrated highly stable signals under continuous laser excitation (Supplementary Figs. 12–14). However, when we simultaneously applied a lateral shear force in addition to the normal force by moving the pressed indenter along the surface of the top layer, the luminescence signal became non-axisymmetric—the shape of the signal is skewed towards the direction of movement (Fig. 1j, and Supplementary Fig. 11c). This suggests that detailed information on the surface-parallel motions of the indenter such as the magnitude of normal force, and the velocity, direction and magnitude of lateral shear force can be extracted by analyzing the non-axisymmetric shape of the luminescence signal. These features induced subtle changes in luminescence intensity, and to effectively observe this aspect, we proposed a tactile sensor based on UCNs instead of the conventional mechanoluminescence-based tactile sensors. Optical tactile sensors using UCNs have better sensitivity and response time compared with mechanoluminescence tactile sensors40,41. By comparing the luminescence shape from the optical tactile sensor, we proposed a method to determine both the direction and magnitude of normal and shear forces in real-time using an ML algorithm for force discrimination (Fig. 1k). By analyzing the luminescence tendency such as axisymmetric and non-axisymmetric for static and dynamic optical tactile sensors, respectively, we proposed ML methodologies to discriminate normal and shear forces in a single image frame in real-time.

Characterization of normal force-induced luminescence for high-sensitivity force measurement

To comprehend the relationship between normal force and luminescence signals, we conducted a systematic analysis of the signal intensity under varying applied normal forces. As the magnitude of the normal force increased from 0 to 5.0 N, we observed a corresponding increase in both luminescence intensity and the pressure distribution area, maintaining an axisymmetric circle (Fig. 2a, b, Supplementary Figs. 15–17, Supplementary Movie 1, and Methods). The stress concentration layer, due to its modulus difference and undulating interface, exhibited a strong and wide luminescence area with enhanced intensity. We established a correlation between grayscale and pressure by analyzing the area beneath the prism (Supplementary Figs. 18a and 19). Our optimized tactile sensor pad (1000 µm thick, including a 200 µm thick stress concentration layer) demonstrated the ability to detect normal forces as low as 0.05 N (Fig. 2b, Supplementary Note 3 for optimization, and Supplementary Figs. 20–22). The characteristic luminescence intensity of yellow UCNs at 530 and 660 nm showed a proportional increase with normal force (Fig. 2c). The consistent relationship between luminescence intensity and the cumulative gray value for each pixel enabled the presentation of a pressure distribution in a color map (Fig. 2d and Supplementary Fig. 23). Our sensor exhibited several notable properties. It showed no hysteresis in luminescence intensity under normal force application, indicating negligible adhesion between the PDMS micro-hemispherical array and the TIR prism (Fig. 2e). Additionally, our sensor demonstrated high stability, maintaining constant luminescence intensity over 5000 cycles of 5.0 N force application, even at high temperatures (Fig. 2f and Supplementary Fig. 24). Our sensor also boasted a fast response time of 9.12 ms, determined using Gaussian fitting. (Fig. 2g, and Supplementary Fig. 25). Remarkably, without the stress concentration layer, our sensor detected pressures as low as 0.02 N, outperforming current optical tactile sensors by an order of magnitude (Supplementary Table 1)15–22,40,41. These results demonstrate the exceptional performance and versatility of our tactile sensor, highlighting its potential for applications requiring precise force measurements and environmental conditions.Fig. 2 Quantitative measurement of static normal force.

a The axisymmetric UCN luminescence image (top) and its pressure distribution profile (bottom) under the normal force using a spherical indenter on the tactile sensor pad. (Scale bar, 2 mm). b Normalized UCN intensity across the centered indented area of the tactile sensor pad with increasing normal force from 0 to 5.0 N. (Scale bar, 2 mm). c Change of upconversion luminescence spectra with increasing normal force from 0 to 5.0 N. d Double y axis graph of integration of gray value of luminescence image (left) and luminescence intensity from 510 to 560 nm wavelength (right). e Loading-unloading hysteresis curves of luminescence signal. f Repeated stability test of luminescence intensity of UCNs-based optical tactile sensor subjected to a 5.0 N normal force over ~5000 cycles. g Response time of our tactile sensor using Gaussian fitting (9.12 ms). Source data are provided as a Source Data file.

Characterization of shear force-induced non-axisymmetric luminescence for real-time force decoupling

We further investigated the temporal change of luminescence signal under applied lateral shear force (Methods in details). Initially, a 1.0 N vertical preload was applied to the tactile sensor pad using a spherical indenter, creating a circular axisymmetric luminescence intensity and pressure contour (Supplementary Note 4). We then accelerated the indenter along the sensor pad surface at a constant rate of 1.0 mm/s2 until the velocity reached 1.0 mm/s. As the indenter began sliding, the axisymmetric luminescence circle stretched into a non-axisymmetric oval along the shear force direction (Fig. 3a). This transformation was captured at 10 frames-per-second (Supplementary Movie 2 middle) and analyzed frames-by-frame. Both the pressure contour color map and the luminescence intensity plot confirmed the change from circular to oval profile, with the center maintaining axisymmetric circular luminescence (Fig. 3a, and Supplementary Fig. 26). Note that the non-axisymmetric luminescence continued to form until the indenter reached a specific velocity (Supplementary Fig. 27 and Supplementary Movie 2). Additionally, the higher applied velocity made the more non-axisymmetric luminescence at the same time (Fig. 3b, c). This non-axisymmetric luminescence profile results from shear force-induced friction between the micro-hemisphere array layer and the TIR prism, causing asymmetric lateral deformation of the individual micro-hemispheres (Supplementary Note 4)42. We verified this unique luminescence profile by reducing friction with silicone oil, which eliminated the non-axisymmetric luminescence profile, maintaining an axisymmetric signal even during acceleration (Supplementary Fig. 28). Using a customized force gauge, we quantitatively measured and recorded the corresponding shear force for each non-axisymmetric luminescence image (Fig. 3a and Supplementary Fig. 29). Notably, our sensor allows immediate determination of shear force direction from a single non-axisymmetric luminescence profile, unlike conventional tactile sensors that requires a coordination analysis of multiple images (Fig. 3d)43,44.Fig. 3 Non-axisymmetric luminescence observation from directional and velocity information.

a Luminescence images (left), pressure distribution profile (middle), and luminescence intensity plots (right) show quantitative changes in the shape of the luminescence signal as the velocity of the spherical indenter increases. (Scale bar, 2 mm). b, c Luminescence images (left), pressure distribution profile (right), and luminescence intensity plots (bottom) at (b) 0 and (c) 0.5 s according to the velocity change of the indenter at 1.0 N of the initial normal force. (Scale bar, 2 mm). d UCN luminescence image (left) and pressure contour map (right) change as the indenter moves in eight different directions (Scale bar, 2 mm). e Calculated skewness for a moving indenter matched the measured shear force. f Overlay of calculated skewness with the experimentally measured shear force. Source data are provided as a Source Data file.

We further analyzed the friction force using a surface forces apparatus (SFA) (Supplementary Fig. 30, and Methods in detail). The kinetic friction coefficient increased from 0.148 to 0.531 as driving velocity increased from 11.6 to 58.0 μm/s (Supplementary Fig. 31). indicating that higher velocity leads to increased frictional forces and micro-hemisphere deformation, affecting the non-axisymmetric shape of the luminescence signal. These velocity-dependent friction coefficients are commonly in complex systems with soft polymers45 or polymer-like surfaces46, contradicting conventional Amontons’ Laws of Friction47,48.

To quantify the non-axisymmetric luminescence characteristic of shear force, the degree of non-axisymmetric luminescence was statistically described using the concept of skewness (γ), which is obtained by calculating the median and mean of the luminescence intensity profile (Supplementary Note 5). For a moving indenter, the calculated skewness matched well with the measured shear force, indicating that the magnitude of the shear force could be quantitatively estimated directly from the non-axisymmetric luminescence profile (Fig. 3e). When we analyzed the change in shear force at a given skewness for various normal forces, skewness decreased when the normal force increased from 0.5 to 1.5 N (Fig. 3f, and Supplementary Figs. 32–34). This is because an increase in normal force decreases the relative contribution of shear force (and thus, skewness value) to the luminescence signal, which is formed from both the normal force that creates the axisymmetric profile, and shear force that creates the non-axisymmetric profile (Fig. 3f, and Supplementary Fig. 35). Based on the single image frame analysis of luminescence intensity profile for static normal force and the directional skewness for dynamic shear force, we successfully quantified the direction and magnitude of both normal and shear forces in real-time from a single image using force discrimination ML algorithm (Fig. 4a, b, Supplementary Tables 2–4, Supplementary Movies 3, and Supplementary Notes 6 and 7). Our model showed high reliability with r2 values of 0.965 and 0.947 for normal and shear forces, respectively. The algorithm analyzed r, g, and b values of detected images to predict the applied normal force and determine the threshold for normal force region contour, enabling successful decoupling of normal and shear force in real-time (Fig. 4c, d, Supplementary Movie 4). Despite shear force attenuation due to frictional force between the sensor pad and TIR prism, our approach relies on the direct correlation between the measured skewness of the luminescence intensity and actual shear force applied to the sensor pad surface. This relationship has been thoroughly validated and used to train our ML model (Supplementary Notes 8). Importantly, the skewness change remained consistent across different time frame rates, suggesting that a cost-effective 10-fps camera is suitable for normal and shear force decoupling (Supplementary Figs. 36 and 37). This finding enhances the practical applicability of our sensor system.Fig. 4 Flowchart illustrating the process of machine learning-based force decoupling system within a single image frame.

a Captured non-axisymmetric luminescence signal induced by dynamic shear force (top) and pressure distribution profile (bottom) (Scale bar, 2 mm). b The process of feature engineering-based standard threshold estimation for force decoupling. Tendentious features of the RGB color space under applied normal force are quantified and utilized to train an SVR model for normal force estimation. Subsequently, the standard threshold for extracting the normal force region is determined by another SVR, which was trained on intensity profiles corresponding to the applied normal force. The estimated threshold and its half value were employed to delineate the normal and shear force regions, respectively. c Demonstration of decoupled normal (left) and shear force (right) in luminescence images (top) and pressure distribution profile (bottom). d Real-time decoupled estimation from a single frame of a non-axisymmetric luminescence signal. Source data are provided as a Source Data file.

Effect of interfacial undulation on enhanced shear force transmission in stress concentration layer

We further examined whether and how the undulated interface between the stiff TPU and soft PDMS in the stress concentration layer of the tactile sensor pad affects the non-axisymmetric luminescence (Supplementary Figs. 38 and 39). When an indenter moved at a velocity of 1.0 mm/s with 1.0 mm/s2 acceleration across a pad without undulation at the TPU-PDMS interface, the maximum value of skewness was 0.1 (Fig. 5a). With undulation, the maximum value of skewness increased to 0.37. When the velocity of the indenter was decreased to velocity of 0.5 mm/s with 0.5 mm/s2, the skewness decreased further to 0.32. Our results indicate that the undulation at the interface and the differences in modulus between TPU and PDMS improved the transmission of the applied shear force to the signal-generating micro-hemisphere layer, which, in turn, increased the non-axisymmetric of the luminescence signal.Fig. 5 Skewness change and FEA analysis of stress concentration layer.

a Experimentally measured Skewness change of four types of the stress concentration layers at 1.0 N of the initial normal force. b Finite element analysis (FEA) calculation of the local stress distribution at the undulating interface and contact width variations on the micro-hemisphere array under applied normal (Fn) and shear (Fs) forces in a 2D model sensor. c Comparison of the experimental luminescence intensity profile (black) and the calculated contact width profile (red) and its fit to the skewed Gaussian (blue). Experimental data was collected when Fs = 0.49 N. d Experimentally calculated skewness values for sensor pads with and without undulating interface under applied normal (Fn = 1.0 N) and shear (Fs = 0.49 N for high Fs and 0.22 N for low Fs) forces. Skewness increased with Fs and was higher in sensor pads with the undulating interface. e Color map at the undulating interface of difference in First Piola–Kirchhoff stress distribution under applied shear force. The dashed line describes the center of applied force. f Normalized plot of First Piola–Kirchhoff stresses of each shear (red) and normal (gray) direction and contact width to experimental intensity when Fs = 0. g FEA calculation of First Piola–Kirchhoff stress distribution between TPU and PDMS interfaces show regions below the interface (black dotted line) is on average 50% higher for the undulated interface. h Comparison plot of first Piola–Kirchhoff stress difference with and without the stress concentration layers at 10 μm below the interface between TPU and PDMS. Source data are provided as a Source Data file.

To understand how the TPU-PDMS interface undulation enhances force transmission, we employed finite element analysis (FEA) to examine the stress distribution in 2D model sensors with and without interface undulation under 1.0 N applied normal force (Fn) with and without shear force (Fs) ranging from 0 to 1.0 N (Fig. 5b, Supplementary Figs. 40–43, and Supplementary Note 9). Because the luminescence signal in our tactile sensor is produced only when UMs contact the TIR prism, we expected the intensity of the luminescence signal in the model sensor to scale with the contact width between the individual UMs and the underlying substrate. Consistent with the experimental intensity profile, our FEA analysis revealed that the contact width between UMs and the underlying substrate (which determines luminescence intensity) is symmetric under Fn alone but skewed in the direction of applied Fs (Fig. 5c and Supplementary Figs. 44 and 45). The calculated skewness, γ, increased with the magnitude of Fs and was substantially higher in the models with an undulating interface (Fig. 5d), corroborating the experimental results showing enhanced Fs transmission in sensors with an undulating interface (Supplementary Figs. 38 and 39). We visualized this enhancement by plotting the stress distribution differences (Δσ) between sensors with (σw) and without (σwo) an undulating interface (Fig. 5e). Notably Δσyx increased strongly near the micro-hemispheres array, indicating more effective Fs transmission due to the undulating interface (Supplementary Fig. 46). While σyy remained symmetric even under Fs, σyx became increasingly asymmetric with higher Fs (Fig. 5f, and Supplementary Fig. 47). Further analysis of σyx near the TPU-PDMS interface showed the regions beneath the undulated interface had, on average 50% higher σyx values (Fig. 5g, h). This phenomenon was independent of the indenter shape, with non-axisymmetric luminescence patterns observed across various indenter shapes and geometries (Supplementary Figs. 48–53). Although stress concentration may complicate precise tip shape determination, we utilized deep learning methods to aid in shape identification. This approach allows our sensor to maintain its effectiveness across a range of indenter shapes and applied forces. (Supplementary Fig. 54 and detailed in Supplementary Note 10).

High-resolution surface texture recognition using a refined thin sensor pad

To expand the versatility of our optical tactile sensor, we developed a thin sensor pad (200 µm thickness) without the stress concentration layer, optimized for precise high-resolution surface inspection (Supplementary Figs. 55–57). This refined design enables exceptional spatiotemporal tactile recognition of 2D surface textures, including multiple objects and fingerprints (Fig. 6a, b). Our sensor achieves a maximum spatial resolution of 100 µm, positioning it at the forefront of optical tactile sensing technology (Supplementary Fig. 58, and Supplementary Table 1)15–22,40,41. This high resolution allows for the differentiation and recognition of various surface textures with remarkable precision. For instance, our sensor visualizes intricate details such as the size and shape of dimples on golf balls and sign of wear with high spatiotemporal resolution (Fig. 6c). We further successfully identified small screws with subtle thread differences, imperceptible to the naked eye, using MATLAB-based automated image analysis (Fig. 6d, e, Supplementary Fig. 59, Supplementary Movie 5, and Supplementary Note 11)49. In addition, we developed a compact BTS system that instantly translates microscopic braille patterns into corresponding sounds using ML (Supplementary Note 12). The system converts topographic patterns of micro-sized braille (e.g., Hello World) into luminescence images and then into audio output by extracting 2D dot patterns (Fig. 6f–h, Supplementary Figs. 60 and 61, and Supplementary Movie 6).Fig. 6 Demonstration of high spatiotemporal optical tactile recognition.

a Tactile recognition of alphabets (Scale bar, 2 mm). b Optical (left) and luminescence (right) images of fingerprints for high spatiotemporal tactile recognition (Scale bar, 2 mm). c Photograph of a golf ball (left), the dimples (right) on its surface, and luminescence images of dimples with new and used golf ball (Scale bar, 5 mm). d Optical (top), luminescence and binary images (middle), and the corresponding intensity profile (bottom) showing the threads of different types of screws. (Scale bar, 1 mm). e Luminescence (top) and 3D color map (bottom) recognition of multiple screw threads (Scale bar, 1 mm). f Extraction of braille dot features from contour maps (bottom). g Schematic diagram of a machine learning workflow for real-time braille-to-speech (BTS). h BTS test using the HELLO WORLD sentence (Scale bar, 2 mm). Source data are provided as a Source Data file.

ML-based biometric handwriting analysis using decoupled normal and shear force data

Most notably, using the ability to simultaneously visualize the direction of movement and the intensity change of both normal and shear force, we developed a ML framework that distinguishes dynamic biometric handwriting of individuals (Fig. 7a, Supplementary Movie 7, and detailed in Supplementary Notes 13–15). Decoupled images represented normal and shear forces, and related features extracted from videos of three different individuals writing the letter ‘e’ were fed into our ML-based linear discriminant analysis (LDA) for writing style classification. LDA clustering successfully discriminated the three different handwritten letters ‘e’ (Fig. 7b, Supplementary Figs. 62–66, Supplementary Tables 6–9, and Supplementary Note 16). We further extracted the averaged velocity-related features from multiple image frames and combined them with normal and shear force features for handwriting analysis. Our single-frame touch signal decoupling-based ML framework showed outstanding performance when compared to analyzing the handwriting using normal forces alone or normal forces with velocity-related features (Fig. 7c, Supplementary Figs. 67 and 68 and Supplementary Table 8). While the average distance within a specific cluster was the same for all analyses, the average distance between different clusters was greater when the analysis included shear force features than when it included velocity features. Comparing the data cluster analyzed using both velocity and shear force features with those analyzed using velocity only and shear force only, we find that the analysis using combined features had a greater impact on the data clusters of velocity only analysis than on shear force only analysis. Further, between velocity-only analysis and shear force-only analysis, we find that shear-force feature-based handwriting analysis forms a more clear-cut data cluster (Supplementary Fig. 66).Fig. 7 Behavioral biometric mining of dynamic forces from luminescence images.

a Schematic diagram of the workflow for discriminating handwriting using a machine learning-based analysis of shear and normal forces extracted from luminescence profiles. Normal force features (red panels) extracted for analysis include maximum intensity from the intensity profile and diameter of the clustering region with maximum intensity (blue circle). Extracted shear force features (green panels) include the length of the major/minor axis of the oval, the ratio of the major to minor axis of the oval, the angle of the major axis, and the statistically estimated shear force. (Scale bar, 2 mm). b Linear discriminant analysis (LDA) clustering using different combinations of extracted features distinguish the handwritten letter ‘e’ of three individuals with different clarity. (Scale bar, 2 mm). c Mean of intra-cluster distance (gray) and mean of inter-cluster distance (orange) for LDA clustering using different combinations of extracted features. d Comparison of LDA clustering performances in terms of Calinski-Harabasz index and Silhouette coefficient for each case. Source data are provided as a Source Data file.

To compare clustering performance, we calculated the Silhouette coefficient, which describes the average distance between points within the same cluster and the average distance between points in the nearest cluster to which the data point does not belong, and the Calinski-Harabasz index, which is the ratio of dispersion within a cluster to dispersion between clusters. Of all the analysis combinations, the normal/shear force combination showed a Silhouette coefficient closest to 1 (0.867) and a remarkable value Calinski-Harabasz index (5.76 × 102) (Fig. 7d, Supplementary Fig. 68, and Supplementary Table 9). Adding the velocity feature to the normal/shear force feature slightly decreased the Silhouette coefficients (0.858) and Calinski-Harabasz index (4.52 × 102), and there is no significant advantage in terms of computational cost. From this point of view, we believe that besides normal force, the shear force feature is a dominant factor for distinguishing writing style. These results demonstrate that normal/shear force-based handwriting analysis represents the most clear-cut cluster and is well classified with clusters of different force combinations. In conventional handwriting authentication, verification is predominantly grounded in the consistency of static handwriting features such as letter size, inclination, and inter-character spacing, rather than dynamic attributes. Remarkably, our study introduces an innovative methodology for handwriting verification, focused on shear force—a parameter not previously explored in this context. This approach encapsulates a wealth of data, encompassing factors such as the exerted pressure during writing and the speed of inscription. These parameters raise the expectation that our framework can offer valuable insights into the writer’s identity, as they exhibit variations correlated with the writer’s age and gender50. Furthermore, our framework possesses the unique capability to decouple normal and shear forces within a single frame, allowing it to provide real-time feedback, unlike previous studies that evaluated handwriting only after all letters were completed. Thus, our system holds the potential to significantly enhance the precision of the handwriting authentication process.

In conclusion, we present an upconversion luminescence-based behavioral biometric optical tactile sensor that simultaneously and quantitatively discriminates velocity, direction, normal, and shear force in a single-frame analysis. Such a unique axisymmetric and non-axisymmetric luminescence signal amplified through the human skin mimetic stress concentration microarchitecture was theoretically studied by FEA simulation, experimentally examined by SFA measurement, and quantitatively analyzed. Our optical sensors are simple to build, and we show they can be used for the tactile recognition of surface textures, fingerprints, and small-scale braille. Most importantly, using ML, our system has enabled accurate dynamic biometric handwriting analysis. We believe our sensor system will open new avenues in facile optical tactile sensors that can intuitively and sensitively display multiple elements of dynamic forces in a single image frame, forming the basis of flexible behavioral biometric optical tactile sensors in wearable devices.

Methods

Materials

NaOH (Aldrich), ethanol (Aldrich, 99%), GdCl3·6H2O (Aldrich, 99.999%), YCl3·6H2O (Aldrich, 99.999%), YbCl3·6H2O (Aldrich, 99.999%), TmCl3·6H2O (Aldrich, 99.9%), NH4F (Aldrich, 99.9%), ErCl3·6H2O (Aldrich, 99.999%), and oleic acid (Aldrich, 90%).

Synthesis of rod-shaped β-NaREF4 (RE = Gd, Y, Yb, Er, Tm) UCNs

Hexagonal rod-shaped UCNs were synthesized using a hydrothermal method38. Briefly, NaOH (0.2 g mL–1), ethanol (10 mL), oleic acid (10 mL), a solution of the lanthanide mixture (0.2 M RECl3, RE = Gd, Y, Yb, Er, and Tm, 4 mL), and 2 M ammonium fluoride (NH4F, 2 mL) were mixed by stirring for 10 min and transferred into a Teflon-lined hydrothermal device. Next, the premixed solution was heated at 200 °C for 3 h. Subsequently, the hydrothermal device was naturally cooled to room temperature (26 °C). The resulting product was collected via centrifugation and washed several times with ethanol and water. Finally, the resulting powder was dispersed in ethanol.

Stamps and screws information

Triangle, rectangle, pentagon, hexagon, alphabet letters, and braille stamps were fabricated using a 3D printer (Form2, Formlabs). The length of one side of the triangle, rectangle, pentagon, and hexagon was 5 mm, and the width of the alphabet stroke on stamps was 400 μm. The diameter and pitch of the dots in braille were 500 and 833.33 μm, respectively. Four types of screw (1/4–20; SS25S050, Thorlabs, M6; SS6MS12, Thorlabs, 8–32; SS8S050, Thorlabs, and M3; SS3S6, Thorlabs) were used for the screening of thread pitches (1/4–20; 1.27 mm, M6; 1.00 mm, 8–32; 0.79 mm, M3; 0.50 mm). Additionally, a golf ball (Titleist Pro V1) was used for surface profiling.

Optical imaging

Optical images for micro-hemisphere structure deformation were captured by a microscopic camera (HA010, Gasworld). To capture these microscopic images through the thick dove prism used for generating TIR, we utilized a long working distance objective lens (Mitutoyo 10×, 0.28 NA) in combination with a macro camera lens (Nikkor 105 mm f/2.8) that served as the tube lens with sCMOS camera (C14440-20UP, Hamamatsu photonics). Luminescence images were captured using a 10-fps CCD camera (DCC3260C, Thorlabs) and a 30-fps CMOS camera (DigiRetina16, Tucsen). Luminescence images were converted to color map contours of the pressure distribution using Origin and MATLAB. The luminescence intensity profile along the centerline was plotted using the ImageJ software.

Measurement of friction force and coefficient of friction using SFA

The friction forces were measured using piezoelectric 3D sensor (upper surface) and biomorph slider (lower surface) attachments in the SFA system (SFA2000, Surforce LLC, USA)51. Supplementary Fig. 31 shows a schematic of the friction force measurement setup. For the upper surface, a PDMS sample with an optical tactile sensor (10 mm × 10 mm) taped onto a flat glass disk was used. For the lower surface, a hemispherical brass with a radius of curvature of 2 cm radius was used. Friction was measured while increasing the load stepwise, at two different driving velocities (11.6 and 58.0 μm/s). The kinetic coefficient of friction was then obtained from the slope of the friction vs. load curve. A constant shear distance (sliding amplitude) (116 μm) and triangular wave signal determined by a function generator was applied during all measurements. The frictional force was measured at least five times for each load.

Characterization of optical signals

Luminescence images of the UMs were taken with a Nikon D810 digital camera on an inverted microscope (Ti-E, Nikon) under irradiation with a 980 nm NIR laser (MDL-F-980-5W, Dragon laser). The UMs were excited using a 980 nm NIR laser. The NIR reflection was removed using a 950 nm cut-on filter (Semrock Inc.). The upconversion luminescence spectra of the UCNs were measured using a modular spectrometer (QE Pro-FL, Ocean Optics). Image frames were extracted using the Adobe Premiere Pro CC.

Characterization of UMs integrated with the stress concentration layer

The geometry of the optical tactile sensor was characterized by field-emission scanning electron microscopy (S-4800, Hitachi) and high-resolution transmission electron microscopy (JEM-1400, JEOL).

Fabrication of the UCN-embedded micro-hemisphere layer assembled with a human skin mimetic stress concentration layer

The synthesized UCNs were dispersed in n-hexane by ultrasonic sonication (VCX 750, Sonics) for 2 h. The UCN solution was added to the PDMS base (Sylgard 184, Dow Corning) in a 2:3 ratio (UCN: PDMS). The UCN/PDMS mixture was dispersed using an ultrasonic processor for 1 h. Subsequently, a PDMS curing agent (1:5 ratio of curing agent to base) was added to the UCN/PDMS mixture depending on the thickness of the sensor. The resulting UCN/PDMS mixture was dispersed using an ultrasonic process for 5 min before pouring into a micro-hemisphere silicon mold (diameter and pitch: 90 and 100 μm, respectively) and cured at 90 °C for 4 h. The dispersion of UCNs and PDMS was observed (Supplementary Fig. 6). The top surface of the UMs array was coated with a Pt layer using an ion sputter (MC1000, Hitachi). To fabricate the stress concentration layer, the pre-cured PDMS solution was coated on a micro-hemisphere silicon mold (diameter and pitch: 20 and 40 μm, respectively) using a spin coater at 750 rpm for 1 min. After curing at 90 °C for 2 h, the obtained undulating microstructured PDMS was attached to a Pt layer. After 1 min of the O2 plasma process (FEMTO SCIENCE, CUTE-1MPR, 100 W, 8 sccm), the TPU solution dissolved in DMF was poured onto the undulating microstructured PDMS surface, and the DMF solvent was evaporated at 60 °C for 12 h.

Normal force and shear force detection device setting

To measure the optical properties of the optical tactile sensor under mechanical forces, the fabricated UMs integrated with a stress concentration layer were placed on the TIR dove prism, where a 980 nm NIR laser (FC-EW-980-40W, Uniotech) was irradiated. A schematic of the normal force detection system is shown in Supplementary Fig. 14. The normal force was measured by a vertically attached force gauge with a spherical indenter to the Z axis liner stage. The force gauge used in our study had a resolution of 0.01 N. A normal force of 0–5.0 N was measured by moving the Z axis linear stage, and movie files of the luminescence profiles were recorded (Supplementary Movie 1). Supplementary Fig. 28 shows a schematic of the shear force detection system. To measure the shear force, a right-angle spherical indenter was fabricated using a 90° tilted digital force gauge, and the shear force applied according to the velocity and acceleration of the spherical indenter was measured. Supplementary Movie 2 shows the change in the luminescence profile as the velocity reaches 1.5, 1.0, and 0.5 mm/s (acceleration: 1.5, 1.0, and 0.5 mm/s2, respectively); the velocity was maintained (1.5 mm/s: 2.3 s, 1.0 mm/s: 4 s, and 0.5 mm/s: 9 s, respectively), and then decelerated to 0 mm/s in 1 s. To obtain the luminescence images and movie file, a CCD camera (DCC3260C, Thorlabs) was placed under a glass dove prism, and a 750 nm cut-off filter (FESH0750, Thorlabs) was used to eliminate 980 nm NIR light.

Handwriting data mining ML

Regression analysis from the normal and shear force predictions for each related feature extraction was conducted based on support vector regression (SVR). SVR is typically used for non-linear systems and predicts the appropriate hyperplane in higher dimensions to fit the input data. Parameters C = 10, degree = 3, epsilon = 0.1, and kernel = ‘rbf’ were selected for the best performance results. The model was trained on 70% of the data randomly split from the data set, and the unused data were used to test the model performance. To evaluate the performance, r2, RMSE, and MAPE parameters were used. LDA, which determines linear combinations of features that characterize or distinguish two or more classes, was used to classify handwriting by lowering the dimensions of the extracted force-related features. In addition, the clustering performance was calculated using the silhouette coefficient and the Calinski-Harabasz index for each class.

Supplementary information

Supplementary information

Peer Review File

Description of Additional Supplementary Files

Supplementary Movie 1

Supplementary Movie 2

Supplementary Movie 3

Supplementary Movie 4

Supplementary Movie 5

Supplementary Movie 6

Supplementary Movie 7

Supplementary Code

Source data

Source data set

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-024-52331-4.

Acknowledgements

This study was supported by the Samsung Science and Technology Foundation (project number SRFC-TA1503-04), National Research Foundation funded by the Korean government (grant numbers 2022R1A2C2008120, 2022R1A2C1013103, 2021R1A2C3009222, 2021M3H4A1A02056025, and 2021M3H4A3A01043764). We thank Dr. Ai Lin Chun for the critical reading of the manuscript.

Author contributions

C.S., Jinyoung K., D.K., S.P., and C.R. contributed equally to this work. C.S. and Jinyoung K. conducted the majority of the experiments, interpreted data, and wrote the manuscript. G.J. and S.J. assisted with the fabrication of hemisphere microarchitectures. C.R., D.B., and S.A. helped with the fabrication of the tactile sensor device. C.L. and D.W.L. conducted the SFA experiment. Y.J. and J.P. assisted with the optical setup and confirmation of frustrated TIR. D. Kang, J.E., J.L., and Jungwook K. conducted theoretical calculations of the force concentration layer, interpreted the COMSOL data, and contributed to manuscript writing. S.P., J.L., and D. Kim interpreted the image data using machine learning methods and contributed to manuscript writing. J.L. and H.K. conceived the project, interpreted the results, wrote the manuscript, and supervised the study. All authors reviewed and approved the manuscript.

Peer review

Peer review information

Nature Communications thanks Guozhen Shen, Junyi Zhai and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

The authors declare that all data supporting the findings of this study are available within the paper and its supplementary information files. All data are available from the corresponding authors upon request. Source data are provided in this paper.

Code availability

The codes used in this article are provided in this paper.

Competing interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

These authors contributed equally: Changil Son, Jinyoung Kim, Dongwon Kang, Seojoung Park, Chaeyeong Ryu.
==== Refs
References

1. Alzubaidi A Kalita J Authentication of smartphone users using behavioral biometrics IEEE Commun. Surv. Tutor. 2016 18 1998 2026 10.1109/COMST.2016.2537748
Alzubaidi, A. & Kalita, J. Authentication of smartphone users using behavioral biometrics. IEEE Commun. Surv. Tutor. 18, 1998–2026 (2016).10.1109/COMST.2016.2537748
2. Frank M Biedert R Ma E Martinovic I Song D Touchalytics: on the applicability of touchscreen input as a behavioral biometric for continuous authentication IEEE Trans. Inf. Forensics Secur. 2012 8 136 148 10.1109/TIFS.2012.2225048
Frank, M., Biedert, R., Ma, E., Martinovic, I. & Song, D. Touchalytics: on the applicability of touchscreen input as a behavioral biometric for continuous authentication. IEEE Trans. Inf. Forensics Secur. 8, 136–148 (2012).10.1109/TIFS.2012.2225048
3. Ha M Bioinspired interlocked and hierarchical design of ZnO nanowire arrays for static and dynamic pressure‐sensitive electronic skins Adv. Funct. Mater. 2015 25 2841 2849 10.1002/adfm.201500453
Ha, M. et al. Bioinspired interlocked and hierarchical design of ZnO nanowire arrays for static and dynamic pressure‐sensitive electronic skins. Adv. Funct. Mater. 25, 2841–2849 (2015).10.1002/adfm.201500453
4. Park J Kim M Lee Y Lee HS Ko H Fingertip skin–inspired microstructured ferroelectric skins discriminate static/dynamic pressure and temperature stimuli Sci. Adv. 2015 1 e1500661 10.1126/sciadv.1500661 26601303
Park, J., Kim, M., Lee, Y., Lee, H. S. & Ko, H. Fingertip skin–inspired microstructured ferroelectric skins discriminate static/dynamic pressure and temperature stimuli. Sci. Adv. 1, e1500661 (2015).26601303 10.1126/sciadv.1500661
5. Lee H-K Chung J Chang S-I Yoon E Normal and shear force measurement using a flexible polymer tactile sensor with embedded multiple capacitors J. Microelectromech. Syst. 2008 17 934 942 10.1109/JMEMS.2008.921727
Lee, H.-K., Chung, J., Chang, S.-I. & Yoon, E. Normal and shear force measurement using a flexible polymer tactile sensor with embedded multiple capacitors. J. Microelectromech. Syst. 17, 934–942 (2008).10.1109/JMEMS.2008.921727
6. Li R Nie B Digiglio P Pan T Microflotronics: a flexible, transparent, pressure‐sensitive microfluidic film Adv. Funct. Mater. 2014 24 6195 6203 10.1002/adfm.201401527
Li, R., Nie, B., Digiglio, P. & Pan, T. Microflotronics: a flexible, transparent, pressure‐sensitive microfluidic film. Adv. Funct. Mater. 24, 6195–6203 (2014).10.1002/adfm.201401527
7. Ramuz M Tee BCK Tok JBH Bao Z Transparent, optical, pressure‐sensitive artificial skin for large‐area stretchable electronics Adv. Mater. 2012 24 3223 3227 10.1002/adma.201200523 22641411
Ramuz, M., Tee, B. C. K., Tok, J. B. H. & Bao, Z. Transparent, optical, pressure‐sensitive artificial skin for large‐area stretchable electronics. Adv. Mater. 24, 3223–3227 (2012).22641411 10.1002/adma.201200523
8. Stassi S Cauda V Canavese G Pirri CF Flexible tactile sensing based on piezoresistive composites: a review Sensors 2014 14 5296 5332 10.3390/s140305296 24638126
Stassi, S., Cauda, V., Canavese, G. & Pirri, C. F. Flexible tactile sensing based on piezoresistive composites: a review. Sensors 14, 5296–5332 (2014).24638126 10.3390/s140305296
9. Viry L Flexible three‐axial force sensor for soft and highly sensitive artificial touch Adv. Mater. 2014 26 2659 2664 10.1002/adma.201305064 24677245
Viry, L. et al. Flexible three‐axial force sensor for soft and highly sensitive artificial touch. Adv. Mater. 26, 2659–2664 (2014).24677245 10.1002/adma.201305064
10. Wang L Beebe DJ A silicon-based shear force sensor: development and characterization Sens. Actuators A: Phys. 2000 84 33 44 10.1016/S0924-4247(99)00342-8
Wang, L. & Beebe, D. J. A silicon-based shear force sensor: development and characterization. Sens. Actuators A: Phys. 84, 33–44 (2000).10.1016/S0924-4247(99)00342-8
11. Wang X Self‐powered high‐resolution and pressure‐sensitive triboelectric sensor matrix for real‐time tactile mapping Adv. Mater. 2016 28 2896 2903 10.1002/adma.201503407 26879844
Wang, X. et al. Self‐powered high‐resolution and pressure‐sensitive triboelectric sensor matrix for real‐time tactile mapping. Adv. Mater. 28, 2896–2903 (2016).26879844 10.1002/adma.201503407
12. Yoon SG Park BJ Chang ST Highly sensitive piezocapacitive sensor for detecting static and dynamic pressure using ion-gel thin films and conductive elastomeric composites ACS Appl. Mater. Interfaces 2017 9 36206 36219 10.1021/acsami.7b11700 28984435
Yoon, S. G., Park, B. J. & Chang, S. T. Highly sensitive piezocapacitive sensor for detecting static and dynamic pressure using ion-gel thin films and conductive elastomeric composites. ACS Appl. Mater. Interfaces 9, 36206–36219 (2017).28984435 10.1021/acsami.7b11700
13. Zheng Q Dynamic real-time imaging of living cell traction force by piezo-phototronic light nano-antenna array Sci. Adv. 2021 7 eabe7738 10.1126/sciadv.abe7738 34039600
Zheng, Q. et al. Dynamic real-time imaging of living cell traction force by piezo-phototronic light nano-antenna array. Sci. Adv. 7, eabe7738 (2021).34039600 10.1126/sciadv.abe7738
14. Jia M Tactile tribotronic reconfigurable pn junctions for artificial synapses Sci. Bull. 2022 67 803 812 10.1016/j.scib.2021.12.014
Jia, M. et al. Tactile tribotronic reconfigurable pn junctions for artificial synapses. Sci. Bull. 67, 803–812 (2022).10.1016/j.scib.2021.12.014
15. Wei XY Dynamic triboelectrification‐induced electroluminescence and its use in visualized sensing Adv. Mater. 2016 28 6656 6664 10.1002/adma.201600604 27213998
Wei, X. Y. et al. Dynamic triboelectrification‐induced electroluminescence and its use in visualized sensing. Adv. Mater. 28, 6656–6664 (2016).27213998 10.1002/adma.201600604
16. Lee B Ultraflexible and transparent electroluminescent skin for real-time and super-resolution imaging of pressure distribution Nat. Commun. 2020 11 663 10.1038/s41467-020-14485-9 32005935
Lee, B. et al. Ultraflexible and transparent electroluminescent skin for real-time and super-resolution imaging of pressure distribution. Nat. Commun. 11, 663 (2020).32005935 10.1038/s41467-020-14485-9
17. Zhao X Self-powered user-interactive electronic skin for programmable touch operation platform Sci. Adv. 2020 6 eaba4294 10.1126/sciadv.aba4294 32832600
Zhao, X. et al. Self-powered user-interactive electronic skin for programmable touch operation platform. Sci. Adv. 6, eaba4294 (2020).32832600 10.1126/sciadv.aba4294
18. Du Y Mechanically excited multicolor luminescence in lanthanide ions Adv. Mater. 2019 31 1807062 10.1002/adma.201807062
Du, Y. et al. Mechanically excited multicolor luminescence in lanthanide ions. Adv. Mater. 31, 1807062 (2019).10.1002/adma.201807062
19. Chen C Creating visible-to-near-infrared mechanoluminescence in mixed-anion compounds SrZn2S2O and SrZnSO Nano Energy 2020 68 104329 10.1016/j.nanoen.2019.104329
Chen, C. et al. Creating visible-to-near-infrared mechanoluminescence in mixed-anion compounds SrZn2S2O and SrZnSO. Nano Energy 68, 104329 (2020).10.1016/j.nanoen.2019.104329
20. Wang Y Enhanced high-resolution triboelectrification-induced electroluminescence for self-powered visualized interactive sensing ACS Appl. Mater. interfaces 2019 11 13796 13802 10.1021/acsami.9b02313 30888789
Wang, Y. et al. Enhanced high-resolution triboelectrification-induced electroluminescence for self-powered visualized interactive sensing. ACS Appl. Mater. interfaces 11, 13796–13802 (2019).30888789 10.1021/acsami.9b02313
21. Wei XY High‐intensity triboelectrification‐induced electroluminescence by microsized contacts for self‐powered display and illumination Adv. Mater. Interfaces 2018 5 1701063 10.1002/admi.201701063
Wei, X. Y. et al. High‐intensity triboelectrification‐induced electroluminescence by microsized contacts for self‐powered display and illumination. Adv. Mater. Interfaces 5, 1701063 (2018).10.1002/admi.201701063
22. Zhuang Y Force-induced charge carrier storage: a new route for stress recording Light Sci. Appl. 2020 9 182 10.1038/s41377-020-00422-4 33133522
Zhuang, Y. et al. Force-induced charge carrier storage: a new route for stress recording. Light Sci. Appl. 9, 182 (2020).33133522 10.1038/s41377-020-00422-4
23. Dvorak N Chung K Mueller K Ku P-C Ultrathin tactile sensors with directional sensitivity and a high spatial resolution Nano Lett. 2021 21 8304 8310 10.1021/acs.nanolett.1c02837 34597518
Dvorak, N., Chung, K., Mueller, K. & Ku, P.-C. Ultrathin tactile sensors with directional sensitivity and a high spatial resolution. Nano Lett. 21, 8304–8310 (2021).34597518 10.1021/acs.nanolett.1c02837
24. Chou H-H A chameleon-inspired stretchable electronic skin with interactive colour changing controlled by tactile sensing Nat. Commun. 2015 6 8011 10.1038/ncomms9011 26300307
Chou, H.-H. et al. A chameleon-inspired stretchable electronic skin with interactive colour changing controlled by tactile sensing. Nat. Commun. 6, 8011 (2015).26300307 10.1038/ncomms9011
25. Haque MA Kurokawa T Kamita G Yue Y Gong JP Rapid and reversible tuning of structural color of a hydrogel over the entire visible spectrum by mechanical stimulation Chem. Mater. 2011 23 5200 5207 10.1021/cm2021142
Haque, M. A., Kurokawa, T., Kamita, G., Yue, Y. & Gong, J. P. Rapid and reversible tuning of structural color of a hydrogel over the entire visible spectrum by mechanical stimulation. Chem. Mater. 23, 5200–5207 (2011).10.1021/cm2021142
26. Liu H Pressure-induced blue-shifted and enhanced emission: a cooperative effect between aggregation-induced emission and energy-transfer suppression J. Am. Chem. Soc. 2020 142 1153 1158 10.1021/jacs.9b11080 31922412
Liu, H. et al. Pressure-induced blue-shifted and enhanced emission: a cooperative effect between aggregation-induced emission and energy-transfer suppression. J. Am. Chem. Soc. 142, 1153–1158 (2020).31922412 10.1021/jacs.9b11080
27. Kim JT Choi H Shin E Park S Kim IG Graphene-based optical waveguide tactile sensor for dynamic response Sci. Rep. 2018 8 16118 10.1038/s41598-018-34613-2 30382147
Kim, J. T., Choi, H., Shin, E., Park, S. & Kim, I. G. Graphene-based optical waveguide tactile sensor for dynamic response. Sci. Rep. 8, 16118 (2018).30382147 10.1038/s41598-018-34613-2
28. Xu Y Glowing stereocomplex biopolymers are generating power: polylactide/carbon quantum dot hybrid nanofibers with high piezoresponse and multicolor luminescence J. Mater. Chem. A 2019 7 1810 1823 10.1039/C8TA08593E
Xu, Y. et al. Glowing stereocomplex biopolymers are generating power: polylactide/carbon quantum dot hybrid nanofibers with high piezoresponse and multicolor luminescence. J. Mater. Chem. A 7, 1810–1823 (2019).10.1039/C8TA08593E
29. Wang Y Dynamic behavior of molecular switches in crystal under pressure and its reflection on tactile sensing J. Am. Chem. Soc. 2015 137 931 939 10.1021/ja511499p 25533888
Wang, Y. et al. Dynamic behavior of molecular switches in crystal under pressure and its reflection on tactile sensing. J. Am. Chem. Soc. 137, 931–939 (2015).25533888 10.1021/ja511499p
30. Zhao W Highly sensitive switching of solid-state luminescence by controlling intersystem crossing Nat. Commun. 2018 9 3044 10.1038/s41467-018-05476-y 30072690
Zhao, W. et al. Highly sensitive switching of solid-state luminescence by controlling intersystem crossing. Nat. Commun. 9, 3044 (2018).30072690 10.1038/s41467-018-05476-y
31. Prevost A Scheibert J Debrégeas G Effect of fingerprints orientation on skin vibrations during tactile exploration of textured surfaces Commun. Integr. Biol. 2009 2 422 424 10.4161/cib.2.5.9052 19907707
Prevost, A., Scheibert, J. & Debrégeas, G. Effect of fingerprints orientation on skin vibrations during tactile exploration of textured surfaces. Commun. Integr. Biol. 2, 422–424 (2009).19907707 10.4161/cib.2.5.9052
32. Aleemardani M Trikić MZ Green NH Claeyssens F The importance of mimicking dermal-epidermal junction for skin tissue engineering: a review Bioengineering 2021 8 148 10.3390/bioengineering8110148 34821714
Aleemardani, M., Trikić, M. Z., Green, N. H. & Claeyssens, F. The importance of mimicking dermal-epidermal junction for skin tissue engineering: a review. Bioengineering 8, 148 (2021).34821714 10.3390/bioengineering8110148
33. Cauna N Nature and functions of the papillary ridges of the digital skin Anat. Rec. 1954 119 449 468 10.1002/ar.1091190405 13283320
Cauna, N. Nature and functions of the papillary ridges of the digital skin. Anat. Rec. 119, 449–468 (1954).13283320 10.1002/ar.1091190405
34. Maeno T Kobayashi K Yamazaki N Relationship between the structure of human finger tissue and the location of tactile receptors JSME Int. J. Ser. C. Mech. Syst. Mach. Elem. Manuf. 1998 41 94 100
Maeno, T., Kobayashi, K. & Yamazaki, N. Relationship between the structure of human finger tissue and the location of tactile receptors. JSME Int. J. Ser. C. Mech. Syst. Mach. Elem. Manuf. 41, 94–100 (1998).
35. Wang F Simultaneous phase and size control of upconversion nanocrystals through lanthanide doping Nature 2010 463 1061 1065 10.1038/nature08777 20182508
Wang, F. et al. Simultaneous phase and size control of upconversion nanocrystals through lanthanide doping. Nature 463, 1061–1065 (2010).20182508 10.1038/nature08777
36. Liu D Emission stability and reversibility of upconversion nanocrystals J. Mater. Chem. C. 2016 4 9227 9234 10.1039/C6TC02990F
Liu, D. et al. Emission stability and reversibility of upconversion nanocrystals. J. Mater. Chem. C. 4, 9227–9234 (2016).10.1039/C6TC02990F
37. Baek D Multi‐color luminescence transition of upconversion nanocrystals via crystal phase control with SiO2 for high temperature thermal labels Adv. Sci. 2020 7 2000104 10.1002/advs.202000104
Baek, D. et al. Multi‐color luminescence transition of upconversion nanocrystals via crystal phase control with SiO2 for high temperature thermal labels. Adv. Sci. 7, 2000104 (2020).10.1002/advs.202000104
38. Lee J Universal process-inert encoding architecture for polymer microparticles Nat. Mater. 2014 13 524 529 10.1038/nmat3938 24728464
Lee, J. et al. Universal process-inert encoding architecture for polymer microparticles. Nat. Mater. 13, 524–529 (2014).24728464 10.1038/nmat3938
39. De Camillis S Controlling the non-linear emission of upconversion nanoparticles to enhance super-resolution imaging performance Nanoscale 2020 12 20347 20355 10.1039/D0NR04809G 33006350
De Camillis, S. et al. Controlling the non-linear emission of upconversion nanoparticles to enhance super-resolution imaging performance. Nanoscale 12, 20347–20355 (2020).33006350 10.1039/D0NR04809G
40. Ma Z Mechanics-induced triple-mode anticounterfeiting and moving tactile sensing by simultaneously utilizing instantaneous and persistent mechanoluminescence Mater. Horiz. 2019 6 2003 2008 10.1039/C9MH01028A
Ma, Z. et al. Mechanics-induced triple-mode anticounterfeiting and moving tactile sensing by simultaneously utilizing instantaneous and persistent mechanoluminescence. Mater. Horiz. 6, 2003–2008 (2019).10.1039/C9MH01028A
41. Wang X Dynamic pressure mapping of personalized handwriting by a flexible sensor matrix based on the mechanoluminescence process Adv. Mater. 2015 27 2324 2331 10.1002/adma.201405826 25711141
Wang, X. et al. Dynamic pressure mapping of personalized handwriting by a flexible sensor matrix based on the mechanoluminescence process. Adv. Mater. 27, 2324–2331 (2015).25711141 10.1002/adma.201405826
42. Mostaghel N Davis T Representations of Coulomb friction for dynamic analysis Earthq. Eng. Struct. Dyn. 1997 26 541 548 10.1002/(SICI)1096-9845(199705)26:5<541::AID-EQE660>3.0.CO;2-W
Mostaghel, N. & Davis, T. Representations of Coulomb friction for dynamic analysis. Earthq. Eng. Struct. Dyn. 26, 541–548 (1997).10.1002/(SICI)1096-9845(199705)26:5<541::AID-EQE660>3.0.CO;2-W
43. Boutry CM A hierarchically patterned, bioinspired e-skin able to detect the direction of applied pressure for robotics Sci. Robot. 2018 3 eaau6914 10.1126/scirobotics.aau6914 33141713
Boutry, C. M. et al. A hierarchically patterned, bioinspired e-skin able to detect the direction of applied pressure for robotics. Sci. Robot. 3, eaau6914 (2018).33141713 10.1126/scirobotics.aau6914
44. Tao J Self‐powered tactile sensor array systems based on the triboelectric effect Adv. Funct. Mater. 2019 29 1806379 10.1002/adfm.201806379
Tao, J. et al. Self‐powered tactile sensor array systems based on the triboelectric effect. Adv. Funct. Mater. 29, 1806379 (2019).10.1002/adfm.201806379
45. Lee DW Banquy X Israelachvili JN Stick-slip friction and wear of articular joints Proc. Natl. Acad. Sci. 2013 110 E567 E574 23359687
Lee, D. W., Banquy, X. & Israelachvili, J. N. Stick-slip friction and wear of articular joints. Proc. Natl. Acad. Sci. 110, E567–E574 (2013).23359687
46. Maksuta D Dependence of adhesive friction on surface roughness and elastic modulus Soft Matter 2022 18 5843 5849 10.1039/D2SM00163B 35900052
Maksuta, D. et al. Dependence of adhesive friction on surface roughness and elastic modulus. Soft Matter 18, 5843–5849 (2022).35900052 10.1039/D2SM00163B
47. Gao, J. et al. Frictional forces and Amontons' law:  from the molecular to the macroscopic scale. J. Phys. Chem. B 108, 3410–3425 (2004).
48. Amontons G De la resistance cause’e dans les machines (1) J. Jpn. Soc. Tribol. 1999 44 229 235
Amontons, G. De la resistance cause’e dans les machines (1). J. Jpn. Soc. Tribol. 44, 229–235 (1999).
49. Guo P Jia M Guo D Wang ZL Zhai J Retina-inspired in-sensor broadband image preprocessing for accurate recognition via the flexophototronic effect Matter 2023 6 537 553 10.1016/j.matt.2022.11.022
Guo, P., Jia, M., Guo, D., Wang, Z. L. & Zhai, J. Retina-inspired in-sensor broadband image preprocessing for accurate recognition via the flexophototronic effect. Matter 6, 537–553 (2023).10.1016/j.matt.2022.11.022
50. No B Choi N Differences in graphomotor skills by the writing medium and children’s gender Educ. Sci. 2021 11 162 10.3390/educsci11040162
No, B. & Choi, N. Differences in graphomotor skills by the writing medium and children’s gender. Educ. Sci. 11, 162 (2021).10.3390/educsci11040162
51. Israelachvili J Recent advances in the surface forces apparatus (SFA) technique Rep. Prog. Phys. 2010 73 036601 10.1088/0034-4885/73/3/036601
Israelachvili, J. et al. Recent advances in the surface forces apparatus (SFA) technique. Rep. Prog. Phys. 73, 036601 (2010).10.1088/0034-4885/73/3/036601
