==== Front Sci Rep Sci Rep Scientific Reports 2045-2322 Nature Publishing Group UK London 78254 10.1038/s41598-020-78254-w Article A missing color area extraction approach from high-resolution statue images for cultural heritage documentation Nasri Adel adel.nasri@whu.edu.cn Huang XianFeng grid.49470.3e0000 0001 2331 6153State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan, 430079 China 14 12 2020 14 12 2020 2020 10 219395 6 2020 17 11 2020 © The Author(s) 2020Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.Ancient statues are usually fragile and have a tendency to deteriorate over time, developing cracks, corrosion, and losing color. Before any intervention on the object of art, a conservator must map degradation and take measurements. Deterioration mapping is an extremely long process, as the conservator or restorer must locate and digitize the damages manually and collect physical measurements from the artwork. Extracting and measuring the deterioration automatically from images is less expensive and aids the digital documentation process, thus reducing the time cost of manual deterioration mapping. In this paper, we propose an effective approach named Missing Color Area Extraction in order to extract and measure missing color areas from high-resolution imagery statues, using a thresholding technique. The conversion from RGB color space to HSV color space is applied, in addition to morphological operations to remove the dust and small objects. Subject terms EngineeringMaterials scienceissue-copyright-statement© The Author(s) 2020 ==== Body Introduction Cultural Heritage is the most common mean of identity in any country. These heritage objects are often fragile, vulnerable, and sometimes even threatened to disappear for various reasons1. There is no doubt that the mission of transmitting cultural heritage to future generations is not limited to publish scientific studies data. If the heritage object as a material entity has value for the scientist in its historical significance, it remains the last trace and the last witness of our predecessor’s passage. Therefore, this legacy must be treated with respect for our human conscience, since non-renewable destruction irreversibly alienates our cultural heritage. We must make sure that they are well documented because the loss of heritage means the loss of a part of our identity. According to2, cultural heritage conservation is considered a vital process in cultural heritage3. Additionally, the documentation process consists of three stages of understanding the importance, policy development, and management. In the fundamental needs of any conservation project and before planning any intervention in an asset of heritage interest, it would be better to have the most complete documentation possible and preferably in digital format to facilitate the management and sharing of the available information. Such documentation corresponds to the current state of the asset but should ideally continue in later phases to assist in monitoring and maintenance tasks. It is hard to obtain such documentation, but it is necessary to help preserve and disseminate tangible cultural heritage. The pictures captured in many areas, such as earth science4, astronomy5, biology6, industry7, etc. helps to solve many of the issues that are hard to solve with traditional methods8. Images have also helped to develop important fields, such as heritage, which allows the assessment and measure of damaged areas without any physical contact9. Hence the rapid development of digital computing has led to a vast expansion of applications for computer vision systems10–12. Before, these processes need to be done manually. These processes are considered very tedious and costly such as feature identification, measurement, and cartographic elaboration. Nowadays, the automated process of feature extractions uses high-performance computers and cameras. The relationship between cultural heritage and new technologies is very complicated13, and it requires a deep understanding and knowledge from multiple subjects. The use of image segmentation is demonstrating to be of prodigious help in the analysis and archival of heritage documentation14. There are many studies and published articles in recent years that propose algorithms to analyze painting images, where researchers provide some specific measurable appearances for cracks detection and removal in digital painting15, as well as for virtual restoration16. Many ancient artworks, especially statues of Mogao Caves in Dunhuang temple, get damaged during the time and suffer from several degradations such as cracks, lacunas, and missing color. The missing Color area is the common damage that refers to the type of degradation and presents the pattern of original color changing that develops across ancient artwork. The missing color area phenomenon is due to the materials used for the artwork and the atmospheric variations that statues have been exposed to. In addition, other reasons such as physical tensions in the structure17, external mechanical factors18 such as human manipulation or storage conditions19. The operation of degradation mapping, measuring, and monitoring cultural heritage is done manually in this field. Such a process is considered long and tedious due to cost and time. Therefore, the use of automatic methods that exploit high-resolution images represents a potentially interesting alternative. Image Segmentation is a process comprising in separating an image into groups of items. The segmentation focuses on the creation of homogeneous colored regions characteristics20. These characteristics are generally defined in computer vision by discontinuities and similarities between intensity values, spectral radiation, or textural patterns within an image21. The result is a collection of pixel clusters where characteristics differ significantly from those of their neighbors in each cluster. The objective of segmentation is to add structure to the data, which allows a faster enabling and more accurate analysis. The application of image segmentation strategy in the heritage domain can be used in order to extract missing color regions22. The knowledge on the exact position or precise volume of missing color and risk areas is capital for conservators to elaborate the degradation mapping, initiate restoration operations by facilitating the delineation of regions of interest, visualizing, assessment, and measuring damaged areas. In the field of heritage, most research studies focus on extracting, measuring, and removing of cracks from image paintings. However, in the literature, we found that the extraction and measurement of damaged areas in statues are not well explored. The images taken by cameras are often processed using different segmentation strategies23. A methodology measurement using a non-contact device (a CMOS digital) used by23. One creative program, for example, based on segmenting the decay zones from images of stone materials24–26. Also An integrated and automated segmentation approach to deteriorated regions recognition for cultural heritage artifacts27,28. Overall, diverse approaches for image processing in the field of cultural heritage information extraction were proposed. The studied papers show that different techniques are used for many kinds of heritage applications29, and they gave different results. Therefore, the proposed methods are not efficient for all applications. In this paper, we propose an approach of missing color area extraction measurement for ancient statues of Mogao Caves in Dunhuang, and we name Missing Color area extraction (for short, MCAE). In addition to a collection of different statue images downloaded from different sources. The main objective of this paper is to develop an overall digital image processing algorithm for automatic missing color area extraction and measurement from statues imagery, which indicates a damaged area on the statues, in addition, helping conservators and restorers to locate and determine precisely the measurement deterioration and its characteristics, as well as for the elaboration of degradation mapping. Moreover, this paper will take advantage not only as document archive and preservation purposes, but also for maintenance, rehabilitation, and restoration. In this paper, we present two main contributions. This research is the first study that has been used for the extraction and measurement of damaged areas from high-resolution statue images for cultural heritage, where there is a lack of studies in this research area. Secondly, we propose MCAE, a simple and efficient methodology for the extraction and measurement of the missing color areas. Materials and methods The ever -increasing value of images and computer vision techniques needs many demands for analysis, and automatic extracting and measuring of information using image segmentation to facilitate and improve the documentation, protection, and restoration process. This paper focuses on the use of images taken by digital cameras. Data In this paper, the datasets provided by Daspatial company for academic research purposes were used in order to achieve our aim. There are two RGB images of statues of Mogao Caves in Dunhuang in this dataset. Mogao Caves is considered one of the most important World Cultural Heritage properties in China according to its archaeological data. The Mogao Caves are a unique site of outstanding value,and they represent an irreplaceable and irreproducible human resource. These caves are inscribed on the World Heritage list since 198730, where we can observe several frescos, mural paintings, and precious statues dated from the fourth to the fifth century, which are located in 492 caves31. The Mogao Caves provides and documents precious and exceptional data for numerous fields such as history, art, and archaeology studies. In addition, the provided data helps to understand the cultural exchange and connection between West and South Asia during the antiquity. After the abandonment of the Silk Route in the thirteenth century, the caves were uninhabited, and the archeological data such as many statues, have deteriorated. The data provided by Daspatial company are two RGB images with a dimension of 5616 × 7344 taken by the Canon EOS-1Ds Mark III Camera. Besides, various RGB statue images are downloaded from different websites. These images were captured with different cameras under different conditions, with different sizes. It is important to mention that the obtention of high-resolution statue images of damaged areas is not easy. As known, the accessibility to cultural heritage data is so limited due to the privacy and data protection issues. Missing color area extraction Image processing methods for missing color area extraction for heritage protection is the term used in this paper. MCAE is defined as a procedure to find the similar region of interest in the pictures. MCAE is based on the similarities in their features, an 8-bit RGB image is the starting point, and the final result is supposed to be a binary image of missing color. In addition to different segmentation techniques, the extraction of missing color area involves the conversion between two model spaces, from RGB to HSV model space, Color-Based segmentation by using the thresholding technique of each component in HSV model space is investigated, in addition to logical and morphological operations. Figure 1 illustrates the main steps of MCAE approach.Figure 1 Illustrates the flowchart of the proposed approach for missing color area extraction. We discuss the different steps used in the proposed approach MCAE in detail. RGB to HSV model space The color range that a camera can see is described in a color space32. The purpose of defining a color model should be qualified and specified in some colors accepted normal manner. It is a collection of codes for every color. Each pixel in a picture has a color in the color space so that pixel labeling can use this color space. All colors can be defined in different ways, so there are diverse color spaces, as well. The size of a color space depends on the main color's number of tones33. The HSV coordinate system presented in34 is based on a hex-cone model, as it is shown in Fig. 233. Hue, Saturation, and Value (HSV) model is widely used for developing image processing algorithms. The Hue is a color attribute that describes the pure color, Saturation is the measure of the degree to which the color described by Hue is diluted by white light, and value is the measure of intensity.Figure 2 Illustration of the HSV color space. HSV is preferred over the red, green, and blue (RGB) for processing purposes because of its effectiveness in describing the color and for feature extraction, and illumination invariance separates the image intensity from the color information and reduces the effect of light changing. There are three values of R, G, B. These values should be between 0 and 1. Each value range from 0 to 255, we divide each value first by 255. The conversion from RGB model space to HSV model space is done by using the following set of equations: The V band of each RGB pixel is given by the Eq. (1) used in35: 1 V=maxR,G,B The S band is calculated by using the Eq. (2), 2 S=V-minR,G,BV-max(R,G,B,V≠0 IfS=0,thenH=0. IfR=V, then, the H band is obtained by using the following equations: 3 H=60G-BV-MinR,G,B,G≥B,360+60(G-B)V-Min(R,G,B),G