
==== Front
Heliyon
Heliyon
Heliyon
2405-8440
Elsevier

S2405-8440(24)12864-2
10.1016/j.heliyon.2024.e36833
e36833
Research Article
Bibliometric analysis of publications that cited the CIOMS 2016 “International ethical guidelines for health-related research involving humans”
Haunschild Robin R.Haunschild@fkf.mpg.de
a⁎
Kays Joanne kaysjs85@gmail.com
b
Rägo Lembit ragol@cioms.ch
c
Kays Mark mkays@interclarity.com
d
a Max Planck Institute for Solid State Research, Heisenbergstr. 1, 70569, Stuttgart, Germany
b Joanne S. Kays Consulting, Carmel, Indiana, USA
c CIOMS, Geneva, Switzerland
d Interclarity, LLC, Carmel, Indiana, USA
⁎ Corresponding author. R.Haunschild@fkf.mpg.de
30 8 2024
15 9 2024
30 8 2024
10 17 e3683318 4 2024
8 8 2024
22 8 2024
© 2024 The Authors
2024
https://creativecommons.org/licenses/by/4.0/ This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
The CIOMS book “International Ethical Guidelines for Health-related Research Involving Humans”, published in 2016 (IEG2016), provides information to assist research ethics committee members and research practitioners with pragmatically implementing ethical considerations while planning and conducting their research. To identify which aspects of research IEG2016 has had the greatest impact since its publication, we analyzed metadata from 942 papers that cited IEG2016 (English language title only) from Web of Science (WoS, Clarivate). Using VOSviewer, we mapped the co-occurrence of keywords to derive the network of all keywords that co-occurred at least five times in the set of citing papers. We found that the keywords ethics, research ethics, informed consent, and clinical trials had high co-occurrence scores in this set of publications. Strong links were also observed between ethics, research ethics, and informed consent. We identified fifteen human-related (HR) keyword nodes in this keyword network. Analysis of the subset of 273 IEG2016-citing articles containing these fifteen HR keywords showed later-date publications were focused on the youngest humans (children, adolescents, young people, minors) and the humans typically responsible for those youngest humans, namely women and parents. Seventy-nine of the 110 networked countries/regions associated with IEG2016-citing articles were home to HR keyword articles. We conclude that IEG2016 has had significant impact in health and medical science literature and has served as a foundation for health-related research around the world in the areas of ethics, informed consent, and research ethics and the linkage of these topics to under-represented populations in such research.

Keywords

Bibliometrics
CIOMS
Ethics
Informed consent
Research ethics
Clinical trials
Children
Women
Adolescents
==== Body
pmc1 Introduction

The Council for International Organizations of Medical Sciences (CIOMS) is an international non-governmental, non-profit organization in official relations with the World Health Organization (WHO) and is an associate partner of the United Nations Educational, Scientific, and Cultural Organization (UNESCO). Founded in 1949, it currently includes 40 international, national, and associate member organizations. The mission of CIOMS is to advance public health through guidance on health research and policy including ethics, medical product development, and safety [1].

Historically, CIOMS has been dealing with a wide range of ethical issues and has issued recommendations that were pioneering and innovative at the time (e.g., on protection of prisoners against torture, medical genetics, research involving animals and clinical research involving humans) [2]. The aim of CIOMS ethical guidelines for research in humans has always been to provide internationally vetted ethical principles together with detailed commentary on how these principles should be applied, with particular attention to low-resource settings. This pragmatic focus on feasibility in difficult circumstances has been appreciated, and the guidelines have been widely used around the world, including in low- and middle-income countries (LMICs) [[3], [4], [5], [6]].

In 2016, CIOMS published the “International Ethical Guidelines for Health-related Research Involving Humans” (IEG2016) – a document that combines the topics of two earlier CIOMS guidelines publications to cover both biomedical research and epidemiological studies including biobanking and research with health-related data [7]. The IEG2016 was developed in collaboration between CIOMS and the World Health Organization (WHO), and in close cooperation with the World Medical Association (WMA). They are based on other authoritative ethical guidance documents, including the WMA's Declaration of Helsinki [8] and UNESCO's Universal Declaration on Bioethics and Human Rights [9], and also consider other documents from UNESCO, WHO, the Council of Europe, as well as various regional and international initiatives that had emerged or changed at the time when the guidelines were drafted [10].

The IEG2016 consists of 25 numbered guidelines on specific topics. Each of these starts with the core principles, followed by extensive, carefully worded commentaries with general considerations, justifications, and conditions of their application. They complement the Declaration of Helsinki, facilitating its implementation. The annexes to the CIOMS guidelines have practical tools, e.g., a list of items to be included in research protocols, and essential information to be provided to prospective participants for their informed consent.

With publication of the IEG2016, CIOMS’ aim is to provide internationally vetted ethical principles combined with detailed commentaries on how these universal ethical principles should be applied in practice. The intended outcome is high impact: a contemporary, practical set of basic principles covering key ethical considerations that will be widely used by key audiences (including ethics committees, researchers, academics, patient organizations and regulatory authorities) in health research. It has been widely accepted that CIOMS guidelines manage to strike a balance between the protection of human participants in health-related research and the promotion of such research activities in an exemplary way [10,11].

To our knowledge there has not been much research trying to assess the impact of ethical guidelines. We are also not aware of any internationally accepted indicators to measure impact of ethical guidelines. We propose that one indicator of impact of ethical guidelines, among others, is the extent to which the document is cited in academic papers. Using raw citation counts is problematic. Normalization of citation counts is not feasible because there is no database of ethical guidelines with their citation counts. Documents that state ethical guidelines are usually not covered by citation databases. Thus, we propose to contextualize the citation information with topics (i.e., keywords from citing papers), authors, their institutions, and countries/regions. More indirect indicators of impact such as the number of languages into which the document is translated also exist. By today the IEG2016 has been translated into all six official UN languages (Arabic, Chinese, English, French, Russian, and Spanish) plus Japanese, Korean, Polish, Portuguese, and Ukrainian. The current research project focused only on publications that cited the English version.

2 Methodology and dataset

2.1 Dataset

We used data from Web of Science (WoS) [12] provided by Clarivate. The IEG2016 is not indexed in WoS. We did not find the IEG2016 as a source item in any other major citation database, either. Thus, we extracted all metadata (with their cited references) from all papers citing the IEG2016 from the web interface of WoS (https://login.webofknowledge.com) via the cited references search with the English title. The following WoS indices were used: SCI-E, SSCI, AHCI, CPCI-S, CPCI-SSH, BKCI-S, BKCI-SSH, ESCI, CCR-E, and IC. However, the latter two did not contribute to the result set. We searched the cited years 2016–2023 for the full title as cited work and cited title. We also searched for the abbreviated version “Int Ethical Guidelin” as cited work during the same time period. In total, 942 papers were downloaded from Web of Science (WoS) on September 05, 2023. As the IEG2016 contains guidelines for health-related research involving humans, this dataset was further refined to a 273-paper dataset which only included papers with at least one of 15 HR keywords (adolescents, adults, children, community, minors, parents, participants, people, populations, pregnant women, research participants, researchers, vulnerable populations, women, and young people). We retrieved these 15 HR keywords from the dataset of all IEG2016 citing papers. To avoid confusion, we took the keywords verbatim and did not attempt to merge keywords that might appear to overlap (e.g., minors and adolescents). We restricted the search for the English book title because WoS has a bias in indexing towards English literature.

2.2 Methods

2.2.1 Co-occurrence maps

We used VOSviewer [13] to map the co-occurrence of keywords (author keywords and keywords plus), authors, affiliations, and countries/regions of authors of the papers citing the book. WoS splits some countries into regions, e.g., the United Kingdom appears as England, Scotland, Wales, and Northern Ireland. We did not merge such splits to avoid discussions about sensitive issues. Network keyword nodes use some basic unification of obvious synonyms (e.g., clinical-trials replaced by clinical trials; risk replaced by risks; etc.; see Supplemental Table S1). The distance between two nodes is determined by the co-occurrence frequency of the terms. The size of the nodes is dependent on the number of papers with a specific keyword, co-author, affiliation, or author country/region. Nodes can be colored according to various characteristics (e.g., cluster assignments, or average publication year). In the results and discussion below, the “average link strength” of any node was calculated by dividing its total link strength by its number of links. In addition to the static VOSviewer maps, we produced interactive versions via VOSviewer Online [14,15].

3 Results

Fig. 1 shows the annual citation profile of IEG2016. There was a bi-phasic distribution of all 942 publications citing IEG2016 from 2017 to 2023. Steady growth in IEG2016-citing articles was seen between 2017 and 2020, followed by a decrease in 2021 and a second and larger peak in 2022. The decline seen in 2023 may be partly explained by having an incomplete year of data (Fig. 1).Fig. 1 Distribution of IEG2016-citing publications from 2017 through September 5, 2023.

Fig. 1

Nearly 70 % (n = 652) of the papers citing the book are articles; the remainder is distributed across the following document types: review (n = 114), editorial material (n = 83), book chapter (n = 36), early access (n = 36), letter (n = 14), and proceedings paper (n = 7). There were 285 unique journals that published at least one IEG2016-citing article. The list of journals with at least ten IEG2016-citing articles is included in Supplemental Table S2.

Country/region co-publication network analysis determined the global distribution of articles citing IEG2016 (Supplemental Fig. S1). Co-publication volume categories of the 110 countries/regions with IEG2016-citing articles are depicted in Fig. 2.Fig. 2 World map of co-publication volumes in countries/regions with IEG2016-citing publications. Derived from the country/region co-publication network (giant component) of all papers citing the IEG2016 (Supplemental Fig. S1). An interactive version of the country/region co-publication network is available at the following URL: https://s.gwdg.de/bcnZ8q.

Fig. 2

3.1 Keyword analysis

Keyword analysis produced 217 nodes (Fig. 3), with 54 nodes comprising the top quartile (Q1) of network nodes (according to average link strength in Table 1 and co-occurrences in Table 2) including 35 nodes having an average link strength greater than the overall average link strength (1.753). Ethics, informed consent, and research ethics were the three top nodes possessing both the keywords' highest average link strength (Table 1) and the keywords’ highest co-occurrences (Table 2).Fig. 3 Co-occurrence network of all keywords that co-occur at least five times in the set of citing papers. Link to the interactive version: https://s.gwdg.de/u6Irkz.

Fig. 3

Table 1 Q1 keyword nodes ranked by average link strength (HR keywords highlighted).

Table 1Rank	Node	Average link strength	
1	Ethics	4.31	
2	Informed Consent	4.18	
3	Research Ethics	4.12	
4	Clinical Trials	2.82	
5	Challenges	2.42	
6	Risks	2.32	
7	Vulnerability	2.25	
8	Clinical Research	2.22	
9	Trials	2.20	
10	Community Engagement	2.14	
11	Health	2.14	
12	Puberty	2.14	
13	Consent	2.12	
14	Bioethics	2.10	
15	Decision Making	2.08	
16	Broad Consent	2.05	
17	Participation	2.02	
18	Ethics Committees	2.01	
19	Care	2.01	
20	Children	1.99	
21	Pregnancy	1.96	
22	Privacy	1.94	
23	Adolescents	1.94	
24	Research	1.94	
25	Capacity	1.92	
26	Dementia	1.92	
27	Issues	1.89	
28	Developing Countries	1.88	
29	People	1.85	
30	Declaration Of Helsinki	1.83	
31	HIV	1.83	
32	Women	1.82	
33	Assent	1.80	
34	CoViD-19	1.80	
35	Vaccines	1.78	
36	Governance	1.75	
37	Biobanking	1.72	
38	Medical-Research	1.71	
39	Health Research	1.69	
40	Safety	1.69	
41	Data Sharing	1.68	
42	Social Value	1.68	
43	Research Participants	1.66	
44	Interventions	1.65	
45	Design	1.65	
46	Health-Care	1.65	
47	Quality	1.64	
48	Attitudes	1.63	
49	Drugs	1.63	
50	Health Policy	1.63	
51	Qualitative Research	1.62	
52	Parents	1.61	
53	Recruitment	1.60	
54	Participants	1.60	

Table 2 Q1 keyword nodes ranked by co-occurrences (HR keywords highlighted).

Table 2Rank	Keyword	Co-occurrences	
1	Ethics	179	
2	Research Ethics	168	
3	Informed Consent	151	
4	Clinical Trials	99	
5	Health	63	
6	Challenges	60	
7	Consent	57	
8	Risks	54	
9	Trials	52	
10	Clinical Research	50	
11	Bioethics	45	
12	Care	45	
13	Children	42	
14	Ethics Committees	42	
15	Vulnerability	41	
16	CoViD-19	38	
17	Participation	32	
18	Guidelines	31	
19	Research	30	
20	Issues	27	
21	Public Health	27	
22	Health-Care	26	
23	Adolescents	25	
24	Community Engagement	25	
25	Women	25	
26	HIV	24	
27	Pregnancy	24	
28	Privacy	24	
29	Interventions	23	
30	Prevalence	23	
31	Experiences	22	
32	Governance	22	
33	Access	21	
34	Health Research	21	
35	Safety	21	
36	Data Sharing	20	
37	Decision Making	20	
38	Vaccines	20	
39	Developing Countries	19	
40	Biomedical Research	18	
41	Capacity	18	
42	Framework	18	
43	Management	18	
44	People	18	
45	Benefits	17	
46	Disease	17	
47	Outcomes	17	
48	Participants	17	
49	Quality	17	
50	Africa	16	
51	Impact	16	
52	Policy	16	
53	Prevention	16	
54	Qualitative Research	16	

3.2 HR keyword analysis

The 217 keywords formed the basis for a network with 4372 pairs of linked nodes. The keyword pair with the highest link strength was informed consent-research ethics, followed by ethics-informed consent (Table 3). Other nodes in the top ten pairs of nodes ranked by link strength were clinical trials, trials, challenges, vulnerability, and risks.Table 3 Top ten keyword pairs ranked by link strength.

Table 3Rank	Keyword Pair	Link strength	
1	Informed Consent	Research Ethics	50	
2	Ethics	Informed Consent	38	
3	Clinical Trials	Ethics	30	
4	Ethics	Research Ethics	29	
5	Clinical Trials	Informed Consent	25	
6	Informed Consent	Trials	25	
7	Challenges	Ethics	24	
8	Clinical Trials	Research Ethics	24	
9	Research Ethics	Vulnerability	21	
10	Ethics	Risks	19	

Within the 217 keyword nodes, nine of the fifteen HR keyword nodes describe specific types of humans; the other six HR keyword nodes refer to more general categories (Table 4). Four specific HR keyword nodes were in the top quartile of nodes ranked by average link strength: children, adolescents, women, and parents (Table 1). Interestingly, children, adolescents, and women were also in the top quartile when ranking by the number of co-occurrences (Table 2).Table 4 HR keyword nodes grouped into specific and general, each sorted alphabetically.

Table 4Specific	General	
Adolescents	Community	
Adults	Participants	
Children	People	
Minors	Populations	
Parents	Research Participants	
Pregnant Women	Vulnerable Populations	
Researchers		
Women		
Young People		

The most frequent pairings of an HR keyword with a non-HR keyword were children-informed consent, adolescents-informed consent and informed consent-people. The HR keyword women was predominantly paired with four non-HR keywords: ethics, pregnancy, research ethics, and risks (Table 5).Table 5 Highest HR keyword to non-HR keyword pairs ordered by link strength.

Table 5Keyword Pair	Link strength	
Children	Informed Consent	13	
Adolescents	Informed Consent	7	
People	Informed Consent	7	
Parents	Informed Consent	6	
Populations	Vulnerability	6	
Pregnant Women	Research Ethics	6	
Women	Ethics	6	
Women	Pregnancy	6	
Women	Research Ethics	6	
Women	Risks	6	
Participants	Ethics	5	
Research Participants	Clinical Trials	5	
Vulnerable Populations	Research Ethics	5	
Adults	Dementia	4	
Participation	Community	4	
Minors	Consent	4	
Young-People	Autonomy	3	
Researchers	Risks	2	
Researchers	Informed Consent	2	

The keyword network refined to the articles containing HR keywords is shown in Fig. 4. There were 273 HR-keyword-containing articles which resulted in 90 keyword nodes co-occurring at least five times within this refined dataset. Of the 273 HR-keyword-containing articles, 193 included only a single HR keyword; 55 had two HR keywords; 23 had three HR keywords; two had four HR keywords. Within this HR-keyword-containing subset, ethics, informed consent, and research ethics remained dominant nodes (see Table 6, Table 7).Fig. 4 All keywords that occur at least five times in the publication set that contain HR keywords. The nodes are colored according to cluster assignment. Link to the interactive version: https://s.gwdg.de/FzKAXP.

Fig. 4

Table 6 Q1 keyword nodes of articles containing HR keywords ranked by average link strength.

Table 6Rank	Keyword	Average link strength	
1	Informed Consent	3.99	
2	Ethics	3.68	
3	Research Ethics	3.31	
4	Community Engagement	2.46	
5	Children	2.38	
6	Clinical Trials	2.37	
7	Challenges	2.28	
8	Vulnerability	2.26	
9	Adolescents	2.24	
10	Women	2.16	
11	Consent	2.15	
12	People	2.14	
13	Health	2.14	
14	Participation	2.07	
15	Assent	2.06	
16	Capacity	2.03	
17	Decision Making	2.02	
18	Clinical Research	1.98	
19	Bioethics	1.97	
20	Care	1.96	
21	Trials	1.93	
22	Participants	1.93	
23	Issues	1.92	

Table 7 Q1 keyword nodes of articles containing HR keywords ranked by co-occurrences.

Table 7Rank	Keyword	Co-occurrences	
1	Research Ethics	65	
2	Informed Consent	62	
3	Ethics	58	
4	Children	42	
5	Clinical Trials	34	
6	Health	30	
7	Challenges	29	
8	Consent	28	
9	Adolescents	25	
10	Community Engagement	25	
11	Women	25	
12	Vulnerability	23	
13	Trials	21	
14	Clinical Research	20	
15	Participation	20	
16	People	18	
17	Risks	18	
18	Care	17	
19	Participants	17	
20	Issues	16	
21	Bioethics	15	
22	Decision Making	14	
23	Assent/Health-Care/HIV/Pregnant Women	13	

Fig. 5 shows the keyword network from articles containing HR keywords with node coloring by average publication year. Citing papers that bear HR keywords had an average publication year of 2020.3. Seven of the fifteen HR keywords had an average publication year prior to 2020.3 (Fig. 6). Of the seven, two were specific HR keywords. Of the eight HR keywords published after 2020.3, seven were specific. Adolescents, minors, and young people were the specific HR keyword nodes published most recently.Fig. 5 Network of all keywords that occur at least five times in the publication set that contain HR keywords. The nodes are colored according to the average publication year of the corresponding articles. Link to the interactive version: https://s.gwdg.de/NQq55N.

Fig. 5

Fig. 6 Timeline of average publication years of HR keyword nodes. The average publication year of all 273 HR keyword articles is 2020.3, designated by the red bar on the timeline. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

Fig. 6

The distribution of the percentage of HR keyword papers per year (Fig. 7) was biphasic with a peak of 56 % in 2019, a trough of 25 % in 2021 which then increased to 50 % in 2022. Except for 2018 and 2021, there were more citing papers that bear specific HR keywords than general HR keywords (Fig. 7 and Supplementary Table S3). Since the trough of HR keyword articles in 2021, it appears that there was a greater focus on publishing articles with specific HR keywords.Fig. 7 HR keyword publications by year (as percentage of all IEG-citing publications)

Fig. 7

During the analyzed time period, the top three specific HR keywords in terms of numbers of articles that cited IEG2016 were children, women, and adolescents. Children and women were the only two of the nine specific HR keyword nodes which had at least one publication in each of the years examined; this was true for four of the six general HR keywords (community, participant, people, and population). The publication year 2019 had the highest number of HR keyword publications (n = 93), while 2022 was the only year that had at least one publication for each of the 15 HR keywords (Supplemental Table S3).

3.3 Author analysis

Fig. 8 depicts the network of all co-authors that co-occur at least twice in the network of IEG2016-citing HR keyword papers. This network contains 116 authors in 34 clusters with 226 links and a total link strength of 382. Authors with four or more IEG2016-citing documents were spread across eight different author clusters (Table 8). Parker, M (Cluster 1, red) and Shah, SK (Cluster 10, salmon) published the highest number of HR keyword IEG2016-citing articles with six each. There were only five of the 116 authors in the network who participated in the development of IEG2016 (i.e., IEG working group authors): Kurihara, C (Cluster 3, two documents), Rid, A (Cluster 10, three documents), van Delden, JJM (Cluster 6, two documents), van der Graaf, R (Cluster 6, four documents), and Wendler, D (Cluster 34, two documents). Taken together, this sub-group of IEG working group authors accounts for 4.3 % of the HR keyword authors, 4 % of the HR keyword documents, and 8.6 % of the total link strength of the HR keyword author network. These data show that IEG2016 had its main impact outside of the IEG working group. Subsequent testing1 [16] provided evidence that authors who participated in the development of IEG2016 did not have an impact significantly different than other authors in terms of production (number of documents, p = 0.4676) or collaborations (number of links; p = 0.4904 or average link strength; p = 0.4173).Fig. 8 Co-authorship network of co-authors that appeared at least twice based on the HR keyword publication set (disconnected nodes are included). Link to the interactive version: https://s.gwdg.de/nqiJmY. An interactive version with average publication year coloring of the nodes is available via this link: https://s.gwdg.de/r3RbNq.

Fig. 8

Table 8 Authors of at least four IEG2016-citing papers with at least one HR keyword.

Table 8Rank	Author	Cluster	Documents	Avg. pub. year	
1	Parker, M	1	6	2019.00	
2	Shah, SK	10	6	2020.17	
3	Desmond, N	4	5	2019.60	
4	Weijer, C	2	5	2019.80	
5	Cheah, PY	1	5	2020.40	
6	Ries, NM	17	5	2020.20	
7	Van Der Graaf, R	6	4	2019.50	
8	Nyirenda, D	4	4	2020.75	
9	Taljaard, M	2	4	2020.00	
10	Slack, C	2	4	2020.75	
11	Knoppers, BM	14	4	2020.75	
12	De Almeida-Neto, PF	8	4	2022.75	

3.4 Institutional analysis

Fig. 9 shows the institution network based on the HR keyword publication set. This network contains 119 institutions in 11 clusters with 482 links and a total link strength of 549.Fig. 9 Institutional network based on the HR keyword publication set with at least two papers per institution. Only the connected nodes (giant component) are shown. Link to the interactive version: https://s.gwdg.de/sFuQu4. An interactive version with average publication year coloring of the nodes is available via this link: https://s.gwdg.de/UdOhpc. The node closest to "univ oxford" is "univ kwazulu natal".

Fig. 9

Oxford University had the highest number of co-published documents associated with HR keywords. The University of KwaZulu-Natal ranked second, representing the only institution from an LMIC2 in the top ten (Supplemental Table S4). Additional institutions from LMICs in the first quartile ranking by number of documents were the University of Witwatersrand, Mahidol University, Makerere University, the University of Cape Town, and the University of Malawi.

Oxford University had the two highest link strengths with Mahidol University and University Liverpool School of Tropical Medicine, respectively. Of the nine institution-pairings that had link strengths of three or higher, Oxford University had two additional links, with the University of KwaZulu-Natal and the University of Malawi (Supplemental Table S5).

The University of KwaZulu-Natal had the highest number of links (36) and highest link strength (50) of all institutions. Oxford University followed with 29 links and a total link strength of 42. Three additional universities from LMICs (The University of Witwatersrand, the University of Cape Town, and the University of Malawi) were in the top ten institutions whether ranked by number of links or by total link strength (Supplemental Table S6).

3.5 Country/region analysis

The country/region network derived from IEG2016-citing HR keyword papers (Fig. 10) contained 79 countries/regions across 11 clusters with 399 links and a total link strength of 639. The 79 countries/regions were distributed as follows: 35 High-Income Countries (HICs), 38 Middle-Income Countries, and 6 Low-Income Countries (i.e., 44 LMICs). The 399 links (pairings) between countries/regions were distributed as follows: 136 were between two HICs, 165 were between one HIC and one LMIC, and the remaining 98 were between two LMICs. Cluster 8 was the only cluster represented entirely by LMICs. This underlines the high impact of IEG2016 on research from LMICs and collaborations between HICs and LMICs. A complete list of country-cluster members is included in Supplemental Table S7.Fig. 10 Country/region network based on the HR keyword publication set with at least one paper per country/region. Link to the interactive version: https://s.gwdg.de/lxD25D. An interactive version with publication year coloring of the nodes is available via this link: https://s.gwdg.de/2ZojaH.

Fig. 10

The USA was the country/region possessing the highest number of co-published documents. South Africa and Malawi were the two LMICs in the top ten countries with the highest number of documents (Supplemental Table S8). In addition, LMICs in the top quartile of number of documents included Brazil, India, Ghana, Kenya, Uganda, and Thailand.

The USA had the highest total number of links to other countries/regions with 45 (Supplemental Table S9). England and France were next with 44 and 31, respectively. Nine LMICs were in the top quartile of links to other countries/regions: India, South Africa, Ghana, Malawi, The Philippines, Kenya, Pakistan, Thailand, and Uganda.

The highest link strength (12) between two HICs was between Canada and the USA. The highest link strength (14) between a HIC and LMIC was shared by the USA and South Africa. The two LMICs with the highest link strength (4) were South Africa and Uganda. See Supplemental Table S10 for additional country pairs by income category.

Fig. 11 updates the global distribution of IEG2016-citing countries in Fig. 2 by differentiating the presence or absence of HR keywords. Of the 110 networked countries/regions with IEG2016-citing publications, fifteen countries/regions had only HR keyword articles; 31 had only non-HR keyword publications; and the remaining 64 had both HR and non-HR keyword publications.Fig. 11 World map of countries specifying the types of IEG2016-citing publications.

Fig. 11

4 Discussion

The purpose of this bibliometric analysis was to use IEG2016 citations as a basis from which to propose a contextualized impact measurement of published ethical guidelines. While developing the proposed methodology, we applied it to the IEG2016. Although the total number of 942 citing publications already indicated a substantial impact, a more detailed analysis measured the specific areas where IEG2016 had its highest impact. We identified emergent themes from keyword networks and how authors, their institutions, and countries/regions that cited IEG2016 were networked. Namely, these emergent themes were related to the keywords ethics, informed consent, and research ethics as well as the intersection of these areas with humans (i.e., HR keywords) that have largely been under-represented in health-related research.

Ethics, informed consent, and research ethics are key concepts and recurring themes in IEG2016. The co-occurrence network of keywords reflected the importance of these three concepts in IEG2016-citing publications with ethics, informed consent and research ethics being the most frequently occurring overall keywords, whether using the entire set of 942 IEG2016-citing publications or the 273 human-related publication subset.

Examining the details some of the IEG2016-citing publications revealed that the scope of IEG2016's influence regarding ethics, informed consent, and research ethics went beyond the settings typically associated with health-related research involving humans. The concepts of ethics and research ethics as applied to health-related research in IEG2016 provided guidance in other areas as diverse as citizen science [17], social work [18], and online research [19]. IEG2016 has also served as a reference document for authors constructing additional ethics guidelines for specific people or disease states. Huria, Palmer [20] was the most subsequently cited publication in the subset of IEG2016-citing articles that contained HR keywords. This paper developed the CONSIDER (“CONSolIDated critERia”) statement, with the goals of strengthening research praxis and advancing the health outcomes of indigenous peoples. Brown and Sugarman [21] devised a set of guidelines specifically for HIV research, referencing IEG2016. Additionally, online training courses covering research ethics and referencing IEG2016 guidelines have been developed [22,23]. Along these same lines, a current CIOMS Working Group is in the process of harmonizing guidelines for education and training syllabi for healthcare professionals globally, including guidance on structuring online courses [24].

IEG2016 strongly recommends expanding/including under-represented/vulnerable people in not only biomedical research, but also in health-related research. Children, women, and adolescents are among the categories of humans considered under-represented, with IEG2016 devoting specific guidelines for including children and adolescents (Guideline 17) and women (Guidelines 18 and 19) as participants in health-related research. Our analysis provides evidence that IEG2016 has had impact on the inclusion of these under-represented humans in recent health-related research, because children, women and adolescents were the HR keywords with the highest ranking based upon node link strength and number of co-occurrences. Furthermore, there is evidence for the intersection of these particular HR keywords with the highest-ranking overall keywords (ethics, informed consent, and research ethics). Ethics and research ethics shared equal link strengths to the HR keyword women (along with pregnancy and risks), while informed consent was the keyword with the highest link strength to HR keywords children and adolescents. The intersection of these HR and overall keywords suggests that IEG2016-citing researchers may recognize issues linked to lower inclusion numbers for these under-represented groups and the lack of evidence-based information regarding health-related interventions for these particular people. Other under-represented groups and areas where information for interventions are lagging are found in publications from the most productive authors of IEG2016-citing papers that contain HR keywords and include pregnant women [25], indigenous peoples [26], children, and “decisionally-vulnerable” adults [27], people with dementia [28], and adolescent athletes [29] in research areas like medication use, genomics studies, and general health care.

Designing and conducting health-related research, especially when including under-represented groups, can face multiple challenges within a given local setting. IEG2016 Guideline 7 is devoted to community engagement and its related components, such as the inclusion of minorities or marginalized groups, individual informed consent, dissemination of data and results outside the community, and building confidence and trust between the community and the researchers. These facets of IEG2016 were seen in the publications of several IEG2016-citing authors which described the need to consider the extent and means by which community/stakeholder engagement (CE) is the most beneficial to guiding research ethics committees before, during, and after conducting a study [[30], [31], [32]]. The two overall highest-ranking institutions by our analyses, Oxford University and the University of KwaZulu-Natal, showed that community engagement in resource-limited settings was a prominent topic in their IEG2016-citing publications [[33], [34], [35], [36], [37], [38], [39], [40], [41], [42]]. A publication where these two institutions collaborated (along with others), drew attention to improved models and practices for CE, including research on factors that contribute to “good” CE (e.g., adapting to local cultural norms, working with local gatekeepers, treating community members with respect, etc.) [43]. These two institutions also joined in research that provided recommendations on ethically redressing the evidence gap around HIV and co-infections for pregnant people [44] and calls for researchers and institutions to transition from individualistic models of autonomy and agency to models based upon interdependencies of people, institutions, and research structures [45]. Such publications illustrate how IEG2016 served as a framework that researchers could use to initiate CE and adapt to meet the needs of the communities in which they were engaged.

Inclusion alone, however, may not be enough to address the lack of sufficient evidence-based information for under-represented groups in health-related research. It may be necessary to enhance the efficiency of biomedical trials by implementing new, and potentially controversial, clinical trial designs [46,47]. More efficient designs could, in theory, collect sufficient data from under-represented groups in a shorter period and provide an opportunity to determine if differences in treatment efficacy and safety exist for those groups. Developing guidelines for these new trial designs could represent an area for future work.

IEG2016 intends to be a document that can provide guidance and serve as a common basis for collaborations to expand health-related research globally. However, special considerations must be taken into account when conducting health-related research in low-resource regions. As stated above, creating an environment of trust and co-operation via community/stakeholder engagement is a critical factor in ensuring equipoise with a research study. Implementing IEG2016 in LMIC health-related research revealed other practical needs requiring separate and more-detailed attention. Several IEG2016-citing publications [[48], [49], [50]] addressed those specialized needs. Recognizing the need to establish a more collaborative research environment in LMICs, CIOMS created a working group to establish recommendations specifically for stakeholders [51]. This working group report provides 20 recommendations divided between three key stakeholder groups (government and regulatory authorities, researchers, and international organizations and funders) to assist these groups in developing and implementing health-related research involving humans in LMICs. The above activities suggest that IEG2016 is a globally recognized basis in the evolving process of developing and conducting ethical health-related research.

Our study is not without limitations. Publications used for this analysis may not represent the whole body of works that utilize the principles of IEG2016 because they were not published in English or did not directly cite IEG2016. Publications that did cite IEG2016 but are not indexed in WoS are also not included in our analysis. The 15 HR keywords were extracted verbatim from the keyword nodes found in Fig. 3. In filtering the 942 articles for HR keywords to create the network found in Fig. 4, some articles contained phrases that included more than the verbatim HR keyword, e.g., “community” and “community-driven”. Such phrases were implicitly included in our analysis to avoid limiting the scope of the verbatim HR keywords.

5 Conclusions

Our analysis of IEG2016-citing articles presents the impact the IEG2016 has had in a contextualized manner. Our results suggest that the IEG2016 has become a standard reference in the field of medical ethics – providing guidance to researchers in the development of new knowledge for ethical principles in health research that involves human subjects. Impact of IEG2016 has been mainly in the areas related to the keywords ethics, informed consent, and research ethics. Importantly, authors extended the development of ethical principles to human subject groups historically under-represented in health-related studies: children, women, and adolescents. IEG2016 has also made an impact on health research publishing from geographic and economic standpoints. While institutions in high-income countries, such as the US and the UK, have been the most involved in producing publications citing IEG2016, institutions in LMICs (especially South Africa) have, individually or in company with HIC institutions, produced research studies citing IEG2016. Overall, we conclude that over the seven years since its publication, IEG2016 has served as a world-wide foundational work regarding medical ethics for researchers conducting biomedical- and health-related studies involving under-represented human subject groups.

Data availability statement

All data were downloaded from WoS (https://www.webofscience.com). The authors do not have permission to share data.

CRediT authorship contribution statement

Robin Haunschild: Writing – review & editing, Writing – original draft, Visualization, Supervision, Software, Project administration, Methodology, Investigation, Formal analysis, Data curation. Joanne Kays: Writing – review & editing, Writing – original draft, Visualization, Methodology, Investigation, Formal analysis. Lembit Rägo: Writing – review & editing, Writing – original draft. Mark Kays: Writing – review & editing, Writing – original draft, Visualization, Supervision, Software, Methodology, Investigation, Formal analysis, Conceptualization.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:Associate Editor for the section Information Sciences of Heliyon: RH.

Employee of CIOMS: LR.

If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A Supplementary data

The following is the Supplementary data to this article:Multimedia component 1

Multimedia component 1

Acknowledgments

The authors thank Dr. Thomas Scheidsteger and Dr. Rieke van der Graaf for helpful discussions during the early stages of this study.

Appendix A Supplementary data to this article can be found online at https://doi.org/10.1016/j.heliyon.2024.e36833.

1 Wilcoxon-Mann/Whitney test.

2 Low-and-Middle Income Countries, Source: World Bank Country and Lending Groups Data, https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups.
==== Refs
References

1 CIOMS CIOMS website [09 February 2024] 2024 Available from: https://cioms.ch
2 CIOMS Bioethics area of work and publications [27 August 2024] 2024 Available from: https://cioms.ch/bioethics/
3 Barugahare J. Kutyabami P. Nature and history of the CIOMS international ethical guidelines and implications for local implementation: a perspective from East Africa Develop. World Bioeth. 20 4 2020 175 183 10.1111/dewb.12249
4 Ward C.L. Shaw D. Sprumont D. Sankoh O. Tanner M. Elger B. Good collaborative practice: reforming capacity building governance of international health research partnerships Glob. Health 14 1 2018 1 10.1186/s12992-017-0319-4
5 Meza M.B. Luengo-Charath X. Arancibia M. Madrid E. Council for international organizations of medical sciences (CIOMS) ethical guidelines: advancements and unsolved topics in 2016 upgrade Medwave 18 2 2018 7 10.5867/medwave.2018.02.7208 PubMed PMID: WOS:000449159800012
6 Mbo F. Mutombo W. Ngolo D. Kabangu P. Valverde Mordt O. Wourgaft N.S. Mwamba E. How clinical research can contribute to strengthening health systems in low resource countries Tropical Medicine and Infectious Disease 5 2 2020 48 10.3390/tropicalmed5020048 PubMed PMID: 32235362
7 CIOMS International Ethical Guidelines for Health-Related Research Involving Humans 2016 CIOMS Geneva https://cioms.ch/publications/product/international-ethical-guidelines-for-health-related-research-involving-humans/
8 WMA Declaration of Helsinki – ethical principles for medical research involving human subjects 2013 https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/
9 UNESCO Universal declaration on Bioethics and human Rights 2005 [09 February 2024]. Available from: https://en.unesco.org/about-us/legal-affairs/universal-declaration-bioethics-and-human-rights
10 van Delden J.J.M. van der Graaf R. Revised CIOMS international ethical guidelines for health-related research involving humans JAMA, J. Am. Med. Assoc. 317 2 2017 135 136 10.1001/jama.2016.18977 PubMed PMID: WOS:000391826200012
11 Ehni H.-J. Wiesing U. Research ethics for a globalised world: the revised CIOMS international guidelines Indian Journal of Medical Ethics 2 3 2017 165 168 10.20529/IJME.2017.046 28285263
12 Birkle C. Pendlebury D.A. Schnell J. Adams J. Web of Science as a data source for research on scientific and scholarly activity Quantitative Science Studies 1 1 2020 363 376 10.1162/qss_a_00018
13 van Eck N.J. Waltman L. Software survey: VOSviewer, a computer program for bibliometric mapping Scientometrics 84 2 2010 523 538 10.1007/s11192-009-0146-3 PubMed PMID: WOS:000278695500019 20585380
14 van Eck N.J. Waltman L. VOSviewer goes online! (Part 1) 2021 Available from: https://www.leidenmadtrics.nl/articles/vosviewer-goes-online-part-1
15 van Eck N.J. Waltman L. VOSviewer goes online! (Part 2) 2021 Available from: https://www.leidenmadtrics.nl/articles/vosviewer-goes-online-part-2 26 October, 2021
16 McGraw-Hill Education MegaStat® (version 10.4, release 1.4.7 windows) nd [20 March http://www.mhhe.com/megastat 2024
17 Ozolinčiūtė E. Bülow W. Bjelobaba S. Gaižauskaitė I. Krásničan V. Dlabolová D.H. Umbrasaitė J. Guidelines for research ethics and research integrity in citizen science Research Ideas and Outcomes 8 2022 10.3897/rio.8.e97122
18 Anastas J.W. Ethics in Research 2024 NASW Press and Oxford University Press https://oxfordre.com/socialwork/display/10.1093/acrefore/9780199975839.001.0001/acrefore-9780199975839-e-135
19 Newman P.A. Guta A. Black T. Ethical considerations for qualitative research methods during the COVID-19 pandemic and other emergency situations: navigating the virtual field Int. J. Qual. Methods 20 2021 16094069211047823 10.1177/16094069211047823
20 Huria T. Palmer S.C. Pitama S. Beckert L. Lacey C. Ewen S. Smith L.T. Consolidated criteria for strengthening reporting of health research involving indigenous peoples: the CONSIDER statement BMC Med. Res. Methodol. 19 1 2019 10.1186/s12874-019-0815-8 PubMed PMID: WOS:000480765000001
21 Brown B. Sugarman J. HIV prevention trials network (HPTN) ethics working group, HPTN Ethics Guidance for Research 2020 [10 July 2024]. Available from: https://www.hptn.org/sites/default/files/inline-files/HPTN%20EWG_FINAL_EGD%20updates2021_02Dec2021_V3.1_0.pdf
22 ENERI ENERI Classroom n.d 10 July https://classroom.eneri.eu 2024
23 Global Health Training Centre Research ethics online training (V2) 2024 [10 July 2024]. Available from: https://globalhealthtrainingcentre.tghn.org/research-ethics-online-training-v2
24 CIOMS Recommended standards of education and training for health professionals participating in medicines development [10 July 2024]. Available from: https://cioms.ch/working_groups/educational-standards 2024
25 van der Graaf R. van der Zande I.S.E. den Ruijter H.M. Oudijk M.A. van Delden J.J.M. Rengerink K.O. Groenwold R.H.H. Fair inclusion of pregnant women in clinical trials: an integrated scientific and ethical approach Trials 19 2018 9 10.1186/s13063-017-2402-9 PubMed PMID: WOS:000423823800003 29304837
26 Tindana P. Molyneux S. Bull S. Parker M. 'It is an entrustment': broad consent for genomic research and biobanks in sub-Saharan Africa Develop. World Bioeth. 19 1 2019 9 17 10.1111/dewb.12178 PubMed PMID: WOS:000461865700003
27 Dalpé G. Thorogood A. Knoppers B.M. A tale of two capacities: including children and decisionally vulnerable adults in biomedical research Front. Genet. 10 2019 12 10.3389/fgene.2019.00289 PubMed PMID: WOS:000463621200001 30761180
28 Ries N.M. Mansfield E. Sanson-Fisher R. Ethical and legal aspects of research involving older people with cognitive impairment: a survey of dementia researchers in Australia Int. J. Law Psychiatr. 68 2020 10.1016/j.ijlp.2019.101534 PubMed PMID: WOS:000518683000010
29 de Almeida-Neto P.F. Da Silva L.F. Miarka B. De Medeiros J.A. de Medeiros R. Teixeira R.P.A. Influence of advancing biological maturation on aerobic and anaerobic power and on sport performance of junior rowers: a longitudinal study Front. Physiol. 13 2022 10.3389/fphys.2022.892966 PubMed PMID: WOS:000805114600001
30 Nyirenda D. Sariola S. Gooding K. Phiri M. Sambakunsi R. Moyo E. 'We are the eyes and ears of researchers and community': understanding the role of community advisory groups in representing researchers and communities in Malawi Develop. World Bioeth. 18 4 2018 420 428 10.1111/dewb.12163 PubMed PMID: WOS:000453595900013
31 Pratt B. Cheah P.Y. Marsh V. Solidarity and community engagement in global health research Am. J. Bioeth. 20 5 2020 43 56 10.1080/15265161.2020.1745930 PubMed PMID: WOS:000531053500017
32 Slack C. Wilkinson A. Salzwedel J. Ndebele P. Strengthening stakeholder engagement through ethics review in biomedical HIV prevention trials: opportunities and complexities J. Int. AIDS Soc. 21 2018 10.1002/jia2.25172 PubMed PMID: WOS:000447650900002
33 Nyirenda D. Sariola S. Kingori P. Squire B. Bandawe C. Parker M. Desmond N. Structural coercion in the context of community engagement in global health research conducted in a low resource setting in Africa BMC Med. Ethics 21 1 2020 10.1186/s12910-020-00530-1 PubMed PMID: WOS:000574270500001
34 Gooding K. Phiri M. Peterson I. Parker M. Desmond N. Six dimensions of research trial acceptability: how much, what, when, in what circumstances, to whom and why? Soc. Sci. Med. 213 2018 190 198 10.1016/j.socscimed.2018.07.040 PubMed PMID: WOS:000445980800021 30142500
35 Mackworth-Young C.R.S. Schneiders M.L. Wringe A. Simwinga M. Bond V. Navigating 'ethics in practice': an ethnographic case study with young women living with HIV in Zambia Global Publ. Health 14 12 2019 1689 1702 10.1080/17441692.2019.1616799 PubMed PMID: WOS:000468656300001
36 Sansom L.J. Minh T.P.N. Hill I.E. Ha Q.N.T. Trong T.D. Vidaillac C. Towards a fair and transparent research participant compensation and reimbursement framework in Vietnam International Health 12 6 2020 533 540 10.1093/inthealth/ihaa066 PubMed PMID: WOS:000593042800006 33165550
37 Guzmán J.A.C. Orlich J. Mendizábal-Cabrera R. Ying A. Vergès C. Espinoza E. Strengthening research ethics governance and regulatory oversight in Central America and the Dominican Republic in response to the COVID-19 pandemic: a qualitative study Health Res Policy Syst 20 1 2022 10.1186/s12961-022-00933-z PubMed PMID: WOS:000903413700001
38 Akuffo A.G. When the researched refused confidentiality: reflections from fieldwork experience in Ghana J. Acad. Ethics 21 4 2023 567 589 10.1007/s10805-023-09471-x PubMed PMID: WOS:000932683200001
39 Singh J.A. Informed consent and community engagement in open field research: lessons for gene drive science BMC Med. Ethics 20 2019 10.1186/s12910-019-0389-3 PubMed PMID: WOS:000477596800001
40 Millum J. Beecroft B. Hardcastle T.C. Hirshon J.M. Hyder A.A. Newberry J.A. Saenz C. Emergency care research ethics in low-income and middle-income countries BMJ Glob. Health 4 2019 10.1136/bmjgh-2018-001260 PubMed PMID: WOS:000500402400001
41 Nkosi B. Zanoni B. Seeley J. Strode A. The ethical-legal requirements for adolescent self-consent to research in sub-Saharan Africa: a scoping review Bioethics 36 5 2022 576 586 10.1111/bioe.12988 PubMed PMID: WOS:000733551900001 34942033
42 Wilkinson A. Slack C. Thabethe S. Salzwedel J. It's almost as if stakeholder engagement is the annoying 'have-to-do Can Ethics Review Help Address the "3 Ts" of Tokenism, Toxicity, and Tailoring in Stakeholder Engagement? Journal of Empirical Research on Human Research Ethics 17 3 2022 292 303 10.1177/15562646221078415 PubMed PMID: WOS:000760030700001
43 Hickey G. Porter K. Tembo D. Rennard U. Tholanah M. Beresford P. What does "good" community and public engagement look like? Developing relationships with community members in global health research Front. Public Health 9 2022 10.3389/fpubh.2021.776940 PubMed PMID: WOS:000753730300001
44 Lyerly A.D. Beigi R. Bekker L.G. Chi B.H. Cohn S.E. Diallo D.D. Ending the evidence gap for pregnancy, HIV and co-infections: ethics guidance from the PHASES project J. Int. AIDS Soc. 24 12 2021 10.1002/jia2.25846 PubMed PMID: WOS:000736096100003
45 Roest J. Nkosi B. Seeley J. Molyneux S. Kelley M. Respecting relational agency in the context of vulnerability: what can research ethics learn from the social sciences? Bioethics 37 4 2023 379 388 10.1111/bioe.13139 PubMed PMID: WOS:000921230600001 36709500
46 Palacios R. Shah S.K. When could human challenge trials be deployed to combat emerging infectious diseases? Lessons from the case of a Zika virus human challenge trial Trials 20 1 2019 10.1186/s13063-019-3843-0 PubMed PMID: WOS:000506890000002
47 Nichols T. The Death of Expertise: the Campaign against Established Knowledge and Why it Matters 2017 Oxford University Press https://academic.oup.com/book/55947?login=true
48 Shah S.K. London A.J. Mofenson L. Lavery J.V. John-Stewart G. Flynn P. Ethically designing research to inform multidimensional, rapidly evolving policy decisions: lessons learned from the PROMISE HIV Perinatal Prevention Trial Clin. Trials 18 6 2021 681 689 10.1177/17407745211045734 PubMed PMID: WOS:000696047100001 34524048
49 Munung N.S. Nembaware V. Osei-Tutu L. Treadwell M. Chide O.E. Bukini D. Assent, parental consent and reconsent for health research in Africa: thematic analysis of national guidelines and lessons from the SickleInAfrica registry BMC Med. Ethics 23 1 2022 10.1186/s12910-022-00843-3 PubMed PMID: WOS:000895906100001
50 Dietrich J.J. Munoz J. Tshabalala G. Makhale L.M. Hornschuh S. Rentas F. A qualitative study of stakeholder and researcher perspectives of community engagement practices for HIV vaccine clinical trials in South Africa J. Community Psychol. 51 3 2023 998 1015 10.1002/jcop.22951 PubMed PMID: WOS:000879487000001 36342974
51 Rägo L. Zweygarth M. CIOMS research guidelines: considering the needs of developing countries Kurihara C. Greco D. Dhai A. Ethical Innovation for Global Health: Pandemic, Democracy and Ethics in Research 2023 Springer Nature Singapore Singapore 175 194 https://link.springer.com/chapter/10.1007/978-981-99-6163-4_10
