
==== Front
Gastroenterol Hepatol Bed Bench
Gastroenterol Hepatol Bed Bench
GHFBB
Gastroenterology and Hepatology From Bed to Bench
2008-2258
2008-4234
Shaheed Beheshti University of Medical Sciences Tehran, Iran

10.22037/ghfbb.v17i3.3005
Systematic Review
Evaluating the effectiveness of citation count as a measure of methodological quality in esophagogastric surgery research: a comparative analysis with the MINORS score and levels of evidence
Ahmad Suhaib JS. 12*
Ahmed Ahmed R 23*
Mohajer-Bastami Ata 4
Moin Sarah 5
Sweetman Benedict 1
Pouwels Sjaak 67
Head Marion 1
Borucki Joseph 8
Lala Anil 1
Yang Wah 9
houlden Christopher John 1
Garsaa Tarek 1
Exadaktylos Aristomenis 2
1 Department of General Surgery, Betsi Cadwaladr University Health Board, Wales, UK
2 Department of Emergency Medicine, Inselspital University of Bern, Bern, Switzerland
3 Department of Bariatric and Metabolic Surgery, Imperial College London, London, UK
4 General Practice, Ealing Hospital, London, UK
5 Department of Surgery, Hillingdon Hospital, London, UK
6 Department of Surgery, Marien Hospital Herne, University Hospital of Ruhr University Bochum, Herne, NRW, Germany
7 Department of Intensive Care Medicine, Elisabeth-Tweesteden Hospital, Tilburg, The Netherlands
8 Norfolk and Norwich University Hospitals NHS Foundation Trust, Norwich, UK
9 Department of Surgery, The First Affiliated Hospital of Jinan University, Guangzhou, China
* Suhaib JS Ahmad and Ahmed R Ahmed contributed equally as a first author.
Reprint or Correspondence: Suhaib JS. Ahmad, Department of General Surgery, Betsi Cadwaladr University Health Board, Wales, UK
2024
17 3 212224
14 4 2024
8 6 2024
© 2024, Gastroenterology and Hepatology From Bed to Bench (GHFBB)
https://creativecommons.org/licenses/by-nc/4.0/ This is an open-access article, distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (https://creativecommons.org/licenses/by-nc/4.0/) which permits others to copy and redistribute the material just in noncommercial usages, provided the original work is properly cited.
Aim:

The primary objective was to assess the relationship between the citation number and the quality of the articles, as compared with the level of evidence and the MINORS score. This study's secondary objective was to characterize the 50 most cited articles in the field of oesophagectomy research.

Background:

There has been an increased need for an evaluation tool to indicate research quality. Available quality assessment tools include the Level of Evidence, the MINORS score, the Cochrane Risk of Bias 2.0 Tool, the Newcastle Ottawa Scale, CASP Appraisal Checklists, and Legend Evidence Evaluation tools.

Methods:

The Web of Science allowed evaluating and comparing articles on oesophagectomy research. The quality of the 50 most cited articles was assessed using the Oxford Centre level of evidence classification and the methodological index for non-randomized studies (MINORS).

Results:

Level of evidence II studies were cited more than level IV (P=0.008). There was a significant positive correlation between citation number and MINORS score (P=0.002). The median MINORS score was highest amongst level II studies, followed by levels III, IV, and I. The median MINORS score for level II evidence was significantly higher than for level IV (P=0.001). The study sample size is associated with higher levels of evidence but does not correlate with the citation number. Female authors contributed to 4 out of 50 articles. Recently published articles tended to be cited more frequently. More authors equated to more citations. Prospective studies are more likely to be cited.

Conclusion:

Citation analysis can be used as an indicator of quality when assessing articles. It should, however, be used with caution as highly cited work, famous authors, and journals are all more likely to be cited. Citation analysis should be used alongside other well-established tools.

Key Words

Oesophagectomy
Oesophageal Resection
Transhiatal Resection
Ivor Lewis
Mckeown
Oesophagus
Oesophageal Cancer
Oesophageal Carcinoma
Oesophageal Adenocarcinoma
Oesophageal Squamous Cell Carcinoma
==== Body
pmcIntroduction

The scientific literature publication rate is increasing year on year (1). This exponential increase in published scientific literature has greatly increased the challenge of identifying relevant and reliable information (2), with only 10-14% being useful (3). As evidence-based medicine relies on high-quality scientific evidence to provide efficacious patient care (4), there is an urgent need for evaluation tools to establish indicators of research performance.

Several methods exist to assess the quality of articles. The concept of Levels of Evidence, in use since 1979 (5) was expanded upon by Sackett in 1989 (6) to create a hierarchy of five discrete levels of evidence according to the probability of bias. Systematic reviews and randomized controlled trials are given the highest level as they are designed to be unbiased and avoid systematic errors. Case series or expert opinions are judged to be at greater risk of bias as there is no control of confounding factors. Importantly, however, the designated level of evidence does not always guarantee the quality of the research.

The Methodological Index for Non-Randomised Studies (MINORS) score is a well-established and recognized assessment tool (7) often used to validate the methodological quality of non-randomized surgical studies. Non-comparative studies are assessed against 8 items, with an additional 4 for comparative studies. Items are scored 0 if not reported, 1 if reported but inadequate; and 2 if reported and adequate.

Citation analysis is a wide-ranging area of bibliometrics that studies citation patterns, allowing the analysis of scholarship influence. A citation is a recognition given to a published or unpublished source. The citation number reflects the intellectual interest in the work and is important for evaluating grant proposals by funding bodies and a mark of prestige. Journals with highly cited articles tend to attract more submissions (8).

We hypothesized that the citation number also directly reflects the quality of the article and aimed to establish the efficacy of using this relationship as a method of assessing research in comparison to the use of levels of evidence and the MINORS score. The fifty most cited articles in the field of oesophagectomy were assessed for the relationship between the citation number, MINORS score, and level of evidence. Our secondary objective was to characterize the fifty most cited articles in oesophagectomy.

As citations accumulate over time, high-quality recent articles are less likely to be included in a list of the most cited articles in a field. By having only the 50 most cited articles in the field of oesophagectomy, several recent articles with a high citation number were not included in this quantitative synthesis.

Comparison between bibliometric indicators from disparate fields presents limitations, as citation numbers vary across disciplines.

We have referred to the original publication during data extraction to mitigate against reported discrepancies between original published articles and versions published in the Web of Science (2). Reports of episodes of missed citations involving articles published before 1980 were also identified limitations with the Web of Science.

Methods

The fifty most cited articles focused on oesophagectomy for oesophageal cancer were identified using the Web of Science. The following databases were used: Medline, Web of Science Core Collection, BIOSIS previews, and SciELo Citation Index (2). Articles published between 1945 and 18th December 2022 were included (Table 1) (Table 2).

The Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) (9) was used to guide the reporting of this systematic review (Figure 1). Reasons for exclusion were the following: conference proceedings, not related to the topic, oesophageal resection for bleeding, peroral endoscopic myotomy, endoscopic submucosal dissection, Heller myotomy resection for achalasia, endoscopic treatment of oesophageal cancer.

For each eligible study, data about the article, author, and journal was collected (Table 3). The MINORS score was calculated to assess the methodological quality of the included studies.

MINORS score

The quality of the 50 most cited articles was assessed using the methodological index for non-randomized studies (MINORS). It is a well-established and recognized assessment tool used to validate the quality of surgical studies, regardless of whether they are comparative or non-comparative (7).

Table 1 PICOS Criteria

Population	The fifty most cited articles with focus on esophagectomy for oesophageal cancer, identified through the Web of Science.	
Intervention	No intervention.	
Comparison	No comparison.	
Outcomes	The relationship between the citation number and article quality, through comparison with the level of evidence and the MINORS score.	
Study	Systematic review	

Table 2 Data collected about each article

	

Figure 1 The PRISMA chart.

Table 3 The fifty most cited articles in oesophagectomy

First Author	Year	Title	Citations (Density)	MINORS Score	
D Cunningham	2006	Perioperative Chemotherapy versus Surgery Alone for Resectable Gastroesophageal Cancer	4240(265)	23	
P Van Hagen	2012	Preoperative chemoradiotherapy for esophageal or junctional cancer	3312(331)	23	
T N Walsh	1996	A comparison of multimodal therapy and surgery for esophageal adenocarcinoma	1553(60)	18	
A M Mandard	1994	Pathologic assessment of tumor regression after preoperative chemoradiotherapy of esophageal carcinoma	1501(54)	17	
J Shapiro	2015	Neoadjuvant chemoradiotherapy plus surgery versus surgery alone for oesophageal or junctional cancer (CROSS): long-term results of a randomised controlled trial	1336(191)	23	
M Ychou	2011	Perioperative chemotherapy compared with surgery alone for resectable gastroesophageal adenocarcinoma: an FNCLCC and FFCD multicenter phase III trial	1291(117)	22	
K M Sjoquist	2011	Survival after neoadjuvant chemotherapy or chemoradiotherapy for resectable oesophageal carcinoma: an updated meta-analysis	1208(110)	15	
J B F Hulscher	2002	Extended transthoracic resection compared with limited transhiatal resection for adenocarcinoma of the esophagus	1154(58)	23	
J Bancewicz	2002	Surgical resection with or without preoperative chemotherapy in oesophageal cancer: a randomised controlled trial	1114(56)	23	
J F Bosset	1997	Chemoradiotherapy followed by surgery compared with surgery alone in squamous-cell cancer of the esophagus	1073(43)	23	
S S A Y Bieree	2012	Minimally invasive versus open oesophagectomy for patients with oesophageal cancer: a multicentre, open-label, randomised controlled trial	1026(103)	22	
D P Kelsen	1998	Chemotherapy followed by surgery compared with surgery alone for localized esophageal cancer	1016(42)	23	
J Tepper	2008	Phase III trial of trimodality therapy with cisplatin, fluorouracil, radiotherapy, and surgery compared with surgery alone for esophageal cancer: CALGB 9781	1006(72)	22	
S G Urba	2001	Randomized trial of preoperative chemoradiation versus surgery alone in patients with locoregional esophageal carcinoma	994(47)	20	
V Gebski	2007	Survival benefits from neoadjuvant chemoradiotherapy or chemotherapy in oesophageal carcinoma: a meta-analysis	947(63)	15	
M Stahl	2005	Chemoradiation with and without surgery in patients with locally advanced squamous cell carcinoma of the esophagus	940(55)	23	
SE Al-Batran	2019	Perioperative chemotherapy with fluorouracil plus leucovorin, oxaliplatin, and docetaxel versus fluorouracil or capecitabine plus cisplatin and epirubicin for locally advanced, resectable gastric or gastro-oesophageal junction adenocarcinoma (FLOT4): a randomised, phase 2/3 trial	864(288)	23	
L Bedenne	2007	Chemoradiation followed by surgery compared with chemoradiation alone in squamous cancer of the esophagus: FFCD 9102	859(57)	22	
R Earlam	1980	Oesophageal squamous cell carcinoma: I. A critical review of surgery	852(20)	15	
N Ando	2012	A randomized trial comparing postoperative adjuvant chemotherapy with cisplatin and 5-fluorouracil versus preoperative chemotherapy for localized advanced squamous cell carcinoma of the thoracic esophagus (JCOG9907)	843(84)	22	
J M Mueller	1990	Surgical therapy of oesophageal carcinoma	771(24)	15	
B H Burmeister	2005	Surgery alone versus chemoradiotherapy followed by surgery for resectable cancer of the oesophagus: a randomised controlled phase III trial	731(43)	22	
H Akiyama	1994	Radical lymph node dissection for cancer of the thoracic esophagus	728(26)	22	
W H Allum	2009	Long-term results of a randomized trial of surgery with or without preoperative chemotherapy in esophageal cancer	691(53)	22	
J D Luketich	2003	Minimally invasive esophagectomy: outcomes in 222 patients	658(35)	19	
M Stahl	2009	Phase III comparison of preoperative chemotherapy compared with chemoradiotherapy in patients with locally advanced adenocarcinoma of the esophagogastric junction	651(50)	22	
F Lordick	2007	PET to assess early metabolic response and to guide treatment of adenocarcinoma of the oesophagogastric junction: the MUNICON phase II trial	596(40)	22	
F Lordick	2016	Oesophageal cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up	574(96)	17	
D E Low	2015	International Consensus on Standardization of Data Collection for Complications Associated with Esophagectomy: Esophagectomy Complications Consensus Group (ECCG)	566(81)	12	
W A Weber	2001	Prediction of response to preoperative chemotherapy in adenocarcinomas of the esophagogastric junction by metabolic imaging	545(26)	18	
J D Luketich	2012	Outcomes after minimally invasive esophagectomy: review of over 1000 patients	540(10)	16	
J R Siewert	2000	Adenocarcinoma of the esophagogastric junction: results of surgical therapy based on anatomical/topographic classification in 1,002 consecutive patients	526(24)	18	
J M T Omloo	2007	Extended transthoracic resection compared with limited transhiatal resection for adenocarcinoma of the mid/distal esophagus: five-year survival of a randomized clinical trial	518(35)	22	
K J Napier	2014	Esophageal cancer: A Review of epidemiology, pathogenesis, staging workup and treatment modalities	515(64)	11	
N Ando	2003	Surgery plus chemotherapy compared with surgery alone for localized squamous cell carcinoma of the thoracic esophagus: a Japan Clinical Oncology Group Study--JCOG9204	498(26)	22	
J D Urschel	2003	A meta-analysis of randomized controlled trials that compared neoadjuvant chemoradiation and surgery to surgery alone for resectable esophageal cancer	476(25)	11	
K Nygaard	1992	Pre-operative radiotherapy prolongs survival in operable esophageal carcinoma: a randomized, multicenter study of pre-operative radiotherapy and chemotherapy. The second Scandinavian trial in esophageal cancer	471(16)	22	
J Lagergren	2017	Oesophageal cancer	459(92)	11	
E L Prise	1994	A randomized study of chemotherapy, radiation therapy, and surgery versus surgery for localized squamous cell carcinoma of the esophagus	459(16)	22	
N Ando	2000	Improvement in the results of surgical treatment of advanced squamous esophageal carcinoma during 15 consecutive years	453(21)	20	
J D Urschel	1995	Esophagogastrostomy anastomotic leaks complicating esophagectomy: a review	431(16)	11	
M B Orringer	1999	Transhiatal esophagectomy: clinical experience and refinements	425(18)	20	
C G Peyre	2008	The number of lymph nodes removed predicts survival in esophageal cancer: an international study on the impact of extent of surgical resection	421(30)	20	
K Isono	1991	Results of a nationwide study on the three-field lymph node dissection of esophageal cancer	407(148)	20	
J R Siewert	2001	Histologic tumor type is an independent prognostic parameter in esophageal cancer: lessons from more than 1,000 consecutive resections at a single center in the Western world	395(19)	18	
J B Hulscher	2001	Transthoracic versus transhiatal resection for carcinoma of the esophagus: a meta-analysis	395(19)	11	
P Flamen	2000	Utility of positron emission tomography for the staging of patients with potentially operable esophageal carcinoma	390(18)	20	
J A Ajani	2019	Esophageal and Esophagogastric Junction Cancers, Version 2.2019, NCCN Clinical Practice Guidelines in Oncology	389(130)	11	
D P Kelsen	2007	Long-term results of RTOG trial 8911 (USA Intergroup 113): a random assignment trial comparison of chemotherapy followed by surgery compared with surgery alone for esophageal cancer	387(26)	22	
C Mariette	2014	Surgery Alone Versus Chemoradiotherapy Followed by Surgery for Stage I and II Esophageal Cancer: Final Analysis of Randomized Controlled Phase III Trial FFCD 9901	381(48)	22	

Table 4 Relationship between the level of evidence and the MINORS score and citation number

 	Number of Studies	Citation Number	MINORS Score	
Minimum	Maximum	Median	Minimum	Maximum	Median	
Level I Evidence	12	389	1208	540.5	11	17	11.5	
Level II Evidence	25	381	4240	940	18	23	22	
Level III Evidence	3	390	728	596	20	22	22	
Level IV Evidence	10	489.5	1501	489.5	16	20	18.5	

In the MINORS score, each article is given a score of 12.

1. A clearly stated aim

2. Inclusion of consecutive patients

3. Prospective collection of data

4. Endpoints appropriate to the aim of the study

5. Unbiased assessment of the study endpoint

6. Follow-up period appropriate to the aim of the study

7. Loss to follow up less than 5%

8. Prospective calculation of the study size

Additional criteria in the case of comparative studies:

9. An adequate control group

10. Contemporary groups

11. Baseline equivalence of groups

12. Adequate statistical analyses

* The items are scored 0 (not reported), 1 (reported but inadequate) or 2 (reported and adequate).

The article's quality can be classified based on the final score out of 12. The ideal score is 16 for non-comparative studies and 24 for comparative studies.

Figure 2 The distribution of the number of citations.

Statistical methods

Data were analyzed using XLSTAT 2022.5.1 (1390). The Primary outcome was analyzed using the Spearman Correlation and the Kruskal-Wallis test. The Secondary outcomes were analyzed using the Mann-Whitney U test, Kruskal-Wallis Test, and Spearman Correlation.

Results

The citation number ranged between 381 and 4240, with a median of 851.52. The MINORS score of the included studies ranged between 11 and 23, with a median of 21. There was a significant positive correlation between citation number and MINORS score (Spearman correlation matrix 0.435/ P=0.002). Studies with level II evidence had significantly higher citation numbers than those with level IV evidence (Kruskal-Wallis test P=0.008) (Table 4).

Studies with level II evidence displayed the highest median MINORS score, followed in descending order by levels III, IV, and I. Studies with level II evidence had a significantly higher MINORS score than both those with level I evidence (Kruskal-Wallis test P=0.0001) and those with level IV evidence (Kruskal-Wallis test P=0.001) (Table 4).

Citation number

The median number of citations was 654.5 (Min 381/Max 4240/ Std.deviation 687.043). Thirteen articles had a citation number greater than 1000, while thirty-four articles had a greater than 500. Citation number values were not normally distributed between studies (Shapiro-Wilk test: W 0.606/P <0.0001) (Figure 2).

Relationship between citation number and the age of the article

All fifty articles were published between 1980 and 2019. The greatest number of articles were published in the 2000s (24 articles), followed by the 2010s (14 articles), 1990s (11 articles), and 1980’s (1 article). This order was reflected in the total number of citations. The 2000s accumulated 19578 citations (46% of the total), the 2010s with 13305 (31%), the 1990s with 8835 (21%), and the 1980s with 852 (2%). There is a weak negative correlation between the age of the article and the number of citations (Spearman correlation matrix -0.051/ P=0.723).

Relationship between citation number and citation density

The citation density was not normally distributed (Shapiro-Wilk test: W 0.698/P <0.0001). The median citation density was 48.9 (Min 15.7/Max 331.3/ Std.deviation 68.965). The citation count and density tend to form a cluster with 3 outliers (Figure 3). There was a significant positive correlation between the number of citations and citation density (Spearman correlation matrix 0.544/ P<0.0001). There was a significant negative correlation between the age of the article and the citation density (Spearman correlation matrix -0.464/ P<0.0001).

Relationship between citation number and gender

Forty-six articles were published by male researchers, with the remaining 4 being published by female researchers. The median citation number for female authors was 1101 (459-1501), compared to male authors with 623.5 (381-4240). There was no significant difference in the citation number (Mann-Whitey test P=0.178) or the citation density (Mann-Whitey test P=0.904) between the two groups. There was no significant difference in the level of evidence of articles published by female and male authors (Mann-Whitey test P=0.790). The age of the articles published by females (Median 24.5/Min 11/Max 28) was not significantly different (Mann-Whitey test P=0.265) compared to the age of the articles published by males (Median 16.5/Min 3/Max 42). The journal quartile in which articles were published was not significantly different between males (Q1-Q3) and females (Q1) (Mann-Whitey test P=0.821) (Table 5).

Relationship between citation number and journal quartile

There was no significant association between the quartile of a journal and the number of citations of articles published in that journal (Table 6).

Relationship between citation number and number of co-authors

The median number of authors was 11.5 (1-58). There was a positive, non-significant correlation between the citation number each article accrues and the number of authors (Spearman correlation matrix 0.258/ P=0.067). There was a negative correlation between the age of the article and the number of authors (Spearman correlation matrix -0.471/ P=0.001).

Relationship between citation number and institute

32 out of the 37 institutions were university hospitals. They contributed to 43 of the 50 most cited articles and accrued 35794 (84%) of the 42570 total citations (Table 7).

Relationship between citation number and country

Europe produced the highest total number of citations (30469), followed by North America (6286), Asia (2929) and Oceania (2886). 31 articles were published by European authors, 11 from North America, 5 from Asia, and 3 from Oceania (Table 8).

Relationship between citation number and field

Articles published in the field of updates in surgery had the highest median citation number, 811.5 (771-852). This was followed by the therapeutic field with 709.5 (381-4240), guidelines with 570 (566-574) updates on oesophageal cancer with 459 (389-515), and diagnostics with 390 (Table 9).

Relationship between citation number and classification (original or review)

Table 5 Publishing journals of the fifty most cited articles in oesophagectomy

Journal	Quartile	Country	Continent	No. of Articles	Average Citation Number	Average Citation Density	Average Age (years)	Average authors per Article	
Journal of Clinical Oncology	Q1	US	NA	12	671	48.9	15	13	
Annals of Surgery	Q1	US	NA	11	518	26	21	6	
The New England Journal of Medicine	Q1	US	NA	6	1353.5	58.7	22	12	
The Lancet Oncology	Q1	UK	EU	5	947	43	15	15	
Lancet	Q1	UK	EU	4	945	97.2	7.5	18	
Cancer	Q1	US	NA	2	980	25.7	28	18	
The British Journal of Surgery	Q1	UK	EU	2	811.5	22.2	37	4	
American Journal of Surgery	Q2	US	NA	1	476	25.1	19	2	
Annals of Oncology	Q1	UK	EU	1	574	95.7	6	6	
Annals of Surgical Oncology	Q2	US	NA	1	843	84.3	10	15	
JNCCN	Q1	US	NA	1	389	129.7	3	38	
Oncology	Q3	CH	EU	1	407	148.1	31	3	
The Annals of Thoracic Surgery	Q2	NL	EU	1	392	18.7	21	4	
World Journal of Gastrointestinal Oncology	Q3	CN	AS	1	515	64.4	8	3	
World Journal of Surgery	Q2	US	NA	1	471	15.7	30	12	

Table 6 Relationship between the journal quartile and the citation number

Kruskal-Wallis Test P Values	Q1 Citations	Q2 Citations	Q3 Citations	
Q1 Citations	1	0.16	0.156	
Q2 Citations	0.16	1	0.736	
Q3 Citations	0.156	0.736	1	

Figure 3 Relationship between the citation number and the citation density.

There were 38 original articles and 12 review articles. The two groups did not have a significant difference in the citation number (Mann-Whitney test P=0.203) or citation density (Mann-Whitney test P=0.158).

Relationship between citation number, prospective, and retrospective

Table 7 Relationship between the article’s institute of origin and citation number

Institution	Sum of Citations	Citation Density (Citations/Year)	Number of Articles	
Royal Marsden Hospital	4931	159	2	
Erasmus University Medical Center	4649	261	2	
University of Sidney	2155	86	2	
University of Amsterdam	2064	37	3	
Technical University of Munich	2062	27	4	
Kliniken Essen-Mitte	1591	53	2	
St James Hospital	1553	60	1	
Centre Francois Baclesse	1501	54	1	
University of Michigan Medical Center	1419	33	2	
University of Montpellier	1291	117	1	
University of Pittsburgh Medical Center	1198	44	2	
RCS Clinical Trials Unit	1114	56	1	
University Hospital J. Minjoz	1073	43	1	
VU University Medical Centre	1026	103	1	
Memorial Sloan Kettering Cancer Centre	1403	55	2	
University of North Carolina School of Medicine	1006	72	1	
Keio University School of Medicine	951	23	2	
Krankenhaus Nordwest	864	288	1	
University Hospital Le Bocage	859	57	1	
University of London	852	20	1	
Tokyo Dental College Ichikawa General Hospital	843	84	1	
University of Cologne	771	24	1	
University of Queensland	731	43	1	
Toranomon Hospital	728	26	1	
University Hospital Leipzig	574	96	1	
Virginia Mason Medical Center	566	81	1	
Texas Tech University	515	64	1	
McMaster University	476	25	1	
Aker Hospital	471	16	1	
Kings College London	459	92	1	
Regional Cancer Institute	459	16	1	
Roswell Park Cancer Institute	431	16	1	
University of Southern California	417	30	1	
Chiba University	407	148	1	
University Hospital Gasthuisberg	390	18	1	
University of Texas	389	130	1	
University Hospital Claude Huriez–Regional	381	48	1	

Table 8 Relationship between article’s country of origin and citation

Author Country	Sum of Citations	Number of articles	
UK	8462	7	
Netherlands	7739	6	
USA	6238	10	
Germany	5862	9	
France	5564	6	
Japan	2929	5	
Australia	2886	3	
Ireland	1553	1	
Canada	476	1	
Norway	471	1	
Belgium	390	1	

There were 33 prospective articles and 17 retrospective articles. The median citation number for prospective articles (Median 843/Min 381/Max 4240) was significantly higher than retrospective articles (Median 515/Min 389/Max 1208) (Mann-Whitney test P=0.020). There was no significant difference in citation density between the two groups (Mann-Whitney test P=0.903).

Relationship between citation number and sample size

Table 9 Relationship between article field and citation number

Field	Sum of Citations	Number of Articles	
Therapeutic	38054	42	
Updates Surgery	1623	2	
Updates on esophageal cancer	1363	3	
Guidelines	1140	2	
Diagnostics	390	1	

Studies of level of evidence I had a significantly larger sample size compared to both level of evidence II (p<0.0001) and level of evidence III (0.003). The study sample size had no significant effect on the citation number (Spearman test P=0.567) or citation density (Spearman test P=1.000).

Discussion

This review demonstrated a positive association between citation number and MINORS score, supporting the hypothesis that citation number reflects the quality of the article. The MINORS score is a well-established and recognized assessment tool that validates surgical studies' methodological quality. Despite its intended use with non-randomized studies, it was found to have a similar ability to identify high-quality randomized studies compared to the established standard for randomized controlled trials, the CONSORT statement (7). This review demonstrated that randomized controlled trials had the highest MINORS score, followed by cohort studies, case series, and case-control studies. The MINORS score for randomized control trials was significantly higher than for systematic reviews. This may be because the MINORS score was created to assess the methodological quality of articles included for systematic reviews or meta-analyses to reduce potential bias and not to assess systematic reviews themselves (7).

In our analysis, we observed a discernible trend in citation frequency relative to the evidence level of articles. Notably, articles categorized under Level II evidence boasted the highest median citation count, surpassing those of Level III, Level I, and Level IV in descending order. A significant difference was particularly evident between Level II and Level IV citations, underscoring the potential of citation frequency to reflect the inherent quality of scholarly articles, which is consistent with previous reports (2, 10). Among the most cited works, twenty-five were attributed to Level II evidence, underscoring the field of minimally invasive oesophagectomy's significant strides, predominantly showcased through randomized control trials. Moreover, the compilation of the fifty most cited articles includes three from Level III and ten from Level IV evidence, suggesting that citation frequency could indicate quality in emerging, albeit not ground-breaking, research.

The apparent anomaly, wherein Level I evidence articles were not the most cited, warrants further contemplation. This phenomenon could be attributed to several factors. For instance, the practical relevance and specific clinical utility of Level II evidence might resonate more with the research community, thereby garnering higher citation rates. Furthermore, ground-breaking discoveries, particularly those presented in Level II studies, capture the academic and public interest to a greater extent, leading to an increased citation footprint. It is also crucial to acknowledge the role of external factors such as the journal's prominence or the authors' renown, which may significantly influence citation practices independent of the article's evidence level (11). This analysis highlights the importance of adopting a multifaceted approach to evaluating scholarly articles, combining citation frequency with other standardized assessment metrics to ensure a comprehensive understanding of an article's quality and impact.

The average number of citations was 654.5, considerably lower than in previous studies (2). The limited number of surgeons who can perform oesophagectomies limits the number of publications and citations within the field. The citation rate for most articles peaks within 3 years of publishing, followed by a decline over time, suggesting citation number to be time-specific. This decline may be due to more advanced research becoming available. Despite this, we found no significant association between citation number and article age, with articles published in the 1980s accumulating the lowest number of citations. Articles published in the 2000s accumulated the highest number of citations, likely reflecting the groundbreaking research into minimally invasive and robotic surgery during this period (12). This review demonstrated a significant positive association between citation number and citation density. Highly cited articles may be perceived as more credible, leading to increased citations.

Level I articles were demonstrated to have significantly larger sample sizes than level II, which had significantly larger sample sizes than level III. Despite these differences, study population size was shown to have no significant correlation with citation number. Statistical tests are designed to handle samples rather than populations. While larger sample sizes increase statistical power, this can result in a tendency to reject the null hypothesis with clinically minimal differences. Exceeding the calculated sample size may also present difficulties, such as a greater financial burden and unethical exposure of patients to unnecessary interventions. The conclusion that sample size is not an accurate indicator of article quality reflects the findings of existing research (13, 14).

Only 8% (4) of the top-cited articles were published by female authors. We found no significant difference in citation number, citation density, or level of evidence between female and male authors. These findings reflect those of articles suggesting that women in surgery exhibit comparable or superior work productivity to their male counterparts (15-20). The pool of surgeons proficient in performing oesophagectomies is limited, and the consultant surgeon demographic, as indicated by the Royal College of Surgeons, reflects a substantial gender imbalance with a male-to-female ratio of 8 to 1. Surprisingly, only 5.6% of women express deterrence due to the scarcity of female representation in surgery, with 75% of women in surgery drawing inspiration from male role models when contemplating their career paths. Studies have demonstrated that almost half of females working in surgery experience some form of discrimination (21). Addressing the pervasive gender disparity and fostering an inclusive environment is imperative for the advancement and diversity of the surgical profession.

This review demonstrated a weak positive, non-significant relationship between the number of citations and the number of authors. Common intuition would suggest research involving more authors would exhibit higher quality and be more likely to be published in esteemed journals. Previous research has also shown a positive association between the collaboration size and the prestige of the journal in which the article was published (22). Our study also demonstrated that recently published articles have more authors (P=0.001), reflecting that collaborative work has recently dominated the research field.

University hospitals contributed to 86% (43) of the most cited articles in the field, accounting for 84% (35794) of citations. With their tripartite aim of clinical care, education, and research, university hospitals have recently dominated the field of research. The financial viability of these institutions enables them to efficiently conduct research and translate the findings into clinical practice (23).

The 50 most cited articles spanned 15 journals, with 8 from the United States and 4 from the United Kingdom. The United States contributed the highest number (10) of articles, followed by Germany (9), the UK (7), France (6), the Netherlands (6), Japan (5), Australia (3), Ireland (1), Canada (1), Norway (1), and Belgium (1). The cumulative citations mirrored the respective publication counts by country. These nations have been prominent in various academic fields, partially attributing their success to substantial government investments in research to drive economic growth (24). Despite a nearly 50% reduction in funding for the USA since the 1960s (25), a weak positive correlation exists between a country's GDP per capita and research output (19). All top fifty articles were authored in English, reflecting the dominance of the language in international journals.

This review demonstrated no significant association between journal quartile and citation number. This suggests research quality is a greater driver for citation number than journal reputation. The instances of articles with high citation numbers published in journals of a lower quartile also support the theory that the journal impact factor is determined by only 30% of published articles, with the remaining 70% receiving relatively few citations (26).

Conclusion and key findings

• A higher citation number is associated with a higher MINORS score.

• Factors affecting citation number include author and journal reputation. Highly cited work is more likely to be cited.

• The citation rate shows an initial peak and then slows.

• Prospective studies have a higher citation number than retrospective studies.

• There is no citation number, density, or count difference between original and review studies.

• Sample size is associated with a higher level of evidence but does not affect citation number.

• While only 8% of articles were published by females, author gender does not affect citation number or citation density.

• University hospitals accounted for 32 of the 37 contributing institutions and 35794 of the 42570 citations.

• English is widely considered in scientific research. 53% (8) of journals originated in the United States, and 27% (8) originated in the United Kingdom,

• The Journal quartile does not affect citation number.

• Most articles originated from the United States (10).

The surgery department contributed to 30 out of the 50 papers (22051/42570 citations). This is because the disease is managed by a multidisciplinary team rather than surgeons alone.

This review suggests citation analysis is a useful indicator of scientific article quality. Citation analysis should be used with other validated tools as the citation number may be affected by famous authors, high-impact journals, and previously highly cited work.

Citation recommendations can be classified as expert recommendations and non-expert recommendations. Experts can cite to improve awareness of publications in their field. This can help the expert develop recommendations that were not initially within their area of focus. Non-experts can cite outside of their area of expertise. Furthermore, citing helps Masters and PhD students learn about the relevant field (25).

Authors should not ask others to cite their work, and editors should avoid influencing authors to cite studies to boost the journal's impact factor (2). To this end, the Web of Science presents citation numbers with self-citations excluded.

Conflict of interests

The authors declare no conflict of interest.
==== Refs
References

1 Larsen PO von Ins M The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index Scientometrics 2010 84 575 603 20700371
2 Ahmad SJ Ahmed AR Kowalewski KF Nickel F Rostami K Stocker CJ Citation classics in general medical journals: assessing the quality of evidence; a systematic review Gastroenterol Hepatol Bed Bench 2020 13 101 14 32308931
3 Weinstein JN Threats to scientific advancement in clinical practice Spine (Phila Pa 1976) 2007 32 58 62
4 Guyatt G Evidence-based medicine ACP J Club 1991 114 16
5 The periodic health examination Canadian Task Force on the Periodic Health Examination Can Med Assoc J 1979 121 1193 254 115569
6 Sackett DL Rules of evidence and clinical recommendations on the use of antithrombotic agents Chest 1989 95 2 4 2909336
7 Slim K Nini E Forestier D Kwiatkowski F Panis Y Chipponi J Methodological index for non-randomized studies (minors): development and validation of a new instrument ANZ J Surg 2003 73 712 6 12956787
8 Gasparyan AY Thoughts on impact factors and editing of medical journals Inflamm Allergy Drug Targets 2010 9 2 5 20230367
9 Moher D Liberati A Tetzlaff J Altman DG PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement PLoS Med 2009 6 1000097
10 Ahmad SS Kohl S Ahmad S Ahmed AR Ahmad SS Reply to the Letter to the Editor: Citation Analysis in Bariatric Surgery Obes Surg 2015 25 2419 20 26277112
11 Ahmad SS Albers CE Büchler L Kohl S Ahmad SS Klenke F The hundred most cited publications in orthopaedic hip research - a bibliometric analysis Hip Int 2016 26 199 208 26951547
12 Ahmad SS Ahmad SS Kohl S Ahmad S Ahmed AR The hundred most cited articles in bariatric surgery Obes Surg 2015 25 900 9 25720513
13 Faber J Fonseca LM How sample size influences research outcomes Dental Press J Orthod 2014 19 27 9
14 Schoonbaert D Roelants G Citation analysis for measuring the value of scientific publications: quality assessment tool or comedy of errors? Trop Med Int Health 1996 1 739 52 8980585
15 Polk HC The declining interest in surgical careers, the primary care mirage, and concerns about contemporary undergraduate surgical education Am J Surg 1999 178 177 9 10527432
16 Evans S Sarani B The modern medical school graduate and general surgical training: are they compatible? Arch Surg 2002 137 274 7 11888448
17 Henningsen JA Why the numbers are dropping in general surgery: the answer no one wants to hear-lifestyle! Arch Surg 2002 137 255 6 11888443
18 Azizzadeh A McCollum CH Miller CC Holliday KM Shilstone HC Lucci A Factors influencing career choice among medical students interested in surgery Curr Surg 2003 60 210 3 14972298
19 Brundage SI Lucci A Miller CC Azizzadeh A Spain DA Kozar RA Potential targets to encourage a surgical career J Am Coll Surg 2005 200 946 53 15922210
20 Wendel TM Godellas C V Prinz RA Are there gender differences in choosing a surgical career? Surgery 2003 134 591 6; discussion 596-8 14605619
21 Gargiulo DA Hyman NH Hebert JC Women in surgery: do we really understand the deterrents? Arch Surg 2006 141 405 7; discussion 407-8 16618901
22 Abramo G D’Angelo CA The relationship between the number of authors of a publication, its citations and the impact factor of the publishing journal: Evidence from Italy J Informetr 2015 9 746 61
23 Raus K Mortier E Eeckloo K Past, present and future of university hospitals Acta Clin Belg 2020 75 177 84 30896377
24 Goolsbee A Does Government R&D Policy Mainly Benefit Scientists and Engineers? American Economic Review 1998 88 298 302
25 Hather GJ Haynes W Higdon R Kolker N Stewart EA Arzberger P The United States of America and scientific research PLoS One 2010 5 12203
26 Seglen PO Why the impact factor of journals should not be used for evaluating research BMJ 1997 314 498 502 9056804
27 Färber M Jatowt A Citation recommendation: approaches and datasets Int J Digit Libr 2020 375 405
