
==== Front
Interv Pain Med
Interv Pain Med
Interventional Pain Medicine
2772-5944
Elsevier

S2772-5944(22)00124-8
10.1016/j.inpm.2022.100128
100128
Educational Supplement
Systematic reviews: Not always a pain
Fogarty Alexandra E. alexandra.fogarty@gmail.com
a∗
Conger Aaron b
Burnham Taylor Taylor.Burnham@hsc.utah.edu
b
a Department of Orthopedic Surgery, Division of Physical Medicine & Rehabilitation, Washington University School of Medicine, St. Louis, MO, USA
b Department of Physical Medicine and Rehabilitation, University of Utah School of Medicine, Salt Lake City, UT, USA
∗ Corresponding author. alexandra.fogarty@gmail.com
15 8 2022
2022
15 8 2022
1 Suppl 2 10012816 5 2022
27 6 2022
© 2022 The Authors
2022
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Systematic reviews analyze the evidence surrounding a specific intervention within a population. High quality systematic reviews can help clinicians and policymakers accurately understand a treatment intervention. This article outlines the basic principles of systematic review development, including assembling a research team, defining the research question, publishing a protocol, designing and executing the search, study selection, extracting the data, assessing risk of bias, synthesizing the data and conducting a certainty assessment. In addition, we will address common pitfalls and highlight special considerations for the field of interventional pain medicine. Understanding systematic review methodology will help investigators improve their primary research and in turn, better primary literature will improve the value of high quality reviews.

Keywords

Systematic review
Methodology
Interventional pain
==== Body
pmc1 Introduction

Systematic reviews of interventions should transparently describe and critically analyze the existing evidence surrounding a specific intervention within a particular population. High quality systematic reviews can help clinicians and policymakers accurately understand the risks, benefits, and harms of particular treatment interventions for a specific patient population. Due to their large scale, they can include a greater range of patients than any single study, thus strengthening or weakening the generalizability of the findings [1]. Systematic reviews of randomized trials have traditionally occupied a high position in the hierarchy of research evidence. Publication volume of systematic reviews has increased by 2728% between 1991 and 2017 [2], which is reflected in the total volume of Pubmed indexed systematic reviews totaling 2500 annually.

The field of interventional pain medicine is rapidly evolving, with the volume of publications increasing dramatically during this same time period [3]. Despite the strength and popularity of systematic reviews, many have been called into question due to redundancy and methodology concerns, such as the failure to develop a protocol, use of inadequate search strategy, and inadequate risk of bias assessements [2,4,5]. As the body of interventional pain medicine research continues to expand, there is a very strong need to characterize the evidence to facilitate sound clinical decision-making [6]. This type of summary data is essential for physicians, policymakers, and insurance companies alike [7]. As the literature in interventional pain medicine research continues to expand, we must strive to produce high quality systematic reviews that answer clinically important questions.

This article outlines the basic principles of systematic review development, addresses common pitfalls, and highlights special considerations for the field of interventional pain medicine. Understanding systematic review methodology will help investigators improve their primary research and in turn, better primary literature will improve the value of high quality reviews. Clinicians versed in what constitutes a high quality review will be better equipped to spot a “lemon” when they see it and less likely to accept erroneous conclusions based on flawed reviews.

2 Building a systematic review

There are varied approaches to designing a systematic review. Some systematic reviews will identify one problem, whereas others will have a more complex approach and address multiple questions, interventions, and outcomes simultaneously [7]. Regardless of the approach, there are seven basic steps required to build a successful review, which are summarized below (Fig. 1) [7,8]. For further reading, we recommend consulting the PRISMA 2020 statement [4] and Cochrane handbook [9].Fig. 1 Fundamental steps for designing and performing a systematic review.

Fig. 1

2.1 Assembling research team and establishing a timeline

It is essential to assemble a professional research team. On average, we recommend that the minimum number of team members is three –of note, it is not methodologically possible to complete a systematic review with a single author –and expect the process to take a minimum of 6–18 months [10].

2.2 Defining the research question

The “Population, Intervention, Comparators and Outcome” (PICO) framework was first introduced in 1995 and has become a well-used clinical tool in systematic reviews of interventions [11,12]. This framework helps authors identify the research question clearly and concisely in advance of their literature search. The PICO acronym serves to remind authors to define the following characteristics:P = Patients and/or populations of interest (relevant patients or groups)

I = Intervention and/or Exposure (diagnostic tests, drugs, procedures)

C = Comparator (none, placebo, other intervention)

O = Outcome (patient-relevant outcomes)

For example, if authors were interested in the development of a systematic review on the efficacy of cervical medial branch radiofrequency ablation on patients with block-confirmed facetogenic pain, the following PICO could be constructed:P = Patients with cervical facet pain

I = Cervical medial branch radiofrequency ablation

C ​= ​Medical management, physical therapy, sham procedure, or none

O = Proportion of patients who experience >50% pain relief at 6 months

Some methodologists support the idea of developing a PICO using a two-step approach. On the first pass, it is recommended to broadly define the four criteria (as above), whereas on the second pass the criteria are expanded with more depth (as below) [11].P ​= ​Adult patients 18 years of age or older with cervical facet pain with documented greater than 80% relief on diagnostic dual medial branch blocks

I = Cervical medial branch radiofrequency ablation using a parallel approach

C ​= ​Medical management, physical therapy, sham procedure, or none

O = Primary: Proportion of patients who experience >50% pain relief at 6 months, pain reduction by numeric rating scale and/or visual analog scale (NRS, VAS), Secondary: functional status per Oswestry Disability Index (ODI), global impression of change (PGIC), analgesic usage.

2.3 Creating and publishing a protocol

Executing a systematic review will require authors to systematically evaluate the contents and quality of the studies yielded from the literature search. To minimize bias, methodological decisions should be made in advance. Since advanced understanding of the literature is often inevitable for clinicians, Cochrane argues that it is important to have a methodologist on the team who is not a content expert [13]. To promote transparency, publishing a protocol is recommended. The goal of the protocol is typically to define the eligibility criteria and to standardize the method for evidence appraisal and synthesis. Publishing a protocol also provides an additional opportunity for peer-review, which can help authors design a more robust product. Protocols for reviews can be published in the Cochrane Library and journals such as BMJ open and Systematic Reviews. Protocols can also be registered on PROSPERO [7].

2.4 Designing and executing the search

Authors should construct a highly sensitive search strategy. Due to the large volume of medical literature and evolving technical expertise required to extract relevant information from databases, working closely with an experienced Healthcare Information Specialist or librarian is recommended. Construction of a sensitive search strategy requires the use of a variety of different search terms, access to multiple databases and search in multiple languages. Generally, both free-text and medical subject headings should be used to construct the search (MeSH terms). It is recommended to search at least three different database (CENTRAL, MEDLINE and Embase) [9]. Utilizing a PICO framework to build the clinical question can assist with optimizing the search terms [14].

2.5 Study selection

Authors will include and exclude studies based on their predefined PICO criteria. However, it is also important to consider the nature of the articles themselves. For instance, the literature search may reveal a variety of manuscripts, including journal articles, conference abstract, and letters to editors, among others. Authors should attempt to anticipate the search output and determine what types of studies to include in the systematic review. Another important consideration is that multiple reports of the same study population should be identified and linked together. For example, this may apply to an extension study of a randomized control trial involving one arm of the original study.

2.6 Extracting the data

In preparation for data extraction, authors are encouraged to develop outlines of the tables and figures to facilitate data collection. These data tables are ideally easy to use and standardized, so that multiple authors can work together in tandem to extract the relevant information. Additionally, this offers authors the opportunity to plan which specific outcomes they are interested in collecting. It is also encouraged that teams identify what data would be required to build a sophisticated meta-analysis in advance.

2.7 Assessing risk of bias

Authors should assess the risk of bias of included studies with a predetermined, validated, and well-established tool that is specific to the type of manuscript being reviewed (i.e., randomized control studies versus observational studies). For example, the Cochrane Risk-of-Bias tool (RoB 2) is frequently used to evaluate the quality of evidence in randomized controlled trials [15]. This tool prompts authors to consider a structured set of domains of bias that address trial design, conduct, and reporting, among others. Authors are then required to make judgements about the risks of bias with help from an algorithm, which enables classification into the following categories: ‘low risk of bias’, ‘high risk of bias’ or ‘some concerns.’ Tools such as the RoB 2 are designed to be implemented by at least two independent reviewers, to ensure reliability.

2.8 Synthesizing the data

Synthesis involves gathering the data from the included studies in order to draw a common conclusion about the evidence. It is recommended that the evidence be summarized in a table that highlights the PICO characteristics of each study, which will assist in determining which studies are similar enough to be grouped. The synthesis itself involves either performing a statistical analysis, or performing a structured reporting of the effects. Synthesis for the purposes of generating a meta-analysis may have advantages, but is only possible if the data and the outcome measures are sufficiently homogenous. When considering meta-analysis, it is critical to involve a collaborator with statistical expertise.

2.9 Certainty assessment

Systematic reviews should provide both an estimate of the treatment effect and a judgment about the certainty or quality of the evidence. Although other systems exist, the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach is widely utilized. It categorizes the evidence quality on four levels: high, moderate, low and very low [16]. These judgments are determined through consideration of a checklist which prompts the reviewers to evaluate the overall bias, inconsistency, indirectness, precision and publication biases. It is recommended to build a ‘summary of findings table’, which has a standardized format developed by the GRADE working group and includes an online tool [17]. This allows the reader to understand the major findings and conclusions from the systematic review.

3 Common pitfalls

3.1 Failure to define the research question & develop a protocol

Authors should seek to establish parameters for the systematic review before performing the literature search. This starts with thoroughly defining the variable within the PICO framework. Having a protocol will reduce the intrinsic bias of the systematic review process and ensure that any subsequent changes to the methodology are transparent and justified [18].

3.2 Inadequate search strategy

Several common mistakes prevent the generation of an exhaustive and reproducible search. For example, authors who search too few databases and restrict language may risk missing important articles. The Cochrane Collaborations Methodological Expectations of Cochrane Intervention Reviews (MECIR) guidelines state that searching MEDLINE, EMBASE and CENTRAL should be considered mandatory [19]. However, a recent study supports that even this strategy may not be sufficient for identifying all important studies; unfortunately, increasing the search by another ten databases increased the output by 2%. This study concluded that researchers should consider manual searches and review of references. [20] Consulting with a healthcare information specialist or librarian may be help ensure adequate search strategy and use of appropriate search terms [7]. A number of recent studies have indicated that librarian involvement improves the reproducibility of literature searching and that librarians should be considered as coauthors in medical literature [10,[21], [22], [23]].

3.3 Inadequate risk of bias assessment

There are several good tools to address the question of the internal validity of included studies. Cochrane has a widely applied tool for randomized control trials [15,24], but other tools address bias and observational studies of different designs [15,16,[25], [26], [27]]. It is important to consider where the tool has been used, prior to selecting it. Common pitfalls include generalizing a risk of bias to an individual study but failing to attribute a risk of bias rating to an outcome across studies. Review authors may confuse imprecision and generalizability with bias, and although these certainly impact evidence quality they do not contribute to “systematic error”. Risk of bias assessment should be applied by at least two different reviewers [7]. Additionally, care should be taken when interpreting studies with immediate cause for concern. For instance, inferior or less-rigorous study designs (i.e. randomized control compared with case report), those with poor reporting, industry funding, or disclosed conflict of interest should be thoroughly evaluated through all domains transparently prior to being rated as a high risk of bias [28].

3.4 Inappropriate meta-analysis and data synthesis

Care should be taken when combining data for a meta-analysis. The potential for a quantitative synthesis should be based on the amount of clinical and methodological homogeneity and assessment of possible biases [29]. For example, a meta-analysis might not be appropriate when analyzing studies of lumbar medial branch radiofrequency ablation if the clinical selection criteria and technical standards are different between the studies. The results of the studies may be significantly different due to those differences alone (and not representative of the effect of the clinical intervention). Similarly, randomized and nonrandomized studies should not typically be combined for analysis [7]. Finally, since methods for data synthesis are sophisticated, and require advanced training, it is recommended to collaborate with a statistician.

3.5 Failure to choose an appropriate review approach

Authors must determine if a systematic review is advisable considering the present state of the evidence. To conduct a systematic review with the potential to inform practice, the topic must be timely and appropriate. Systematic reviews can be inadvisable if the topic is extremely new, whereby there are too few relevant published papers. Conversely, if several high-quality systematic reviews already exist on the topic, the utility of repeating a systematic review may be low [10]. This can be prevented by developing a strong PICO, briefly searching the literature, and having an appropriate team with a content expert and librarian.

Authors may also fail to recognize that a systematic review is not the best design to answer their question. or scoping reviews have a more expansive inclusion criteria and differ from systematic reviews in their overriding purpose [30]. Their purpose is to identify the types of evidence in a field, to clarify key concepts in the literature, to examine how research is conducted on a certain topic, to identify gaps in the knowledge base, or to serve as a precursor to a systematic review. The “population, concept, and context” (PCC) model is used to guide question development, rather than the PICO model[30].

4 Special considerations for pain medicine

A systematic review's conclusion is established by the quality of evidence, which is determined by the reviewers' confidence in the causal relationship of variables across multiple studies. Generally, the causal relationship of an independent variable/exposure/intervention (e.g., radiofrequency ablation (RFA)) and a dependent variable/outcome (e.g., post-RFA pain reduction), is established when variables are tested in a specific population free of bias, confounders, or chance. The most effective way to establish a causal relationship or interventional efficacy is in a large, blinded, randomized controlled trial (RCT) or a large, well-controlled cohort study. An RCT significantly reduces bias and confounding effects and makes the reader more confident in the relationship between an intervention and outcome. A positive systematic review that reported strong evidence of a particular intervention would comprise multiple extensive, non-funded randomized clinical trials reporting consistent effect sizes. Alternatively, a systematic review of low quality studies will result in low confidence in the causal relationship's confidence and the interaction between the independent and dependent variables. That is not to say the intervention is ineffective, but the quality of evidence is low and the true relationship or association between the variables may be different than what is currently published. There are particular factors within interventional pain medicine and topics of misunderstanding that reduce the quality of primary literature and affect the conclusions of systematic reviews.

4.1 Study design

As previously mentioned, large, blinded, placebo controlled RCTs are often considered the gold standard of study design due to highest methodological rigor, less tendency for bias due to blinding, and more control of confounding factors [31,32]. However, given the expense of performing large RCTs, many of the existing studies in the field of interventional pain medicine are non-randomized, single group observational studies. There is debate about the merit of including observational studies in systematic reviews across many disciplines. Observational designs lack blinding and random allocation to an intervention which increases the risk of bias in the reported results [31]. However, methodologists have argued that a well-conducted prospective observational study will yield similar estimates of effect compared with RCTs [32]. In interventional pain medicine, where a novel therapy may be associated with a small body of literature, including prospective observational studies into a systematic review can be important to avoid underrepresenting the effect of an intervention.

In addition, including unpublished, non-peer reviewed data introduces the risk of bias into the systematic review. However, in a rapidly evolving field such as pain medicine, it is important to consider the ramifications of excluding unpublished data. If authors feel confident in assessing the quality of this literature, it could be reasonable to consider their inclusion. However, this decision should be outlined in the methods and discussed in the paper for it may represent a limitation to the generalizability of the results. Studies that include this literature should attempt to use a gray-literature appraisal tool [33]. Unpublished or gray data acquired via industry sponsorship should be carefully considered as it may compound the risk of bias.

4.2 Outcomes measures that matter in interventional pain medicine

Authors of systematic reviews may be confronted with studies that utilize subpar outcome measures, report data poorly and employ inappropriate statistical approaches. This important flaw should be considered when rating individual studies. When employing the Cochrane risk of bias tool, for example, authors are prompted to rate the appropriateness of the outcome measure and statistical methods applied to the data [15]. Categorical data, which groups participants into predefined categories of pain relief (i.e. 50% pain reduction or 80% pain reduction) are the most powerful, as these enable the authors to independently calculate the 95% confidence interval, which (1) empowers the reader to understand the nature of the reported effect [34,35],(2) enables the systematic review author to consider sophisticated meta-analysis [36]. Continuous data, when fully reported, can also be valuable when the scale of choice is widely accepted and validated (i.e. subject scores on the visual analog scale (VAS) or numeric rating scale (NRS)). However, studies that share limited continuous data without commenting on the distribution of the sample, means or standard deviations, should raise concern. This data may preclude meta-analysis and can be subject to inappropriate statistical tests (i.e. t-tests to yield p-values on pre-vs post-numerical scores that are performed on data that does not follow a normal distribution) [37].

4.3 Industry sponsorship

The field of pain medicine must recognize the positive role private companies play in developing novel therapeutics, particularly since device and drug development is associated with tremendous cost and medical research funding is increasingly difficult to acquire [38,39]. However, when designing a successful systematic review, authors must consider the limitations of industry sponsored trials in light of the multiple stakeholders and direct financial consequences for private companies and hospitals. Additionally, private financing of clinical research often includes incentives for academic investigators, which can create a conflict of interest with the potential to bias both methodology and results. This compensation can come in form of direct financing, shares, options, or paid positions on scientific committees and advisory boards [40].

The consequences of industry sponsorship may extend to study design and directly and indirectly influence results. It is well documented in the literature that when compared to non-industry funded studies of the same intervention, industry funded studies trend toward more positive results on average [41]. One group determined that there was a statistically significant association between the source of funding and the outcome of the study [42]. There are many possible explanations for this. For instance, critics have called into question RCT designs that appear to have non-equivalent arms at baseline and lack of blinding or open-label designs. In addition, a review of industry sponsored RCTs showed a consistent trend towards explanatory (placebo-controlled) designs rather than pragmatic designs which are not designed to show whether the index treatment is successful, but rather are built to inform the consumer on the contribution of non-specific effects to the results [41]. Selective reporting bias and publication bias may also contribute to this observed discrepancy.

5 Conclusions

High-quality systematic reviews are an essential part of pain medicine literature, but their success depends on careful planning and execution. Authors must consider the anticipated benefit of their proposed manuscript on the existing body of literature. Prospective application of the PICO principles is critical when designing a rigorous review. While many of the guidelines for systematic review development are uniform across medical specialities, special attention should be paid to the scope of the inclusion criteria, role of industry sponsorship, and appropriateness of outcomes measures when contemplating the field of interventional pain medicine.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
==== Refs
References

1 Vetter T.R. Systematic review and meta-analysis: sometimes bigger is indeed better Anesth Analg 128 3 2019 575 583 10.1213/ANE.0000000000004014 30649072
2 Ioannidis J.P.A. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses Milbank Q 94 3 2016 485 514 10.1111/1468-0009.12210 27620683
3 Huygen F. Kallewaard J.W. van Tulder M. Evidence-based interventional pain medicine according to clinical diagnoses”: update 2018 Pain Pract 19 6 2019 664 675 10.1111/papr.12786 30957944
4 Page M.J. McKenzie J.E. Bossuyt P.M. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement J Clin Epidemiol 134 2021 103 112 10.1016/j.jclinepi.2021.02.003 33577987
5 Fontelo P. Liu F. A review of recent publication trends from top publishing countries Syst Rev 7 1 2018 147 10.1186/s13643-018-0819-1 30261915
6 Cohen S.P. Gallagher R.M. Ethical conundrums in pain medicine: the intersection of industry sponsorship, fee-for-service interventions, and access to care Pain Med 18 9 2017 1629 1630 10.1093/pm/pnx199 28926657
7 Bero L. Getting the systematic review basics right helps clinical practice: 4 common pitfalls for systematic review authors to avoid Br J Sports Med 53 1 2019 6 8 10.1136/bjsports-2017-098239 29217534
8 Ghogomu E.A.T. Maxwell L.J. Buchbinder R. Updated method guidelines for cochrane musculoskeletal group systematic reviews and metaanalyses J Rheumatol 41 2 2014 194 205 10.3899/jrheum.121306 24293581
9 Lefebvre C. Glanville J. Briscoe S. Cochrane information retrieval methods group. technical supplement to chapter 4: searching for and selecting studies Cochrane Handbook for Systematic Reviews of Interventions 2019 1 94 Published online
10 Phillips V. Barker E. Systematic reviews: structure, form and content J Perioperat Pract 31 9 2021 349 353 10.1177/1750458921994693
11 Speckman R.A. Friedly J.L. Asking structured, answerable clinical questions using the population, intervention/comparator, outcome (PICO) framework PM&R 11 5 2019 548 553 10.1002/pmrj.12116 30729707
12 Richardson W.S. Wilson M.C. Nishikawa J. Hayward R.S. The well-built clinical question: a key to evidence-based decisions ACP J Club 123 3 1995 A12 A13 https://www.ncbi.nlm.nih.gov/pubmed/7582737
13 Chapter 1: starting a review https://training.cochrane.org/handbook/current/chapter-01
14 Schardt C. Adams M.B. Owens T. Keitz S. Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions BMC Med Inf Decis Making 7 2007 16 10.1186/1472-6947-7-16
15 Sterne J.A.C. Savović J. Page M.J. RoB 2: a revised tool for assessing risk of bias in randomised trials BMJ 366 2019 l4898 10.1136/bmj.l4898 31462531
16 Guyatt G.H. Oxman A.D. Vist G.E. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations BMJ 336 7650 2008 924 926 10.1136/bmj.39489.470347.AD 18436948
17 GRADEpro http://www.gradepro.org
18 Ahmadi A. Yazdizadeh B. Doshmangir L. Majdzadeh R. Asghari S. PROTOCOL: systematic review of methods to reduce risk of bias in knowledge translation interventional studies in health-related issues Campbell Systematic Reviews 18 2 2022 10.1002/cl2.1236
19 MECIR manual https://community.cochrane.org/mecir-manual
20 Aagaard T. Lund H. Juhl C. Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol 16 1 2016 161 10.1186/s12874-016-0264-6 27875992
21 Hameed I. Demetres M. Tam D.Y. An assessment of the quality of current clinical meta-analyses BMC Med Res Methodol 20 1 2020 105 10.1186/s12874-020-00999-9 32380945
22 Koffel J.B. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors PLoS One 10 5 2015 e0125931 10.1371/journal.pone.0125931
23 Rethlefsen M.L. Farrell A.M. Osterhaus Trzasko L.C. Brigham T.J. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews J Clin Epidemiol 68 6 2015 617 626 10.1016/j.jclinepi.2014.11.025 25766056
24 Higgins J.P.T. Altman D.G. Gøtzsche P.C. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials BMJ 343 2011 d5928 10.1136/bmj.d5928 22008217
25 Boyles∗ A.L. Thayer K.A. Rooney A.A. Assessing risk of bias in observational studies for systematic reviews: applying the OHAT method in two case studies ISEE Conference Abstracts 2014 1 2014 2497 10.1289/isee.2014.p2-437
26 Morgan R.L. Thayer K.A. Santesso N. Evaluation of the risk of bias in non-randomized studies of interventions (ROBINS-I) and the “target experiment”concept in studies of exposures: rationale and preliminary instrument development Environ Int 120 2018 382 387 https://www.sciencedirect.com/science/article/pii/S0160412017321967?casa_token=kE2XE8sBmnoAAAAA:58l6HVFRGGOScrutd6pfyLjskTUB_CoE5X7L1RzAYq6FCzQam6V1ZEABtDS8uUIuHdKzehWldaCV 30125855
27 Sterne J.A. Hernán M.A. Reeves B.C. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions BMJ 355 2016 i4919 10.1136/bmj.i4919 27733354
28 Viswanathan M. Ansari M.T. Berkman N.D. Assessing the risk of bias of individual studies in systematic reviews of health care interventions. Agency for healthcare research and quality methods guide for comparative effectiveness reviews 2012 AHRQ Methods for Effective Health Care Published online https://hero.epa.gov/hero/index.cfm/reference/details/reference_id/3507859
29 Metelli S. Chaimani A. Challenges in meta-analyses with observational studies Evid Base Ment Health 23 2 2020 83 87 10.1136/ebmental-2019-300129
30 Munn Z. Peters M.D.J. Stern C. Tufanaru C. McArthur A. Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach BMC Med Res Methodol 18 1 2018 1 7 10.1186/s12874-018-0611-x 29301497
31 Stroup D.F. Berlin J.A. Morton S.C. Meta-analysis of observational studies in epidemiology: a proposal for reporting JAMA 283 15 2000 2008 2012 10.1001/jama.283.15.2008 10789670
32 Shrier I. Boivin J.F. Steele R.J. Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles Am J Epidemiol 166 10 2007 1203 1209 10.1093/aje/kwm189 17712019
33 Tyndall J. How low can you go? Towards a hierarchy of grey literature Published online https://www.semanticscholar.org/paper/2e5b9b797afac94056f16292320c4bcc09cfb743 2008
34 Speckman R.A. Summary measures and measures of effect: summarizing and comparing outcomes in rehabilitation research. Part 1: overview Pharm Manag PM R 11 8 2019 888 894 10.1002/pmrj.12195
35 Speckman R.A. Burnham T.R. Summary measures and measures of effect: summarizing and comparing outcomes in rehabilitation research. Part 2: binary outcomes PM&R 12 9 2020 933 939 10.1002/pmrj.12457 32710509
36 Bogduk N. Stojanovic M. Group data or categorical data for outcomes of pain treatment? Pain Med 21 10 2020 2046 2052 10.1093/pm/pnaa286 33118600
37 Bogduk N. Kennedy D.J. Vorobeychik Y. Engel A. Guidelines for composing and assessing a paper on treatment of pain Pain Med 18 11 2017 2096 2104 10.1093/pm/pnx121 28633460
38 Duval G. The benefits and threats of research partnerships with industry Crit Care 9 4 2005 309 310 10.1186/cc3539 16137365
39 Bodenheimer T. Uneasy alliance — clinical investigators and the pharmaceutical industry N Engl J Med 342 20 2000 1539 1544 10.1056/nejm200005183422024 10816196
40 Chopra S.S. Industry funding of clinical trials: benefit or bias? JAMA 290 1 2003 113 114 10.1001/jama.290.1.113 12837722
41 Djulbegovic B. Lacevic M. Cantor A. The uncertainty principle and industry-sponsored research Lancet 356 9230 2000 635 638 10.1016/S0140-6736(00)02605-2 10968436
42 Davidson R.A. Source of funding and outcome of clinical trials J Gen Intern Med 1 3 1986 155 158 10.1007/BF02602327 3772583
