
==== Front
BMC Med Res Methodol
BMC Med Res Methodol
BMC Medical Research Methodology
1471-2288
BioMed Central London

2332
10.1186/s12874-024-02332-0
Research
Response is increased using postal rather than electronic questionnaires – new results from an updated Cochrane Systematic Review
Edwards Phil Phil.Edwards@LSHTM.ac.uk

Perkins Chloe
https://ror.org/00a0jsq62 grid.8991.9 0000 0004 0425 469X Data Collection Methodology Group, Department of Population Health, Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, Keppel Street, London, WC1E 7HT UK
16 9 2024
16 9 2024
2024
24 2096 11 2023
6 9 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by/4.0/ Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Background

A decade ago paper questionnaires were more common in epidemiology than those administered online, but increasing Internet access may have changed this. Researchers planning to use a self-administered questionnaire should know whether response rates to questionnaires administered electronically differ to those of questionnaires administered by post. We analysed trials included in a recently updated Cochrane Review to answer this question.

Methods

We exported data of randomised controlled trials included in three comparisons in the Cochrane Review that had evaluated hypotheses relevant to our research objective and imported them into Stata for a series of meta-analyses not conducted in the Cochrane review. We pooled odds ratios for response using random effects meta-analyses. We explored causes of heterogeneity among study results using subgroups. We assessed evidence for reporting bias using Harbord’s modified test for small-study effects.

Results

Twenty-seven trials (66,118 participants) evaluated the effect on response of an electronic questionnaire compared with postal. Results were heterogeneous (I-squared = 98%). There was evidence for biased (greater) effect estimates in studies at high risk of bias; A synthesis of studies at low risk of bias indicates that response was increased (OR = 1.43; 95% CI 1.08–1.89) using postal questionnaires. Ten trials (39,523 participants) evaluated the effect of providing a choice of mode (postal or electronic) compared to an electronic questionnaire only. Response was increased with a choice of mode (OR = 1.63; 95% CI 1.18–2.26). Eight trials (20,909 participants) evaluated the effect of a choice of mode (electronic or postal) compared to a postal questionnaire only. There was no evidence for an effect on response of a choice of mode compared with postal only (OR = 0.94; 95% CI 0.86–1.02).

Conclusions

Postal questionnaires should be used in preference to, or offered in addition to, electronic modes.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-024-02332-0.

Keywords

Data collection
Questionnaires and surveys
Survey methodology
Nonrespondents
Self report
NIHR Evidence Synthesis ProgrammeNIHR133238 NIHR133238 issue-copyright-statement© BioMed Central Ltd., part of Springer Nature 2024
==== Body
pmcIntroduction

Rationale

When collecting information from large, geographically dispersed populations, a self-administered questionnaire is usually the only financially viable option [1]. Non-responses to questionnaires reduce effective sample sizes, reducing study power, and may introduce bias in study results [2]. The Cochrane Methodology Review of methods to increase response to self-administered questionnaires has provided a much-used scientific evidence base for effective data collection by questionnaire since the publication of the first version of the review in 2003 which focused on postal questionnaires [3].

A decade ago paper-and-pencil administration of questionnaires in epidemiological studies was twenty times more common than electronic administration [4], but increased Internet access and decreasing volumes of mailed letters suggests that electronic administration has gained favour [5–7]. Researchers planning to collect data from participants using a self-administered questionnaire need to know how will the proportion of participants responding to a questionnaire administered electronically compare with one administered by post? We conducted further analyses of the trials included in the recently updated Cochrane Review [8] to answer this question.

Objective

To assess whether response rates to questionnaires administered electronically differ to those of questionnaires administered by post.

Methods

Data sources/measurement

We exported data of randomised controlled trials included in the updated Cochrane Review [8] from RevMan and imported them into Stata for a series of meta-analyses not conducted in the Cochrane review.

Comparisons

We focused on data from trials included in three comparisons in the Cochrane Review that had evaluated hypotheses relevant to our research objective:

Postal vs. electronic questionnaire (Cochrane Comparison 81).

Electronic questionnaire only vs. choice (postal or electronic) (Cochrane Comparison 84).

Choice (electronic or postal) vs. postal questionnaire only (Cochrane Comparison 82).

These comparisons assess: response to questionnaires administered by post compared with questionnaires administered electronically, response to a questionnaire administered electronically compared with response when including a postal response option, and response when including an electronic response option compared with response to a questionnaire administered by post only, respectively.

Data items

Outcome measures

The data obtained from each trial included the numbers of participants randomised to each arm of the trial with the numbers of completed, or partially completed questionnaires returned after all mailings (for trials including a postal questionnaire), and the numbers of participants randomised to each arm of the trial with the numbers of participants submitting the completed, or partially completed online questionnaires after all contacts (electronic questionnaire).

Other variables

Additional data were also extracted on the:

Year of publication of the study.

Risk of bias in each included study (a judgment - high, low, or unclear); we assessed the overall risk of bias in each study using the Cochrane Collaboration’s tool [9].

Effect measures and synthesis

For each of the three comparisons (2.1.1 above), we pooled the odds ratios for response in each included study in a random effects meta-analysis (to allow for heterogeneity of effect estimates between studies) using the metan command in Stata [10]. This command also produced a forest plot (a visual display of the results of the individual studies and syntheses) for each comparison. We quantified any heterogeneity using the I2 statistic that describes the percentage of the variability in effect estimates that is due to heterogeneity [11].

Subgroup analyses

We explored possible causes of heterogeneity among study results by conducting subgroup analyses according to two study-level factors: Year of study publication, and risk of bias in studies. We used a statistical test of homogeneity of the pooled effects in subgroups to assess evidence for subgroup differences. The statistical test of homogeneity used is Cochran’s Q test, where the Q statistic is distributed as a chi-square statistic with k-1 degrees of freedom, where k is the number of subgroups. If there was evidence for subgroup differences provided by the test of homogeneity, we chose the ‘best estimate of effect’ as the estimate from the subgroup of studies with low risk of bias, or the subgroup of studies published after 2012. If there was no evidence for subgroup differences, we chose our best estimate of effect based on the synthesis of all studies.

Year of study publication

From 2012, household access to a computer exceeded 40%: [5] As the odds ratios for response to questionnaires administered electronically may be associated with household access to a computer, we analysed trial results in two subgroups – before 2012 and after 2012, where we used the year of publication as an approximation of the year of study conduct.

Risk of bias

The odds ratios for response estimated in the included studies may be associated with trial quality. [12, 13] For this reason we analysed trial results in two subgroups – trials judged to be at low and at high risk of bias.

Reporting bias assessment

We assessed evidence for reporting bias using Harbord’s modified test for small-study effects implemented in Stata using the metabias command [14]. This test maintains better control of the false-positive rate than the test proposed by Egger at al [14].

Results

Study characteristics

Thirty-five studies [15–49] reported 45 independent trials included in one or more of the three comparisons (Table 1). The studies were conducted in the US (n = 20), Europe (n = 13), and Australasia (n = 2). The studies included between 133 and 12,734 participants and were published between 2001 and 2020. Eight studies were judged to be at high risk of bias [16, 19, 33, 34, 42, 43, 45, 46].

Table 1 Characteristics of 35 included studies

First author	Year	Setting	Comparison	Study size	Participants	Risk of Bias	Topic	Questionnaire description	
						SG±	AC∀	BPP¥	BOO§	IOD&	SR€	OSBα			
Akl	2005	US	1. Postal vs. Electronic questionnaire	202	Residents and faculty of the University at Buffalo Internal Medicine Residency program	Unclear	Unclear	Unclear	Low	Low	Low	Low	Academic issues	28 questions (residents), 23 (faculty).	
Basnov	2009	Denmark	1. Postal vs. Electronic questionnaire	782	Women referred for mammography	Unclear	Unclear	Unclear	Low	High	High	Low	Anxiety and Depression	Short Form Health Survey and Hospital Anxiety and Depression Scale − 17 pages, 119 items.	
Bech	2009	Denmark	1. Postal vs. Electronic questionnaire	9,900	People aged 50–75 years drawn from the Central National Population Register	Unclear	Unclear	Unclear	Low	Low	Low	Low	Design of nursing homes and facilities	35 questions; 16 pages	
Beebe	2018	US	1. Postal vs. Electronic questionnaire	342	Primary care clinicians, including physicians, nurse practitioners, and physician assistants	Unclear	Unclear	Unclear	Low	Low	Low	Low	Clinician knowledge, barriers, and perceived parental barriers regarding HPV vaccination.	Questionnaire not described	
Bergeson	2013	US	1. Postal vs. Electronic questionnaire	9,313	Patients receiving care at 6 clinics in Minnesota.	Unclear	Unclear	Unclear	Low	High	Low	Low	Patient experiences with care.	CAHPS hybrid survey, 22 questions	
Bjertnaes	2018	Norway	1. Postal vs. Electronic questionnaire;

2. Electronic only vs. Choice (postal or electronic);

3. Choice (electronic or postal) vs. postal only

	1,696	Parents of children with type 1 diabetes in the Norwegian Childhood Diabetes Registry with at least one outpatient consultation in the previous year.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Parent experiences with their child’s diabetes outpatient care.	40 questions, 4 pages.	
Bray	2017	UK	2. Electronic only vs. Choice (postal or electronic)	8,765	Young people in the Avon Longitudinal Study of Parents and Children (ALSPAC) birth cohort study	Low	Low	Low	Low	Low	Low	Low	Gambling, Self-Harm, Employment, Education & Training, Tobacco & Alcohol.	A5 booklet of 44 pages; Online questionnaire designed to be similar to the paper questionnaire.	
Brøgger	2007	Norway	3. Choice (electronic or postal) vs. postal only	4,213	People aged 20–40 years from the Norwegian Central Population Registry.	Unclear	Unclear	Low	Low	Low	Low	Low	Asthma and allergies	one-page, 40 questions	
Clark	2011	US	1. Postal vs. Electronic questionnaire	205	Director of Nursing and Administrator in nursing homes with at least 30 beds.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Q55uestionnaire not described	Questionnaire not described	
Cobanoglu	2001	US	1. Postal vs. Electronic questionnaire	194	Hospitality professors with email addresses from members of the Council on Hotel, Restaurant, and Institutional Education.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Hospitality education.	Questionnaire not described	
Fluss	2014	Scotland, UK	1. Postal vs. Electronic questionnaire	4,417	Residents in Grampian (north of Scotland, UK) aged 25+.	Low	Low	Unclear	Low	Low	Low	Low	Pain and pain management	20-pages.	
Fowler	2019	US	1. Postal vs. Electronic questionnaire	4,323	Patients age 18 + with at least one primary care visit during previous 6 months, Boston.	Unclear	Unclear	Unclear	Low	High	Low	Low	Medical care experiences.	56 questions, standard CAHPS survey, 12 pages.	
Hardigan	2012	US	1. Postal vs. Electronic questionnaire;

2. Electronic only vs. Choice (postal or electronic);

3. Choice (electronic or postal) vs. postal only

	3,953	Dentists from the Florida Department of Health, Board of Dentistry.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Tobacco use by dental patients	28 questions; 4 pages.	
Hardigan	2016	US	1. Postal vs. Electronic questionnaire	4,661	Practicing pharmacists.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Job satisfaction.	60 questions; 4 pages.	
Hohwu	2013	Denmark	2. Electronic only vs. Choice (postal or electronic);

3. Choice (electronic or postal) vs. postal only

	1,573	Parents of children aged 2–17 years	Unclear	Unclear	Unclear	Low	Low	Low	Low	Children’s health and welfare	NordChild questionnaire, 28 pages, 73 questions.	
Iversen	2020	Norway	2. Electronic only vs. Choice (postal or electronic)	6,454	Patients aged 16 + registered with a GP.	Unclear	Unclear	Unclear	Low	Low	Low	Low	User experiences with healthcare.	47 questions on 6 pages.	
Jacob	2012	US	1. Postal vs. Electronic questionnaire	556	High school principals, Michigan.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Online learning and virtual education	42 questions; 15 min completion time	
Lagerros	2012	Sweden	1. Postal vs. Electronic questionnaire	393	People from the population register.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Physical activity.	4 questions.	
Leece	2004	US	1. Postal vs. Electronic questionnaire	442	Surgeon-members of the Orthopaedic Trauma Association	High	High	Unclear	Low	Low	Low	Low	Operative treatment for femoral neck fractures.	8 pages	
Mauz	2018	Germany	1. Postal vs. Electronic questionnaire;

3. Choice (electronic or postal) vs. postal only;

	6,194	Children and adolescents participating in the third wave of the ‘German Health Interview and Examination Survey for Children and Adolescents’.	Unclear	Unclear	Unclear	Low	High	Low	Low	Health care utilisation, living conditions, and environmental determinants of health.	Questionnaire not described	
Millar	2011	US	1. Postal vs. Electronic questionnaire;

2. Electronic only vs. Choice (postal or electronic);

3. Choice (electronic or postal) vs. postal only

	1,345	Undergraduate students at the main campus of Washington State University.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Experiences at University.	36 questions.	
Millar	2019	US	1. Postal vs. Electronic questionnaire	470	Adults with colorectal, breast, prostate, ovarian, and multiple myeloma cancers, reported in the Utah Cancer Registry.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Cancer recurrence and willingness to participate in cancer research.	35 questions.	
Murphy	2020	US	2. Electronic only vs. Choice (postal or electronic);	12,734	Physicians from the American Medical Association MasterFile, from haematology or oncology specialties.	Low	Low	Low	Low	Low	Low	Low	Referral and recruitment of patients to clinical trials and barriers to trial accrual.	5-min survey	
Reinisch	2016	US	1. Postal vs. Electronic questionnaire	309	Members of the American Association of Plastic Surgeons	Unclear	Unclear	Unclear	Low	Low	Low	Low	Authorship issues.	Questionnaire not described	
Sakshaug	2019	Germany	1. Postal vs. Electronic questionnaire	8,996	Public & private establishments that had previously participated in a Job Vacancy Survey and were registered as employing staff in at least one of 25 target professions.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Factors that influence establishments’ decision-making process for filling job vacancies.	Questionnaire not described	
Schmuhl	2010	US	1. Postal vs. Electronic questionnaire;

2. Electronic only vs. Choice (postal or electronic);

3. Choice (electronic or postal) vs. postal only

	1,178	Emergency medical services (EMS) providers with current certification in Utah.	Unclear	Unclear	Low	Low	Low	Low	Low	Utah Emergency Services for Children needs assessment.	31 questions.	
Schwartzenberger	2017	US	1. Postal vs. Electronic questionnaire	646	Patients who underwent endoscopic carpal tunnel release	Unclear	Unclear	Unclear	Low	Low	Low	Low	Patient-reported outcomes 1 year following carpal tunnel release	23 questions	
Scott	2011	Australia	2. Electronic only vs. Choice (postal or electronic)	1,741	Doctors undertaking clinical practice drawn from a national directory of doctors.	Low	Low	High	Low	Low	Low	Low	Balancing Employment & Life -Workforce participation and its determinants among Australian doctors.	58 questions, 8 page booklet (specialists in training); 87 questions, 13 page booklet (specialists).	
Sebo	2017	Switzerland and France	1. Postal vs. Electronic questionnaire	3,400	Community-based GPs with a valid and available email address	Low	Low	Unclear	Low	High	Low	Low	GPs’ preventive care activities.	37 questions.	
Taylor	2019	US	1. Postal vs. Electronic questionnaire	133	Individuals aged 6–24 years with smartphones, recruited via Internet to participate in a study about mood and mobile game habits.	Unclear	Unclear	Low	Low	Low	Low	Low	Mood, mobile games, and the weather.	6 daily multiple-choice questions	
van den Berg	2011	Holland	2. Electronic only vs. Choice (postal or electronic)	277	Female survivors of childhood cancer	High	High	Unclear	Low	Low	Low	Low	Reproductive function, ovarian reserve, and risk of premature menopause in female childhood cancer survivors.	Department of Epidemiology of the Netherlands Cancer Institute Questionnaire in a cohort study on long-term effects of ovarian stimulation for in vitro fertilization.	
Weaver	2019	US	1. Postal vs. Electronic questionnaire	636	Physicians from the Minnesota Board of Medical Practice with both a postal and email address listed.	Unclear	Unclear	Unclear	Low	High	Low	Low	Factors that influence physicians’ willingness to disclose medical errors and adverse events to patients and their families.	Questionnaire not described	
Whitehead	2011	New Zealand	1. Postal vs. Electronic questionnaire	1,914	Students randomly selected from a database containing all university students.	Low	Low	Unclear	Low	Low	Low	Low	Anxiety and depression in non-psychiatric populations.	41 items.	
Yetter	2010	US	1. Postal vs. Electronic questionnaire	780	School psychologists with an email address.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Perceptions of Pre-referral Intervention Teams for addressing children’s school-related academic and behavior difficulties.	66 items.	
Ziegenfuss	2010	US	3. Choice (electronic or postal) vs. postal only	770	Randomly selected Olmsted County residents aged 25–65.	Unclear	Unclear	Unclear	Low	Low	Low	Low	Bowel disease.	16 pages	
Risk of Bias key: ± Sequence generation ∀ Allocation concealment ¥ Blinding of participants and personnel § Blinding of outcome assessment & Incomplete outcome data € Selective reporting α Other sources of bias

Results of syntheses

Comparison 1 - Postal vs. electronic questionnaire

Twenty-seven trials (66,118 participants) evaluated the effect on questionnaire response of postal administration compared with electronic. [15–20, 23–28, 31–36, 38–41, 43, 44, 46–48] The odds of response were increased by over half (OR 1.76; 95% CI 1.34 to 2.32) using a postal questionnaire when compared with an electronic one (Fig. 1). There was considerable heterogeneity between the trial results (I-squared = 98%), but most of the studies showed response was greater with postal questionnaires than with electronic questionnaires, and the high I-squared is due to differences in the size of the benefit for postal questionnaires, rather than being due to an even spread of results between those favouring postal and those favouring electronic questionnaires.

Fig. 1 Effect on response of mode of administration

Comparison 2 - electronic questionnaire only vs. choice (postal or electronic)

Ten trials (39,523 participants) evaluated the effect on questionnaire response of providing a choice of response mode (postal or electronic) compared to an electronic questionnaire only [20, 21, 27, 29, 30, 35, 37, 40, 42, 45]. The odds of response were increased by over half when providing a choice of response mode (OR 1.63; 95% CI 1.18 to 2.26; Fig. 2). There was considerable heterogeneity between the trial results (I-squared = 97.1%), but again, most of the studies favoured giving people the choice of response mode rather than electronic questionnaire only, and the high I-squared is due to differences in the size of the benefit for choice, rather than being due to an even spread of results between those favouring choice and those favouring electronic only.

Fig. 2 Effect on response of choice of response mode compared with electronic only

Comparison 3 - choice (electronic or postal) vs. postal only

Eight trials (20,909 participants) evaluated the effect of providing a choice of response mode (electronic or postal) compared to postal response only [20, 22, 27, 29, 34, 35, 40, 49]. There was no evidence for an effect on response of providing a choice (OR 0.94; 95% CI 0.86 to 1.02; Fig. 3). There was moderate heterogeneity among the trial results (I-squared = 50.9%).

Fig. 3 Effect on response of choice of response mode compared with postal only

Results of subgroup analyses

Table 2 presents the results of subgroup analyses according to the two study-level factors (forest plots of these subgroup analyses are included in supplementary figures).

Table 2 Results of subgroup analyses of according to two study-level factors

Comparison 1		Number of studies	OR	95%CI	I-squared	
Subgroups	All studies	27	1.76	1.34–2.32	98%	
Year of publication	before 2012	10	1.85	1.12–3.06	97.5%	
	after 2012	17	1.70	1.19–2.43	98.3%	
Risk of Bias	High	7	3.24	1.68–6.25	99%	
	Low	20	1.43	1.08–1.89	96.8%	
Comparison 2		Number of studies	OR	95%CI	I-squared	
Subgroups	All studies	10	1.63	1.18 to 2.26	97.1%	
Year of publication	before 2012	4	1.22	0.93–1.61	69.6%	
	after 2012	6	2.02	1.30–3.13	98.2%	
Risk of Bias	High	2	1.08	0.43–2.71	84.2%	
	Low	8	1.77	1.23–2.55	97.7%	
Comparison 3		Number of studies	OR	95%CI	I-squared	
Subgroups	All studies	8	0.94	0.86–1.02	50.9%	
Year of publication	before 2012	4	0.85	0.73–0.98	48.5%	
	after 2012	4	1.01	0.93–1.08	7.0%	
Risk of Bias	High	1	0.98	0.88–1.08	-	
	Low	7	0.92	0.83–1.03	57.1%	

Comparison 1 - postal vs. electronic questionnaire

Year of publication

A third of studies were published before 2012 [15–17, 23, 24, 33, 35, 40, 47, 48]. In this subgroup of studies the odds of response were 85% greater (OR 1.85; 95% CI 1.12 to 3.06) with a postal questionnaire compared with an electronic one. In the subgroup of studies published after 2012 the effect was lower (OR 1.70; 1.19 to 2.43), consistent with our concern (Sect. 2.4.1) that higher household access to a computer from 2012 may have improved preference for electronic questionnaires, however the statistical test of homogeneity of the pooled effects in these two subgroups was not significant (p = 0.788), indicating no evidence from these studies for different effects by year of study (Supplementary Fig. 1a).

Risk of bias

Seven of the trials [16, 17, 26, 33, 34, 43, 46] were judged to be at high risk of bias and for these trials the odds of response were more than tripled (OR 3.24; 95% CI 1.68 to 6.25) using a postal questionnaire when compared with an electronic one. There was considerable heterogeneity between the trial results (I-squared = 99%).

When only the 20 trials deemed to be at low risk of bias were synthesised, the odds of response were increased by two-fifths (OR 1.43; 95% CI 1.08 to 1.89). There was also considerable heterogeneity between these trial results (I-squared = 96.8%).

The statistical test of homogeneity of the pooled effects in these two subgroups (p = 0.025) provides some evidence for greater effect estimates in studies at high risk of bias (Supplementary Fig. 1b). Our best estimate of the effect on response of mode of administration is hence from a synthesis of the studies at low risk of bias (OR 1.43; 95% CI 1.08 to 1.89). Results overall were thus confounded by risk of bias, but this did not explain the between study heterogeneity.

Comparison 2 - electronic questionnaire only vs. choice (postal or electronic)

Year of study

Half of studies were published before 2012 [35, 40, 42, 45]. In this subgroup of studies there was no evidence for an effect on response of providing a postal response option (OR 1.22; 95% CI 0.93 to 1.61). In the subgroup of studies published after 2012 there was evidence for an effect on response of providing a postal response option (OR 2.02; 95% CI 1.30 to 3.13). The statistical test of homogeneity of the pooled effects in these two subgroups was significant (p = 0.057), indicating some evidence from these studies for different effects by year of study (Supplementary Fig. 2a). This apparent preference for a postal response option in studies published after 2012 was counter to our concern (Sect. 2.4.1) that higher household access to a computer from 2012 would improve preference for electronic questionnaires. There was considerable heterogeneity between the trial results (I-squared = 98.2%), but most of the studies favoured giving people the choice of response mode rather than electronic questionnaire only, and the high I-squared is due to differences in the size of the benefit for choice, rather than being due to an even spread of results between those favouring choice and those favouring electronic only.

Risk of bias

Two of the trials were judged to be at high risk of bias [42, 45]. There was no evidence for an effect on response of a postal option in these studies (OR 1.08; 95% CI 0.43 to 2.71). When only the 8 trials deemed to be at low risk of bias were synthesised, there was evidence that the odds of response were increased when providing a postal response option (OR 1.77; 95% CI 1.23 to 2.55). There was considerable heterogeneity between these trial results (I-squared = 97.7%). The statistical test of homogeneity of the pooled effects in these two subgroups (p = 0.326) provides no evidence for different effects by risk of bias (Supplementary Fig. 2b). Our best estimate of the effect on response of providing a postal response option is hence from a synthesis of all of these studies (OR 1.63; 95% CI 1.18 to 2.26).

Comparison 3 - choice (electronic or postal) vs. postal questionnaire only

Year of study

In the subgroup of studies published before 2012 there was very weak evidence that the odds of response were lower with an electronic option (OR 0.85; 0.73 to 0.98), whereas in studies published after 2012 there was no evidence for a difference between an electronic option and postal only – perhaps due to electronic methods being more acceptable with increased computer access. The results in both subgroups were more homogeneous (I-squared = 48.5% and 7.0% respectively). The statistical test of homogeneity of the pooled effects in these two subgroups (p = 0.04) provides some evidence for different effects by year of study (Supplementary Fig. 3a). If we consider the most recent trials to better represent the situation today (i.e., greater access to computers than prior to 2012), then our best estimate of the effect on response of providing an electronic response option is from a synthesis of the studies published after 2012 (OR 1.01; 95% CI 0.93 to 1.08), i.e., no evidence for an effect.

Risk of bias

There was one study at high risk of bias [34]. Its results were entirely consistent with the results of the seven studies at low risk of bias (the statistical test of homogeneity of the pooled effects in these two subgroups was not significant (p = 0.454), Supplementary Fig. 3b).

Results of assessments of evidence for reporting bias

Comparison 1 - postal vs. electronic questionnaire

There was no evidence for small study effects (Harbord’s modified test p = 0.148).

Comparison 2 - electronic questionnaire only vs. choice (postal or electronic)

There was no evidence for small study effects (Harbord’s modified test p = 0.841).

Comparison 3 - choice (electronic or postal) vs. postal questionnaire only

There was no evidence for small study effects (Harbord’s modified test p = 0.139).

Discussion

General interpretation of the results in the context of other evidence

This study has shown that response to a postal questionnaire is more likely than response to an electronic questionnaire. It has also shown that response is more likely when providing the option for postal response with an electronic questionnaire. It has further shown that providing an electronic response option with a postal questionnaire has no effect on response. Response is thus increased using postal rather than electronic questionnaires.

A previous meta-analysis of 43 mixed-mode surveys from 1996 to 2006 also found paper and postal administration produced greater response than electronic administration [50]. Our result that providing an electronic response option to postal administration does not increase response is consistent with a previous meta-analysis of randomised trials that found that mailed surveys that incorporate a concurrent Web option have significantly lower response rates than those that do not [51].

We suggest two possible reasons for these results:

Paper questionnaires are more accessible than electronic questionnaires.

Although access to the Internet increased over the period during which the studies included in this study were conducted [5, 52], a ‘digital divide’ [53] persists in many populations where completion of a paper questionnaire may be possible, but completion of an electronic one may not.

Paper questionnaires are more personal than electronic questionnaires.

Personalised materials have been shown to increase response [54]. If participants perceive a paper questionnaire with a return envelope to be more ‘personal’ than a request to go to a website to answer some questions, we should expect a higher response with paper.

Strengths and limitations

The main strengths of this study are that our results are based on syntheses of the results of 45 randomised controlled trials that span two decades, and most of which were judged to be at low risk of bias.

There was, however, considerable heterogeneity between the results of the included studies. Our subgroup analyses did not identify any causes of heterogeneity among study results, but they did reveal confounding of the pooled result for postal versus electronic questionnaires. The unexplained heterogeneity means that we cannot be confident about the magnitude of the effects on response using postal rather than electronic questionnaires. However, from inspection of the forest plots we can be confident about the direction of these effects.

The evidence included in this review addresses ‘unit’ non-response only (i.e., return of questionnaires). ‘Item’ response (i.e., completion of individual questions) may be greater with electronic methods, but this was not addressed in this review and requires investigation in the future.

We assessed evidence for reporting bias using Harbord’s modified test for small-study effects and found no evidence for bias. This test may not be reliable given the substantial heterogeneity between the results of the included trials [55].

Due to the nature of this study (secondary analysis of a published review), there is no pre-registered protocol for the subgroup analyses provided in this study.

Implications for practice, policy, and future research

These results will help researchers and healthcare providers to improve data collection from study participants and patients, helping to maintain study power and reduce bias due to missing data in research studies. In addition to the methods already known to be effective in increasing questionnaire response [8, 56], postal questionnaires should be used in preference to, or offered in addition to, electronic modes as this will help to increase the proportion of participants that responds. It should be noted, however, that the evidence upon which this recommendation is based is from studies published between 2001 and 2020, and this may change in the future as access to the Internet increases and more people become ‘tech-savvy’. Furthermore, we consider that the certainty of the evidence provided in this study is “Moderate”, due to the unexplained heterogeneity between the results of the included studies.

Future research

Evidence on effective data collection in low- and middle-income settings is needed. Research centres such as LSHTM can embed studies within trials (SWATs) in their research in these settings to help to increase the evidence base [57].

Participation rates for epidemiologic studies have been declining [58]. Our study has presented evidence that postal questionnaires are preferable to electronic questionnaires to improve participation, but it does not tell us why. Research is still needed to advance sociological and psychological theories of participation in data collection procedures [59].

Electronic administration provides benefits for researchers over paper administration which have not been addressed by this study: A well-designed Web questionnaire can control skip patterns, check for allowable values and ranges and response consistencies, and it can include instructions and explanations about why a question is being asked [60]. These options could help to improve the completeness and quality of self-administered data collection, maintaining study power, reducing the risk of bias in study results, and saving study resources. Further research into the cost-effectiveness of electronic administration compared with postal administration in different settings will be needed to inform practice [61].

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Supplementary Material 3

Supplementary Material 4

Supplementary Material 5

Supplementary Material 6

Supplementary Material 7

Acknowledgements

Not applicable.

Author contributions

PE independently screened the search results and the full-text reports, and extracted data from eligible studies. He assisted with data entry into RevMan 5. He exported data into Stata and he conducted the meta-analyses and other statistical analyses. He drafted the manuscript. PE is guarantor for the paper.CP ran the electronic searches, independently screened the search results and the full-text reports, and extracted data from eligible studies. CP assisted with data entry into RevMan 5, and commented on the manuscript.

Funding

This work was supported by the NIHR Evidence Synthesis Programme [Grant NIHR133238]. Conduct of the work was entirely independent of the funder.

Data availability

Data extracted from included studies will be available in the forthcoming update on the Cochrane Library.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Armstrong BK White E Saracci R Principles of exposure measurement in Epidemiology 1992 New York Oxford University Press
Armstrong BK, White E, Saracci R. Principles of exposure measurement in Epidemiology. New York: Oxford University Press; 1992.
2. Greenland S Response and follow-up bias in cohort studies Am J Epidemiol 1977 106 184 7 10.1093/oxfordjournals.aje.a112451 900117
Greenland S. Response and follow-up bias in cohort studies. Am J Epidemiol. 1977;106:184–7.900117 10.1093/oxfordjournals.aje.a112451
3. Edwards PJ, Roberts IG, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix L, Pratap S. Methods to increase response rates to postal questionnaires. Cochrane Database of systematic reviews 2007, issue 2. Art No : MR000008 10.1002/14651858.MR000008.pub3
4. van Gelder MMHJ Reini W Bretveld RW Nel Roeleveld N Web-based questionnaires: the Future in Epidemiology? Am J Epidemiol 2010 172 1292 8 10.1093/aje/kwq291 20880962
van Gelder MMHJ, Reini W, Bretveld RW, Nel Roeleveld N. Web-based questionnaires: the Future in Epidemiology? Am J Epidemiol. 2010;172:1292–8.20880962 10.1093/aje/kwq291
5. Alsop T. Share of households with a computer at home worldwide from 2005 to 2019 https://www.statista.com/statistics/748551/worldwide-households-with-computer/ [Accessed 4 May 2024].
6. Mazareanu E. United States Postal Service - mail volume. https://www.statista.com/statistics/320234/mail-volume-of-the-usps/#:text=United%20States%20Postal%20Service%20%2D%20mail%20volume%202004%2D2020&text=After%20reaching%20a%20peak%20of,to%20just%20129.2%20billion%20units [Accessed 4 May 2024].
7. BBC. What is happening to the Royal Mail? http://news.bbc.co.uk/2/hi/business/8304722.stm [Accessed 4 May 2024].
8. Edwards PJ Roberts I Clarke MJ DiGuiseppi C Woolf B Perkins C Methods to increase response to postal and electronic questionnaires Cochrane Database Syst Rev 2023 11 11 MR000008 10.1002/14651858.MR000008.pub5 38032037
Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Woolf B, Perkins C. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2023;11(11):MR000008. 10.1002/14651858.MR000008.pub5.38032037 10.1002/14651858.MR000008.pub5
9. Higgins JP Altman DG Gøtzsche PC Jüni P Moher D Oxman AD Savovic J Schulz KF Weeks L Sterne JA Cochrane Bias Methods Group; Cochrane Statistical Methods Group The Cochrane collaboration’s tool for assessing risk of bias in randomised trials BMJ 2011 343 d5928 10.1136/bmj.d5928 22008217
Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA, Cochrane Bias Methods Group; Cochrane Statistical Methods Group. The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. 10.1136/bmj.d5928.22008217 10.1136/bmj.d5928
10. Harris RJ, Deeks JJ, Altman DG, Bradburn MJ, Harbord RM, Sterne JAC. Metan: fixed- and Random-effects Meta-Analysis. Stata J. 2008).;8(1):3–28. 10.1177/1536867X0800800102.
11. Higgins JPT Thompson SG Deeks JJ Altman DG Measuring inconsistency in meta-analyses BMJ 2003 327 557 60 10.1136/bmj.327.7414.557 12958120
Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–60.12958120 10.1136/bmj.327.7414.557
12. Schultz Empirical evidence of Bias - dimensions of Methodological Quality Associated with estimates of Treatment effects in controlled trials JAMA 1995 273 5 408 12 10.1001/jama.1995.03520290060030 7823387
Schultz, et al. Empirical evidence of Bias - dimensions of Methodological Quality Associated with estimates of Treatment effects in controlled trials. JAMA. 1995;273(5):408–12.7823387 10.1001/jama.1995.03520290060030
13. Sterne JA, Jüni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med. 2002;21(11):1513-24. 10.1002/sim.1184. PMID: 12111917.
14. Harbord RM Harris RJ Sterne JA Updated tests for small-study effects in meta-analyses Stata J 2009 9 2 197 210 10.1177/1536867X0900900202
Harbord RM, Harris RJ, Sterne JA. Updated tests for small-study effects in meta-analyses. Stata J. 2009;9(2):197–210.10.1177/1536867X0900900202
15. Akl EA Maroun N Klocke RA Montori V Schünemann HJ Electronic mail was not better than postal mail for surveying residents and faculty J Clin Epidemiol 2005 58 4 425 9 10.1016/j.jclinepi.2004.10.006 15862729
Akl EA, Maroun N, Klocke RA, Montori V, Schünemann HJ. Electronic mail was not better than postal mail for surveying residents and faculty. J Clin Epidemiol. 2005;58(4):425–9.15862729 10.1016/j.jclinepi.2004.10.006
16. Basnov M, Kongsved SM, Bech P, Hjollund NH. Reliability of short form-36 in an Internet- and a pen-and-paper version. Inform Health Soc Care. 2009;34(1):53 – 8. doi: 10.1080/17538150902779527. PMID: 19306199.
17. Bech M Kristensen MB Differential response rates in postal and web-based surveys among older respondents Surv Res Methods 2009 3 1 1 6
Bech M, Kristensen MB. Differential response rates in postal and web-based surveys among older respondents. Surv Res Methods. 2009;3(1):1–6.
18. Beebe TJ Jacobson RM Jenkins SM Lackore KA Rutten LJF Testing the impact of mixed-Mode designs (mail and web) and multiple contact attempts within Mode (Mail or web) on Clinician Survey Response Health Serv Res 2018 53 Suppl 1 3070 83 10.1111/1475-6773.12827 29355920
Beebe TJ, Jacobson RM, Jenkins SM, Lackore KA, Rutten LJF. Testing the impact of mixed-Mode designs (mail and web) and multiple contact attempts within Mode (Mail or web) on Clinician Survey Response. Health Serv Res. 2018;53(Suppl 1):3070–83. Epub 2018 Jan 22. PMID: 29355920; PMCID: PMC6056581.29355920 10.1111/1475-6773.12827
19. Bergeson SC, Gray J, Ehrmantraut LA, Laibson T, Hays RD. Comparing Web-based with Mail Survey Administration of the Consumer Assessment of Healthcare Providers and Systems (CAHPS R) Clinician and Group Survey. Primary health care: open access. 2013;3.
20. Bjertnaes O Iversen HH Skrivarhaug T A randomized comparison of three data collection models for the measurement of parent experiences with diabetes outpatient care BMC Med Res Methodol 2018 18 1 95 10.1186/s12874-018-0557-z 30236067
Bjertnaes O, Iversen HH, Skrivarhaug T. A randomized comparison of three data collection models for the measurement of parent experiences with diabetes outpatient care. BMC Med Res Methodol. 2018;18(1):95. 10.1186/s12874-018-0557-z. PMID: 30236067; PMCID: PMC6149010.30236067 10.1186/s12874-018-0557-z
21. Bray I Noble S Robinson R Molloy L Tilling K Mode of delivery affected questionnaire response rates in a birth cohort study J Clin Epidemiol 2017 81 64 71 10.1016/j.jclinepi.2016.09.004 27663611
Bray I, Noble S, Robinson R, Molloy L, Tilling K. Mode of delivery affected questionnaire response rates in a birth cohort study. J Clin Epidemiol. 2017;81:64–71.27663611 10.1016/j.jclinepi.2016.09.004
22. Brøgger J Nystad W Cappelen I Bakke P No increase in response rate by adding a web response option to a postal population survey: a randomized trial J Med Internet Res 2007 9 5 e40 10.2196/jmir.9.5.e40 18174120
Brøgger J, Nystad W, Cappelen I, Bakke P. No increase in response rate by adding a web response option to a postal population survey: a randomized trial. J Med Internet Res. 2007;9(5):e40.18174120 10.2196/jmir.9.5.e40
23. Clark M Rogers M Foster A Dvorchak F Saadeh F Weaver J A randomized trial of the impact of survey design characteristics on response rates among nursing home providers Eval Health Prof 2011 34 4 464 86 10.1177/0163278710397791 21411474
Clark M, Rogers M, Foster A, Dvorchak F, Saadeh F, Weaver J, et al. A randomized trial of the impact of survey design characteristics on response rates among nursing home providers. Eval Health Prof. 2011;34(4):464–86.21411474 10.1177/0163278710397791
24. Cobanoglu C Moreo PJ Warde B A comparison of mail, fax and web-based Survey methods Int J Market Res 2001 43 4 1 15 10.1177/147078530104300401
Cobanoglu C, Moreo PJ, Warde B. A comparison of mail, fax and web-based Survey methods. Int J Market Res. 2001;43(4):1–15.10.1177/147078530104300401
25. Fluss E, Bond CM, Jones GT, Macfarlane GJ. The effect of an internet option and single-sided printing format to increase the response rate to a population-based study: a randomized controlled trial. BMC Med Res Methodol 2014;14.
26. Fowler FJ Jr Cosenza C Cripps LA Edgman-Levitan S Cleary PD The effect of administration mode on CAHPS survey response rates and results: a comparison of mail and web-based approaches Health Serv Res 2019 54 3 714 21 10.1111/1475-6773.13109 30656646
Fowler FJ Jr, Cosenza C, Cripps LA, Edgman-Levitan S, Cleary PD. The effect of administration mode on CAHPS survey response rates and results: a comparison of mail and web-based approaches. Health Serv Res. 2019;54(3):714–21.30656646 10.1111/1475-6773.13109
27. Hardigan PC Succar CT Fleisher JM An analysis of response rate and economic costs between mail and web-based surveys among practicing dentists: a randomized trial J Community Health 2012 37 2 383 94 10.1007/s10900-011-9455-6 21858591
Hardigan PC, Succar CT, Fleisher JM. An analysis of response rate and economic costs between mail and web-based surveys among practicing dentists: a randomized trial. J Community Health. 2012;37(2):383–94.21858591 10.1007/s10900-011-9455-6
28. Hardigan PC Popovici I Carvajal MJ Response rate, response time, and economic costs of survey research: a randomized trial of practicing pharmacists Res Social Administrative Pharm 2016 12 1 141 8 10.1016/j.sapharm.2015.07.003
Hardigan PC, Popovici I, Carvajal MJ. Response rate, response time, and economic costs of survey research: a randomized trial of practicing pharmacists. Res Social Administrative Pharm. 2016;12(1):141–8.10.1016/j.sapharm.2015.07.003
29. Hohwu L, Lyshol H, Gissler M, Jonsson SH, Petzold M, Obel C. Web-based Versus Traditional Paper questionnaires: a mixed-Mode Survey with a nordic perspective. J Med Internet Res 2013;15(8).
30. Iversen HH Holmboe O Bjertnaes O Patient-reported experiences with general practitioners: a randomised study of mail and web-based approaches following a national survey BMJ open 2020 10 10 e036533 10.1136/bmjopen-2019-036533 33051230
Iversen HH, Holmboe O, Bjertnaes O. Patient-reported experiences with general practitioners: a randomised study of mail and web-based approaches following a national survey. BMJ open. 2020;10(10):e036533.33051230 10.1136/bmjopen-2019-036533
31. Jacob RT Jacob B Prenotification, incentives, and Survey Modality: an experimental test of methods to increase Survey Response Rates of School principals J Res Educational Eff 2012 5 4 401 18
Jacob RT, Jacob B. Prenotification, incentives, and Survey Modality: an experimental test of methods to increase Survey Response Rates of School principals. J Res Educational Eff. 2012;5(4):401–18.
32. Lagerros YT Sandin S Bexelius C Litton JE Löf M Estimating physical activity using a cell phone questionnaire sent by means of short message service (SMS): a randomized population-based study Eur J Epidemiol 2012 27 7 561 6 10.1007/s10654-012-9708-4 22744733
Lagerros YT, Sandin S, Bexelius C, Litton JE, Löf M. Estimating physical activity using a cell phone questionnaire sent by means of short message service (SMS): a randomized population-based study. Eur J Epidemiol. 2012;27(7):561–6.22744733 10.1007/s10654-012-9708-4
33. Leece P Bhandari M Sprague S Swiontkowski MF Schemitsch EH Tornetta P Internet versus mailed questionnaires: a randomized comparison (2) J Med Internet Res 2004 6 3 26 33 10.2196/jmir.6.3.e30
Leece P, Bhandari M, Sprague S, Swiontkowski MF, Schemitsch EH, Tornetta P, et al. Internet versus mailed questionnaires: a randomized comparison (2). J Med Internet Res. 2004;6(3):26–33.10.2196/jmir.6.3.e30
34. Mauz E Hoffmann R Houben R Krause L Kamtsiuris P Gößwald A 2018 Mode Equivalence of Health Indicators Between Data Collection Modes and Mixed-Mode Survey Designs in 10.2196/jmir.7802 PMID: 29506967; PMCID: PMC5859740
Mauz E, Hoffmann R, Houben R, Krause L, Kamtsiuris P, Gößwald A. Population-Based Health Interview Surveys for Children and Adolescents: Methodological Study. J Med Internet Res. 2018;20(3):e64. 10.2196/jmir.7802. PMID: 29506967; PMCID: PMC5859740. Mode Equivalence of Health Indicators Between Data Collection Modes and Mixed-Mode Survey Designs in.10.2196/jmir.7802
35. Millar MM Dillman DA Improving response to web and mixed-Mode surveys Pub Opin Q 2011 75 2 249 69 10.1093/poq/nfr003
Millar MM, Dillman DA. Improving response to web and mixed-Mode surveys. Pub Opin Q. 2011;75(2):249–69.10.1093/poq/nfr003
36. Millar MM Elena JW Gallicchio L Edwards SL Carter ME Herget KA Sweeney C The feasibility of web surveys for obtaining patient-reported outcomes from cancer survivors: a randomized experiment comparing survey modes and brochure enclosures BMC Med Res Methodol 2019 19 1 208 10.1186/s12874-019-0859-9 31730474
Millar MM, Elena JW, Gallicchio L, Edwards SL, Carter ME, Herget KA, Sweeney C. The feasibility of web surveys for obtaining patient-reported outcomes from cancer survivors: a randomized experiment comparing survey modes and brochure enclosures. BMC Med Res Methodol. 2019;19(1):208.31730474 10.1186/s12874-019-0859-9
37. Murphy CC Craddock Lee SJ Geiger AM Cox JV Ahn C Nair R Gerber DE Halm EA McCallister K Skinner CS A randomized trial of mail and email recruitment strategies for a physician survey on clinical trial accrual BMC Med Res Methodol 2020 20 1 123 10.1186/s12874-020-01014-x 32429848
Murphy CC, Craddock Lee SJ, Geiger AM, Cox JV, Ahn C, Nair R, Gerber DE, Halm EA, McCallister K, Skinner CS. A randomized trial of mail and email recruitment strategies for a physician survey on clinical trial accrual. BMC Med Res Methodol. 2020;20(1):123.32429848 10.1186/s12874-020-01014-x
38. Reinisch JF, Yu DC, Li WY. Getting a Valid Survey Response From 662 Plastic Surgeons in the 21st Century. Ann Plast Surg. 2016;76(1):3–5. 10.1097/SAP.0000000000000546. PMID: 26418779.
39. Sakshaug JW Vicari B Couper MP Paper, e-mail, or both? Effects of contact mode on participation in a web survey of establishments Social Sci Comput Rev 2019 37 6 750 65 10.1177/0894439318805160
Sakshaug JW, Vicari B, Couper MP. Paper, e-mail, or both? Effects of contact mode on participation in a web survey of establishments. Social Sci Comput Rev. 2019;37(6):750–65.10.1177/0894439318805160
40. Schmuhl P Van Duker H Gurley KL Webster A Olson LM Reaching emergency medical services providers: is one survey mode better than another? Prehospital Emerg Care 2010 14 3 361 9 10.3109/10903121003760184
Schmuhl P, Van Duker H, Gurley KL, Webster A, Olson LM. Reaching emergency medical services providers: is one survey mode better than another? Prehospital Emerg Care. 2010;14(3):361–9.10.3109/10903121003760184
41. Schwartzenberger J Presson A Lyle A O’Farrell A Tyser AR Remote Collection of patient-reported outcomes following outpatient hand surgery: a Randomized Trial of Telephone, Mail, and E-Mail J Hand Surg 2017 42 9 693 9 10.1016/j.jhsa.2017.05.002
Schwartzenberger J, Presson A, Lyle A, O’Farrell A, Tyser AR. Remote Collection of patient-reported outcomes following outpatient hand surgery: a Randomized Trial of Telephone, Mail, and E-Mail. J Hand Surg. 2017;42(9):693–9.10.1016/j.jhsa.2017.05.002
42. Scott A, Jeon SH, Joyce CM, Humphreys JS, Kalb G, Witt J et al. A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC Med Res Methodol 2011;11.
43. Sebo P, Maisonneuve H, Cerutti B, Fournier JP, Senn N, Haller DM, Rates. Delays, and Completeness of General Practitioners’ Responses to a Postal Versus Web-Based Survey: A Randomized Trial. J Med Internet Res 2017;19(3).
44. Taylor S Ferguson C Peng F Schoeneich M Picard RW Use of In-Game rewards to Motivate Daily Self-Report Compliance: Randomized Controlled Trial J Med Internet Res 2019 21 1 e11683 10.2196/11683 30609986
Taylor S, Ferguson C, Peng F, Schoeneich M, Picard RW. Use of In-Game rewards to Motivate Daily Self-Report Compliance: Randomized Controlled Trial. J Med Internet Res. 2019;21(1):e11683. 10.2196/11683. PMID: 30609986; PMCID: PMC6682282.30609986 10.2196/11683
45. van den Berg MH, Overbeek A, van der Pal HJ, Versluys AB, Bresters D, van Leeuwen FE et al. Using web-based and paper-based questionnaires for Collecting Data on Fertility Issues among Female Childhood Cancer survivors: differences in response characteristics. J Med Internet Res 2011;13(3).
46. Weaver L Beebe TJ Rockwood T The impact of survey mode on the response rate in a survey of the factors that influence Minnesota physicians’ disclosure practices BMC Med Res Methodol 2019 19 1 73 10.1186/s12874-019-0719-7 30940087
Weaver L, Beebe TJ, Rockwood T. The impact of survey mode on the response rate in a survey of the factors that influence Minnesota physicians’ disclosure practices. BMC Med Res Methodol. 2019;19(1):73.30940087 10.1186/s12874-019-0719-7
47. Whitehead L Methodological issues in internet-mediated research: a randomized comparison of internet versus mailed questionnaires J Med Internet Res 2011 13 4 e109 10.2196/jmir.1593 22155721
Whitehead L. Methodological issues in internet-mediated research: a randomized comparison of internet versus mailed questionnaires. J Med Internet Res. 2011;13(4):e109.22155721 10.2196/jmir.1593
48. Yetter G Capaccioli K Differences in responses to web and paper surveys among school professionals Behav Res Methods 2010 42 1 266 72 10.3758/BRM.42.1.266 20160305
Yetter G, Capaccioli K. Differences in responses to web and paper surveys among school professionals. Behav Res Methods. 2010;42(1):266–72.20160305 10.3758/BRM.42.1.266
49. Ziegenfuss JY Beebe TJ Rey E Schleck C Locke GR 3rd Talley NJ Internet option in a mail survey: more harm than good? Epidemiology 2010 21 4 585 6 10.1097/EDE.0b013e3181e09657 20539115
Ziegenfuss JY, Beebe TJ, Rey E, Schleck C, Locke GR 3rd, Talley NJ. Internet option in a mail survey: more harm than good? Epidemiology. 2010;21(4):585–6. 10.1097/EDE.0b013e3181e09657.20539115 10.1097/EDE.0b013e3181e09657
50. Shih TH Fan X Response Rates and Mode preferences in Web-Mail mixed-Mode surveys: a Meta-analysis Int J Internet Sci 2007 2 1 59 82
Shih TH, Fan X. Response Rates and Mode preferences in Web-Mail mixed-Mode surveys: a Meta-analysis. Int J Internet Sci. 2007;2(1):59–82.
51. Medway RL Fulton J When more gets you less - a Meta-analysis of the effect of concurrent web options on Mail Survey Response Rates Pub Opin Q 2012 76 4 733 46 10.1093/poq/nfs047
Medway RL, Fulton J. When more gets you less - a Meta-analysis of the effect of concurrent web options on Mail Survey Response Rates. Pub Opin Q. 2012;76(4):733–46.10.1093/poq/nfs047
52. Petrosyan A. Global internet access rate 2005–2022. https://www.statista.com/statistics/209096/share-of-internet-users-worldwide-by-market-maturity/#:text=As%20of%202022%2C%2066%20percent,access%20rate%20was%2066%20percent [Accessed 4 May 2024].
53. Vestberg H. How can we bring 2.6 billion people online to bridge the digital divide? World Economic Forum Annual Meeting; Jan 2024. https://www.weforum.org/agenda/2024/01/digital-divide-internet-access-online-fwa/ [Accessed 4 May 2024].
54. Scott P Edwards P Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials BMC Health Serv Res 2006 6 111 10.1186/1472-6963-6-111 16953871
Scott P, Edwards P. Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials. BMC Health Serv Res. 2006;6:111. 10.1186/1472-6963-6-111.16953871 10.1186/1472-6963-6-111
55. Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443-57. 10.1002/sim.2380. PMID: 16345038.
56. Edwards P Cooper R Roberts I Frost C Meta-analysis of randomised trials of monetary incentives and response to mailed questionnaires J Epidemiol Community Health 2005 59 11 987 99 10.1136/jech.2005.034397 16234429
Edwards P, Cooper R, Roberts I, Frost C. Meta-analysis of randomised trials of monetary incentives and response to mailed questionnaires. J Epidemiol Community Health. 2005;59(11):987–99. 10.1136/jech.2005.034397.16234429 10.1136/jech.2005.034397
57. Treweek S Bevan S Bower P Campbell M Christie J Clarke M Trial Forge Guidance 1: what is a study within a trial (SWAT)? Trials 2018 19 1 139 10.1186/s13063-018-2535-5 29475444
Treweek S, Bevan S, Bower P, Campbell M, Christie J, Clarke M, et al. Trial Forge Guidance 1: what is a study within a trial (SWAT)? Trials. 2018;19(1):139. 10.1186/s13063-018-2535-5.29475444 10.1186/s13063-018-2535-5
58. Galea S Tracy M Participation rates in epidemiologic studies Ann Epidemiol 2007 17 9 643 53 10.1016/j.annepidem.2007.03.013 17553702
Galea S, Tracy M. Participation rates in epidemiologic studies. Ann Epidemiol. 2007;17(9):643–53.17553702 10.1016/j.annepidem.2007.03.013
59. Dillman DA Brenner PS Towards survey response rate theories that no longer pass each other like strangers in the night Understanding survey methodology - sociological theory and applications 2020 Switzerland Springer Nature 15 44
Dillman DA. Towards survey response rate theories that no longer pass each other like strangers in the night. In: Brenner PS, editor. Understanding survey methodology - sociological theory and applications. Switzerland: Springer Nature; 2020. pp. 15–44.
60. Griffin DH, Fischer DP, Morgan MT. Testing an Internet Response Option for the American Community Survey. Washington: U.S. Bureau of the Census, 2001. https://www.census.gov/library/working-papers/2001/acs/2001_Griffin_01.html [Accessed 27 October 2023].
61. Sinclair M O’Toole J Malawaraarachchi M Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches BMC Med Res Methodol 2012 12 132 10.1186/1471-2288-12-132 22938205
Sinclair M, O’Toole J, Malawaraarachchi M, et al. Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches. BMC Med Res Methodol. 2012;12:132. 10.1186/1471-2288-12-132.22938205 10.1186/1471-2288-12-132
