
==== Front
Digit Health
Digit Health
DHJ
spdhj
Digital Health
2055-2076
SAGE Publications Sage UK: London, England

10.1177/20552076241258419
10.1177_20552076241258419
Original Research Article
Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing
https://orcid.org/0000-0003-0284-0575
Semonella Michelle 1
Marchesi Gloria 2
Castelnuovo Gianluca 23
Andersson Gerhard 45
https://orcid.org/0000-0002-5911-5748
Pietrabissa Giada 23
1 Department of Psychology, 26731 Bar-Ilan University , Ramat Gan, Israel
2 Department of Psychology, 9371 Catholic University of Milan , Milan, Italy
3 Psychology Research Laboratory, 9354 IRCCS Istituto Auxologico Italiano , Milan, Italy
4 Department of Behavioural Science and Learning, Department of Biomedical and Clinical Sciences, 4566 Linköping University , Linköping, Sweden
5 Department of Clinical Neuroscience, Karolinska Institute, Stockholm, Sweden
Giada Pietrabissa, Department of Psychology, Catholic University of Milan, Largo A. Gemelli, 1–20123, Milan, Italy. Email: g.pietrabissa@auxologico.it
11 9 2024
Jan-Dec 2024
10 2055207624125841910 11 2023
14 5 2024
© The Author(s) 2024
2024
SAGE Publications Ltd, unless otherwise noted. Manuscript content on this site is licensed under Creative Commons Licenses
https://creativecommons.org/licenses/by-nc-nd/4.0/ This article is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 License (https://creativecommons.org/licenses/by-nc-nd/4.0/) which permits non-commercial use, reproduction and distribution of the work as published without adaptation or alteration, without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.sagepub.com/en-us/nam/open-access-at-sage).
Introduction

Internet-based self-help interventions have the potential to help people address their emotional needs at relatively low costs. However, if the system does not offer optimal functions, it could reduce end-user adherence and satisfaction with treatment and compromise the effectiveness of the program. This study evaluated the usability of an Internet-based self-help intervention for emotional self-management among the general population of Italy during the COVID-19 pandemic.

Methods

A balanced sex-age sample of 10 individuals who met the inclusion criteria were consecutively recruited online. The think-aloud testing method, the system usability scale and an ad hoc semi-structured interview were used to determine the overall system usability.

Quantitative data were analyzed using descriptive statistics and qualitative data were analyzed using thematic analysis.

Results

The participants were mostly satisfied with the usability of the program. However, older users (<45 years) encountered some problems, which took longer, made more mistakes, and needed more help in performing the tasks than their younger counterparts. The analysis of the interviews revealed three central themes: general thoughts about the platform, weaknesses of the platform and difficulties encountered while navigating and completing tasks, and strengths of the platform.

Discussion

Based on the results of this study, important improvements will be made before the RinasciMENTE program is tested under real-world conditions. Conducting usability testing is a crucial step at an early stage of the development process of an Internet-based self-help intervention to identify potential usability problems with the system.

Internet
usability testing
mental health
self-care
COVID-19
typesetterts19
cover-dateJanuary-December 2024
==== Body
pmcIntroduction

Internet-delivered self-help interventions and COVID-19

Internet-delivered interventions, also known as online interventions or e-Health interventions, have been extensively studied since their emergence in the late 1990s. Research has consistently shown that these interventions can be equally effective as face-to-face 1 in addressing a wide range of mental health disorders,2,3 not only in controlled trial settings but also in the context of routine psychiatric care.4,5

Internet-delivered interventions offer several advantages, including increased accessibility, flexibility in delivery, scalability, cost-effectiveness, and the ability to reach people who may not have access to traditional face-to-face interventions due to geographical, logistical, or stigma-related barriers. 6

Research has also documented the impact of the degree of involvement of the therapist, ranging from “pure” to “guided” self-help, on treatment outcomes. Self-help interventions involve individuals advancing independently through a diverse range of therapeutic material delivered via the Internet at their own convenience and speed.7–10 These interventions utilize various modalities, including AI chatbots, 11 and interventions featuring automated feedback websites.6,12 As this means that no human professionals are involved, Internet-delivered self-help interventions ensure discreet access to therapy, and it is easier and cheaper to widely implement. 13

This approach can be an effective treatment option for various psychological problems, including anxiety14,15 and depressive symptoms, 16 eating disorders, 17 and adjustment disorders. 18

Internet-based self-help interventions have gained increasing relevance and popularity in light of the COVID-19 pandemic, and research suggests that interest and utilization of such solutions continue to remain substantial. 19 With the increasing demand for mental health support, Internet-based self-help interventions offer a potential solution to these challenges.20–23 Their accessibility and effectiveness make self-help interventions valuable tools for providing support to individuals facing unprecedented difficulties, especially while adhering to social distancing measures.24–26

Furthermore, previous studies have consistently highlighted that Internet-based self-help interventions commonly rely on evidence-based techniques to help people cope with mental health disorders. Well-established methods such as traditional cognitive behavioral therapy (CBT) and more recent variations such as third-wave CBT, which includes mindfulness, acceptance, and commitment therapy, have received strong support as evidence-based techniques to provide Internet-based self-help interventions targeting mental health issues like anxiety and depression.27–29

Despite these advantages, compared to Internet-based interventions delivered through some form of guidance, self-guided programs exhibit low levels of adherence20,30 and higher dropout rates. 31

Since adherence is related to the outcome, it is important to consider all factors that could improve the completion of self-guided Internet-based interventions. 32 This starts with designing and evaluating the usability of the system during the development process.33,34

The need for usability assessment

Usability is defined as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” (p. 3). 35 Usability testing is a critical step in the development of Internet interventions as it solicits end-user feedback about what works, what does not, and where gaps in information or functionality exist to improve the website interface and content.36–40

Problems with the usability of technology, such as ineffective system design, lack of ease of use and convenience of access, and a mismatch between the system features and the needs, expectations, and characteristics of users.41,42 They are also likely to result in frustration and reduced user satisfaction when interacting with systems, 43 a higher probability of committing errors, and suboptimal clinical outcomes or refusal of the intervention. 44 This is particularly important for healthcare technologies, as their usability can affect the quality and effectiveness of treatment. 45 Even if other Internet-based self-management programs have conducted usability studies to help refine their systems,46,47 the number of Internet-based programs that publish their usability evaluation results remains scarce. 36 To identify issues related to system design and functionalities, this study evaluates the usability of the RinasciMENTE Internet-based self-help intervention.

Overview of the RinasciMENTE e-platform to promote emotional self-management among the community during the COVID-19 pandemic

This program has previously been described. 48 In summary, it consists of an Internet-based self-help intervention, based on CBT principles and techniques, specifically developed to address immediate distress and prevent the long-term psychological consequences of the COVID-19 pandemic among Italians living within the country and abroad. The content is organized into eight weekly e-sessions over a period of two months. After logging into the e-platform, users are asked to complete self-report measures, and a brief online meeting with a mental health professional is then scheduled. Subsequently, users can access contents that promote: (1) information about possible psychological consequences of COVID-19 and how CBT works, (2) understanding the connection between feelings and behaviors, (3) understanding how negative thoughts can influence mood and the use of alternative thinking strategies to manage cognitive traps, (4) acceptance, (5) stress management, (6) problem resolution, and (7) a plan of completion and maintenance. In addition, an eighth module is tailored to the specific needs of the users: (a) emotional education, (b) anxiety and exposition, (c) enduring anxiety, (d) social anxiety, (e) panic attack, (f) sleep quality, (g) perfectionism, (h) relaxation, and (i) management of default memories.

The e-platform proactively sends out scheduled emails to inform users that new content is available, and a weekly reminder containing a brief encouraging message is emailed to those who do not access the material.

Objective

This study aims to assess the usability of a self-guided Internet-provided intervention for emotional self-management in terms of user performance and satisfaction with the content, interface, and functionality of the program. The main goals were to: (1) collect qualitative and quantitative data on usability; and (2) identify usability problems.

Methods

Study design

The study was carried out between January 2021 and March 2021. It employed a mixed methods sequential explanatory design approach, comprising the following end-user testing: (1) collection and analysis of performance measures employing the think-aloud method, 49 (2) collection and analysis of quantitative data through the use of a System Usability Scale (SUS),50,51 and (3) collection and analysis of qualitative data through the use of an ad hoc semistructured interview to further explain the results obtained in the earlier stages. The study adhered to the Good Reporting of A Mixed Methods Study in health services research criteria. 52 Supplemental File 1 reports the COnsolidated criteria for REporting Qualitative research Checklist.

Ethical statement

The study was reviewed and approved by the Ethics Committee of the Catholic University of Milan (ID: 32–22). All procedures were carried out following the ethical standards of the institutional and/or national research committee and with the Helsinki Declaration and its subsequent amendments or comparable ethical standards. Participants gave their informed written consent to participate in this study.

Recruitment

Ten individuals who showed interest in participating in the RinasciMENTE project by writing to the email address behind the official website (https://www.iterapi.se/sites/rinascimente/) were consecutively asked to participate in the usability test based on inclusion criteria. The study was carried out using the platform that was specifically developed for the project. Participants were stratified according to sex and age to help ensure that the study sample was representative of the larger population studied (demographics are reported in Table 1). No participants declined to participate or withdrew from the study.

Table 1. Sociodemographic characteristics of participants.

Characteristic	Value	
Age (years), M (SD); range		
	40.7 (15.36)	
	20–70	
Sex, n (%)		
Male	5 (50%)	
Female	5 (50%)	
Education, n (%)		
Elementary school	1 (10%)	
Middle school	1 (10%)	
High school	2 (20%)	
University	6 (60%)	
n (%)		
Self-reported good computer skills	6 (60%)	
Self-reported good Internet skills	7 (70%)	
Prior usability test, n (%)		
Yes	3 (30%)	
No	7 (70%)	

To be eligible, participants had to (1) be ≥18 years, (2) be able to read and speak Italian, (3) have basic computer skills and Internet access, (4) own a computer or compatible device, and (5) provide informed consent to participate. Participants were excluded if they presented self-reported visual, hearing, or cognitive impairments that prevented them from following instructions.

Procedures

As a result of the COVID-19 pandemic, all procedures were performed online (completion time: 45  minutes, on average). First, each participant received an email with (a) the link to the online meeting, including the date and time, and (b) the link and access credentials of the RinasciMENTE platform. Before starting the usability test, the researcher provided detailed information about the test procedures and described the purpose of the computer-based self-management system. The participants were then asked to give their informed consent, provide demographic information, and report on their level of comfort with computers and the Internet.

They were also directly asked to self-report their computer and Internet skills using an ad hoc question and to tell if they had already taken a usability test before.

All participants were assigned a unique ID not related to their identity. Personal data was linked only to this unique ID and stored on a secure online platform accessible only by the research team and protected by a password.

No relationship between the researcher and the participants was established prior to the start of the study. The participants’ understanding of the researcher was limited to the reasons behind conducting the research.

Think-aloud and tasks competition

Participants were instructed to log into the system; and then perform a series of tasks on the platform while providing continuous commentary following the think-aloud method. 49 This is a common approach to usability testing that allows for the evaluation of the ease with which a system is learned. 53 Unlike a formal qualitative coding process that involves a thematic analysis to discover the main findings, 54 the think-aloud process was used to gather feedback on the prototype of the platform.

A facilitator guided each participant through three goal-oriented tasks (see Textbox 1) by presenting them and reminding (if needed) the participants to provide reactions to the features of the platform, navigation, and perceived utility by continuously speaking their thoughts and actions out loud. 55 Only in the case of failure to verbalize their thoughts aloud, the facilitator reminds the user, as studies have shown that active solicitation during the think-aloud process can interfere and have a strong impact on the reliability and validity of the results.56,57 Also, if the participant asked a question, the facilitator remained neutral and replied, “What do you think?” or “What would you do?” This allowed participants to explain their thoughts as they tried to work through the intervention and obtain real-time feedback and emotional responses to the intervention.

Textbox 1. Tasks.

Task 1	Task 2	Task 3	
Go to the “troubleshooting” treatment module, select “exercise 1” and verbalize what you are doing, thinking, and feeling while completing the task.	Go to the “dysfunctional thoughts” treatment module, select “exercise 1” and verbalize what you are doing, thinking, and feeling while completing the task.	Go to the “dysfunctional thoughts” treatment module, select “exercise 2” and verbalize what you are doing, thinking, and feeling while completing the task.	

Several performance measures were collected, including task completion rate, task completion time, error frequency, and frequency of help. The completion rate was defined as the percentage of participants who completed the task. The task completion time was the average time it took to complete the task. The amount of time that participants had to complete the tasks was not limited, but they were instructed to try their best to perform the tasks. They were also asked to report to the researcher/facilitator if they were unable to complete the tasks. The frequency of error was defined as the total number of errors made on the task by all of the participants who went through the task (errors included choosing a wrong button, being unable to find and interpret the information correctly, etc.). The participants were corrected and asked to try again when they made an error. The frequency of help was defined as the total number of times that all participants needed help with the task. The facilitator video-recorded the interviews and took systematic notes on a Word document while the participant provided feedback.

Before starting the observation with the test participants, a pilot test was conducted to verify any issues in the execution of tasks or other problems that could be resolved before involving the participants.

Questionnaire administration

At the end of the session, the Italian version SUS50,51,58,59 was administered to assess the perceived usability of the platform by users, with a particular focus on their satisfaction with the system (see Supplemental File 2). The SUS is a 10-item questionnaire with response options ranging from 1 (strongly disagree) to 5 (strongly agree). The total score varies between 0 and 100, with higher overall scores indicating a higher quality of usability and an overall usability score below 68 showing poor usability of the evaluated system. 60 Based on different studies in the literature, the SUS is an easy-to-use and reliable tool for distinguishing between usable and non-usable systems, even with small sample sizes. 61

User perceptions (semistructured interviews)

Lastly, ad hoc semistructured interviews with seven questions (see Textbox 2) were conducted by author MS to further explore the experiences of the participants with the platform—including their general opinions about the system, any difficulties encountered while navigating and completing tasks, and what they believed to be the strengths and weaknesses of the platform. The interview lasted about 15  minutes. The researcher recorded the interviews on video and made field notes during the interview. Transcripts were not returned to participants for comment, corrections, or feedback on the findings.

Textbox 2. Semistructured interviews.

1. Is the platform easy to navigate?	
2. Is the platform design appealing?	
3. Is the content presented in an engaging, interesting, and simple way?	
4. Is the information provided clear and appropriate?	
5. Does the platform provide relevant and interesting activities?	
6. What do you like most about the platform?	
7. What do you like least about the platform?	

Data analyses

The data was analyzed by two researchers (authors MS and GM) using both quantitative and qualitative methods. Quantitative data were analyzed using SPSS Version 24.0. 62 To summarize and describe the main characteristics of the sample, descriptive statistics, including means and standard deviations, were first used. Following, data from the performance measures (think-aloud) were extracted from the videos, and the means/frequencies were examined. The semistructured interviews were then recorded and transcribed verbatim. After transcription, researchers likely engaged in the thematic analysis of the interview content,63,64 thus identifying key themes, concepts, or patterns that emerged from participants’ responses to gain insight into their experiences, opinions, or behaviors. More in detail, the thematic analysis process began with data familiarization. Once researchers became familiar with the interview data by reading and rereading transcripts, they started the coding procedure by systematically identifying and labeling interesting patterns within the data. This involved highlighting text segments that were relevant to the research question or objectives. The codes were grouped based on similarities or relationships, forming potential themes. The researchers systematically searched for patterns, connections, and meanings within the coded data. Once potential themes have been identified, the researchers reviewed and refined them. Subsequently, they defined and clearly described each theme based on the patterns observed in the data, also providing illustrative examples. Each theme was given a descriptive and meaningful name.

Sample size determination

The sample size was based on considerations by Nielsen 65 and Virzi 66 according to which involving the first five users in usability testing is generally sufficient to identify a significant portion (95%) of usability problems. Then, these problems are expected to be confirmed by subsequent users, suggesting that the initial five users are representative enough to capture most of the issues. Furthermore, considering various factors such as the complexity of the system being tested, the diversity of the user population, and the specific goals of the testing process, a larger number of participants were included in the study to identify most of the usability problems. 67 The researchers reached a consensus on data saturation following their discussion.

Results

Characteristics of the participants

The participants were evenly distributed by sex, with a mean age of 40.7 years (SD = 15.36). They had a high level of education and most of the respondents self-reported good computer and Internet skills.

Performance measures

Half of the users (n = 5) did not experience difficulties in identifying the icon to access the intervention, while the remaining five users were confused in distinguishing between the command “Welcome” and “Login.” Further negative judgments were associated with the platform home pages, as follows: “I am confused, I don't understand the difference between the home page and the beginning of the intervention”; or “The beginning is not very clear, as an introductory part seems to be missing” (Figure 1). In contrast, positive judgments were made about the presentation of the treatment modules: “The page that presents the treatment modules is clear and intuitive” (Figure 2).

Figure 1. The RinasciMENTE welcome page.

Figure 2. The RinasciMENTE modules.

Task completion rate

The completion rate reflects the achievement of all points related to each exercise. The results reveal that only three participants fully completed all navigation tasks. Specifically, 30% of the users did not complete Task 1; 100% of the participants completed Task 2; and 90% of the sample completed Task 3. On average, a success rate equal to 73% was achieved in completing all three activities.

Task completion time

The average completion time of the three tasks was about 20  minutes, and the results showed that the ones that took the longest were users aged ≥45 years.

Frequency of error

Concerning the insertion of more answer options in a single line or the erroneous compilation of the tables required by the tasks, these errors were made by only three users (30% of the sample).

Frequency of help

Similarly, to the frequency of error, the finding revealed that only Tasks 1 and 3 produced a call for help (n = 1), which was made uniquely by users older than 45 years. Performance measures are reported in Table 2 below:

Table 2. Performance measures as assessed via three tasks.

Task	N	Task completion rate (%)	Mean task completion time (min)	Frequency of error nerror (n) a	Frequency of help nhelp (n) b	
1	10	70%	7.3	1 (2)	1 (1)	
2	10	100%	4.7	0	0	
3	10	90%	6.4	2 (1)	1 (1)	
a nerror(n) represents the number of times an error was made and n represents the number of people who made the error.

b nhelp(n) represents the number of times help was given and n represents the number of people who needed help.

Perceived usability outcomes

The results of perceived usability (as measured by the SUS) are presented in Table 3. Data analysis showed that the RinasciMENTE platform achieved an average score of 68.25, meaning that the platform does not present serious usability problems that need to be addressed. 60 The highest score was 77.5, reported by user #3, while the lowest score of 42.5 was achieved by user #9.

Table 3. Result of the System Usability Scale.

Item	P1 a	P2	P3	P4	P5	P6	P7	P8	P9	P10	
1	4	3	3	4	1	3	4	3	4	3	
2	2	3	1	3	1	1	2	2	3	2	
3	4	4	4	2	4	3	3	4	2	4	
4	3	3	1	1	1	1	4	4	4	3	
5	5	3	3	2	1	1	5	4	4	4	
6	2	2	2	1	3	2	1	2	3	2	
7	5	4	5	3	5	5	4	4	4	3	
8	3	1	3	3	1	1	2	2	4	1	
9	3	4	4	4	3	3	4	3	1	5	
10	1	1	1	2	1	1	1	2	4	1	
Odd score	21	18	19	15	14	15	20	18	15	19	
#X	16	13	14	10	9	10	15	13	10	14	
Even score	11	10	8	10	7	6	10	12	18	9	
Y	14	15	17	15	18	19	15	13	7	16	
X + Y = Z	30	28	31	25	27	29	30	26	17	30	
Z × 2.5	75	70	77.5	62.5	67.5	72.5	75	65	42.5	75	
a P1–10: participant 1–10; Item: 1. I think that I would like to use this intervention; 2. I found the intervention unnecessarily complex; 3. I found the intervention easy to use; 4. I think I would need support from a technical person to use this intervention; 5. I found the various functions in this intervention were well integrated; 6. I thought there was too much inconsistency in this intervention; 7. I would imagine that most people would learn to use this intervention quickly; 8. I found the intervention very cumbersome to use; 9. I felt very confident using the intervention; 10. I needed to learn a lot of things before I could start using this intervention.

According to the classification of Bangor et al.’s, 68 60% of the respondents scored higher than 70, thus rating the platform as “good”; 30% of the users achieved an SUS total score ranging from 62.5 to 67.5, suggesting that usability improvements are needed; and only 10% of the respondents (n = 1) classified the platform as “poor” (SUS score ≤ 51). Those who achieved the higher scores had a mean age of 37.5 (range 22–66) years and were equally distributed between males and females. The lower score was, instead, shown by a female respondent aged 61 years.

Of the 10 participants, seven indicated the RinasciMENTE platform as “easy to use,” and eight respondents stated that “most people would learn to use the intervention quickly” (see Figure 1).

Regarding the need for technical support to use the platform, the results showed good usability of the system, as seven out of the 10 respondents disagreed or strongly disagreed with the need for help; and six users indicated that the system was not complex or cumbersome. Still, two respondents did not “feel very confident using the intervention.” Remarkably, none of the participants was “strongly in agreement” with the negatively worded questions.

Responses to the semistructured interview

The qualitative analysis of the semistructured interviews was conducted, and three main themes were derived from the data: general thoughts about the platform; weaknesses of the platform and difficulties encountered while navigating and completing tasks, and strengths of the platform.

Theme 1—General thought about the platform

In general, the platform was perceived as clear by users; however, certain aspects revealed areas for improvement. Excerpts of dialog below illustrate instances where users identified limitations of the platform. Doubts were mostly related to the amount of information, which appeared excessively noisy for each module, but scant in the introductory part, where participants expected to receive more details.The platform is clear and seems easy to use; just some points are not perfectly comprehensible, but overall—I find it a useful program.

The first page is not very clear, there is no introductory part that I expected, but overall, the platform is not difficult to use. I find it rich in information, but I would only remember the main information. Anyway, I find it to be functional (the program) for its purpose if used consistently.

It seems incomplete; I was expecting more information mainly on the homepage in the ‘Welcome’ section.

Theme 2: Weaknesses of the platform and difficulties encountered while navigating and completing tasks

The main difficulties concerned the first web page, for which doubts emerged with the login and the homepage.I found it difficult to navigate, I did not understand the difference between ‘Go to the module’ and ‘Homepage’ commands after reading the instructions.

At the beginning, I couldn't find the ‘login’ command, then I had trouble understanding what the ‘Selected’ entry meant. Therefore, I reread the instructions but I did not understand them, and I asked for help. Furthermore, I did not know how to complete the exercise because I did not understand the example.

Theme 3—Strengths of the platform

From the interviews, it emerged how important the graphic part is, including colors and images. In terms of content, the presence of examples also helped to make the platform more usable.It is a useful platform for its functionalities; I find the setup of the treatment modules clear, especially for the use of images; in general, I think the platform is very intuitive. It is easy to understand how to use it.

I appreciated the clarity of the content and the presence of examples that help to understand how to complete the exercises.

In my opinion, the positive aspects of the platform are its colors, and graphics that are simple and clean, so they can help people focus on appropriate information and not irrelevant details.

Discussion

Principal results

This study investigated the usability of the RinasciMENTE platform by evaluating the interaction between users and the system through a set of quantitative–qualitative tools. 69 According to Nielsen's assumption, 65 the main problems with the system were encountered by the first five users, while subsequent users only confirmed them, and no further difficulty was detected.

Overall, the results show that most of the participants were able to perform the tasks without additional time and resource expenditures. This reflects the “practice of simplicity” prerequisite of an Internet-based intervention, which is to be as simple as possible, allowing users to find what they are looking for quickly and logically. 65

However, older users (45 years) experienced more difficulties, as they took longer, made more mistakes and needed help in carrying out the tasks compared to younger people. These findings are consistent with previous studies investigating the influence of age on usability testing.70–72 Van Deursen's investigations73,74 showed that older people display lower operational and navigational skills more often than their younger counterparts. This discrepancy could be attributed to the lack of practical digital skills of older participants rather than to the design of the platform and could be overcome by offering them introductory sessions on the platform to help them through the program for the first time. 72

In this study, the participants were very positive about the aesthetics and layout. This is an important outcome, as studies claimed that visual elements, including graphics, are the first things that grab users’ attention by creating their first impression.75,76 However, some areas that could be improved to further enhance the system were identified. First, the homepage appeared unclear and confusing to some respondents. The participants also indicated that they felt overwhelmed by the amount of information reported on the homepage. In fact, text-heavy web pages can make content difficult to process, acting as a barrier to intervention adherence, 77 and comprehensibility of website design should be considered to “reduce the cognitive effort required to effectively use eHealth applications.” 78 Ultimately, the SUS revealed that the program was rated “good,” indicating that the intervention was considered useful.

These findings are similar to those of other usability studies of digital interventions that employed CBT techniques to address stress due to COVID-19.79–82 Specifically, a study aimed to evaluate the feasibility, acceptance, and usability of a resilience app on a sample of 75 students revealed a satisfactory degree of usability. 83 Analogous results were obtained by a previous test of a digital mental health tool for essential workers and those unemployed due to COVID-19. 81 Moreover, the results of a recent meta-analysis of 16 studies indicate that digital mental health treatments could be feasible and beneficial for various groups, including the general public, those at risk, and individuals with existing mental health conditions in both high- and middle-income nations. 84 Still, studies evaluating the usability of digital tools differ in the technologies employed—such as mobile apps, web-based programs, virtual reality environments, or wearable devices—the methods used to assess their usability, including user surveys, interviews, usability testing, or qualitative analysis of user feedback, and outcome measures (e.g. efficiency, effectiveness, learnability, memorability, errors, and subjective perceptions of usability). Other sources of variability are represented by the sample size and characteristics, such as the clinical or nonclinical population, age group, levels of digital literacy, technological preferences, access to digital devices or Internet connectivity, or different aspects of the user experience being investigated (e.g. ease of navigation, clarity of instructions, visual design, interactivity, responsiveness, and overall satisfaction).

Given this variability, the findings may lead to different recommendations to improve the usability of digital interventions, such as redesign of the interface, customization of content, technical support, or integration with existing healthcare services.

Multi-method usability testing is, therefore, essential to ensure the relevance of the content and discover intervention improvement needs (e.g. more clear buttons) and for content inadequacy (e.g. inconsistencies, lengthy text).

Participants saw value in the RinasciMENTE intervention and confirmed their interest in its use to self-treat symptoms once it was available.

Improvements to the usability of the internet-based intervention will be made before going to the full trial.

Strengths and limitations

A strength of this study is the use of a mixed-methods design to identify potential usability problems more reliably. 85 Furthermore, for the present usability study, the methodology was carefully set based on systematic usability test guidelines and empirical research.65,86 Therefore, this study provides an effective method to identify elements that require modification for solutions delivered through the Internet. These results, indeed, further highlight the importance of performing a usability test during a system development process to avoid drawing incorrect conclusions about the effectiveness of a given intervention. Internet-based self-management programs have the potential to improve the quality of care at a relatively low cost, but, if their content and/or design are perceived as unpleasant by users, this might negatively impact health outcomes regardless of the quality of the intervention provided. Despite the small sample, the number of participants meets the feasibility study recommendations for feasibility studies, 65 and it can be deemed representative of the population of potential end users of the RinasciMENTE program.

It is also important to acknowledge the limitations of this study. First, only participants from the general population were included in the study. In addition to testing the platform with a community sample, providing valuable insights from a range of perspectives, which can help identify usability issues that may not be apparent when testing with a clinical population alone, the inclusion of both clinical and nonclinical populations would have enhanced the validity, generalizability, and inclusivity of findings, ultimately improving the effectiveness of the online platform for mental health support across diverse user groups. Another limitation of this study is that, due to its nature, it did not allow the inclusion of subjects who did not have computer and Internet access at home, which could affect the representativeness of the sample. Furthermore, since usability testing was done online, it is important to take into account the inability to account for the environmental factors that vary between testing locations. 87 User characteristics (e.g. age, sex, and computer experience) may also have affected measurement outcomes.

Future research and implications

The feasibility and effectiveness of the RinasciMENTE program will be evaluated in a randomized controlled trial design. 48 A community sample that experienced mild to moderate psychological problems during the COVID-19 pandemic will be selected for inclusion based on the selected criteria. Individuals will be randomly assigned to a group receiving the CBT intervention or placed on a waiting list control. The effect of the RinasciMENTE program on selected primary and secondary psychological outcomes will be tested at the end of the intervention (two months) and at six- and 12-month follow-ups.

Conclusions

Internet-based technologies can play an important role in helping people better manage their psychological symptoms. However, to increase end-user satisfaction with the interventions and prevent them from committing errors or having difficulties using the system, the program must be tailored to the characteristics, needs, and preferences of the end-users. The methods used in the present usability study were effective in identifying elements that needed modification, and significant improvements will be made before the RinasciMENTE program is deemed ready for testing under real-world conditions to go full trial.

Supplemental Material

sj-pdf-1-dhj-10.1177_20552076241258419 - Supplemental material for Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing

Supplemental material, sj-pdf-1-dhj-10.1177_20552076241258419 for Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing by Michelle Semonella, Gloria Marchesi, Gianluca Castelnuovo, Gerhard Andersson and Giada Pietrabissa in DIGITAL HEALTH

sj-pdf-2-dhj-10.1177_20552076241258419 - Supplemental material for Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing

Supplemental material, sj-pdf-2-dhj-10.1177_20552076241258419 for Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing by Michelle Semonella, Gloria Marchesi, Gianluca Castelnuovo, Gerhard Andersson and Giada Pietrabissa in DIGITAL HEALTH

Acknowledgments

We thank all of those with whom I have had the pleasure to work during this and other related projects.

Authors contribution: MS and GP conceptualized and designed. MS coordinated the study's recruitment and conducted interviews. GM and MS coded and analyzed the interviews under the supervision of GP. MS and GP wrote the manuscript. All authors participated in the critical review of the manuscript, and read and approved its final version.

Conflict of interest: The authors declare that the research was conducted in the absence of any commercial or financial relationship that could be construed as a potential conflict of interest.

Data availability statement: The dataset for this study is available upon request to interested researchers.

Ethical approval: The study was reviewed and approved by the Ethics Committee of the Catholic University of Milan (ID: 32–22). All procedures were carried out following the ethical standards of the institutional and/or national research committee and with the Helsinki Declaration and its subsequent amendments or comparable ethical standards.

Funding: The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Italian Ministry of Health - Ricerca Corrente.

Guarantor: Giada Pietrabissa.

Supplemental material: Supplemental material for this article is available online.

ORCID iDs: Michelle Semonella https://orcid.org/0000-0003-0284-0575

Giada Pietrabissa https://orcid.org/0000-0002-5911-5748
==== Refs
References

1 Hedman-Lagerlöf E Carlbring P Svärdman F , et al. Therapist-supported internet-based cognitive behaviour therapy yields similar effects as face-to-face therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. World Psychiatry 2023; 22 : 305–314.37159350
2 Andersson G Titov N Dear BF , et al. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry 2019; 18 : 20–28.30600624
3 Folker AP Mathiasen K Lauridsen SM , et al. Implementing internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv 2018; 11 : 60–70.30135761
4 Hedman E Ljotsson B Kaldo V , et al. Effectiveness of internet-based cognitive behaviour therapy for depression in routine psychiatric care. J Affect Disord 2014; 155 : 49–58.24238951
5 Mathiasen K Riper H Andersen TE , et al. Guided internet-based cognitive behavioral therapy for adult depression and anxiety in routine secondary care: observational study. J Med Internet Res 2018; 20 : e10927.
6 Wang Q Zhang W An S . A systematic review and meta-analysis of internet-based self-help interventions for mental health among adolescents and college students. Internet Interv 2023; 34 : 100690.38023965
7 Soklaridis S Lin E Lalani Y , et al. Mental health interventions and supports during COVID-19 and other medical pandemics: a rapid systematic review of the evidence. Gen Hosp Psychiatry 2020; 66 : 133–146.32858431
8 Wang C Pan R Wan X , et al. Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China. Int J Environ Res Public Health 2020; 17 .: 1-25
9 Wind TR Rijkeboer M Andersson G , et al. The COVID-19 pandemic: the ‘black swan’ for mental health care and a turning point for e-health. Internet Interv 2020; 20 : 100317.32289019
10 Andersson G Titov N . Advantages and limitations of internet-based interventions for common mental disorders. World Psychiatry 2014; 13 : 4–11.24497236
11 Liu H Peng H Song X , et al. Using AI chatbots to provide self-help depression interventions for university students: a randomized trial of effectiveness. Internet Interv 2022; 27 : 100495.35059305
12 Kim H Choi Y . A practical development protocol for evidence-based digital integrative arts therapy content in public mental health services: digital transformation of mandala art therapy. Front Public Health 2023; 11 : 1175093.: 1-1537841732
13 Barak A Klein B Proudfoot JG . Defining internet-supported therapeutic interventions. Ann Behav Med 2009; 38 : 4–17.19787305
14 van Ballegooijen W Riper H Klein B , et al. An internet-based guided self-help intervention for panic symptoms: randomized controlled trial. J Med Internet Res 2013; 15 : e154.
15 Kahlke F Berger T Schulz A , et al. Efficacy of an unguided internet-based self-help intervention for social anxiety disorder in university students: a randomized controlled trial. Int J Methods Psychiatr Res 2019; 28 : e1766.
16 Cuijpers P Donker T Johansson R , et al. Self-guided psychological treatment for depressive symptoms: a meta-analysis. PLoS One 2011; 6 : e21274.
17 Kass AE Trockel M Safer DL , et al. Internet-based preventive intervention for reducing eating disorder risk: a randomized controlled trial comparing guided with unguided self-help. Behav Res Ther 2014; 63 : 90–98.25461783
18 Eimontas J Rimsaite Z Gegieckaite G , et al. Internet-based self-help intervention for ICD-11 adjustment disorder: preliminary findings. Psychiatr Q 2018; 89 : 451–460.29124500
19 Dominiak M Gedek A Antosik AZ , et al. Prevalence, attitudes and concerns toward telepsychiatry and mobile health self-management tools among patients with mental disorders during and after the COVID-19 pandemic: a nationwide survey in Poland from 2020 to 2023. Front Psychiatry 2023; 14 : 1322695.38260801
20 Newman MG Szkodny LE Llera SJ , et al. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: is human contact necessary for therapeutic efficacy? Clin Psychol Rev 2011; 31 : 89–103.21130939
21 Clarke G Kelleher C Hornbrook M , et al. Randomized effectiveness trial of an internet, pure self-help, cognitive behavioral intervention for depressive symptoms in young adults. Cogn Behav Ther 2009; 38 : 222–234.19440896
22 Pietrabissa G Simpson SG . Psychological consequences of social isolation during COVID-19 outbreak. Front Psychol 2020; 11 : 2201.33013572
23 Giusti EM Pedroli E D'Aniello GE , et al. The psychological impact of the COVID-19 outbreak on health professionals: a cross-sectional study. Front Psychol 2020; 11 : 1684.32754102
24 Semonella M Andersson G Dekel R , et al. Making a virtue out of necessity: COVID-19 as a catalyst for applying internet-based psychological interventions for informal caregivers. Front Psychol 2022; 13 : 856016.35465576
25 Castelnuovo G Pietrabissa G Manzoni GM , et al. Fighting the COVID-19 pandemic using the technology-based second-line in Italy and Lombardy: the urgent need of home-based remote monitoring systems to avoid the collapse of the hospital-centred first line. J Glob Health 2020; 10 : 010371.
26 Simpson S Richardson L Pietrabissa G , et al. Videotherapy and therapeutic alliance in the age of COVID-19. Clin Psychol Psychother 2021; 28 : 409–421.33037682
27 Waters L Algoe SB Dutton J , et al. Positive psychology in a pandemic: buffering, bolstering, and building mental health. J Posit Psychol 2022; 17 : 303–323.
28 Heber E Ebert DD Lehr D , et al. The benefit of web- and computer-based interventions for stress: a systematic review and meta-analysis. J Med Internet Res 2017; 19 : e32.
29 Harty S Enrique A Akkol-Solakoglu S , et al. Implementing digital mental health interventions at scale: one-year evaluation of a national digital CBT service in Ireland. Int J Ment Health Syst 2023; 17 : 29.37817270
30 Rodrigues TR C Reijnders T Breeman LD , et al. Use intention and user expectations of human-supported and self-help eHealth interventions: internet-based randomized controlled trial. JMIR Form Res 2024; 8 : e38803.
31 Karyotaki E Kleiboer A Smit F , et al. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol Med 2015; 45 : 2717–2726.25881626
32 Kazlauskas E Eimontas J Olff M , et al. Adherence predictors in internet-delivered self-help intervention for life stressors-related adjustment disorder. Front Psychiatry 2020; 11 : 137.32231597
33 Tindale WB Dimitri P . Medtech innovation across the life course - the importance of users and usability. J Med Eng Technol 2022; 46 : 427–432.36074637
34 Stoddard JL Augustson EM Mabry PL . The importance of usability testing in the development of an internet-based smoking cessation treatment resource. Nicotine Tob Res 2006; 8 : S87–S93.
35 ISO 9241-11. Ergonomics of Human-System Interaction - Part 11: Usability: Definitions and Concepts. International Organization for Standardization. 2018. https://www.iso.org/standard/63500.html [accessed 2023-01-10].
36 Maramba I Chatterjee A Newman C . Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019; 126 : 95–104.31029270
37 Chang H . Evaluation framework for telemedicine using the logical framework approach and a fishbone diagram. Healthc Inform Res 2015; 21 : 230–238.26618028
38 Sadegh SS Khakshour Saadat P Sepehri MM , et al. A framework for m-health service development and success evaluation. Int J Med Inform 2018; 112 : 123–130.29500009
39 Harte R Glynn L Rodríguez-Molinero A , et al. A human-centered design methodology to enhance the usability, human factors, and user experience of connected health systems: a three-phase methodology. JMIR human Factors 2017; 4 : e8.
40 National Institute of Standards and Technology. NIST guide to the processes approach for improving the usability of electronic health records. CreateSpace Independent Publishing Platform, 2010, pp. 17–27.
41 Galavi Z Montazeri M Ahmadian L . Barriers and challenges of using health information technology in home care: a systematic review. Int J Health Plann Manage 2022; 37 : 2542–2568.35643906
42 Wallin E Norlund F Olsson EMG , et al. Treatment activity, user satisfaction, and experienced usability of internet-based cognitive behavioral therapy for adults with depression and anxiety after a myocardial infarction: mixed-methods study. J Med Internet Res 2018; 20 : e87.
43 Bevan N Kirakowski J Maissel J . What is usability? In:Proceedings of the 4th International Conference on HCI; Stuttgart, Germany: Proceedings of the 4th International Conference on HCI, 1991.
44 Jacko JA Sears A Sorensen SJ . Framework for usability: healthcare professionals and the internet. Ergonomics 2001; 44 : 989–1007.11693249
45 Wilks CR Yin Q Zuromski KL . User experience affects dropout from internet-delivered dialectical behavior therapy. Telemed J E Health 2020; 26 : 794–797.31502945
46 Kenter RMF Schonning A Inal Y . Internet-delivered self-help for adults with ADHD (MyADHD): usability study. JMIR Form Res 2022; 6 : e37137.
47 Nitsch M Dimopoulos CN Flaschberger E , et al. A guided online and mobile self-help program for individuals with eating disorders: an iterative engagement and usability study. J Med Internet Res 2016; 18 : e7.
48 Bertuzzi V Semonella M Andersson G , et al. Study protocol for a randomized controlled trial evaluating the effectiveness of an internet-based self-help intervention to cope with psychological distress due to COVID-19 in the Italian general population: the RinasciMENTE project. Trials 2022; 23 : 801.36153586
49 Someren M Barnard Y Sandberg J . The think aloud method - a practical guide to modelling cognitive processes. London, England: Academic Press, 1994.
50 Brooke J . SUS-A quick and dirty usability scale. In Usability evaluation in industry. London, England: CRC Press, 1996; 189: 4–7.
51 Mobilio V . La sperimentazione di piattaforme. Innovazione 2006; 10 : 7–8.
52 O'Cathain A Murphy E Nicholl J . The quality of mixed methods studies in health services research. J Health Serv Res Policy 2008; 13 : 92–98.
53 Brinck T Gergle D Wood SD . Usability for the web: designing web sites that work. San Francisco, CA: Elsevier, 2001.
54 Nielsen Norman Group. Thematic analysis of qualitative user research data. https://www.nngroup.com/videos/thematic-analysis-qualitative-user-research-data/ [accessed 2023-01-04]
55 Fan M Lin J Chung C , et al. Concurrent think-aloud verbalizations and usability problems. ACM Trans Comput-Hum Interact 2019; 26 : 1–35.
56 Boren T Ramey J . Thinking aloud: reconciling theory and practice. IEEE Trans Profess Commun 2000; 43 : 261–278.
57 Nisbett RE Wilson TD . Telling more than we can know: verbal reports on mental processes. Psychol Rev 1977; 84 : 231–259.
58 Gruppo di Lavoro per l'Usabilità. eGLU 1.0 - Protocollo per l'esplorazione dei siti web delle Pubbliche Amministrazioni 2013.
59 Federici S Borsci S Meloni F . Le misure dell'usabilità: sstudio sulle caratteristichepsicometriche del QUIS e del SUMI nella versioneitaliana. Giornale di Psicologia 2009; 3 : 233–253.
60 Bangor A Kortum PT Miller JT . An empirical evaluation of the system usability scale. Int J Hum-Comput Interact 2008; 24 : 574–594.
61 System Usability Scale. Usability.Gov. https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html [accessed 2024-01-04].
62 IBM Corp . IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp., 2016.
63 Braun V Clarke V . Using thematic analysis in psychology. Qual Res Psychol 2006; 3 : 77–101.
64 Braun V Clarke V . Successful qualitative research: a practical guide for beginners. London, England: SAGE Publications Ltd 2013: 1-400.
65 Nielsen J . Usability engineering. U.S.: Morgan Kaufmann, 1994.
66 Virzi RA . Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors 1992; 34 : 457–468.
67 Faulkner L . Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35 : 379–383.14587545
68 Bangor A Kortum P Miller J . Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4 : 114–123.
69 Church A Fish R Haines-Young R , et al . UK National Ecosystem Assessment Follow-On. Work Package Report 5: Cultural ecosystem services and indicators; Cambridge, UK 2014.
70 Sonderegger A Schmutz S Sauer J . The influence of age in usability testing. Appl Ergon 2016; 52 : 291–300.26360221
71 Sauer J Sonderegger A Schmutz S . Usability, user experience and accessibility: towards an integrative model. Ergonomics 2020; 63 : 1207–1220.32450782
72 van der Vaart R van Driel D Pronk K , et al. The role of age, education, and digital health literacy in the usability of internet-based cognitive behavioral therapy for chronic pain: mixed methods study. JMIR Form Res 2019; 3 : e12883.
73 van Deursen AJ van Dijk JA . Internet skills performance tests: are people ready for eHealth? J Med Internet Res 2011; 13 : e35.
74 van Deursen AJAM van Dijk JAGM . Internet skill levels increase, but gaps widen: a longitudinal cross-sectional analysis (2010–2013) among the Dutch population. Information. Commun Soc 2015; 18 : 782–797.
75 Liu CH Chiu HP Hsieh CL , et al. Optimizing the usability of mobile phones for individuals who are deaf. Assist Technol 2010; 22 : 115–127.20698429
76 Gronier G . Measuring the first impression: testing the validity of the 5 second test. J Usability Stud 2016; 12 : 8-25
77 Eccles H Nannarone M Lashewicz B , et al. Barriers to the use of web-based mental health programs for preventing depression: qualitative study. JMIR Form Res 2021; 5 : e16949.
78 Rotondi AJ Spring MR Hanusa BH , et al. Designing eHealth applications to reduce cognitive effort for persons with severe mental illness: page complexity, navigation simplicity, and comprehensibility. JMIR Hum Factors 2017; 4 : e1.
79 Semonella M Marchesi G Andersson G , et al. Usability study of SOSteniamoci: an internet-based intervention platform to support informal caregivers in Italy. Digit Health 2024; 10 : 20552076231225082.38235418
80 Newton A Bagnell A Rosychuk R , et al. A mobile phone-based app for use during cognitive behavioral therapy for adolescents with anxiety (MindClimb): user-centered design and usability study. JMIR Mhealth Uhealth 2020; 8 : e18439.
81 Mata-Greve F Johnson M Pullmann MD , et al. Mental health and the perceived usability of digital mental health tools among essential workers and people unemployed due to COVID-19: cross-sectional survey study. JMIR Ment Health 2021; 8 : e28360.
82 Minian N Gayapersad A Saiva A , et al. An e-mental health resource for COVID-19-associated stress reduction: mixed methods study of reach, usability, and user perceptions. JMIR Ment Health 2022; 9 : e39885.
83 Nicolaidou I Aristeidis L Lambrinos L . A gamified app for supporting undergraduate students’ mental health: a feasibility and usability study. Digit Health 2022; 8 : 20552076221109059.35756831
84 Zhong S Yang X Pan Z , et al. The usability, feasibility, acceptability, and efficacy of digital mental health services in the COVID-19 pandemic: scoping review, systematic review, and meta-analysis. JMIR Public Health Surveill 2023; 9 : e43730.
85 Jaspers MW . A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 : 340–353.19046928
86 Yen PY Bakken S . Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 : 413–422.21828224
87 Tuckson RV Edmunds M Hodgkins ML . Telehealth. N Engl J Med 2017; 377 : 1585–1592.29045204
