German Version of the Telehealth Usability Questionnaire and Derived Short Questionnaires for Usability and Perceived Usefulness in Health Care Assessment in Telehealth and Digital Therapeutics: Instrument Validation Study

Kopka M., Slagman A., Schorr C., Krampe H., Altendorf M., Balzer F., Bolanaki M., Kuschick D., Möckel M., Napierala H., Scatturin L., Schmidt K., Thissen A., Schmieding M.L.

Tacke T., Nohl-Deryk P., Lingwal N., Reimer L.M., Starnecker F., Güthlin C., Gerlach F.M., Schunkert H., Jonas S., Müller A.

Objective In Germany, only a few standardized evaluation tools for assessing the usability of mobile Health apps exist so far. This study aimed to translate and validate the English patient version for standalone apps of the mHealth App Usability Questionnaire (MAUQ) into a German version. Methods Following scientific guidelines for translation and cross-cultural adaptation, the patient version for standalone apps was forward and back-translated from English into German by an expert panel. In total, 53 participants who were recruited as part of the beta testing process of the recently developed mHealth app HerzFit, answered the questions of the German version of the MAUQ (GER-MAUQ) and the System Usability Scale. Subsequently, a descriptive as well as a psychometric analysis was performed to test validity and reliability. Results After conducting three cognitive interviews, five items were modified. The values for Cronbach alpha for the entire questionnaire and the three subscales (0.966, 0.814, 0.910, and 0.909) indicate strong internal consistency. The correlation analysis revealed that the scores of the GER-MAUQ, the subscales and the SUS were strongly correlated with each other. The correlation coefficient of the SUS and the GER-MAUQ overall score was r = 0.854, P < 0.001 and the coefficients of the subscales and the SUS were r = 0.642, P < 0.001; r = 0.866, P < 0.001 and r = 0.643, P < 0.001. Conclusions We have developed a novel German version of the MAUQ and demonstrated it as a reliable and valid measurement tool for assessing the usability of standalone mHealth apps from the patients’ perspective. The GER-MAUQ allows a new form of standardized assessment of usability of mHealth apps for patients with cardiovascular disease in Germany. Further research with a larger sample and other samples is recommended.

Santos M.R., Malaguti C., Cabral L.A., Soares A.L., Neves L.H., Sena L.D., Parmanto B., Sauers A.L., José A., Oliveira C.C.

Moorthy P., Weinert L., Harms B.C., Anders C., Siegel F.

Background Good usability is important for the adoption and continued use of mobile health (mHealth) apps. In particular, high usability can support intuitive use by patients, which improves compliance and increases the app’s effectiveness. However, many usability studies do not use adequate tools to measure perceived usability. The mHealth App Usability Questionnaire (MAUQ) was developed specifically for end users in a medical context. MAUQ is a relatively new but increasingly used questionnaire to evaluate mHealth apps, but it is not yet available in German. Objective This study aims to translate MAUQ into German and determine its internal consistency, reliability, and construct validity. Methods This validation study was conducted as part of a usability evaluation project for an mHealth app used as a therapy support tool during breast cancer chemotherapy. MAUQ was translated into German through a rigorous forward-backward translation process, ensuring semantic and conceptual equivalence. Patient responses to MAUQ and System Usability Scale (SUS) were analyzed for validation. Descriptive analysis was performed for the MAUQ subscales and SUS standard scores. Significance tests and correlation coefficients assessed the relationship between the SUS and MAUQ results, confirming construct validity. Internal consistency was assessed for item reliability and consistency in measuring the target construct. Free-text questions assessed translation comprehensibility, with responses analyzed descriptively and qualitatively using content analysis. Results In this study, 133 participants responded to the questionnaire, and the validation analysis showed substantially positive correlations between the overall MAUQ score and its subscales: ease of use (r=0.56), interface and satisfaction (r=0.75), and usefulness (r=0.83). These findings support the construct validity of MAUQ and emphasize the importance of these subscales in assessing the usability of the Enable app. The correlation coefficients ranging from 0.39 to 0.68 for the items further validate the questionnaire by aligning with the overall score and capturing the intended concept. The high internal consistency reliability of MAUQ (Cronbach α=.81) and its subscales further enhances the instrument’s robustness in accurately evaluating the usability of mHealth apps. Conclusions We successfully validated the German translation of the MAUQ for stand-alone apps using a standardized approach in a cohort of patients with breast cancer. In our validation study, MAUQ exhibited strong internal consistency reliability (Cronbach α=.81) across its subscales, indicating reliable and consistent measurement. Furthermore, a significant positive correlation (P<.001) was found between the subscales and the overall score, supporting their consistent measurement of the intended construct. Therefore, MAUQ can be considered a reliable instrument for assessing the usability of mHealth apps among German-speaking adults. The availability of the German version of MAUQ will help other researchers in conducting usability studies of mHealth apps in German-speaking cohorts and allow for international comparability of their results.

Brix T.J., Janssen A., Storck M., Varghese J.

The System Usability Scale (SUS) is a reliable tool for usability measurement and evaluation. Since its original language is English, a translation is required before a target group can answer it in their native language. The challenge of translating questionnaires lies in the preservation of its original properties. Different versions of a German SUS have been proposed and are currently in use. Objective of this work is to find and compare available German translations. Four versions were found and compared in terms of the translation process and the exact wording of the translation. Only the version of Gao et al. has been systematically validated, but has an unnatural wording. Although not validated yet, the proposed version of Rummel et al. is a good compromise between wording and methodically clean development. The version of Lohmann and Schäffer is the close runner up, as it may improve the wording at the expense of methodological accuracy. Since the version of Rauer gives no information about its translation process, it is considered least preferred of the four compared translations.

Zimmermann J., Heilmann M.L., Fisch-Jessen M., Hauch H., Kruempelmann S., Moeller H., Nagel L., Nathrath M., Vaillant V., Voelker T., Deckers M.J.

Pediatric palliative home care (PPHC) provides care for children, adolescents, and young adults with life-limiting illnesses in their own homes. Home care often requires long travel times for the PPHC team, which is available to the families 24/7 during crises. The complementary use of telehealth may improve the quality of care. In this pilot study we identify the needs and concerns of patients, teams, and other stakeholders regarding the introduction of telehealth. As a first step, focus groups were conducted in three teams. For the second step, semi-structured interviews were conducted with patients and their families (n = 15). Both steps were accompanied by quantitative surveys (mixed methods approach). The qualitative data were analyzed using content analysis. A total of 11 needs were identified, which were prioritized differently. Highest priority was given to: data transmission, video consultation, access to patient records, symptom questionnaires, and communication support. The concerns identified were related to the assumption of deterioration of the status quo. Potential causes of deterioration were thought to be the negative impact on patient care, inappropriate user behavior, or a high level of technical requirements. As a conclusion, we define six recommendations for telehealth in PPHC.

von Rohr S., Knitza J., Grahammer M., Schmalzing M., Kuhn S., Schett G., Ramming A., Labinsky H.

AbstractWe aimed to investigate (1) student-led clinics and (2) electronic patient-reported outcomes (ePROs) to accelerate diagnosis and treatment of patients with axial spondyloarthritis (axSpA). Patients with suspected axSpA completed an initial student-led clinic visit (T-1) prior to their planned actual rheumatologist visit (T0). Acceleration of patient appointment and NSAID therapy start, availability of diagnostic findings, and treatment response at T0 were evaluated. Beginning at T-1, patients completed electronic BASDAI questionnaires every 2 weeks. Concordance of paper-based and electronic BASDAI was evaluated. Patient acceptance of ePRO reporting and student-led clinics was measured using the net promoter score (NPS). 17/36 (47.2%) included patients were diagnosed with axSpA. Student-led clinics (T-1) significantly accelerated patient appointments by more than 2 months (T0, T-1, p < 0.0001) and axSpA guideline-conform NSAID treatment (p < 0.0001). At T0, diagnostic workup was completed for all patients and 7/17 (41.2%) axSpA patients presented with a clinically important improvement or were in remission. 34/36 (94.4%) patients completed at least 80% of the ePROs between T-1 and T0. Electronic and paper-administered BASDAI correlated well (r = 0.8 p < 0.0001). Student-led clinics and ePROs were well accepted by patients with NPS scores of + 62.0% (mean ± SD 9.2/10.0 ± 0.9) and + 30.5% (mean ± SD 8.0/10.0 ± 1.7), respectively. In conclusion, student-led clinics and ePRO monitoring were well accepted, accelerated diagnostic workup and treatment in patients with axSpA.

Kopka M., Slagman A., Schorr C., Krampe H., Altendorf M.B., Balzer F., Bolanaki M., Kuschick D., Möckel M., Napierala H., Scatturin L., Schmidt K., Thissen A., Schmieding M.

Background: The use of mobile health applications is increasingly common among the general public. More recently, healthcare systems incorporate such tools in their service workflows, be it as guidance to the most appropriate care setting or as prescription digital therapeutics. With such apps percolating into the classic healthcare sector, the necessity of sound and standardized evaluation grows. The mHealth App Usability Questionnaire (MAUQ) provides a novel and custom-tailored psychometrically validated instrument to capture users’ perception of the usefulness and usability of an mHealth application. So far, this questionnaire is only available in English, Malay and Chinese.Objective: The aim of this study was to translate and validate a German version of the MAUQ (G-MAUQ). Further, we developed a short scale with 6 items (G-MAUQ-S) in German. Methods: We used the Translation, Review, Adjudication, Pretest and Documentation (TRAPD) method to translate the MAUQ. Subsequently, we assessed content validity with 15 expert ratings and face validity with 15 German speaking mHealth users. To further validate the questionnaire, we used data from 148 participants of an RCT examining symptom checkers in the Emergency Department to assess convergent validity by correlating the G-MAUQ with the German version of the System Usability Scale and discriminant validity by correlating the G-MAUQ with other unrelated questionnaires. Lastly, we developed a short version by assessing item discrimination, factor loadings, correlation with the full scale and construct validity.Results: All but one item showed sufficient content validity with item-level content validity index values between CVI-I = 0.8 and 1.0. Face validity was excellent with item-level face validity index values ranging from FVI-I = 0.87 to 1. Convergent validity was sufficient with r = 0.769, and discriminant validity was sufficient with values between r = -0.014 and r = 0.220. An internal consistency of Cronbach’s α = 0.93 demonstrated high reliability. The short scale showed sufficient convergent validity (r = 0.762) and discriminant validity (r between –0.012 and 0.201). Conclusions: A validated and reliable G-MAUQ can be used by researchers and practitioners to assess the usability of mHealth interventions. We also provide the German mHealth App Usability Questionnaire – Short Version (G-MAUQ-S) with six questions to quickly assess the usability of an intervention.

Zhu Y., Zhao Z., Guo J., Wang Y., Zhang C., Zheng J., Zou Z., Liu W.

The COVID-19 pandemic has significantly impacted the healthcare industry, especially public health resources and resource allocation. With the change in people’s lifestyles and increased demand for medical and health care in the post-pandemic era, the Internet and home healthcare have rapidly developed. As an essential part of Internet healthcare, mobile health (mHealth) applications help to fundamentally address the lack of medical resources and meet people’s healthcare needs. In this mixed-method study, we conducted in-depth interviews with 20 users in China (mean age = 26.13, SD = 2.80, all born in China) during the pandemic, based on the unified theory of acceptance and use of technology 2 (UTAUT-2) mode, and identified four dimensions of user needs in mHealth scenarios: convenience, control, trust, and emotionality. Based on the interview results, we adjusted the independent variables, deleted the hedonic motivation and the habit, and added the perceived trust and perceived risk as the variables. Using a structural equation model (SEM), we designed the questionnaire according to the qualitative results and collected data from 371 participants (above 18 years old, 43.9% male) online to examine the interrelationships these variables. The results show that performance expectancy (β = 0.40, p < 0.001), effort expectancy (β = 0.40, p < 0.001), social influence (β = 0.14, p < 0.05), facilitating condition (β = 0.15, p < 0.001), and perceived trust (β = 0.31, p < 0.001) had positive effects on use intention. Perceived risk (β = −0.31, p < 0.001) harmed use intention, and price value (β = 0.10, p > 0.5) had no significant effects on use intention. Finally, we discussed design and development guidelines that can enhance user experience of mHealth applications. This research combines the actual needs and the main factors affecting the use intention of users, solves the problems of low satisfaction of user experience, and provides better strategic suggestions for developing mHealth applications in the future.

Kodjebacheva G.D., Tang C., Groesbeck F., Walker L., Woodworth J., Schindler-Ruwisch J.

This qualitative study surveyed caregivers regarding their perspectives on the benefits of, challenges with, and suggestions for improving telehealth during the COVID-19 pandemic. Caregivers who had the responsibility for caring for at least one child aged under 18 years of age in Genesee County, MI, participated. The caregivers were biological parents, stepparents, foster parents, adoptive parents, and guardians. A total of 105 caregivers completed a survey with open-ended questions via Qualtrics. Two independent coders developed themes based on the responses using grounded theory. Participants were primarily biological parents and non-Hispanic White and African Americans. According to the participants, the benefits of telehealth included prevention of exposure to the COVID-19 virus, quality communication with physicians, savings in travel time, and cost-effective methods to receive care. The challenges included a lack of in-person interaction, fear of compromised confidentiality, and the potential for misdiagnosis. Suggestions for improvement by caregivers included increasing access to telehealth for poorer families, offering a media educational campaign to promote telehealth use, and creating a universal platform to share patient information. Future studies may test the effectiveness of interventions such as those suggested by caregivers in this study to improve telehealth.

Labinsky H., Ukalovic D., Hartmann F., Runft V., Wichmann A., Jakubcik J., Gambel K., Otani K., Morf H., Taubmann J., Fagni F., Kleyer A., Simon D., Schett G., Reichert M., et. al.

Treat-to-target (T2T) is a main therapeutic strategy in rheumatology; however, patients and rheumatologists currently have little support in making the best treatment decision. Clinical decision support systems (CDSSs) could offer this support. The aim of this study was to investigate the accuracy, effectiveness, usability, and acceptance of such a CDSS—Rheuma Care Manager (RCM)—including an artificial intelligence (AI)-powered flare risk prediction tool to support the management of rheumatoid arthritis (RA). Longitudinal clinical routine data of RA patients were used to develop and test the RCM. Based on ten real-world patient vignettes, five physicians were asked to assess patients’ flare risk, provide a treatment decision, and assess their decision confidence without and with access to the RCM for predicting flare risk. RCM usability and acceptance were assessed using the system usability scale (SUS) and net promoter score (NPS). The flare prediction tool reached a sensitivity of 72%, a specificity of 76%, and an AUROC of 0.80. Perceived flare risk and treatment decisions varied largely between physicians. Having access to the flare risk prediction feature numerically increased decision confidence (3.5/5 to 3.7/5), reduced deviations between physicians and the prediction tool (20% to 12% for half dosage flare prediction), and resulted in more treatment reductions (42% to 50% vs. 20%). RCM usability (SUS) was rated as good (82/100) and was well accepted (mean NPS score 7/10). CDSS usage could support physicians by decreasing assessment deviations and increasing treatment decision confidence.

Labinsky H., Gupta L., Raimondo M.G., Schett G., Knitza J.

Abstract Mobile health applications and digital therapeutics (DTx) aim to improve current patient care. Real-world data on DTx are, however, scarce. The aim of this study was to evaluate the adherence, acceptance, and efficacy of DTx in a clinical routine rheumatology setting. We conducted a prospective observational cohort study assessing the use, adherence, acceptance, and efficacy of the DTx DiGA (Digitale Gesundheitsanwendungen) by survey over 12 weeks. Patients included had to have a rheumatic disease and had been prescribed a DiGA. Acceptance was assessed using the Net promoter score (NPS). 48 patients were prescribed DiGA. Of these, 39/48 (81%) completed the follow-up survey. 21/39 (54%) patients downloaded the DTx and 20/39 (51%) used the DTx at least once. 9/39 (23%) of patients stopped quickly afterward and 5/39 (13%) reported having completed the whole DTx program. Lack of time and commitment were reported as the main reasons for non-use. Overall acceptance of DiGA was high (Net promoter score (NPS) mean (SD) 7.8/10 (2.3)). While the majority of patients (60%) reported no improvement, one subgroup of patients (7/20, 35%) who regularly used an exercise-based DTx for back pain reported symptom improvement. Acceptance of DTx in patients with rheumatic diseases is high, however onboarding to DTx use and adherence to DTx is still challenging in patients with rheumatic diseases. In a subgroup of patients with back pain, however, the use of an exercise-based DTx led to symptom improvement.

Li M.M., Rising K.L., Goldberg E.M.

• Given the surge in telehealth due to the COVID-19 pandemic, evaluation is critical. • Evaluation can be divided into access, cost, experience, effectiveness domains. • Existing survey tools can be used to evaluate telehealth outcome domains. • The telehealth needs of priority populations should be addressed to ensure equity. Telehealth use has surged since the onset of the coronavirus disease 2019 (COVID-19) pandemic, but the evaluation of telehealth outcomes and performance has not necessarily matched the pace of its uptake. In this article we aim to guide the design of a telehealth evaluation system encompassing all four domains of the outcome measurement framework developed by the National Quality Forum (NQF) – access to care, cost, experience, and effectiveness. We aim to achieve this through proposing survey items that can be distributed to patients or clinicians as a questionnaire and providing suggestions on areas of focus for evaluation studies. Using PubMed and Google Scholar search engines, we performed a literature review of articles related to the evaluation of telehealth outcomes that were published in English since 2000. We found existing survey tools to assist the development of an evaluation questionnaire, and categorized items into the four NQF outcome domains. For each outcome domain, we also summarize existing work on evaluation and make recommendations on areas for future assessment. In particular, we found that telehealth accessibility and accommodations have been historically under-studied and provide tools to address this. Evaluating telehealth outcomes is critical to ensure efficient and high-quality care delivery, and we believe establishing an evaluation system will help practices assess and improve their telehealth systems as well as their ability to use telehealth to respond to the diverse needs of patients. Since the start of the coronavirus disease 2019 (COVID-19) pandemic, telehealth use has been on the rise. Evaluating outcomes related to telehealth is critically important, but given the urgency of telehealth uptake, many health systems and practices may not yet have evaluation systems in place. This article guides the design of a telehealth evaluation system by proposing several validated and novel survey questions that can be used as part of a patient or clinician questionnaire and suggesting important measures of outcome for evaluation studies to assess across the four domains of telehealth quality as outlined by the National Quality Forum (NQF) – access to care, cost, experience, and effectiveness. We present tools to reach priority populations who often lack access to remote care, including older adults, underrepresented minorities, and people with disabilities.

Adams C., Walpola R., Schembri A.M., Harrison R.

Background Patient experience is a complex phenomenon that presents challenges for appropriate and effective measurement. With the lack of a standardized measurement approach, efforts have been made to simplify the evaluation and reporting of patient experience by using single-item measures, such as the Net Promoter Score (NPS). Although NPS is widely used in many countries, there has been little research to validate its effectiveness and value in the healthcare setting. The aim of this study was to systematically evaluate the evidence that is available about the application of NPS in healthcare settings. Methods Studies were identified using words and synonyms that relate to NPS, which was applied to five electronic databases: Medline, CINAHL, Proquest, Business Journal Premium, and Scopus. Titles and abstracts between January 2005 and September 2020 were screened for relevance, with the inclusion of quantitative and qualitative studies in the healthcare setting that evaluated the use of NPS to measure patient experience. Results Twelve studies met the inclusion criteria. Four studies identified benefits associated with using NPS, such as ease of use, high completion rates and being well-understood by a range of patients. Three studies questioned the usefulness of the NPS recommendation question in healthcare settings, particularly when respondents are unable to select their service provider. The free-text comments section, which provides additional detail and contextual cues, was viewed positively by patients and staff in 4 of 12 studies. According to these studies, NPS can be influenced by a wide range of variables, such as age, condition/disease, intervention and cultural variation; therefore, caution should be taken when using NPS for comparisons. Four studies concluded that NPS adds minimal value to healthcare improvement. Conclusion The literature suggests that many of the proposed benefits of using NPS are not supported by research. NPS may not be sufficient as a stand-alone metric and may be better used in conjunction with a larger survey. NPS may be more suited for use in certain healthcare settings, for example, where patients have a choice of provider. Staff attitudes towards the use of NPS for patient surveying are mixed. More research is needed to validate the use of NPS as a primary metric of patient experience. Patient or Public Contribution Consumer representatives were provided with the research findings and their feedback was sought about the study. Consumers commented that they found the results to be useful and felt that this study highlighted important considerations when NPS data is used to evaluate patient experience.

Ejdys J.

With the growing interest in e-education, particularly in the context of the pandemic, more scientific studies have been undertaken recently to analyze and identify factors influencing e-learning acceptance. Indeed, e-learning acceptance depends on many different factors, but no consensus has been reached on factors that contribute most to the acceptance of e-learning solutions. Consequently, this article ascertains the factors and their relationships behind the satisfaction and the future intention to use e-learning among Polish university students. From among the factors analyzed in the literature, the author examined the relationship between computer self-efficacy (CSE), facilitating conditions (FC), satisfaction (S) and the future intention to use e-learning (FI). Data were gathered using structured questionnaires and computer-assisted web interviewing (CAWI). Students at Bialystok University of Technology (Poland) were sent an electronic link to the questionnaires using the internal e-mail system. Eight hundred three forms were returned fully filled out. Aiming to ascertain the extent to which measured variables describe the number of constructs, the author made the Confirmatory Factor Analysis (CFA). The Generalized Least Squares (GLS) estimator was used to calculate the values of model parameters.The results confirmed that higher computer self-efficacy and better facilitation conditions result in greater user satisfaction with e-learning. However, facilitating conditions impact user satisfaction more than computer self-efficacy construct variables. Based on the findings, user satisfaction is a strong anticedents of the future intention to use e-learning.

Albrecht A., Taubmann J., Minopoulou I., Hatscher L., Kleinert S., Mühlensiepen F., Welcker M., Leipe J., Schulz N., Klemm P., Hueber A., Schett G., Kuhn S., Labinsky H., Knitza J.

Prescribable digital health applications (DiGAs) present scalable solutions to improve patient self-management in rheumatology, however real-world evidence is scarce. Therefore, we aimed to assess the effectiveness, usage, and usability of DiGAs prescribed by rheumatologists, as well as patient satisfaction. The DiGAReal registry includes adult patients with rheumatic conditions who received a DiGA prescription. Data at baseline (T0) and the 3-month follow-up (T1) were collected through electronic questionnaires. Study outcomes included DiGA-specific outcome assessments as well as generic outcome assessments, including the Patient Global Impression of Change (PGIC), Patient Activation Measure (PAM®), and the German Telehealth Usability and Utility Short Questionnaire (TUUSQ). Changes between T0 and T1 were analyzed using descriptive statistics and paired tests. A total of 191 patients were included between June 2022 and April 2023. Of these, 127 completed the 3-month follow-up, and 114 reported using the prescribed DiGA, with 66% reporting weekly use and 15% completing the full DiGA program. The most commonly prescribed DiGAs targeted pain management (53%). Symptom improvement was reported by 51% of patients using a DiGA, with significant reductions in exhaustion levels (p = 0.03). Significant DiGA-specific improvements were observed for DiGAs addressing back pain (p = 0.05) and insomnia (p = 0.006). However, no overall significant changes were detected in patient activation, health literacy, pain, overall health, or disease activity. Back pain and weight management DiGAs were the most effective, frequently used, and best-rated DiGAs, with symptom improvements reported by 50% to 82% of patients. The findings suggest that DiGAs can improve symptom management in rheumatic patients, especially for conditions like back pain and weight control. Further real-world evidence is needed and may support value-based digital health efforts and reimbursement frameworks.