Electronic Collection of Multilingual Patient-Reported Outcomes across Europe

Abstract

Background Patient-reported outcomes (PROs) are information provided directly by patients that helps in improving patient diagnosis and treatment. Validated translations of PROs can be used to treat international patients. Electronic systems and especially mobile devices provide a great opportunity for their collection; however, these systems are normally study-oriented and therefore single language, not scalable, and not interoperable.

Objectives This article reports the development of a multicenter, multilingual, and interoperable electronic PRO (ePRO) system and evaluates its user satisfaction in an international clinical study.

Methods The ePRO named “MoPat2” was developed using Java 8 and jQuery Mobile 1.4.5. The system was evaluated in the context of the European dermatology project “European Network on Assessment of Severity and Burden of Pruritus”(PruNet), which aimed to unify the assessment of itch in routine dermatological care in Europe. Twenty-six clinicians and 468 patients from 8 European clinical centers were asked to complete a user satisfaction questionnaire regarding the use of MoPat2 with a tablet personal computer. The results were then analyzed and correlated with the age, gender, and language of the respondents.

Results MoPat2 was enhanced with multilingual capabilities and is now able to conduct surveys in several languages, as well as store and display the results in the local language. The interviewed clinicians rated the system with an average score of 2.0 (“good”) in a 1 to 5 Likert scale. Note that 93.9% of the patients (439 of 468) reported having got on well using the system and 88.9% (416 of 456) would be willing to further use it. The age of the patients not willing to further use MoPat2 was, in average, considerably higher than the age of patients willing to use the system.

Conclusions This study represents the first use of an ePRO system for the collection of multilingual PROs in an international, multicenter setting. MoPat2 has been evaluated by both clinicians and patients in the context of a European dermatological study, resulting in a high user satisfaction. The system will be further developed to include new features such as patient follow-ups outside of the clinical setting.

Keywords

patient-reported outcome measures - multilingualism - mobile health

Introduction

Health data are nowadays typically collected and stored electronically. The amount of these data is increasing exponentially, thus producing new opportunities for clinical research.[1] Electronic health records (EHRs) improve health care efficiency and safety,[2] [3] as well as enable rapid methods for the secondary use of clinical records. Unfortunately, the collaboration and sharing of information between outpatient clinics and hospitals is still difficult due to legislation, security concerns, incompatibility of the systems, and lack of standards in use.

The medical informatics community is conducting several initiatives to solve the lack of interoperability between electronic health information systems. One of the most widely accepted solutions is the use of standards that enable the classification of elements in a way that several systems are compliant with, including transport protocols and security layers to preserve data integrity and privacy. The adoption of these standards is an ongoing task and not always straightforward. The large variety of standards and a lack of global decisions on which to use (especially regarding transport protocols and formats) produce a lack of consensus and interoperability problems.

Patient-reported outcomes (PROs) are defined by the Food and Drug Administration of the United States as “any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else.”[4] PROs have become a valuable tool for improving patient diagnosis and treatment, as well as the secondary use of health data for clinical research, although their selection needs to be handled carefully.[5] The validation of PROs and their translations, defined by Acquadro et al as “the ability to express and investigate equivalent concepts across all language versions,” ensures their correctness and enables their use in clinical routine and research practice.[6]

Language barriers affect quality of care around the world, hampering comprehension and adherence as well as patient and provider satisfaction.[7] With increasing migration in Europe,[8] physicians often need to treat patients who do not speak the local language(s), which hampers patient diagnosis and treatment. A solution could be the engagement of translators, but they are not always available or even requested.[9] Translated and validated PROs can be used to gather patient information regardless of the local language.

Electronic systems and mobile devices provide vital opportunities for the collection of PROs. Electronic PRO (ePRO) systems can lead to more accurate and complete data, improved protocol compliance, avoidance of secondary data entry errors, easier implementation of skip patterns, less administrative burden, high respondent acceptance, reduced sample size requirements, and potential cost savings.[10] In addition, ePROs could be used to collect patient data in any language and the results be visualized in the local language. This would support patient diagnosis for international patients when a translator is not an option and enable the easy analysis of clinical data in international clinical studies. So far, the majority of ePROs have been single purpose: they were developed for a certain study, and thus they are generally neither multilingual, multicenter, scalable, nor interoperable.

There is therefore a need for an ePRO that allows for the collection of patient data in several centers across different countries and in several languages, being able to export the data in different formats and standards. Furthermore, the system should be evaluated and accepted by both clinicians and patients.

The Institute of Medical Informatics (IMI) of the University of Münster started in 2010 the development of the ePRO MoPat,[11] which was solely implemented for use at a single center (the Center for Chronic Pruritus at the University Hospital of Münster) in a single language (German). MoPat was newly developed (MoPat2) in 2014 and is being routinely updated.[12] MoPat2 is a Web-based ePRO with a user-friendly interface that enables patients to answer medical questionnaires. Patient answers are encapsulated into encrypted messages, either in Health Level-7 or Operational Data Model (ODM) format, and securely transferred to EHRs and/or research databases.[13] MoPat2 is a generic ePRO that includes several functions such as the import of questionnaires from ODM files, creation of new questionnaires and surveys, generation of user statistics, and a score calculator based on questionnaire responses.

Objectives

The aim of this research is to report the development of a multicenter, multilingual, and interoperable ePRO and the evaluation of its user satisfaction in an international setting.

Methods

Development: Process Steps and Technologies

The development process started with the requirements acquisition and design of the software architecture. In this initial process step, existing solutions were explored and their reuse was analyzed.

The second step involved the development of the system. MoPat2 was developed using the Java 8 programming language, utilizing the frameworks jQuery Mobile 1.4.5, Spring 4, and Maven 3. The resulting functionality will be shown in detail in the “Results” section. Its development was completed with automatic and semiautomatic tests using jUnit and the Selenium plugin for Mozilla Firefox.

Evaluation

The newly developed functionality was evaluated by the European Network on Assessment of Severity and Burden of Pruritus (PruNet), an expert group funded by the European Academy of Dermatology and Venereology (EADV). The aim of the said expert group is to harmonize the assessment of itch in routine dermatological care in Europe[14] and establish a consensus among several experts on which PROs best serve this purpose. The selected PROs[14] were then translated and validated in several languages following their selection. The validation of these PROs required a multilingual functionality that allows patients from different countries to answer the same set of questions in their native language. For a better and easier analysis of the results, the data collected had to be stored in a central system and remain exportable in a format compliant with statistic tools.

Organizational Setting

The PruNet study was led by the Center for Chronic Pruritus and the IMI at the University Hospital of Münster. The multilingual ePRO MoPat2 was evaluated in eight centers in eight different countries: the Policlinico A. Gemelli, Catholic University of Sacred Heart, Rome, Italy; the Medical University of Graz, Austria; the University Hospital of Brest, France; the Wroclaw Medical University, Poland; the Adnan Menderes University, Aydın, Turkey; the Kantonsspital Aarau, Switzerland; the Hospital Sant Pau, Barcelona, Spain; and the University Hospital of Münster, Germany.

Data Collection

Data were collected in 1-month periods, from July 2015 until March 2016, in each clinic. During this time, all dermatological patients who visited these clinical institutions and clinicians working at the clinics involved in the PruNet project were asked to complete an electronic survey including a user satisfaction questionnaire using MoPat2.

The PRO validation for the PruNet study[14] included two MoPat2 patient surveys: the first survey was completed by patients following their arrival at the clinical institution, and the second survey after their visit with the physician. The first survey contained 67 questions and included the PROs requiring validation and a MoPat2 user satisfaction questionnaire.

Two questionnaires were created by the IMI team to assess the user satisfaction of MoPat2, with one addressing clinicians and the other patients. The clinician questionnaire contains seven Boolean questions (yes/no), one multiple-choice question with five possible answers, and four free-text questions. In contrast, the patient questionnaire contains eight Boolean, two multiple-choice, and four free-text questions. Although MoPat2 can be used in every tablet personal computer (PC) running on Android v4.0, iOS v7.0, or Windows Mobile v8.0 or higher; only iPad 2 tablet devices running on iOS v9.0 were used in this study for the purpose of data collection. As no instructions are provided to the users regarding the use of MoPat2, the satisfaction questionnaires contain questions concerning the general usability of the iPad. MoPat2 was not referred to, since users are unaware of its name and/or role.

The questions in the satisfaction questionnaires are meant to assess the user satisfaction of MoPat2 (questions 1, 2, 8, and 9 in the patients' questionnaire; 1, 2, 3, and 8 in the clinicians') and the acceptance of the MoPat2 PROs (questions 3–7 in the patients' questionnaire and 4–7 in the clinicians'). The analysis will be focused on the former. The user satisfaction questionnaires have been included as an additional file in the [Supplementary Appendix A].

Data Analysis

A descriptive and explorative analysis of the results was performed using R and the corrected contingency coefficient K*∈[0,1] between patient responses and age, gender, and language of the respondents calculated.

Results

Development of ePRO Multilingual Functions

Multilingual capabilities for the administration and user functions of MoPat2 have been developed. Since it was previously only possible to store a single question text, the database required enhancement. The single column storing the corresponding text was changed to a multivalue attribute that stores the combined language code according to International Organization for Standardization (ISO) 639, the specific country code according to ISO 3166, and the bidirectionally translated text for the defined language. The administration interface was extended to allow the addition of all languages available in Java 8 Locale class. To improve the simplicity of the user interface, the input fields for single languages have been made collapsible and are marked with colors to indicate whether they have been included.

This functionality allows users to create questionnaires and surveys in all known languages. The graphical user interface (GUI) was translated in seven different languages: English, French, German, Italian, Polish, Spanish, and Turkish.

The workflow for conducting a survey using MoPat2 ([Fig. 1]) is as follows: (1) The clinician selects a language for the GUI and logs into MoPat2. (2) The clinician enters patient's case number, pseudonym, or study identification number. (3) The clinician selects a questionnaire bundle (the set of questionnaires that conforms the electronic survey in MoPat2) and the patient's language. If the questionnaire bundle is in a language not yet supported by the general GUI, the survey (answers and questions) will be in the patient's language and the general GUI (buttons next, previous, etc.) will remain in the currently displayed language. (4) The patient receives the tablet PC and completes the survey, handing it back to the clinician once the survey has been completed.

Fig. 1 Process steps to be followed to perform a multilingual survey using MoPat2.

As a Web-service, MoPat2 is available through the Internet. A Web-clip with MoPat2's URL was installed in the devices used for surveying the patients. Once a patient completes a survey, responses are securely sent via a Secure Sockets Layer to x4T: an electronic data capture system located in Münster.[15] x4T was also used for subject recruitment and survey completeness tracking. All values received in x4T apart from free-text fields were displayed in the home language.

User Satisfaction

The user satisfaction evaluation included the clinicians involved in the use of MoPat2 and the patients visiting the clinic that were recruited for the study and signed the agreement form.

Clinician User Satisfaction

Twenty-eight clinicians completed the user satisfaction survey. The MoPat2-relevant results are as follows:

1. Ninety-three percent of clinicians (26 out of 28) think that patient data collection with MoPat2 is acceptable.

2. Ninety-six percent of clinicians (27 out of 28) think that MoPat2 is an appropriate tool for recording pruritus in patients.

3. On a scale from 1 (“very good”) to 5 (“very bad”), the participating clinicians rated the usefulness of the iPad system (MoPat2) with an average score of 2 (“good”).

Patient User Satisfaction

A total of 495 patients participated in the PruNet study and 468 (94.5%) completed the user satisfaction questionnaire. Participants completed the survey including the satisfaction questionnaire in 18 minutes and 2 seconds in average with a median value of 9 minutes and 20 seconds.

The age distribution by survey language selected in MoPat2 is depicted in [Fig. 2]. The participants who completed the survey in Turkish are, on average, slightly younger (45 years) than the general patient population (52 years), whereas Italian-speaking participants were found to be slightly older (58 years).

Fig. 2 Distribution of population by language and age: tr_TR (Turkish-Turkey), fr_FR (French-France), pl_PL (Polish-Poland), de_DE (German-Germany), es_ES (Spanish-Spain) and it_IT (Italian-Italy). One patient in Spain preferred to complete the survey in English (en_GB). The dashed line represents the inclusion criterion of a minimum age of 18 years. The solid blue lines represent the median and quartiles of their collective ages. The dashed line represents the inclusion criterion of a minimum age of 18 years. The solid blue lines represent the median and quartiles of their collective ages.

The results of the patient user satisfaction questionnaire are shown in [Table 1].

Table 1 Patient responses to the user satisfaction questionnaire

Question	True	False	NA	K* Gender[a]	K* Language[a]
1. Do you feel you have been given sufficient information about the purpose of the iPad survey?	452 (96.6%)	14 (3%)	2 (0.4%)	0.00	0.31
2. Did you get on well using the iPad to complete these questionnaires?[b]	439 (93.9%)	27 (5.8%)	2 (0.3%)	0.04	0.33
2a. I generally don't get on well with iPads/tablets[c]	20 (74%)	7 (26%)	–	0.34	0.27
2a. The display/the font is too small[c]	4 (15%)	23 (85%)	–	0.16	0.31
2a. I had problems selecting the answer/number on the display and had to make frequent corrections[c]	7 (26%)	20 (74%)	–	0.15	0.50
2a. I find it hard to enter data into the iPad because of my manual/visual limitations (e.g., osteoarthritis of the hands, rheumatism, eye disease, or similar)[c]	9 (33%)	18 (67%)	–	0.37	0.26
2b. Other (free text)[c]	0 (0%)	27 (100%)	–
3. Did you find the individual questions easy to understand?	451 (96.4%)	14 (3%)	3 (0.6%)	0.00	0.26
4. Did you find it helpful to have to answer the questions in the order they were given?	437 (93.4%)	27 (5.8%)	4 (0.8%)	0.17	0.17
5. Were there any questions where you had difficulty in choosing an answer?	152 (32.5%)	314 (67.1%)	2 (0.4%)	0.13	0.24
6. Did the survey on the iPad seem too long?	74 (15.9%)	392 (83.8%)	2 (0.3%)	0.03	0.25
7. Would you be willing to use the iPad if it was used for routine surveys in an outpatient clinic?[b]	416 (88.9%)	50 (10.7%)	2 (0.4%)	0.01	0.27
7a. Is answering questions on the iPad too much effort for you?[c]	21 (42%)	29 (58%)	–	0.42	0.82
7a. Do you have difficulty using an iPad?[c]	21 (42%)	29 (58%)	–	0.25	0.65
7b. Other (free text)[c]	11 (22%)	39 (78%)	–	0.05	0.74
8. Do you feel that the iPad is a suitable tool for collecting data on itching?	420 (89.8%)	44 (9.5%)	4 (0.9%)	0.14	0.32
8a. Why not? (free text)[c]	27 (61%)	17 (39%)	–
9. Did anything else come to mind? (free text)	130 (27.8%)	338 (72.3%)	–

^a The corrected contingency coefficient K*∈[0,1] in the last two columns is used as a measure of association where higher values mean higher disassociation.

^b For subquestions note that in question 2, there is no “false,” i.e., no data in subquestions for language tags “en_GB,” “es_ES,” and “pl_PL,” and in question 7, none for “en_GB.”

^c The questions 2a, 2b, 7a, 7b, and 8a were shown only when the questions 2, 7, and/or 8 were answered with a “No,” respectively.

Twenty-seven patients expressed a hindered ability to use the iPad to complete the questionnaire ([Table 1]), being “I generally don't get on well with iPads” the most common reason (20 out of 27).

Fifty patients reported they would be unwilling to use the survey app (MoPat2) in an outpatient clinic ([Table 1]). These patients were then asked for the reason why in a multiple-choice format and free-text subquestion. The effort involved with using an iPad and the difficulties in using one were the most common reasons given (both with 21 out of 50 responses). Two of the 50 refused to answer when asked to provide a reason for their skepticism. Five participants had concerns regarding both the effort needed to answer questions and the difficulties using an iPad. Another 16 participants specified the effort needed to answer questions and the same number reported difficulties using an iPad as their only concern. Eleven selected “other reasons.” An analysis of the free-text responses was performed using Google Translate for the unknown languages and did not reveal any issues regarding the MoPat2.

The responses to the patient user satisfaction survey were also correlated with the language and gender of the patients. The highest disassociations were found on the correlation with language groups, especially on the answers to the subquestion 7 “Is answering questions on the iPad too much effort for you?” (K* = 0.84) and “Do you have difficulty using an iPad?” (K* = 0.65) ([Table 1]). Those associations were, in most cases, overrepresentations of participants who answered the questionnaires in Turkish (and in a few cases those speaking Italian).

The results for question 7 were also compared with the age of the respondents ([Fig. 3]). A difference of 9 years was observed among the median age of the positive and negative respondents. The median age of the group not willing to further use MoPat2 was 61 years, while the median age of the remaining participants was 52.

Fig. 3 Responses to the question “Would you be willing to use the iPad if it were used for routine surveys in an outpatient clinic?” correlated with the age of the respondents.

The form of associations was analyzed via the inspection of (extended) mosaic plots, but is not detailed in this section as the authors did not consider them significant. The complete analysis can be found in the [Supplementary Appendix B].

Discussion

Multilingual and multicenter capabilities were implemented for the ePRO MoPat2. The system was successfully evaluated by 28 clinicians and 468 patients in 8 European clinical centers and represents the first report on the use and evaluation of an electronic system for the collection of multilingual PROs.

The majority of the interviewed clinicians (96%) stated that the iPad is a valid tool for measuring pruritus and reported a high user satisfaction of the survey system (overall score of 2.0, “good”) despite being unable to perceive the benefits of using MoPat2 for their clinical practice, as they did not have access to the patient responses (the data collected was solely used for the validation of the PRO translations). This might have negatively affected the system's overall score. Among the patients, the responses were positive when asked if they got on well using the iPad and most of them would be willing to further use the system on an outpatient clinic. These findings suggest a high user satisfaction for MoPat2.

The patients unwilling to use MoPat2 for completing surveys in an outpatient care center are, in average, 9 years older than the ones willing to use it ([Fig. 3]). Other studies on this specific issue found similar results,[16] although trends could be evolving toward more acceptance of mobile-based systems by the elderly,[17] meaning that a future evaluation of the system could result in a higher MoPat2 acceptance by the elderly and the gap between the age of the users willing to use the system could be reduced.

To avoid fatigue and study “drop outs” due to too many questions in the survey[18] as well as reducing to the minimum the completion time of the survey before the clinical examination, the number of questions in the survey was limited to the PruNet study-oriented PROs and a single user satisfaction questionnaire. The use of standard tools such as the System Usability Scale[19] could enhance the scalability of the study, but would provide less specific information about the system and the iPad for the collection of PROs.

When an ePRO is needed, it must be decided whether a new one has to be implemented, as there are already several systems available and some offer functions similar to those of MoPat2.[20] [21] [22] [23] LimeSurvey[24] offers an extensive spectrum of possibilities for multilingual online questionnaires but does not include EHR linkage capabilities and data protection may be an issue. REDCap is one of the best-known data capture systems, but is mainly clinical research-oriented as it does not include a connection with the EHR.[25] Other ePROs such as PatientViewpoint and Patient Care View are able to export the data to an EHR, but in a single format and language.[26] The Computer-based Health Evaluation Software[27] is a multilingual ePRO that can be used for data collection in multicenter clinical studies. The two most important mobile device operative systems also have their own initiatives to support the electronic collection of PROs: ResearchKit (Apple)[28] and ResearchStack (Android)[29] provide a framework for the easy creation of ePROs. Other initiatives such as C3-PRO try to establish a framework for integration of ePROs.[30] However, none of these ePROs has the functional capabilities and versatility of MoPat2, especially with regard to its multilingualism and interoperability.

Since the use of MoPat2 for data collection within the PruNet, its development and use has continued and its general GUI has been translated into the following languages (apart from the seven previously reported): Arabic, Dari, Farsi, Hindi, Kurdish, Dutch, Norwegian, Portuguese, Russian, Albanese, and Swedish.

MoPat2 is currently only usable at the clinical site. Remote capabilities should be implemented so that MoPat2 can be used by patients outside of a clinical setting. A future evaluation of these features should include a standard measurement tool for results scalability, as well as possible differences when using the system at home or at the clinical setting.

Conclusions

The successful implementation of multilingual functions for an ePRO and its use in a multicountry setting for the collection of anonymous data for a clinical study have been demonstrated. MoPat2 can now be used for the collection of clinical routine and study data in an international environment. The user satisfaction evaluation of the system reveals a general acceptance of the system by both clinicians and patients from several countries. Future work should include the possibility to use MoPat2 outside of a clinical setting.

Conflict of Interest

None.

Acknowledgments

The authors would like to thank the institutions involved in the use and evaluation of MoPat2. They also thank Esther Serra-Baldrich, Franz Legat, Simone Garcovitch, Laurent Misery, Adam Reich, Ekin Savk, and Markus Streit who permitted and helped with the data collection at their centers; and Anna Post, Tanja Winter, Juliane Mozar, Friederike Elisabeth Vietmeier, and Julia Kristin Funke, who collected the data at the clinical centers.

Note

The PruNet study that includes the satisfaction evaluation was registered in the Deutsches Register Klinischer Studien No. DRKS00007958.

Authors' Contributions

ISR led the development of the electronic PROs and the satisfaction evaluation and wrote the manuscript. MR built the statistics. PB was in charge of the research database and made the exports. CZ arranged the translations. CR helped with the data collection. SS tested and corrected the electronic PROs and helped to collect data. SS led the clinical group and helped to design the study. MD led the technical group and helped to design the study. MS led the development of MoPat and supervised the drafting of the manuscript. All of the authors approved and helped to draft the manuscript.

• References

• 1 Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Health Inf Sci Syst 2014; 2: 3
• 2 Hillestad R, Bigelow J, Bower A. , et al. Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Aff (Millwood) 2005; 24 (05) 1103-1117
• 3 Shekelle P, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology. Rockville, MD: Agency for Healthcare Research and Quality (US); 2006
• 4 U.S. Department of Health and Human Services FDA Center for Drug Evaluation and Research; U.S. Department of Health and Human Services FDA Center for Biologics Evaluation and Research; U.S. Department of Health and Human Services FDA Center for Devices and Radiological Health. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health Qual Life Outcomes 2006; 4: 79 Doi: 10.1186/1477-7525-4-79
• 5 Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why?. Qual Life Res 2009; 18 (01) 115-123
• 6 Acquadro C, Bayles A, Juniper E. Translating patient-reported outcome measures: a multi-step process is essential. J Bras Pneumol 2014; 40 (03) 211-212
• 7 Schwei RJ, Del Pozo S, Agger-Gupta N. , et al. Changes in research on language barriers in health care since 2003: a cross-sectional review study. Int J Nurs Stud 2016; 54: 36-44
• 8 United Nations. International Migration Report. Available at: http://www.un.org/en/development/desa/population/migration/publications/migrationreport/docs/MigrationReport2017_Highlights.pdf . Accessed September 11, 2018
• 9 Karliner LS, Pérez-Stable EJ, Gildengorin G. The language divide. The importance of training in the use of interpreters for outpatient practice. J Gen Intern Med 2004; 19 (02) 175-183
• 10 Coons SJ, Eremenco S, Lundy JJ, O'Donohoe P, O'Gorman H, Malizia W. Capturing patient-reported outcome (PRO) data electronically: the past, present, and promise of ePRO measurement in clinical trials. Patient 2015; 8 (04) 301-309
• 11 Fritz F, Ständer S, Breil B, Riek M, Dugas M. CIS-based registration of quality of life in a single source approach. BMC Med Inform Decis Mak 2011; 11: 26
• 12 Storck M, Trinczek B, Dugas M, Fritz F. Towards a trial-ready mobile patient questionnaire system. Stud Health Technol Inform 2014; 205: 768-772
• 13 Soto-Rey I, Dugas M, Storck M. Implementation of an ODM and HL7 compliant electronic patient-reported outcome system. Stud Health Technol Inform 2016; 228: 421-425
• 14 Ständer S, Zeidler C, Riepe C. , et al. European EADV network on assessment of severity and burden of Pruritus (PruNet): first meeting on outcome tools. J Eur Acad Dermatol Venereol 2016; 30 (07) 1144-1147
• 15 Bruland P, Forster C, Breil B, Ständer S, Dugas M, Fritz F. Does single-source create an added value? Evaluating the impact of introducing x4T into the clinical routine on workflow modifications, data quality and cost-benefit. Int J Med Inform 2014; 83 (12) 915-928
• 16 Kim BY, Lee J. Smart devices for older adults managing chronic disease: a scoping review. JMIR Mhealth Uhealth 2017; 5 (05) e69 Doi: 10.2196/mhealth.7141
• 17 Plaza I, Martín L, Martin S, Medrano C. Mobile applications in an aging society: Status and trends. J Syst Softw 2011; 84 (11) 1977-1988
• 18 Rolstad S, Adler J, Rydén A. Response burden and questionnaire length: is shorter better? A review and meta-analysis. Value Health 2011; 14 (08) 1101-1108
• 19 Brooke J. SUS-A quick and dirty usability scale. Usabil Eval Industry 1996; 189 (194) 4-7
• 20 Steele Gray C, Gill A, Khan AI, Hans PK, Kuluski K, Cott C. The electronic patient reported outcome tool: testing usability and feasibility of a mobile app and portal to support care for patients with complex chronic disease and disability in primary care settings. JMIR Mhealth Uhealth 2016; 4 (02) e58
• 21 Pinney S, Otobo E, Freeman R. , et al. Use of electronic patient reported outcomes and automated devices for heart failure disease management. IPROC 2017; 3 (01) e24
• 22 Hartkopf AD, Graf J, Simoes E. , et al. Electronic-based patient-reported outcomes: willingness, needs, and barriers in adjuvant and metastatic breast cancer patients. JMIR Cancer 2017; 3 (02) e11
• 23 Epis OM, Casu C, Belloli L. , et al. Pixel or paper? Validation of a mobile technology for collecting patient-reported outcomes in rheumatoid arthritis. JMIR Res Protoc 2016; 5 (04) e219
• 24 LimeSurvey. The most popular FOSS survey tool on the web; 2016:36:30. Available at: https://www.limesurvey.org/de . Accessed September 11, 2018
• 25 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
• 26 Bennett AV, Jensen RE, Basch E. Electronic patient-reported outcome systems in oncology clinical practice. CA Cancer J Clin 2012; 62 (05) 337-347
• 27 Holzner B, Giesinger JM, Pinggera J. , et al. The Computer-based Health Evaluation Software (CHES): a software for electronic patient-reported outcome monitoring. BMC Med Inform Decis Mak 2012; 12: 126 Doi: 10.1186/1472-6947-12-126
• 28 ResearchKit; 2016 :09:21. Available at: http://researchkit.org/ . Accessed September 11, 2018
• 29 ResearchStack; 2016 :22:06. Available at: http://researchstack.org/ . Accessed September 11, 2018
• 30 Pfiffner PB, Pinyol I, Natter MD, Mandl KD. C3-PRO: connecting Researchkit to the health system using i2b2 and FHIR. PLoS One 2016; 11 (03) e0152722