Implementation and Feasibility of Electronic Patient-Reported Outcome (ePRO) Data Entry in the PRAEGNANT Real-Time Advanced and Metastatic Breast Cancer Registry

Abstract Purpose Patient-reported outcomes (PROs) have been incorporated into clinical trials for many symptoms and medical conditions. A transition from paper-based capture of PROs to electronic PROs (ePROs) has recently started. This study reports on the feasibility of ePRO assessment in a prospective registry including molecular data for patients with advanced breast cancer. Methods As part of the PRAEGNANT network, patients were invited by clinical trial staff, physicians, and nurses to complete three standardized Internet-based questionnaires (EQ 5D 5 L, CES-D and IPAQ). Feasibility was assessed by the staff members who assigned the user accounts by the patients. The completeness of the questionnaires was also assessed. Results Fifteen of 17 patients who were asked agreed to participate to complete the PRO questionnaires (EQ-5D-5L and CES-D). However, the IPAQ (physical activity) questionnaire was only validly completed by 9 patients. Feasibility was ranked better by the physicians and dedicated clinical trial staff than by the nursing staff. Conclusions Incorporating ePRO questionnaires into an advanced breast cancer registry is feasible, and no major hurdles were reported. Involving stakeholders from the start, the application is tailored to the capacities and abilities of both patients and clinical staff. The patientsʼ compliance was better with some questionnaires, but others may present difficulties.


Introduction
Over the last 10 years, it has been growing increasingly important to take patientsʼ well-being into account and to assess the effects of treatment on their quality of life. The development of novel therapies and new diagnostic tools have altered treatment approaches in both early and advanced breast cancer. Modern molecular tests have now made it possible to identify women who are unlikely to benefit from chemotherapy, as they have a favorable prognosis [1,2]. In addition, the development of new drugs has led to new drug toxicity profiles that are rarely seen in patients receiving conventional chemotherapy or antihormonal therapy [3]. Examples of these include treatment with mechanistic target of rapamycin (mTOR) inhibitors, which is associated with clinically relevant stomatitis [4], and phosphoinositide 3-kinase (PI3K) inhibitors, with some a dose-limiting toxicity has been found to involve psychiatric disorders [5]. It is therefore becoming more and more important to register the side effects of drugs and the pa-tientsʼ quality of life as accurately as possible and to include several methods of measuring these [6,7]. This study reports on the methods used to incorporate reporting of specific patient-reported outcomes (PROs) into a real-time registry including molecular data for patients with advanced breast cancer.
Adverse events and specific symptoms are common in patients with advanced breast cancer, but they are not always detected in clinical routine work or in clinical trials [8,9]. Capturing data on treatment outcomes and quality of life for advanced breast cancer patients during clinical routine work and in clinical trials is challenging. The paper-based methods currently used for recording PROs require costly staff resources, and they may lead to the pa-tientsʼ point of view being underrepresented. With the growing demand for better information provision and communication with patients, on the one hand, and the increasing use of information technology by health-care providers on the other, the potential for developing "eHealth" solutions in health-care research is becoming increasingly clear [10].
In closely related areas such as research on health-related quality of life (HRQoL), satisfaction with care, and drug adherence, PROs are already a standard method of collecting data [11][12][13]. The Food and Drug Administration (FDA) in the United States has therefore issued draft guidance for the pharmaceutical industry establishing PRO measures as a standard for evaluating symptom end points [14]. The feasibility of direct symptom reporting by patientseven those with terminal cancer and a severe symptomatic burdenhas been shown in several studies [13,15,16].
In addition, the assessment of adverse events by health-care professionals may be subject to considerable interobserver variation [17]. In this context, PROs are substantially more reliable and practicable. Most patients are willing and able to self-report their experience of treatment [18]. In the United States, a recent study of the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) reported that the tool had favorable validity, reliability, and responsiveness [19].
More and more PRO surveys are being used in routine outpatient cancer care. Current data show that among breast cancer patients, there was a high level of monthly compliance with completing questionnaires at home using Internet-based reportingjustifying strategies for enhancing compliance in routine care settings [16].
In particular, it has also been reported that patient self-reporting via web-connected devices during routine chemotherapy care at clinic visits is feasible, with a high level of patient satisfaction members who assigned the user accounts by the patients. The completeness of the questionnaires was also assessed.
Results Fifteen of 17 patients who were asked agreed to participate to complete the PRO questionnaires (EQ-5D-5L and CES-D). However, the IPAQ (physical activity) questionnaire was only validly completed by 9 patients. Feasibility was ranked better by the physicians and dedicated clinical trial staff than by the nursing staff.
Conclusions Incorporating ePRO questionnaires into an advanced breast cancer registry is feasible, and no major hurdles were reported. Involving stakeholders from the start, the application is tailored to the capacities and abilities of both patients and clinical staff. The patientsʼ compliance was better with some questionnaires, but others may present difficulties.
Schlussfolgerung Die Benutzung der ePRO-Fragebögen ist in der PRAEGNANT Registry grundsätzlich durchführbar. Die Machbarkeit hängt maßgeblich von den Fähigkeiten und Kapazitäten der beteiligten Patientinnen und Personalgruppen ab. Die Compliance und Vollständigkeit war nicht bei allen ePRO-Fragebögen gleich gut und ergab teilweise Schwierigkeiten. and good usability with the systems, even for patients unfamiliar with the Internet and elderly patients [20,21]. Research has been carried out on identifying subgroups of patients who may or may not be willing to use web-based questionnaires. Barriers to ePRO usage were identified in particular in older patients and subgroups with a poorer quality of life [22].
Basch et al. have recently described qualitative and quantitative differences between the PRO-CTCAE and classic HRQoL surveys. Qualitatively, the PRO data captured with the PRO-CTCAE have greater depth and less scattered detail. The PRO-CTCAE is also intended for continuous everyday use, in comparison with the before-after comparisons recorded using the HRQoL [23].
Our research group recently established a real-time registry including molecular data for patients with advanced breast cancer [24]. In view of the extensive published data reporting the advantages of electronic capture of PROs, the aims of the present study were to assess the feasibility of using dedicated study staff to initiate patient data entry; to investigate the patientsʼ own views on the feasibility of the process; and to assess the quality of the patient-reported outcome data.

The PRAEGNANT network and the PRO subprotocol
The PRAEGNANT network is a real-time registry including molecular data for patients with advanced breast cancer [24] that serves not only to collect and analyze data, but also as a health-care tool for identifying patients who may be eligible for inclusion in clinical trials [25]. The present study is based on a subprotocol within the main study, focusing on the implementation of electronic capture of three different validated questionnaires. In order to assess feasibility in the clinical setting, three different groups of staff were involved in including patients in the study and explaining to them how to fill out the electronic forms. The study was designed to be completed when five patients for each staff group had logged into the electronic data capture system. The three designated groups were: 1. the dedicated clinical trial unit (CTU) staff, trained to carry out documentation in the PRAEGNANT network; 2. the resident physicians treating the patients; and 3. the nurses involved in actual cancer care and treatment.

Procedures for patient inclusion and data capture
After being included in the main PRAEGNANT study, patients were asked whether they would be willing to take part in the PRAEG-NANT-PRO subproject. A total of 18 patients were asked at study entry, one patient had to be excluded because of withdrawal of informed consent and 17 of 18 patients agreed to complete three validated questionnaires through the studyʼs remote data entry system (▶ Fig. 1). The patients were provided with two links: one leading to a form for changing their individual login credentials and one for logging on to the data capture dashboard to access the questionnaires (▶ Fig. 2). After logging on to the system, the patients then had to select the relevant form to proceed to the questions (▶ Fig. 2). The order in which the questionnaires were displayed was: 1. the European Quality of Life Five Dimensions, Five Levels (EQ-5D-5L) scale for health status; 2. the Center for Epidemiological Studies of Depression (CES-D) scale for depression status; and 3. the International Physical Activity Questionnaire (IPAQ) for physical activity.
After completing their data entry, the patients had to click on "Save" to submit the results. Optional "on-site completion" (completing the questionnaires on a computer at the hospital) was provided for patients who did not have devices available at home.

Feasibility assessment of PRO capture by patients and staff
Two different feasibility questionnaires were also provided for the patients and clinical staff. The questions for clinical staff included the estimated time needed to set up the patientsʼ accounts, the feasibility of the workflow on a scale of 1 to 5, and an assessment of possible problems in setting up the patientsʼ accounts. The feasibility questions for patients included the frequency with which they used a computer, tablet, or smartphone in everyday life, the ease of answering the questions on a scale from 1 to 5, the ease of responding to the ePRO questions electronically on a scale from 1 to 5, the time needed to complete the electronic questionnaires, and a question on whether needing to use ePRO might prevent them from taking part in a future clinical trial. The complete study design is shown in ▶ Fig. 1.

General data collection in PRAEGNANT
Additional data on patients, diseases, and treatments were collected by trained and dedicated staff [24]. The data for the study are monitored using automated plausibility checks. Data that are routinely documented in the patient charts or electronic medical records are transcribed into purpose-designed electronic case report forms. In addition, data that are not usually documented as part of routine clinical work are collected prospectively on paper using structured questionnaires, once the patient has been registered in the study. These data consist of epidemiological data such as family history, cancer risk factors, quality of life, nutrition and lifestyle items, and psychological health. These data are also transcribed into the electronic case report forms.
Documentation in PRAEGNANT is similar to the type of documentation used in standard phase III registration trials. All of the data are pseudonymized, and only the physician treating the patient at a specific study site is aware of the patientʼs identity. For systematic data queries, the datasets are additionally anonymized.

Statistical considerations
The patient characteristics, feasibility questions, and EQ-5D-5L QoL data recorded are presented as means and standard deviations (SD) for continuous variables and as absolute and relative frequencies for categorical variables. For quality control purposes, age, body mass index (BMI), and the scores in the questionnaires used were tested for associations using Spearmanʼs correlation coefficient. Differences in feasibility assessments by the staff group were tested using the Kruskal-Wallis test. All statistical analyses and summary statistics were carried out using IBM SPSS Statistics, version 21.0 (IBM Corporation, Armonk, New York, USA).

Patient demographics and technical skills
Fifteen of the 17 patients who agreed to take part in this feasibility study completed at least part of one questionnaire. The study was designed in such a way that for each of three groupsphysicians, nurses, and dedicated study personnelexactly five patients were to complete the questionnaire (▶ Fig. 1). Data are presented here for the 15 patients who completed the questionnaire. Only data from the feasibility questionnaire addressed to the clinical staff are reported for the set of all 17 user account assignments.
The patientsʼ mean age was 56.9 years (± 14 years) and their mean BMI was 22.8 kg/m 2 (± 4.3 kg/m 2 ). Almost half of the patients were included before the start of first-line treatment in the advanced therapy setting (n = 7, 47 %). At the time when they were included in the ePRO PRAEGNANT study, most of the patients stated that they used a computer, tablet, or smart phone daily or at least once a week (n = 12, 80 %). The patient characteristics are listed in ▶ Table 1.   Fig. 2 The patient data entry form (translated from the German version).

Patient reported outcome data
The individual results for the participating patients are shown in ▶ Table 2. All of the 15 patients completed the EQ-5D-5L questionnaire; the IPAQ was completed by 14 patients; and the CES-D was completed by all 15. The IPAQ results from five patients had to be removed, as the patients stated that they spent either less than 10 minutes (n = 1) or more than 960 minutes per week (n = 4) in physical activity, and these values are regarded as outliers in the IPAQ guidelines for data processing and analysis [26]. As one patient did not complete the IPAQ questionnaire, there were only nine patients for whom valid metabolic equivalents of task (METs) per week could be calculated. The EQ-5D-5L index values (calculated with reference to Germany population) ranged from 0.127 to 0.999, the IPAQ METs/week ranged from 0 to 5742, and the CES-D scores ranged from 10 to 36 (▶ Table 2). For quality control purposes, age and BMI were correlated with the results of the questionnaires. Although only one correlation was statistically significant (IPAQ METs/week versus CES-D, n = 12, p = 0.049), there were negative correlations between BMI and IPAQ (ρ = − 0.211) and between age and CES-D (n = 15, ρ = − 0.296). It should be noted that there was no correlation between age and BMI in this small group of participants (▶ Table 3).
▶ * Patient stated that he did less than 10 min exercise weekly and had to be removed in accordance with IPAQ analysis guidelines.
**Patients stated that they did more than 960 min exercise weekly. # Calculated with reference to Germany.
▶ Table 3 Correlation between selected parameters for quality control purposes, showing Spearman correlation coefficients and numbers of patients for correlation ρ (n). One of the correlations was statistically significant (IPAQ MET/week versus CES-D, p = 0.049).

Feasibility analysis of clinical staff
With regard to the assessment by the clinical staff of the feasibility of assigning data accounts to the patients (▶ Table 4), most of the hospital personnel stated that it was either easy or very easy (n = 13, 76 %). None of the participating staff stated that it was very difficult, but two (12 %) described it as difficult. The average time needed to assign the accounts was 13 min (± 12 min). When the staff groupsʼ assessments of the degree of difficulty of assigning data accounts to patients were compared, it was found that the physicians and dedicated study personnel described it as low, in comparison with higher difficulty levels reported by nurses (▶ Table 5).

Patient preferences regarding paper-versus electronic-based assessment
When initially asked about their preference, only three patients stated that they wanted to use a hospital device/computer to complete the questionnaires (▶ Table 4). However, seven patients completed the questionnaires in the hospital later on.
With regard to ease of use, most patients stated that they felt comfortable or very comfortable with the ePRO questionnaires (n = 11, 74 %) (▶ Table 6) and rated the questions as being quite easy or very easy to answer (n = 11, 74 %). The average time needed to complete the three questionnaires was 15 min (SD ± 11 min). Five patients stated that they would prefer to use ePRO rather than paper; three patients had no preference with regard to subsequent questionnaires, and seven of fifteen patients in total stated that they would not prefer an electronic questionnaire. Three of these patients, who would prefer pPRO, had no access at home to an electronic device (e.g., a computer, smartphone, or tablet). Only one patient who had access to an electronic device would not want to take part in an ePRO study similar to the PRAEGNANT study if electronic questionnaires were to become mandatory.

Discussion
The responses from 15 of the 17 patients included show that it appears to be feasible to allow patients to complete standardized electronic questionnaires about quality of life, depression, and physical activity. This is also reflected in the fact that the majority of the patients (88 %) stated that they were comfortable with completing electronic questionnaires. However, 40 % of the patients who had everyday access to computers (4/10) still said they would prefer a paper questionnaire. There do not appear to be any issues involving a potentially lower recruitment rate when electronic questionnaires are offered.
Although electronic self-reporting is increasingly being used in the administration of clinical trials and in other studies [22], there are only limited data about the feasibility of electronic self-reporting for patients and study personnel. Studies investigating the feasibility of self-reporting via web devices during hospital visits for routine chemotherapy care have reported high mean compliance rates, ranging from 75 to 85 % [20,21]. Results published by our own group have also shown that ▶ ▶ Table 5 Feasibility of registering a patient for the electronic patient-reported outcome (ePRO) questionnaires and assigning a user account. Feasibility was assessed using a numbered scale, with 1 representing very easy and 5 very difficult. Figures represent the number of staff assessments. None of the differences were statistically significant (p = 0.047, Kruskal-Wallis test). electronic screening provides a time-saving option, with excellent data quality [27]. The implementation of ePRO questionnaires in the PRAEG-NANT network is therefore being closely monitored, and the feasibility of the procedure is being assessed as part of a research program. This feasibility study shows that the physicians and trained research staff involved found that the technical process of assigning ePRO accounts to patients was easier than dedicated cancer care nurses perceived it to be. It may be that eHealth facilities need to be addressed for this health-care group as part of future educational programs, or that the development and implementation of eHealth facilities might require completely new health-care personnel. In connection with obstacles to implementation, possible support services for participating nurses might promote willingness, compliance, and elimination of technical hurdles, as this group was found to have reservations regarding ePRO. Several programs concerned with supporting the role of nurses in relation to eHealth are available [28][29][30].
Although 67 % (n = 10) of the patients were daily users of electronic devices (such as computers, smartphones, or tablets), 33 % (n = 5) did not use them or used them only infrequently.
Other ePRO applications have been reported to be easily usable and acceptable, even among patients unfamiliar with the Internet and among elderly patients [20,21]. Our research group is conducting a trial investigating the willingness of patients with advanced breast cancer to respond to electronic questionnaires. This has shown that patients who would prefer a paper-based survey (pPRO) a priori tend to be older (ePRO 53 years vs. pPRO 62 years; p = 0.0014) and typically have lower educational levels (p = 0.0002) and are in poorer health (p = 0.0327) [22]. Previous experience with technology also appears to be a key factor, as it was found that there are more ePRO users than pPRO users among the advanced and professional groups (n = 27, 52 % vs. n = 17, 6 %). Willingness to use ePRO in the future was lower among older patients and subgroups of patients who had a poorer quality of life [22]. This is in stark contrast to previously published data and underlines the need for tailored interventions and support services for older, less technologically skilled patients. However, most of the published trials have been based in the United States, which might explain the higher prevalence of users familiar with the technology needed. Overall, randomized PRO trials in oncology have reported that ePRO is well received among patients in comparison with pPRO [31,32]. It can therefore be anticipated that using electronic PRO questionnaires will be feasible in age groups comparable with that in the average breast cancer population.
Although the associations tested between patient characteristics and questionnaire outcomes were only exploratory, serving for quality control purposes, several associations in the expected directions were observed. Age correlated inversely with the depression score, as has also been reported in other studies [33][34][35]. BMI also correlated inversely with METs/week, as has also previously been reported [36,37]. There were no correlations between BMI and age in the present study. This might be one reason why there was not an inverse correlation between age and METs/ week [38][39][40].
The study has several limitations. Firstly, it is a feasibility assessment in a prospective investigation of the implementation of electronic patient-completed validated ePRO questionnaires. The study thus only has a small sample size, and the EQ-5D-5L, IPAQ, and CES-D values obtained should be treated with caution. However, the ranges do not indicate any outliers or unusual values. The IPAQ questionnaires produced valid MET calculations in only nine cases. Although the questionnaire has only been validated for patients up to the age of 69 and the present patient group included had three participants over that age, one of the three did not start the IPAQ at all and the other two did not enter any data in it, resulting in elimination of the calculated MET data. Another reason for the exclusion of these IPAQ values was the question-▶ * Three of these patients said they never used a computer/tablet/ smartphone at all, and four patients stated that they used them daily. **Two of these patients said they never used a computer/tablet/ smartphone at all. naireʼs guidelines for data processing and analysis, which provide rules for excluding outliers [26]. However, the five patients who had to be removed from the IPAQ scoring were contacted and the extent of weekly physical activity was confirmed in one patient. Future research will need to address the issue of differences between the groups of patients who prefer electronic assessment and those who prefer nonelectronic assessment. The importance of this lies in the need to understand the difference between these preference groups and whether their responses to pPRO or ePRO questionnaires would be discordant. It might also be possible that the same patient might choose different responses to the same question, depending on whether the recording method is ePRO or pPROperhaps in connection with assessing mood or depression. Although the patients needed an average of about 15 minutes to complete each of the three questionnaires, many studies have included additional PRO questionnaires to assess and monitor the complex area of HRQoL. For longitudinal data capture, ePRO applications might be superior to link-based platforms regarding patient adherence. Future studies will therefore need to focus also on the quality of data relative to the time needed to complete the questionnaire and relative to potential bias resulting from the order in which the questionnaires are displayed and answered.

Conclusions
Overall, the study shows that it appears to be feasible to use data entered electronically by patients themselves to register patientreported outcomes in an advanced breast cancer registry. No major problems occurred and no major hurdles were reported. One aim of the study was to make the implementation process more transparent, in order to enable other researchers to retain crucial elements of it when implementing ePRO in other oncological settings. Involving stakeholders from the start, the application is tailored to the capacities and abilities of both patients and clinical staff. The patientsʼ compliance was better with some questionnaires, but others may present difficulties.