Virtual Consultations —“ Should We Be Using Them? ” : A Two-Cycle Clinical Audit

Objective In response to the coronavirus disease 2019 pandemic, the use of virtual appointments increased substantially. An audit was designed to evaluate the ef ﬁ cacy of virtual appointments at Alder Hey Children ’ s Hospital. Patients and clinicians satisfaction of virtual appointments were also recorded. Materials and Methods Patients having virtual appointments were identi ﬁ ed, and data were recorded, including demographics and outcome of the appointment. A pilot study was conducted to identify any necessary modi ﬁ cations to the data collection process. Data were collected over two cycles, and the results were analyzed. Results The ﬁ rst cycle was performed retrospectively and there were 158 patients identi ﬁ ed who had a virtual appointment. An outcome could be drawn from the virtual appointment in over 70% of cases. Recommendations following the ﬁ rst cycle were implemented. A second cycle was then completed prospectively. A total of 145 patients were identi ﬁ ed. An outcome could be drawn in 51% of cases. Patient and clinician feedback on virtual appointments was generally positive. Conclusion This study demonstrated that a successful outcome could be drawn in the majority of virtual appointments, and clinicians and patients had positive experiences regarding virtual clinics. This highlights the usefulness of this service, which is particularly relevant due to the increased reliance on virtual clinics during the pandemic.


Introduction
Due to the emergence of the coronavirus disease 2019 (COVID- 19) pandemic, various measures were implemented to cope with changes in the delivery of healthcare. Virtual appointments by video or telephone have been used to reduce the number of patients attending secondary or tertiary care. Use of virtual appointments aims to reduce footfall through hospitals, while still providing essential and routine care for patients.
providing recommendations for future practice, and to identify areas for development within respective departments. [4][5][6] Research has found that virtual appointments may provide a promising alternative for remote assessment, screening, advice, monitoring, referrals, and/or prescriptions to patients for dental problems. [7][8][9][10][11] A recent study performed in a dental hospital in East Surrey explored the patient satisfaction and acceptance of teledentistry services. 12 This led the authors to undertake this quality improvement project at Alder Hey Children's Hospital, to identify how virtual appointments are being used at a secondary/ tertiary care center, and the usefulness of virtual appointments to patient and provider. Alder Hey Children's Hospital is a children's hospital that provides secondary/tertiary dental care. Among the services provided include cleft, craniofacial, orthodontics, oral and maxillofacial surgery (OMFS), oral surgery, and pediatric dentistry. This quality improvement project was designed to identify and gather information regarding the provision of virtual appointments across all dental specialties at Alder Hey Children's Hospital.
The aims of the audit were to determine if a successful outcome could be drawn at the end of the virtual appointment, and patients and clinicians perspectives of the virtual appointments. Further aims were to identify the Did Not Attend (DNA) rate of virtual appointments, to determine the reasons for DNA, and to determine the call quality of virtual appointments.

Pilot Study
A pilot study was conducted for the duration of 1 week in September 2020 to identify the various clinics to be included in the study and to collect sample data. The study did not require ethical approval and was validated by the Ethics and Quality Improvement Committee at Alder Hey Children's Hospital. Data collected was analyzed and changes were made to the data collection process. Following the completion of the pilot study, the first cycle was commenced.

Standards
The following standards were set: 1. A successful outcome recorded from 75% of virtual appointments. This was set based on opinion from consultants in all involved specialties.
2. The DNA rate for virtual appointments should be comparable to the DNA rate in a similar time period in 2019 (8.5%). 3. The call quality to be recorded in 100% of cases.

Data Collection
All video and telephone consultations across all dental specialties at Alder Hey Children's Hospital were identified and analyzed. Specialties assessed were cleft, craniofacial, orthodontics, OMFS/oral surgery, and pediatric dentistry. Data collected from virtual appointments included patient age, gender, date of consultation, clinician grade and specialty, attended/DNA, type of appointment (video/telephone), outcome of appointment, reason for outcome, and call quality. In the second cycle, patient and operator satisfaction was also recorded. Data was recorded on Microsoft Excel for analysis and tabulation.

First Cycle
The first cycle of the audit ran for 5 weeks from September 28, 2020 to November 6, 2020 when services commenced after the lockdown in June 2020. Data from virtual consultations was recorded retrospectively. Virtual appointments were categorized according to being "successful" or "unsuccessful" following the criteria specified by the audit team. Criteria is defined as shown in ►Table 1. Results from the first cycle were then presented at the departmental clinical governance meeting in December 2020, to discuss and disseminate results with recommendations to be implemented within each department. The following recommendations were made to be implemented prior to the commencement of the second cycle: 1. Call quality to be recorded for video appointments. 2. A clinician-led decision at the end of the appointment should be made, to determine if the outcome was successful or not. 3. Data capture forms were generated to be filled in by clinicians in the second cycle. 4. Patient and operator satisfaction to be recorded.

Second Cycle
Data was collected prospectively for patients who had a virtual consultation from January 15, 2021 to February 19, 2021. Data capture forms were used and filled in by clinicians at the time of the virtual consultation. Outcomes of virtual appointment were standardized according to results from the first cycle for ease of data tabulation and analysis. The results were transferred to Microsoft Excel for analysis and dissemination.

First Cycle
A total of 158 patients were identified in the first cycle. The age of patients ranged from 2 months to 36 years old, and 82.9% (n ¼ 131) of patients were 16 years old and under. Nearly 57% (n ¼ 89) were male. Of the virtual appointments, 85.4% (n ¼ 135) were conducted by video, and the remaining 14.6% (n ¼ 23) via telephone consultation. When assessed by specialty, 34.2% of consultations were cleft, 22.8% orthodontics, 21.5% oral surgery/OMFS, 17.1% pediatric dentistry, and 4.4% craniofacial. Review appointments accounted for 46.2% of virtual clinics, followed by new patient consultations (28.5%) and multidisciplinary team meetings (MDTs) (25.3%).
The DNA rate was 9.5% (n ¼ 15). The majority of cases did not provide a reason for DNA. Among some of the reasons obtained were that the patient was unwell, the parent/guardian forgot the appointment, the appointment letter was not received, and a possible safeguarding concern. Three patients were also attempted to be contacted via telephone after failing to attend their video consult; however, clinicians were still unable to contact the patient/guardian.
The call quality was not mentioned in 64% of cases. In cases where the call quality was mentioned, 19.3% had good call quality, 11.1% with issues to video/sound; however, consultation could still be performed, and 2.2% had poor video/sound leading to the consultation being postponed or changed (see ►Fig. 1).
A successful outcome could be drawn in 70.9% of virtual consultations (see ►Fig. 2). When assessing new patient consultations via virtual appointments, 51.1% had a successful outcome. In virtual MDT clinics, 72.5% had a successful outcome. In review patients, 82.2% had a successful outcome.

Second Cycle
A total of 145 patients were identified in the second cycle. The age of patients ranged from 1 month old to 43 years old, with 90% (n ¼ 130) of patients aged 16 years and under. Eighty-nine (61.4%) patients identified were male.
Of the virtual appointments, 87% (n ¼ 101) were by video, with the remaining via telephone. When assessing the specialties, 40.7% were cleft, 11% oral surgery/OMFS, 15.2% orthodontics, and 33.1% pediatric dentistry. Review appointments accounted for 25.5% of appointments, while MDT and new patient consultations were 40.7 and 33.8%, respectively.
The DNA rate was 21% (n ¼ 30). In patients that DNA, 36.7% were subsequently discharged due to multiple previous DNAs, 56.7% were rebooked for a virtual appointment, 1 patient was sent a letter to contact the department, and 1 patient was rebooked for a face-to-face (FTF) appointment.
The call quality was recorded in 99.1% (n ¼ 114) of cases. In appointments where the call quality was recorded, it was acceptable in 34.8% of cases, 42.6% were good, 15.7% were intermittently poor, and 6.1% were poor leading to the appointment being postponed or changed. The call quality was not recorded in one appointment (►Fig. 3).
In patients who attended a virtual appointment, 11.4% were subsequently discharged, 52.6% had a subsequent FTF appointment, 12.3% required a virtual follow-up, and 23.7% were placed under "watchful waiting" (see ►Fig. 4). The overall successful outcome of virtual appointments was 55.9%. In new patient consultations, 43.8% had a successful outcome. In virtual MDT appointments, 67.8% had a successful outcome. In review patients, 54.1% had a successful outcome.
In patients who required a subsequent FTF appointment, 35.7% required clinical examination/assessment, while 32.1% returned for special investigations and 32.1% required FTF appointments for treatment.
The majority of patients (65.2%) were satisfied with the virtual appointment, while 13.9% preferred a FTF appointment (see ►Fig. 5). When assessing the operator satisfaction of the virtual appointment, 36.7% felt "neutral," with 18.3% being extremely satisfied and 10.1% not satisfied at all (see ►Fig. 6).

Discussion
Multiple studies have reported benefits of virtual consultations including improving patient access to healthcare and Fig. 1 Recording of call quality of virtual appointment in first cycle. increasing engagement in patients with complex medical needs. [13][14][15] This study aimed to explore the outcomes of virtual appointments from a secondary/tertiary care provider and to assess patient follow-up after the appointment. By gaining an insight into the patient pathway and journey, this not only provides the operator with greater clarity into how virtual appointments are currently being used but also allows the healthcare provider to ensure patient-centered care.
Results from our clinical audit demonstrate the majority of patients had a successful outcome from the virtual appointment (see ►Table 2). "Successful" outcomes of virtual appointment were defined in cases where the virtual appointments were used as a substitute instead of a supplement to FTF appointments (see ►Table 1). It is important to note that the measure of success is subjective and varies according to the provider/ specialty and aims of the virtual appointment. In our study, there was a reduction in "successful" outcomes in the second cycle as majority of patients who were categorized as "unsuccessful" required a further appointment for special investigations, for example, impression taking, radiographs, or photographs. This, in our opinion, does not showcase a reduction in standard or quality of care; however, it highlights the importance of systematically triaging referrals for suitability as virtual appointments. However, the outcome from cleft patients undertaking virtual appointments was successful in 83.2% of cases that met the set standard. This indicates that some specialties in dentistry may be able to utilize it more successfully than others. There were no changes in the success of outcomes when assessed by the ages of the patient. Patients who were placed under "watchful wait" or "follow-up virtual consultation" highlighted the success of the virtual consultation in providing routine dental care for patients.
The DNA rate of virtual appointments in the first cycle was 9.5, and 21% in the second cycle. The DNA rate of the first cycle was consistent with the DNA rate from a similar time period in 2019 (8.5%); however, there was an unexpected increase in DNA rate during the second cycle. The majority of patients did not provide a reason or explanation for failing to attend the virtual appointment and perhaps this is an area that requires further development and improvement. Further insight will be helpful to identify the various reasons leading to reduced patient engagement, although reasons are often multifactorial. [16][17][18][19] However, one would assume with the lockdown measures in place currently in the United Kingdom at the time this article was written, with majority of the workforce working from home (unless unable to do so); this would allow for greater flexibility and compliance in attending virtual consultations.
When assessing the patient's experience of the virtual appointments at Alder Hey Children's Hospital, 65% of patients were satisfied with the consultation and would attend a virtual consultation again. This was reinforced with the clinicians' experiences, as the majority felt satisfied with the outcome   Fig. 3 Recording of call quality of virtual appointment in second cycle. and usage of the virtual consultation (50.5%). This was consistent with previous studies, with research recommending triaging by clinicians to ensure patients suitable for a remote consultation are offered one to ensure the efficient allocation of resources. 20,21 Interestingly, a large proportion of clinicians (36.7%) was neither satisfied nor dissatisfied with the virtual appointment, with 10.1% not satisfied with the virtual appointment. It would be worthwhile to study in greater depth the clinician/operator's perspective of the virtual appointment to identify and assess reasons for deviations in satisfaction of usage of virtual clinics when compared with the patient's outcomes and their reported satisfaction.
As dentistry involves examination and assessment of the oral cavity, limitations to this study include basic limitations with virtual appointments. 22,23 Patients with complex dental conditions often required a further FTF appointment to allow in-depth clinical examination, and this would have limited the usefulness of the virtual appointment. Further limitations of this study were basic questions in regard to the patient and clinicians' experience of the virtual consultation. Another limitation to this study was the parameters deemed a successful and unsuccessful outcome by the audit team may have not been agreed upon from the point of view of the operator.
This study sought to explore how virtual appointments are being used across different dental specialties in a secondary/ tertiary care provider, and what the outcomes were from these appointments. From our study, we identified that virtual appointments are being used as an adjunct to FTF appointments in most dental specialties at Alder Hey Children's Hospital. Virtual appointments can be utilized when a thorough clinical examination is not indicated, for example, providing oral hygiene advice and toothbrushing advice, follow-up of patients postoperatively, updating medical and social history, discussing likely treatment options and benefits and risks of treatment, and building rapport with anxious patients. 24 The benefits of teledentistry for new patient consultations are uncertain, which suggests further insight and development into triaging of new patients referred to the various specialties to determine the suitability for virtual consultation. Virtual appointments offer numerous possibili-ties beyond the current COVID-19 pandemic, where integration of these systems will allow us to develop a more standardized approach to healthcare. There were some obvious benefits of virtual appointments: 1. Outreach clinics continued virtually that otherwise would have been discontinued. 2. Clinicians who were shielding could participate in clinics. 3. Patients who were shielding could attend virtual appointments. 4. Virtual clinics did not add to the waiting room footfall providing the flexibility to see more patients.

Conclusion
The rise of virtual appointments in response to the COVID-19 pandemic has presented certain challenges to the routine delivery of dental care. Results from our study show a successful outcome could be drawn from the majority of virtual appointments. Clinicians and patients were generally satisfied with using the "Attend Anywhere" virtual platform. It is the decision of the individual units and clinicians to consider advantages and disadvantages of virtual clinics in different specialties in dentistry. It makes it challenging to ascertain if the outcome was successful or unsuccessful in certain consultations. It is interesting to note that subsequent to this audit and with further experience with virtual clinics, clinicians are continuing to use the virtual clinics in 2021.
Authors' Contribution YWJL lead the study. YWJL, AP, and SA undertook the data collection. YWJL, AP, and SA drafted a paper, and MS contributed to the final draft. All authors gave their final approval and agreed to be accountable for all aspects of the work.

Funding
The authors of this manuscript declare that they have no conflicts of interest. No funding was received for the study.