Background and Significance
Part C of the Individuals with Disabilities Education Act[1 ] authorizes states to create early intervention (EI) programs, which currently serve
3.7% of U.S. children ages 0 to 3 years old.[2 ] EI programs aim to offer family-centered services in the child's natural environment
to improve child and family outcomes, such as increasing family access to valued activities
for their child to participate and develop skills.[3 ] However, children and families experience disparities in their access to and use
of EI services that have historically relied on face-to-face processes of care.[4 ]
[5 ]
[6 ] Practice-based strategies for improving equitable EI processes, such as those proposed
for EI referral and evaluation,[7 ] hold promise for improving EI service quality.
The use of patient-reported outcomes (PROs) is endorsed as a practice-based strategy
for advancing value-based services for children and families.[8 ] Electronic patient-reported outcomes (e-PROs) can serve as an informatics tool for
including the child and family perspective regarding their needs and priorities when
designing and evaluating services.[9 ]
[10 ] In EI, e-PROs can be used in direct service provision to consider family needs and
priorities for tailoring the development of the individualized family service plan
(IFSP) that should include functional goals and strategies for goal attainment based
on family-identified priorities and inclusive of their expertise. Some e-PROs can
also be useful for tracking progress to guide shared and data-driven decisions about
tailoring interventions with families.[10 ] In both cases, e-PROs support equitable services via quantifiable estimates from
the family that are consistent in the information they capture and minimize the need
for clinician interpretation (and implicit bias).[10 ] Beyond their benefits to individual-level services, e-PROs yield data that can be
aggregated for conducting robust health services and implementation research, to demonstrate
the value of EI services and guide decisions for improving these services.[10 ]
[11 ] While there are challenges to appraising and implementing e-PROs in pediatric contexts
like EI,[11 ]
[12 ]
[13 ]
[14 ]
[15 ] they hold promise for accelerating quality improvement of these services in programs
with electronic data capture systems.
The Young Children's Participation and Environment Measure (YC-PEM) is a promising
e-PRO informatics tool that offers an evidence-based strategy to deliver family-centered
EI in programs with electronic data capture systems.[16 ]
[17 ]
[18 ]
[19 ]
[20 ]
[21 ]
[22 ]
[23 ]
[24 ] Its completion takes approximately 20 to 30 minutes[18 ] and yields a summary report for caregivers to share with their EI service providers.[25 ] EI stakeholders have participated in research to develop, evaluate, and implement
it per best practice standards.[26 ]
[27 ]
[28 ] Specifically, the YC-PEM e-PRO and its companion goal setting application were developed
primarily in partnership with EI providers[17 ]
[29 ] and EI families,[18 ]
[23 ]
[30 ]
[31 ]
[32 ] and EI program leadership (PL) partnered in developing organizational infrastructure
needed to incentivize EI provider involvement in research on this topic.[33 ] Together, these EI stakeholders informed decisions about how and when to introduce
the YC-PEM e-PRO option for inclusive service design toward meaningful EI outcomes,
including their child's participation in valued activities.[34 ]
[35 ] For example, these EI stakeholders have shaped decisions to explore implementing
the YC-PEM e-PRO as part of the child's service visit and annual evaluation of progress.[17 ]
[18 ]
[36 ]
To date, EI stakeholders representing differing perspectives about information exchange
during family-provider encounters[37 ] and at all levels of an EI organization (i.e., families, EI providers and service
coordinators (SC), and PL) have been engaged to develop the YC-PEM e-PRO option (e.g.,
layout of the summary report) and select implementation strategies (e.g., research
group infrastructure, timing of its implementation). However, they have not yet been
engaged to identify the full range of relevant supports and barriers to implementing
the YC-PEM e-PRO option across EI programs, for improving both individual-level information
exchanges within routine EI workflows and for guiding decisions for quality improvement
of EI services.[38 ]
[39 ]
Methods
Study Design and Setting
This cross-sectional study employed an explanatory-sequential (quan> QUAL) mixed-methods
study design.[41 ] This study is part of a larger implementation research project that follows a hybrid
type-1 effectiveness-implementation research approach.[42 ]
[43 ] A mixed-methods study design is appropriate when one research approach is insufficient
for addressing the main research problem under investigation.[41 ] The choice to mix quantitative and qualitative methods during data collection was
deemed most appropriate based on the breadth and nature of stakeholder input needed.
Choices for sequencing and emphasizing qualitative data collection were made a priori
by the corresponding author who had prior experience in using this particular mixed-methods
study design (M.A.K.),[29 ]
[30 ] to further explain a subset of quantitative findings concerned with pilot implementation
of the intervention.[41 ] This research took place at a large, urban, and non-university affiliated EI program
in the Denver Metro catchment of Colorado. Ethical approval was obtained by the Institutional
Review Board at the University of Illinois (protocol #2020–0555) and the University
of Colorado (protocol #20–2380). We registered the Parent-Reported Outcomes to Strengthen
Partnership within the Early Intervention Care Team (PROSPECT) trial at clinicaltrials.gov (NCT04562038) and published protocol details.[40 ]
Participants
During the implementation trial period, three groups of stakeholders were recruited
from a large, cosmopolitan EI program: Families, SC, and PL. This EI program was comprised
of more than 45 EI service providers (e.g., SC, skilled therapists, special educators)
serving over 2,200 children and families. Families varied on sociodemographic factors
(e.g., 51.7% with Medicaid and 46.5% as Non-Hispanic White) (Jodi Litfin-Dooling,
personal communication, June 28, 2022): Six families were assigned to the intervention
group during the PROSPECT trial and indicated interest in participating in a follow-up
interview or focus group. Nine SC were enrolled, including five SC who were engaged
in conducting the larger trial and four SC who were not involved in the trial. PL
(n = 7) comprised of directors, supervisors, and data and program managers. SC and PL
stakeholders with at least 2 years of EI experience were recruited by email. Families
were included if they: (1) were at least 18 years old; (2) identified as a parent
or legal guardian of a child receiving EI; (3) were able to read, write, and speak
English; (4) had internet and telephone access; and (5) had a child 0 to 3 years old
who received EI for 3 or more months. Each participant was issued a $20 electronic
gift card.
Data Collection and Procedures
SC and PL stakeholders completed a demographic questionnaire online and prior to qualitative
data collection. Quantitative data on implementation were collected as part of a larger
pragmatic trial on the effectiveness of the YC-PEM e-PRO, when paired with a co-designed,
program-specific decision-support guide, on select outcomes.[40 ] These quantitative data on implementation were summarized and used to prompt stakeholders
to explain findings during qualitative data collection.[40 ] Specific data on sample characteristics and implementation included: (1) items administered
to caregivers online (e.g., “How helpful was the [instructional] video for completing
the YC-PEM e-PRO questionnaire?” [from 1 = not helpful to 4 = very helpful]); and
(2) items administered to SC online during their monthly meetings (e.g., “How helpful
were monthly meetings with other SC in the intervention group” [from 1 = not helpful
to 4 = very helpful]; “How helpful were the YC-PEM e-PRO results within your IFSP
documentation to guide your conversation with parents during the annual evaluation
of progress” [from 1 = not helpful to 4 = very helpful]).
The qualitative data collection phase included six families each taking part in a
30 to 60-minute semi-structured virtual interview, and two in-person focus groups
(3–4 SC and PL per group) at an off-site location ([Supplementary Appendix A ], available in the online version). Additionally, one SC was rescheduled for a virtual
interview. Individual interviews were chosen for families to gain deeper insight into
the study topic, and focus groups for SC and PL were used to encourage discussion
by employees of the organization.[44 ] Sessions were co-facilitated by two of four study staff (V.C.K., J.G.C.S., N.J.M.,
and M.A.K.), most of whom had prior qualitative and/or mixed-methods research experience
for examining mechanisms of EI care coordination and/or further developing the PEM
approach for use in EI. These study staff used semi-structured interview guides, one
version of which was piloted with a caregiver of a child who had received EI services.
During each session, participants (families, SC, and PL) were presented with data
displays (bar graphs and tables) of quantitative findings and then asked to explain
the results shown. We also asked all stakeholders to elaborate or explain facilitators
and barriers to implementation that had surfaced during monthly SC meetings (e.g.,
optimal timing of YC-PEM e-PRO completion for use at the IFSP meeting). In-person
focus groups and virtual interviews were audio and video recorded.
Data Analysis
Quantitative findings were analyzed descriptively and presented as percentages to
all stakeholders during the qualitative data collection phase. Each completed session
yielded a recording that was transcribed verbatim by a staff member (J.G.C.S.) and
imported into NVivo 13.0[45 ] for analyses. Study staff performed content analyses on these transcripts, listening
to audio and viewing video recordings as needed for coding accuracy.
For Aim 1, two of three study staff (V.C.K., J.G.C.S., S.R.) concurrently analyzed
each of three sets of transcripts for a stakeholder group (i.e., families, EI service
providers, and PL). Study staff first used deductive content analysis[46 ] to independently code data to established constructs for four of five domains in
the Consolidated Framework for Implementation Research analytic framework (CFIR),[47 ] reaching a Kappa of 0.86 across all stakeholders and CFIR domains. The CFIR is an
established determinant framework that is appropriate for examining multiple levels
of influence within the organizational context.[48 ] The CFIR domain “characteristics of individuals” was excluded from the codebook
because this study involved an established High Value EI Research Group with known
characteristics. Coding discrepancies were resolved through discussion to achieve
consensus on final constructs, combining select CFIR constructs with similar coded
content for ease of interpretation. The coded data for each CFIR construct (original
or combined) were then inductively analyzed to generate categories that were further
collapsed into themes within original or combined CFIR constructs. These themes were
presented using frequency counts per stakeholder group, and deidentified excerpts
from stakeholders' full transcripts to demonstrate frequency, extensiveness, and specificity
of concepts or themes.[49 ]
For aim 2, we shared created themes specific to SC and PL perspectives with six national
early childhood advisory group members to check their relevance across multiple EI
programs. These advisors have professional histories including EI practice experience
and contribute to improving EI quality in multiple states, but they have limited knowledge
of the YC-PEM e-PRO that has been previously designed and tested in close partnership
with caregivers of young children with developmental need.[18 ]
[23 ]
[30 ]
[31 ]
[32 ] Therefore, geographically diverse professional expertise was prioritized in this
study phase to help identify scalable supports and strategies given the state-to-state
variability in EI service provision. During a 30-minute virtual session, two staff
(S.R. and V.C.K.) co-facilitated with advisory leadership (A.S., L.J., and K.B.),
to solicit advisors' ratings on main findings according to: (1) the importance of
the support, strategy, or barrier for implementation and (2) how easy this strategy,
support, or barrier would be to achieve during implementation. Each item response
was assigned a weighted percentage for importance and difficulty (i.e., 25% = not
important at all/very difficult to 100% = very important/very easy) and multiplied
by the number of respondents who indicated each response option. Item responses were
used to confirm thematic findings (e.g., each main thematic finding was perceived
to be important to one or more advisors) and informed the ranking of the CFIR constructs
within CFIR domains. SC and PL perspectives were centered in this study because caregiver
perspectives were emphasized in prior phases of implementation research on this topic,
and because of this study's focus on systemic implementation.[18 ]
Results
As shown in [Table 1 ], participants were six family caregivers (four female, two male) of children receiving
EI services, nine female SC, and seven female PL. Families differed according to their
child's developmental need, but each had completed graduate coursework or a degree,
and most families earned above the U.S. median household income.[50 ] Most families (83.3%) perceived technology to be very important for working with
their EI team. SC and PL had 2 to 15 years of EI experience and diverse fields of
degree.
Table 1
Early intervention stakeholder characteristics
Stakeholder
Caregiver race
Caregiver role
Family annual income ($)
Child diagnosis/delay
F1
Caucasian
Mother
>100,000
Bilateral hearing loss
F2
Caucasian
Father
90,001–100,000
Congenital diaphragmatic hernia, tracheostomy, gastrojejunostomy tube, global delays
F3
Caucasian
Father
40,001–50,000
Developmental delay (no diagnosis)
F4
Caucasian
Mother
>100,000
Childhood apraxia of speech
F5
Caucasian
Mother
>100,000
Hearing loss
F6
Black or African American
Mother
70,001–80,000
Developmental delay (no diagnosis)
Stakeholder
Years of service in organization
Years of EI experience
Field of degree
SC1
2.5
2.5
Human development
Early childhood education
SC2
2
2
Early childhood education
SC3
2
2
Speech and language pathology
SC4
16
14
Social work
SC5
2
2
Not disclosed
SC6
2
2
Social work
SC7
8
8
Human development
Family studies
SC8
4
4
Education
Human development
SC9
11
15
Sociology
Stakeholder
Years of service in organization
Years of EI experience
Field of degree
PL1
3
3
Sociology
Women's studies
PL2
12
12
Psychology
PL3
2.5
2.5
Social work
PL4
2
15
Speech and language pathology
PL5
12
14
Social Work
PL6
4
4
Education
PL7
5
7
Social Work
Early childhood education
Abbreviation: EI, early intervention; F, family; PL, program leadership; SC, service
coordinator.
As part of Aim 1, all three stakeholders (i.e., families, SC, and PL) identified supports
and barriers across multiple constructs for each of four CFIR domains ([Fig. 1 ]). We report descriptive quantitative results using percentages and frequency counts
(n ) and exemplar quotes for 19 themes associated with original or combined CFIR constructs
([Table 2 ]). Findings within each CFIR domain are presented in rank order based on advisory
feedback obtained as part of Aim 2.
Fig. 1 Qualitative findings by consolidated framework for implementation research (CFIR)[47 ] domains and constructs. EI, early intervention; e-PRO, electronic patient-reported
outcomes; IFSP, individualized family service plan; YC-PEM, Young Children's Participation
and Environment Measure.
Table 2
Consolidated framework for implementation research (CFIR)[47 ] domains and constructs with exemplar quotes
CFIR domain
CFIR construct as addressed by EI stakeholder group(s)
Exemplar quotes
Construct importance and ease (%)
Intervention characteristics
Complexity, design quality and packaging
Data-driven programmatic decision-making
“…I mean that [aggregate of what the families are needing] could help with us thinking
even about hiring and staffing and training […] to improve the quality and accessibility
of our intervention services and [what things] were really prevalent in our community
as needs. And we could think about like, ‘Oh, is there a resource out there that we
can engage and help our families to know about?’.” (PL2)
Easy to complete with adequate resources
“It was pretty straightforward to me, but I think, in my case, I was willing to devote
time to it. It was just that, the, the day that I did it turn into a really busy day,
and then I was just trying to catch up to the rest of the day. Um, but it really just
came down to the fact that the survey let me do that, it let me keep going. Right,
moving quickly.” (F3)
80%
Adaptability and trialability
Starting small with EI provider engagement
“I think we were definitely taking more of a thoughtful approach [by starting with
a smaller group]. We're kind of finding what's working, what's not working, and tweaking
things […] with the intention of also adding more people, adding their input and sort
of, you know, continue to grow it from there.” (SC7)
“I think having people that have already been more familiar with it and then slowly
integrating into the rest would be helpful. Because if you, do it all at once, there's
going to be a lot of questions and a lot of things that might not go smoothly. But
this way, like the smaller group is more controlled, and […] as you add people, there's
people that can ask or answer those questions and help fix anything” (SC6)
Diversifying by language and mode of administration
“I'm wondering if .. if there was a button that somebody, you know, prerecorded if
the family needed help or they didn't quite understand […]. Kind of giving them like
a face with the voice. And I also think about families with different learning styles.
Um, if it's all reading that might not be as accessible […]. So, if we had like a
text to speech option on it, and some more of those example piece. I think that would
be beneficial” (PL6)
75%
Relative advantage
Capturing family priorities and doing so, ahead of time
“It's a lot of extemporaneous [information] during the meeting […] and so I think
it was […] helpful to remember like to think about sort of what were the things we
wanted from the service that we were getting [to be] better prepared” (F2)
“[It] is an important tool because it helps us to say, here are the things that you
want to focus on […] for your plan.” (SC8)
“I think the YC-PEM tool is unique to help the family get started and thinking ahead
of time […] ‘How have my routines changed in the last six months?’ and ‘What might
I want to work on?’ Instead of just at the end of the meeting being like, ‘So [we]
have this goal.’ […] I think it's a really great opportunity for families to get to
have some reflection outside of that meeting, without people staring at them or feeling
pressured to answer questions. Which just makes the meeting in my opinion just more
authentic and […] very genuine.” (PL5)
Gaining efficiency for IFSP development
“[…] the question that was asked which was very narrow and specific and so there was
a ton of questions about a few very specific activities that he's never been involved
in and so kind of felt like that made the whole thing look like he wasn't really involved
in life at all in ways that we didn't feel like were representative of our experience.” (F2)
73.5%
Intervention source
N/A (no data mapped to this construct)
Evidence strength and quality
N/A (no data mapped to this construct)
Cost
N/A (no data mapped to this construct)
Process
Engaging
Affluence and linguistic diversity
“[This was] made available for families who […] self-select if they speak English.
And so when I'm looking at this I'm also thinking about how families, Hispanic, LatinX,
or Spanish origin, if they're like, ‘Oh my goodness, this is going to be fully in
English’. They might not feel comfortable enough to proceed.” (PL5)
Provider engagement and identity
“[The] podcast was such a great motivator [...]and so I think that speaks volumes.
When we had talked about the podcast and then that great feedback the family gave
us about feeling more prepared for their meeting […] I think it spoke to [the new
hire] and now that she [is part of the project and] is going through grad school she
wants to be better connected to research. And so this is fabulous, really.” (PL1)
“I love the way that we also divided up the work and that we had [a family engagement
specialist] there who was our outreach specialist and additional layer of support
because I could go to her and bounce ideas off to make sure that our staff understood
what those next steps were. And I think also the level of communication [and accessibility
of team members] has been really valuable as well, because when I did run into a stumbling
block, I'm like' this is going to impact what we're doing', I could go to call [the
project coordinator], and we could brainstorm. I didn't need a scheduled meeting with
[the project coordinator] every month. I just knew I needed to know that I can reach
out yeah exactly.” (PL1)
80%
Reflecting and evaluating
Protected space for provider reflection and learning
“I definitely think at first it would be helpful to have like, like the monthly meetings
that we had or like biweekly […] like even if I felt like I didn't always have questions
going into the meeting, I definitely felt like it was always helpful to hear what
other people had to say, and just kind of have that time set aside to be able to […]
think about it and talk about it.” (SC5)
Instructional video for family reflection
“Video probably captures the widest audience and that, you know, allows for those
who […] vary in literacy or ability to read through the material to have a clear video.
So I think from that perspective, having a video like that is probably still the best
option. There may be some who prefer to take that in other ways like reading, or simply
listening. But I think to be honest for the, the audience that this likely targets
that's probably the high yield mechanism.” (F4)
“I feel like I do need the visual and if it's in a video it's definitely more helpful
and easier to understand than just, you know, a questionnaire, or it being written.
So I feel like this is pretty accurate to how my families would react to it.” (SC6)
“[…]but also maybe for parents who are more often, or more intensely distracted by
their kids. And, and have a hard time just sitting down and reading something to have
something playing that they can listen to can be helpful.” (SC8)
71.3%
Planning and executing
Timing and automating of processes
“it may not have actually been that far that long time, but it felt like it was longer
than it should be. To really remember the answers.” (F3)
“a month could be huge in development, the life of a child […] your kids like starts
daycare something you know like their lives are completely different.” (SC3)
“if I knew that this was serving the purpose of informing decision making for like
what our services are going to look like, then I think it would make it more important.
I guess like more just meaningful.” (F5)
Optional or required
“I also feel like fine about it being like an optional thing because it's like the
parents that have the time and want to do all this extra stuff to really get the most
out of this experience, like good for them, and then for the parents who like, it's
like an accomplishment for them to just show up to their therapy session […] I feel
like I go back and forth between like maybe the people that it would help the most
would be the people that would be less likely to volunteer.” (SC5)
68.3%
Inner setting
Culture, network and communication
Education and practice for clear communication
“… in the past, I could have looked at some parents differently than others, because
I thought some parents were going to come to the meeting prepared and some weren't
and it's really just the biases or deficit thinking could get in the way.” (F5)
“I needed somebody to translate it to me, yeah, but I do feel like when we were then
implementing like you gave me the YC-PEM and I could see it and I walked through it
as though I was the parent, … boosted the confidence of our staff to do it and myself.” (PL1)
92.5%
Structural characteristics and Implementation Climate
Increasing and changing workload demands
“… as a service coordinator, it does take that extra step to prepare and make sure
that we're reading that. And there's sometimes, as I'm sure that N and A can attest
to, where you just don't have that extra time. So having that extra step could make
it more difficult to make sure the tools being more of a help, than it is not.” (SC1)
“And I have to say that at this point, for example, I'm handling 110 cases. It wouldn't
be feasible for me to add another layer. I'm not discarding that it's just at this
moment it wouldn't be practical and feasible, at least from my perspective. I already,
on top of the parents [I am] like, ‘oh please sign this document, please’.” (SC2)
“[if you] say that that research is important, then you're just going to implement
it as part of your workflow. And you're going to demonstrate it in active ways to
your staff by having them be a part of it or at least dip their toe and try it.” (PL1)
Incentivizing organizational change
“if I'm being 100% honest, if I had not been offered an incentive and looked at that
survey and it was optional, I probably would not have taken the time to fill it out.” (F1)
“some incentive. And, like, probably stronger, more like people who've been there
for a while, I guess.” (SC3)
76.3%
Outer setting
Cosmopolitanism
Improving family-centered pathways into EI
“I want a whole team of navigators, and like every touch point that we have, [so]
we're getting [in] there, like coming in at the connection from referral to evaluation
[…] and the Denver Health Navigator now coming in, when the pediatrician is doing
the screening to help you explain to the family like what EI is [and] understanding
the model, I think at our touch points.” (PL4)
“The Head Start [is] for kids that are over three so it's the Early Head Start programs.
They're in the same facilities [and] we work really closely with [them]. […] If we
were able to […] market to gynecologist, right, to OB-GYNs […]. She [the gynecologist]
didn't have a clue that [EI] was even available […]. So I think there are so many
places that we can try to […] get our, our message out there where we can start when
they're pregnant. Because, yeah, I was blown away that [my gynecologist] did not have
any idea what early intervention was.” (PL5)
Improving family-centered pathway within and across EI programs statewide
“So I'm really excited to see like as they [the state] continue to revise this tool,
and if we can use it for those ARs [annual reviews…] I would want to keep an eye on
that […] as like an additional quality tool [to] see, like are we writing better plans.” (PL1)
70%
Patient needs and resources
Meeting needs of families accessing EI
“Part of the point of early intervention is like meeting the family where they're
at […]. Sometimes they can't even afford rent and that's what they're thinking about
is like […] we always talked about like the Maslow's hierarchy of needs” (SC5)
“I think it really shows that issue of access. There […] are barriers to EI but it's
available to everybody. […] seeing who engages, who enrolls and who was served is
a reflection of kind of the issues that might be out there […]. Non-White and lower
income families were not making it into early intervention like at the same rate […].
This is very consistent with that data.” (PL6)
65%
External policies and incentives
COVID Restrictions Impacting YC-PEM e-PRO Completion and Family Enrollment
“There are a lot of questions about how do you participate in activities in the community…He
doesn't because we've been locked down for 15 months […] as a family have been particularly
cautious because I'm lucky enough to work from home and my wife wasn't working.” (F3)
“We're really focusing on trying to get families […] to actively engage in services
when the pandemic first hit and we had to go completely telehealth.[…] if you're going
back to just trying to meet your basic needs, you're not going to feel capable to
engage.” (PL1)
“And they're doing what they can to be getting services for their child […] it's very
stressful especially now with COVID still happening and everything. I feel like parents
are feeling more isolated just […] because they have been in their house for so long
and they don't feel comfortable with their kids going to daycare maybe or they've
lost their job because of everything that's going on […]. It's a lot of outside factors
that they just can't […] add one more thing to their plate.” (SC6)
“That's kind of the time we're in right now […]. People haven't had a lot of interactions
with adults, and you ask them one question and they fill out their entire life story
[…] because you haven't seen adults, like face to face, for quite some time […] some
people need that interaction.” (SC6)
50%
Peer pressure
N/A (no data mapped to this construct)
N/A
Abbreviations: CFIR, consolidated framework for implementation research; EI, early
intervention; e-PRO, electronic patient-reported outcome; F, family; IFSP, individualized
family service plan; PL, program leadership; SC, service coordinator; YC-PEM, Young
Children's Participation and Environment Measure.
CFIR Domain: Intervention Characteristics
EI stakeholders discussed two themes in each of three original and combined constructs:
(1) “complexity, design quality, and packaging,” (2) “adaptability and trialability,”
and (3) “relative advantage” ([Fig. 1 ]).
CFIR Constructs: Complexity, Design Quality, and Packaging
Data-Driven Programmatic Decision-Making
PL (n = 2) recognized the potential of the YC-PEM e-PRO to inform data-driven, programmatic
decision-making by being “really helpful for us [EI program] to just have a good sense in aggregate of what
the families were needing help with” (PL2). PL further described how this aggregated information could help them improve
quality and accessibility of their services.
Easy to Complete with Adequate Resources
All stakeholder groups (families [n = 2], SC [n = 1], PL [n = 1]) found the YC-PEM e-PRO to be clear and easy to navigate. Some families (n = 3) and SC (n = 2) recognized potential challenges to YC-PEM e-PRO completion by families without
“active internet access […], intact vision and hearing ” (F2), English proficiency, and/or adequate time. SC also reflected on time pressure,
sharing that SC could become overwhelmed when integrating information from a YC-PEM
e-PRO report during their IFSP meeting and wanting to “make sure that you're kind of hitting all those points that the family, kind of wanted
to improve and work on” (SC3). They suggested increasing use of skip logic and/or administering it in modules.
CFIR Constructs: Adaptability and Trialability
Starting Small with EI Provider Engagement
All SC (100%) in the intervention group perceived the benefit to starting with a smaller
provider group when adapting and testing the implementation of the YC-PEM e-PRO in
their EI program. In explaining this result, they described being able to engage in
trial and error and access peer support by creating “that kind of community […] to bounce off of each other if we need support” (SC8), collectively and collaboratively brainstorming ideas.
Diversifying by Language and Mode of Administration
All three stakeholder groups (i.e., families [n = 6], SC [n = 4], and PL [n = 4]) further described the benefit to offering the YC-PEM e-PRO in different languages
and expanding modes of administration, suggesting online or phone support (“complete a quick call with them ” [SC1]) and the option to “click on the audio [and have someone] read the question [to] you” (F6).
CFIR Construct: Relative Advantage
Capturing Family Priorities and Doing So, Ahead of Time
All SC perceived the YC-PEM e-PRO as very helpful (67%) or helpful (33%) in representing
family priorities for goal setting. These results reveal the YC-PEM e-PRO offers performance
advantages, which were further explained by most stakeholders (i.e., families [n = 5], SC [n = 7], and PL [n = 7]) as helping families to reflect on their needs and priorities prior to the IFSP
meeting, to strengthen meaningful discussions and create quality goals for the service
plan during the meeting.
In addition to relieving families of conceptualizing and articulating their priorities
on the spot at the IFSP meeting, the YC-PEM e-PRO option afforded SC “to see what's in the environment [and] read for what the family might want or need (SC1)” due to the coronavirus disease (COVID) restrictions that hindered their access to
this information via home visits.
Gaining Efficiency for IFSP Development
SC reported that the YC-PEM e-PRO is helpful (67%) or somewhat helpful (33%) to guide
the family-professional exchanges during IFSP development. In explaining these results,
some families recognized that information generated by the YC-PEM e-PRO underlies
its benefit to IFSP development, depending on family expectations (e.g., information
is less helpful for those activities that are less valued by families [n = 2] or not deemed to be problematic owing to COVID restrictions [n = 3]).
Several SC (n = 3) shared how YC-PEM e-PRO results help them to facilitate more efficient IFSP
meetings, “because you're not going back and asking them to repeat information” (SC1) and, in turn, can free up resources so that “we can do more for families utilizing it” (SC6).
CFIR Domain: Process
For this domain, EI stakeholders described two themes for each of these original and
combined constructs: (1) “engaging,” (2) “reflecting and evaluating,” and (3) “planning
and executing” ([Fig. 1 ]).
CFIR Construct: Engaging
Affluence and Linguistic Diversity
When reflecting on participant engagement, one member of each stakeholder group mentioned
our inclusion criteria, with respect to “a selection bias process […] the people who answered the survey are the ones who didn't
have barriers that stop them from answering” (F2). Most PL (n = 5) explained the value of expanding engagement to Latinx families and those with
undocumented status and who may not identify as English speaking. Similarly, two PL
mentioned a tendency of SC with higher education levels to volunteer as early adopters
and implementers of the YC-PEM e-PRO.
Provider Engagement and Identity
SC and PL (n = 4) suggested strategies for engaging providers as early adopters. Strategies ranged
from sharing information (e.g., family quotes, their podcast on the history of this
research partnership)[51 ] to teamwork (i.e., designating project coordinators at research and EI sites, designating
EI staff to screen and recruit families, recruiting a family engagement specialist,
and organizing team experiences to train and troubleshoot). Several PL (n = 3) valued their transition to including all new hires to their “High Value EI Research
Group” as part of their onboarding process and recognized their need to avoid calling
it a “writing group,” a misleading informal group name risking to “keep them [SC] from taking that next step because they're thinking that they're […]
not strong in that area” (SC4).
CFIR Construct: Reflecting and Evaluating
Protected Space for Provider Reflection and Learning
Most SC assigned to the intervention group perceived their monthly meetings as very
helpful (33%) or helpful (67%). Some PL (n = 2) and one SC elaborated on the importance of creating an environment where team
members “[come] together, virtually or in some sort of chat” (PL1) to share their concerns, generate ideas, and exchange suggestions. “That's when people realize oh, you're experiencing the same thing” (PL1). However, SC recognized the risk to discussing specific nuances of “some cases […] and other SC [not having] that kid on their caseload” (PL3) and how that might be less relevant to all group members.
Instructional Video for Family Reflection
Most families (89%) reported watching the optional YC-PEM e-PRO instructional video
prior to completion. While most families found this implementation strategy to be
very helpful (35%) or helpful (47%), some families rated it as somewhat helpful (12%)
or not helpful (6%). Most families (n = 4) and some SC (n = 2) explained these findings by describing its benefits including (1) an overview
of what to expect, (2) a resource to refer to as needed, (3) encouraging respondents
to take their time, (4) supports self-reflection, and (5) supports families that need
visual support and help to focus when surrounded by distraction(s).
They explained that the video might have been less helpful to those who “didn't really find the instructions complicated” (F3), those who lacked time given the amount of information families receive before
an IFSP, and/or those who are in an inconvenient location to watch it. Most SC (n = 5) and one family emphasized the value of giving families options to counter such
issues, “besides the video, something that I can follow through at my own pace ” (SC2).
CFIR Constructs: Planning and Executing
Timing and Automating of Processes
All three stakeholders (families [n = 2], SC [n = 8], PL [n = 6]) elaborated on the importance of creating a consistent and compatible timeline
for implementing the YC-PEM e-PRO in their workflow. SC (n = 7) and families (n = 2) preferred the YC-PEM e-PRO report being sent to the SC well ahead of the scheduled
IFSP meeting, “like maybe a week ” (SC1), to ensure its integration in their meeting preparation. However, some stakeholders
(families [n = 1], SC [n = 2]) recognized the reduced value when YC-PEM e-PRO results are generated too far
in advance of the meeting.
Most SC (n = 6) and some PL (n = 2) described instituting flexible timelines, such as asking for the report within
“about 24 hours or 48 hours” (SC9) of the IFSP meeting and then allowing those families a second option to “login like 10 or 15 minutes early for the annual” (SC5) to complete the YC-PEM e-PRO during this first part of their IFSP meeting time.
Despite these differences, all three stakeholders (families [n = 2], SC [n = 8] and PL [n = 2]) spoke to the value of having these results “fresh in your mind and having it right when you go into the meeting” (SC6) to ensure their meaningful integration into the IFSP meeting. Some caregivers
(n = 2) described wanting more information and instruction to complete the YC-PEM e-PRO
ahead of the IFSP meeting. SC (n = 3) and PL (n = 5) suggested further automating processes in ways that promote YC-PEM e-PRO use
(e.g., send automated meeting reminders and prompts to complete the YC-PEM e-PRO ahead
of the IFSP meeting and “resent to them [the copy of the report] the day before [the meeting] or something” [SC3]).
Optional or Required
Most SC (n = 5) believed optional completion would be more family-centered but recognized the
potential in providing choice such as in the timing of completing the YC-PEM e-PRO
(e.g., “optional filling it out early but mandatory filling it out”; “they could complete
it as part of the IFSP” [SC6]), particularly for families who they anticipate may not opt to complete it
but are those that might benefit most from doing so.
CFIR Domain: Inner Setting
For this domain, stakeholders discussed one theme related to the constructs of “culture,
network, and communication” and two themes related to the constructs of “structural
characteristics and implementation climate” ([Fig. 1 ]).
CFIR Constructs: Culture, Network, and Communication
Education and Practice for Clear Communication
Most families (n = 5) and SC (n = 6), and a PL (n = 1) described the value of YC-PEM e-PRO results being understood so that interactions
and decisions made during IFSP meetings could be justified. Most SC (n = 5) recognized the need for increased practice with the tool, particularly when
“coming in as a new staff, you have no idea about it [implementing the YC-PEM e-PRO
into your workflow]” (SC3).
CFIR Constructs: Structural Characteristics and Implementation Climate
Incentivizing Organizational Change
A climate that fosters tangible and intangible incentives to its staff (e.g., awards,
recognition via performance reviews, salary increases, increased respect) and families
(e.g., gift cards) were described by families (n = 1) and SC (n = 4) as creating a strong implementation climate.
Increasing and Changing Workload Demands
Most SC (n = 4) further described their anticipation of tensions that may arise by implementing
the YC-PEM e-PRO into their existing workflow, considering available resources (e.g.,
“space or time for this to happen” [SC4]).
Two PL echoed the difficulties accompanying workflow changes in a program with high
productivity demands (e.g., “Even if things are going well, there's just lots of demands” [PL1] and “[losing] a huge number of our SC for turnover” [PL1]) yet recognized the importance of taking part in projects to demonstrate their
program's value of research to improve practice.
CFIR Domain: Outer Setting
For this domain, EI stakeholders described themes for the following original and combined
constructs: (1) two themes for “cosmopolitanism,” (2) one theme for “patient needs
and resources,” and (3) one theme for “external policies and incentives” ([Fig. 1 ]).
CFIR Construct: Cosmopolitanism
Improving Family-Centered Pathways into EI
PL (n = 3) reflected on the value of YC-PEM e-PRO data for communicating to EI referral
sources in ways that improve pathways into EI. They described the use of navigators
using data to explain to families the purpose of EI.
Improving Family-Centered Pathway within and across EI Programs Statewide
Most PL (n = 4) elaborated on how the YC-PEM e-PRO illustrates compatibility with state-level
measures and initiatives for ensuring quality IFSP development, and strategies for
expanding its adoption that they “theoretically […] could do […] with the ARs [annual reviews] right now” (PL2).
CFIR Construct: Patient Needs and Resources
Meeting Needs of Families Accessing EI
Two stakeholders (i.e., SC [n = 6] and PL [n = 5]) explained results depicting trial enrollment for example, mostly White (73%)
and non-Hispanic (78%) families with diverse education levels (e.g., 29% earned less
than a college degree). PL explained that these demographics aligned with a “pretty large shift in the demographic of Denver as a city” (PL4), which was perceived to affect EI access and use and, in turn, the types of
families experiencing the YC-PEM e-PRO. They suggested use of a Spanish version to
improve reach.
CFIR Construct: External Policies and Incentives
COVID Restrictions Impacting YC-PEM e-PRO Completion and Family Enrollment
While families (n = 4) expressed difficulty rating their children's participation in community activities
when completing the YC-PEM e-PRO during COVID, because of their limited access to
those community activities, PL (n = 3) noted increased skills with technology use during COVID.
SC (n = 2) and PL (n = 4) described how COVID and the transition to telehealth impacted family engagement
in EI, and consequently, their exposure to the YC-PEM e-PRO and time needed for meaningful
engagement due to social isolation.
Discussion
Family engagement is pivotal to the development and implementation of quality e-PROs
that can facilitate family-centered service plan development and service improvement.
This study elicited perspectives of three key EI stakeholder groups (i.e., families
of young children with developmental needs who are enrolled in EI services, EI SC,
and EI PL) to fill clinically important knowledge gaps about the full range of considerations
for implementing the YC-PEM e-PRO, when paired with a program-specific decision-support
tool, to support family-centered service plan development and service improvement.
Nineteen themes were identified across all four CFIR domains, revealing relevant supports,
barriers, and strategies to implementing this option across EI program workflows.
Intervention and Process Considerations for YC-PEM e-PRO Implementation
Most themes addressed “intervention characteristics” (n = 6) and “processes” (n = 6). For longer-term implementation, EI stakeholders prioritized: (1) its design
(e.g., needing resources to tailor and support easy completion, and shaping its potential
for data-driven programmatic decision-making); and (2) expanding family and provider
engagement (e.g., Latinx families and those with undocumented status and who may not
identify as English speaking).
The need and benefits of easy and tailored YC-PEM e-PRO administration as described
by stakeholders are critical to informatics tools and consistent with prior work.[17 ]
[18 ] For example, caregivers have appreciated the comprehensiveness of the YC-PEM e-PRO
while recognizing the risk to feeling overwhelmed.[17 ] Stakeholders specified new ways to tailor YC-PEM e-PRO implementation, by explicitly
making the option for completing it and timing its administration to obtain accurate
information that can be integrated into service plans. While existing implementation
strategies simplified instruction content (e.g., introductory video for families,
semi-structured case-based monthly research group meetings) to enable stakeholders
to focus and reflect on how they complete or integrate YC-PEM e-PRO results into the
IFSP meeting, EI stakeholder groups introduced new ideas about automating processes
(e.g., increasing automated reminders for completion, automatically resending a report
copy the day before the IFSP meeting). Their ideas yield initial core requirements
for the design and development of YC-PEM e-PRO interface(s) with customizable dashboards
tailored to the distinct informational needs of each EI stakeholder group.[40 ]
[52 ] For example, they proposed custom content displays (item-level vs. aggregate YC-PEM
e-PRO results) and tailored timing of reminders and report information. Human factors
and health informatics expertise may ensure that these interfaces include human-centered
data entry and easy-to-comprehend visualizations through co-design and iterative usability
evaluation and refinement.[53 ]
[54 ]
[55 ]
[56 ]
[57 ] Interventions that are tailored appropriately can better fit into workflow and more
effectively support user needs and activities, to be responsive to health literacy
and cultural needs that can shape patient portal use.[58 ]
The need to continue expanding engagement to less socially privileged families is
also consistent with prior work.[59 ] To ensure equitable use of the YC-PEM e-PRO, developments are underway to: (1) facilitate
cultural adaptations of the tool and its related products (e.g., introductory video),
with and without language translation and per best practice guidelines[59 ]; (2) improve engagement of racially minoritized families to upgrade select content
(e.g., centering the expertise of Black and African-American identifying families
to introduce anti-racist terms within the YC-PEM e-PRO); and (3) extend its functionality
by pairing it with the PEM + . PEM+ is a goal-setting application that is being automated
and personalized to enhance user navigation support online for goal-setting purposes.
It has similar advantages to the YC-PEM e-PRO for families,[31 ]
[32 ]
[60 ] such as the relative advantage for capturing family priorities ahead of an IFSP
meeting, and can be expected to guide a more efficient and meaningful meeting experience.
Inner and Outer Setting Considerations for YC-PEM e-PRO Implementation
There were relatively fewer themes specific to “inner setting” (n = 3) and “outer setting” (n = 4). For longer-term implementation, stakeholders prioritized: (1) culture, network
and communication (e.g., education and practice for clear communication), and (2)
cosmopolitanism (e.g., improving family-centered pathways into EI and within and across
EI programs statewide). Taken together, stakeholders were prompted to consider ways
to better communicate with families receiving EI services and those who might benefit
from these services. This finding may represent opportunities to broaden the value
of the YC-PEM e-PRO for prospective and current families across EI programs. Prior
evidence shows that the YC-PEM e-PRO option is viable for use across EI programs with
varying levels of research culture, but this evidence is restricted to its use with
current families in a single service catchment.[27 ]
[61 ] Since EI programs vary considerably in their workflows, available resources, and
readiness for change, implementing participation-focused innovations like the YC-PEM
e-PRO may benefit from formal assessment of organizational culture and readiness for
change, availability of resources for research partnership, and caseload and time
constraints within the organization and broader service catchment.[33 ]
[34 ] We therefore can and should anticipate ample scope for further work to design hybrid
type-2 and -3 approaches for testing the implementation of a YC-PEM e-PRO option within
a broader EI service system, guided by systematic baseline assessment of each participating
EI program and according to: (1) its research culture (e.g., Community Impacts of
Research Oriented Partnerships [CIROP][62 ]; and (2) its practice behaviors such as adoption of participation-focused practices
that are reinforced by tools like the YC-PEM e-PRO option (e.g., Method for using
Audit and feedback in Participation implementation[63 ]
[64 ]; Professional Evaluation and Reflection on Change Tool[65 ]). Such programmatic assessment could help with testing how well the YC-PEM e-PRO
performs in different conditions, how to leverage available resources to incentivize
and prepare an EI site for implementation, and how to monitor service outcomes.
Work is also underway to examine mechanisms of family-centered service coordination
in broader EI service catchments across multiple states, to further elucidate the
range of EI workflows to be assessed in future trials.[3 ] In the larger trial and related work to examine family-centered EI care mechanisms,
we intentionally focus on strengthening strategies for recruiting, retaining, and
describing underrepresented families in EI who enroll in our research, to support
a more inclusive EI research experience.[66 ] We anticipate its increasing value in future research phases, due to mixed evidence
about the role of health literacy in patient portal use.[58 ]
[67 ]
Limitations
There are study limitations related to the timing of data collection during the COVID
2019 pandemic. First, this pandemic prolonged pragmatic trial data collection, resulting
in the need to present interim quantitative data on implementation to drive qualitative
data collection with the resources available for this project. While final distribution
of quantitative estimates does not significantly differ from the distribution of interim
estimates presented to stakeholders, using interim results during data collection
could have underestimated the magnitude of findings that stakeholders were prompted
to explain in the qualitative phase. Second, we included a small number of participants
and a lower percentage of families relative to prior research,[18 ] thereby potentially limiting saturation of some thematic findings. Families may
have experienced increased response burden while being recruited during their trial
enrollment lasting up to 12 months. Despite this limitation, most themes reached saturation.
