Int J Sports Med 2021; 42(02): 112-121
DOI: 10.1055/a-1212-0649
Review

Evaluating the Implementation of Injury Prevention Strategies in Rugby Union and League: A Systematic Review using the RE-AIM Framework

1  Department for Health, University of Bath, Bath, United Kingdom of Great Britain and Northern Ireland
,
1  Department for Health, University of Bath, Bath, United Kingdom of Great Britain and Northern Ireland
,
2  Institute for Sport and Exercise Medicine, Division of Orthopaedics, Department of Surgical Science, Faculty of Medicine and Health Sciences, Stellenbosch University, Stellenbosch, South Africa
3  IOC Research Centre, Stellenbosch University, Stellenbosch, South Africa
,
1  Department for Health, University of Bath, Bath, United Kingdom of Great Britain and Northern Ireland
4  Rugby Football Union, Twickenham, London, United Kingdom of Great Britain and Northern Ireland
,
Carly D. McKay
1  Department for Health, University of Bath, Bath, United Kingdom of Great Britain and Northern Ireland
5  Centre for Motivation and Health Behaviour Change, University of Bath, Bath, United Kingdom of Great Britain and Northern Ireland
› Author Affiliations
 

Abstract

Rugby (union and league) has come under intense scrutiny due to its injury risk. Various interventions have been introduced to protect players from injury, with many deemed efficacious and advocated for use across various worldwide contexts. However, their implementation is less clear. The objective of this systematic review was to determine whether injury prevention interventions in rugby have evaluated their ‘reach’, ‘effectiveness’, ‘adoption’, ‘implementation’ and ‘maintenance’ as per the RE-AIM Multi-Dimension Item Checklist. Six electronic databases were searched in November 2019. Inclusion criteria included: English language, peer-reviewed journal article, original research, field-based rugby code, prospective intervention. Of the 4253 studies identified, 74 met the full inclusion criteria. Protective equipment, predominately mouthguards, was the intervention of interest in 44 studies. Other interventions included multimodal national injury prevention programmes, law changes and neuromuscular training programmes. ‘Effectiveness’ was the highest scoring RE-AIM dimension (55%), followed by ‘reach’ (26%). All other RE-AIM dimensions scored below 20%. Research currently focuses on determining intervention ‘effectiveness’. For injury prevention strategies to have their desired impact, there must be a shift to address all determinants associated with implementation. Consideration should be given to how this can be achieved by adopting specific reporting checklists, research frameworks and study designs.


#

Background

Rugby (union and league) is popular worldwide with millions of participants [1] [2]; however, the sport has recently come under scrutiny for its injury risk and resultant public health burden [3] [4]. A meta-analysis of time-loss match injuries in professional men’s rugby union reported an injury incidence rate of 81 per 1000 player-hours (/1000 h) [5]. In men’s community rugby union, the incidence is estimated at 43/1000 h [6], with similar rates in rugby league [7]. In the 2010 women’s Rugby World Cup, there was an injury incidence rate of 36/1000 h [8], whilst a systematic review of studies in youth rugby union and league had a pooled injury incidence of 27/1000 h [9]. Whether these rates are considered high, or indeed acceptable [4] [10] [11], it is in the interest of all stakeholders to make rugby as safe as possible for all players.

Various sports have successfully decreased injury risk through preventative measures such as protective equipment [12], neuromuscular training programmes [13], and law changes [14]. Whilst many of these interventions are well received and widely disseminated, most do not have their intended effect in the ‘real world’ owing to poor implementation. For example, the Nordic hamstring curl can reduce hamstring injuries by 51% [15], yet only 17% of professional football clubs use this exercise despite the high prevalence of hamstring injuries in the sport [16]. Similarly, the 11+ warm-up programme demonstrated a 25% injury reduction in youth and adult football [17]. However, there has been poor programme adoption worldwide despite the football world governing body developing and disseminating the programme [18] [19].

Implementation, or the uptake of research findings into practice [20], is complex and requires a thorough understanding of end-user behaviour along with environmental, social and delivery factors that may facilitate or hinder successful outcomes [21]. Unfortunately, evaluating this critical step is often overlooked in the sports injury prevention field. Instead, the focus is often upon developing efficacious interventions, and programmes may subsequently be promoted without recognition of the complexities of successful implementation [22]. If researchers and national governing bodies first sought to understand the contextual determinants of successful implementation, they may be empowered to develop specific strategies to effectively design and deliver interventions for widespread use. The term scale-up has been applied to this practice, and is defined as ‘enabling interventions to have maximum impact’ [23].

Numerous theories and frameworks have been developed to evaluate the impact of public health interventions [24]. RE-AIM (‘reach’, ‘effectiveness’, ‘adoption’, ‘implementation’, ‘maintenance’) is an evaluation framework [25] designed to assess outcomes and processes across different levels. Each dimension has a specific operational definition ([Table 1]), identified by ‘italics’ and should not be confused with similar terms used colloquially. The framework has been applied to physical activity, disease management, substance abuse [26], policy change [27], and the planning of behaviour change interventions [28]. More recently the framework has been used in sports injury prevention studies to assess the implementation of neuromuscular training programmes [29] and perceptions towards preventive interventions [30]. Within RE-AIM, an intervention’s overall impact is dependent on addressing all five dimensions [25]. For example, if an intervention has high ‘effectiveness’ but low ‘adoption’, it will have limited impact [31].

Table 1 Definition of the 5 dimensions of the RE-AIM framework [25] [83].

Dimension

Definition

Reach’ (R)

Proportion and representativeness of targeted population that participate in the intervention.

Effectiveness’ (E)

Intervention success or failure rate if implemented as per guidelines.

Adoption’ (A)

Proportion and representativeness of settings and delivery agents willing to adopt the intervention.

Implementation’ (I)

The extent to which the intervention is delivered as intended (fidelity, consistency, cost, time) in the real world.

Maintenance’ (M)

The extent to which the intervention is sustained over time at the institutional and individual level.

O’Brien and Finch [32] conducted a systematic review to evaluate the implementation of neuromuscular training programmes in team ball sports. They adopted the 31-item RE-AIM Model Dimensions Item Checklist (MDIC) to comprehensively assess the level of reporting for each dimension within the literature [33]. Large knowledge gaps existed regarding ‘adoption’ and ‘maintenance’ of interventions, with research focused heavily upon ‘effectiveness’ [32]. This is concerning given that interventions require ‘adoption’ and ‘maintenance’ to have their intended outcome [25]. These knowledge gaps are a call to action to evaluate and report determinants which influence implementation. Achieving this will enable stakeholders to adapt new or existing interventions and scale-up their implementation.

There has been a plethora of injury prevention studies in response to the reported injury rates in rugby. Yet, as promising interventions are rapidly being rolled out in individual rugby unions and around the world, it is unclear whether implementation efforts have been research-informed. Therefore, the objective of this systematic review was to assess whether implementation factors, categorised and appraised using the RE-AIM framework, have been evaluated in the rugby injury prevention literature across all levels of the game. Identification of implementation knowledge gaps around existing primary injury prevention interventions is critical in ensuring that researchers, policy makers, and governing bodies are able to address potential barriers to intervention scale-up.


#

Materials and Methods

Search strategy

This review was registered and conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [34]. A meta-analytic approach was not deemed appropriate given the review sought to evaluate implementation factors, rather than outcome effects, of injury prevention interventions. The review was conducted in accordance with the ethical standards of the International Journal of Sports Medicine [35].

In consultation with a health subject librarian, six electronic databases (PubMed, Embase, SPORTDiscus, PsycINFO, Web of Science, Scopus) were searched from their date of inception. Additional references were obtained through journal searches, exploring reference lists and contacting experts in this field. Four key search themes were identified (rugby, injury, prevention, RE-AIM) and expanded to maximise the number of retrieved articles. Where applicable, medical subject headings (MeSH) were used along with text words (full search strategy in online resource 1).

On completion of each database search, duplicates were removed and two pairs of reviewers screened titles and abstracts of the remaining articles. Reviewers screened the articles independently before reaching consensus within their pair (inter-rater reliability – Pair 1: 94% agreement, κ=0.56; Pair 2: 93% agreement, κ=0.56). If consensus could not be reached, a reviewer from the other pair made a final decision. For abstracts progressing to full text review, each pair reviewed half of the papers and through discussion reached consensus on final inclusion.


#

Eligibility criteria

Articles were included provided they were (1) in the English language, (2) peer-reviewed, (3) based on original research, (4) involved field-based contact rugby players (all codes), and (5) investigated a prospective primary prevention intervention. An intervention was defined as an intentional action or process designed to prevent or reduce injury risk in rugby (i. e. primary prevention). Studies were excluded if they assessed only injury risk factors (i. e. no intervention), reported duplicate data from another study, or did not report outcomes relating to the RE-AIM framework. Interventions focused on secondary or tertiary injury prevention were excluded.


#

Study design and quality appraisal

Studies were classified by their overall design according to the classification proposed by Curran et al [36]. Non-hybrid designs included either efficacy (explanatory trials conducted under ideal conditions), effectiveness (pragmatic trials conducted in ‘real-world’ settings) or full implementation studies [36] [37] based on their stated objectives. The primary aim of Type I hybrid effectiveness studies is to establish intervention effectiveness; the secondary aim is to understand implementation context. Co-primary aims of Type II hybrid effectiveness studies are to determine both effectiveness and utility of an intervention strategy with equal emphasis. Type III hybrid effectiveness studies aim to determine the utility of an implementation strategy with the secondary aim of assessing clinical outcomes [36]. The methodological quality of quantitative studies was assessed using a modified Downs and Black (DB) quality assessment tool (online resource 2) [38]. The score for each article was calculated to a maximum of 28 points and subsequently given a rating based on the total score: poor (≤14), fair (15–19), good (20–25), or excellent (26–28) [39]. Levels of evidence were categorised according to the guidelines of the Oxford Centre of Evidence-Based Medicine (online resource 3) [40]. Qualitative studies were assessed using the National Institute for Care and Excellence (NICE) Quality Appraisal Checklist for Qualitative Studies (online resource 4) [41]. The lead author conducted all quality appraisals and categorised all study designs.


#

Data extraction

On completion of full text screening, the lead author extracted the following data from each study: year of publication, participant demographics, country of investigation, intervention type, and intervention delivery agent (individuals who delivered intervention components). Playing age was categorised into youth (participants competing at under-21 and below) or senior (no age restriction). Playing standard was divided into amateur (youth, community, university, ‘semi-professional’) or professional. For the purpose of this review, named intervention programmes that were developed and endorsed by national governing bodies, typically including a variety of intervention strategies, were termed multimodal national injury prevention programmes.

Data pertaining to the RE-AIM framework were extracted using the 31-item RE-AIM MDIC (online resource 5) [33] which determined whether each item was addressed (answered ‘yes’, ‘no’, ‘inappropriate’, or ‘not applicable’). The level of reporting for each item or dimension was calculated as a proportion of ‘yes’ answers for eligible items (‘yes, ‘no’, ‘inappropriate’) [32]. A MDIC impact score was calculated for each study using the same method. For example, an article achieving 12 ‘yes’ answers, 10 ‘no’ answers, 8 ‘yes – inappropriate’ answers, and 1 ‘not applicable’ answer would achieve a score of ((12)/(12+10+8) × 100)=40%.


#
#

Results

Search strategy

Initial database searches yielded 4253 articles, with 3091 potentially relevant articles to be screened following duplicate removal ([Fig. 1]). Of these, 2766 were excluded during title and abstract screening, leaving 325 articles for full-text review. Two hundred and forty-three articles were excluded at this stage whilst 8 full texts could not be retrieved. A total of 74 studies were retained for synthesis.

Zoom Image
Fig. 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart for rugby injury prevention literature retrieval. ‘Ineligible rugby format’ category included all non-field-based and/or non-contact rugby.

#

Study characteristics

The earliest published study was in 1977, with 34% of articles published between 2010–2019 ([Table 2]). Most studies were conducted in Australia (n=21), followed by New Zealand (n=17) and South Africa (n=12). Rugby union was the code of interest in 67 studies, six studies investigated rugby league and one study included both codes (online resource 6).

Table 2 Study level of evidence [40] and methodological quality rating [38].

OCEBM Model Level

Studies (n) per DB Score Rating (%)

Median DB Score (Range)

Mean EV Score

Mean IVScore

MDIC Dimensions Addressed

Mean MDIC Impact Score

Good (20–25)

Fair (15–19)

Poor (≤14)

1 (n=7)

3 (4%)

2 (3%)

2 (3%)

19 (9–23)

1.7/3

7.9/13

57%

24%

2* (n=31)

6 (8%)

9 (12%)

13 (18%)

16 (11–21)

2.3/3

6.3/13

86%

19%

3 (n=33)

2 (3%)

17 (23%)

17 (23%)

15 (8–21)

2.2/3

5.8/13

100%

19%

4 (n=2)

-

-

2 (3%)

12 (12–12)

2.0/3

6.5/13

43%

16%

Total

11 (15%)

28 (38%)

34 (47%)

15 (8–23)

2.2/3

6.1/13

-

-

Note: OCEBM, Oxford Centre for Evidence-Based Medicine Model (online resource 3); DB, Downs and Black checklist; EV, external validity (DB items 11–13); IV, internal validity (DB items 14–26); MDIC, RE-AIM Model Dimension Items Checklist. *In the absence of specific guidelines, quasi-experimental studies were classified as level 2b evidence given their non-randomised design.

Protective equipment was the most frequently investigated intervention (n=44), followed by national injury prevention programmes (n=9), law changes (n=7), neuromuscular training programmes (n=7), and law enforcement (n=3). Other interventions included training load manipulation, stress management, injury prevention education, and a “multidisciplinary approach to reducing injury risk” in a professional team [42] (all n=1). Of the 44 equipment studies, mouthguards were the intervention of interest in 37 studies and headgear in 12 studies. A range of other protective equipment was investigated (all n=≤5).

Forty-one studies included only men, ten included men and women, and one study included only women. Participant gender in the remaining studies was either unreported (n=19) or deemed not applicable (n=3) given that researchers investigated general policies rather than targeted populations. Sixteen studies assessed youth rugby players, 12 assessed senior players, and 19 studies included both. One study did not describe participant age.

Efficacy/effectiveness were the most common study design (n=27), and there were 12 implementation studies. Type II hybrid designs (n=16) were more common than Type I (n=9) and type III hybrid designs (n=10). No delivery agents were identified in 36 cases, predominately in observational studies assessing players’ use of equipment. Referees (n=13) and coaches (n=11) were most frequently identified as delivery agents, followed by national governing bodies (n=9) and club sports science or medical staff (n=8). Other delivery agents included community-based dentists (n=6), individual rugby club staff (n=2), workshop instructors from governing bodies and research staff (both n=1), podiatrists (n=1) and parents (n=1).


#

Quality and level of evidence

Cross-sectional (Oxford level 3c) studies were the most frequent (n=31), followed by cohort studies (level 2b; n=27) and randomised controlled trials (level 1b; n=7). The median DB score was 15 (8–23) ([Table 2]). One study was qualitative in design, scoring ++ on the NICE quality assessment checklist for qualitative studies. The methodological quality score and level of evidence for each study can be found in online resource 6.


#

Reporting of RE-AIM dimensions

Most studies addressed at least one item pertaining to ‘effectiveness’ (96%) and 57% of studies reported information relating to ‘reach’. ‘Adoption-setting level’ details were reported in 41% of studies, but only 14% addressed ‘adoption-delivery agent level’. ‘Implementation’ was addressed in 31% of studies, with ‘maintenance’ (‘individual’ and ‘setting-level’) reported in 12% and 13% of studies respectively.

As per the RE-AIM MDIC, ‘effectiveness’ was the dimension with the highest level of reporting (55%), more than double the next nearest dimension (‘reach’, 26%). All remaining dimensions scored below 20% ([Fig. 2]).

Zoom Image
Fig. 2 Level of reporting per RE-AIM MDIC dimension (all papers pooled). REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimensions Item Checklist.

#

Level of reporting of individual RE-AIM MDIC items

The level of reporting by individual RE-AIM MDIC item is presented in [Fig. 3]. Sixty-two studies reported a primary outcome measure (item 5), with 11 studies failing to report broader outcome measures (item 6). No studies reported the characteristics of intervention delivery agents (item 16) or the long-term attrition (>6 months) of participants (item 26). Only 11 studies provided information relating to adherence (item 18) and one study reported changes that were made to the intervention during delivery (item 19). Qualitative methods were rarely used or reported across dimensions (items 4, 9, 17, 22, 27, 31), with levels of reporting ranging from 3–7% for these items.

Zoom Image
Fig. 3 Level of reporting for each RE-AIM MDIC item (all papers pooled). REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimensions Item Checklist.

#

Intervention types

Of the five intervention types found in more than one study, national injury prevention programme studies [30] [43] [44] [45] [46] [47] [48] [49] [50] had the greatest mean MDIC impact score, addressing all RE-AIM MDIC dimensions ([Table 3]). Law change studies [51] [52] [53] [54] [55] [56] [57] failed to report information relating to the ‘implementation’ and ‘adoption delivery-agent’ of such laws and also had the lowest mean DB score. Of protective equipment studies, 70% used ‘adoption’ as an outcome measure. However, these studies scored poorly on both ‘adoption’ dimensions because they omitted information relating to settings and delivery agents. ‘Maintenance’ of protective equipment use was not reported in any paper. Studies investigating NMT training programmes [58] [59] [60] [61] [62] [63] [64] had the highest median DB score but all failed to report programme ‘maintenance’.

Table 3 RE-AIM MDIC dimension score and quality assessment score per intervention type.

Intervention

REA

EFF

A-SL

A-DAL

IMP

M-IL

M-SL

Mean MDIC Impact Score

Median DB Score

Equipment (n=44)

23%

49%

13%

7%

13%

0%

0%

17%

15

National IPP (n=9)

33%

62%

28%

19%

13%

14%

22%

27%

16.5

Law Change (n=7)

17%

64%

75%

0%

0%

25%

29%

26%

14

NMT (n=7)

48%

56%

26%

4%

19%

0%

0%

22%

20

Law Enforcement (n=3)

10%

17%

22%

0%

33%

11%

44%

20%

18

Note: Mean level of reporting per RE-AIM MDIC dimension is reported by intervention type. NMT, neuromuscular training; IPP, injury prevention programme; REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimension Items Checklist; DB, Downs and Black Checklist.


#

National injury prevention programmes

Multimodal national injury prevention programmes were often addressed in multiple papers, each evaluating different outcome measures (e. g. not reporting duplicate data). BokSmart (South Africa) was the only programme to address all seven dimensions of the RE-AIM MDIC ([Table 4]), scoring on 19 out of the 31 items collectively across seven related papers [30] [43] [44] [45] [49] [55] [65]. RugbySmart (New Zealand) has been evaluated across five dimensions [48] [50] [54] [66], scoring on 12 questions. Three dimensions were addressed for the Rugby Injury Performance Project (RIPP; New Zealand) [66] [67] [68] [69] [70] and one paper presented data from both RugbySmart and RIPP [66]. Other national injury prevention programmes were identified, but related studies investigated only one specific intervention and were not classed as multimodal; for instance, the MayDay safety procedure for SmartRugby [46] [47] (Australia) and the Activate programme of Rugby Safe [59] [63] (England).

Table 4 Aspects of the RE-AIM framework addressed by national injury prevention programmes.

National IPP

REA

EFF

A-SL

A-DAL

IMP

M-IL

M-SL

Dimensions Addressed

MDIC Answered ‘Yes’

BokSmart

100%

19/31

RugbySmart

71%

12/31

RIPP

43%

7/31

Note: IPP, injury prevention programme; RIPP, Rugby Injury Performance Project; REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimension Items Checklist.


#
#

Discussion

This is the first systematic review to evaluate the implementation of injury prevention strategies in rugby. Following PRISMA guidelines, 74 studies were identified and assessed using the 31-item RE-AIM MDIC. Studies routinely reported information regarding ‘effectiveness’, with little consideration given to reporting levels of ‘reach’, ‘adoption’, ‘implementation’, or ‘maintenance’.

It is perhaps not surprising that ‘effectiveness’ had the highest level of reporting. Systematic reviews conducted in a variety of contexts, including sports injury prevention [32] and behaviour change in public health [26] [71] reported similar findings. Research emphasis on ‘effectiveness’ as a primary outcome measure is not a recent trend, but notably the terms ‘efficacy’ and ‘effectiveness’ are often used interchangeably, despite different meanings [37]. Within the RE-AIM framework itself, both concepts are included in the ‘effectiveness’ domain, which has contributed to persistent assumptions that efficacy automatically leads to effectiveness. In 2011, Kessler and Glasgow [72] called for a 10-year moratorium on randomised controlled trials in healthcare, stating that efficacy studies were, problematically, becoming the only type of evidence considered by researchers and clinicians. This statement was reinforced by findings of a systematic review of sports injury prevention, where 420 efficacy studies were identified compared to just 32 effectiveness studies [73]. Whilst this paper is now dated and there have since been numerous calls for greater use of different study types to aid implementation [22] [74], it is evident the sports injury prevention field still has a long way to go in achieving this shift.

National injury prevention programmes had the greatest impact score of commonly used interventions, being the only intervention type for which all dimensions have been addressed. Boksmart evaluated the greatest number of items and dimensions, despite being the newest national injury prevention programme, whilst also being one of the only interventions supported by qualitative research. Law change studies also fared well despite 6 of the 7 studies failing to score on any ‘adoption’ items. It is plausible that because law changes are enforced across entire populations, reporting of participant characteristics or the delivery agents enforcing these laws are viewed as less important than ‘effectiveness’ outcomes. The same may also hold for the ‘implementation’ dimension, although results from law enforcement studies included in this review suggest tackle [65] [75] and ruck [76] laws may not be implemented as intended. There was no consideration of ‘maintenance’ of neuromuscular training programmes, but given three studies were published in 2017, there may be ongoing research into this dimension. It is also possible that the studies were stopped after efficacy was evaluated, which is a common limitation in the literature. Despite a disproportionate emphasis in the literature, protective equipment studies had the poorest RE-AIM MDIC score. Given the equivocal findings around protective equipment in rugby [77] [78], researchers should consider whether it is beneficial to direct efforts towards addressing implementation knowledge gaps in this context.

Whilst many randomised controlled trials report efficacious results, their highly controlled nature and failure to address the complexity of implementation [22] means that many interventions developed solely under this approach do not have their intended effect in the real world. This is in part associated with the poor external validity of these studies [79], reinforced by the results of the present review. Many studies failed to report the representativeness of the included settings, participants, and delivery agents. As a result, it would be challenging for readers and researchers to generalise the findings. To address this issue, Glasgow et al. [80] updated author guidelines, regardless of study type, to improve the reporting of external validity and aid readers when selecting interventions. However, this recommendation does not appear to have been routinely regarded in studies included in this review.

Interventions are often trialled in a controlled environment and then subsequently in a real-world setting [21]. However, this process is vastly time-consuming and in some healthcare fields it has been reported there is a 30-year lag between the discovery of an efficacious intervention and widespread adoption [81]. To accelerate this process, one approach could be to focus less upon the use of level one evidence, instead utilising pragmatic and/or qualitative study designs [33]. Curran et al. [36] have proposed the use of hybrid study designs with a dual focus of assessing effectiveness alongside implementation to expedite research into practice. Findings from the present review suggest that whilst hybrid designs are being utilised, there is still a bias towards establishing intervention efficacy. Furthermore, studies which were Type III hybrid designs or implementation studies commonly used surveys to assess barriers and facilitators rather than assessing true implementation or scale-up factors. Proctor et al. [82] have suggested that implementation research should focus on eight outcome measures (acceptability, adoption, appropriateness, implementation cost, feasibility, fidelity, penetration, and sustainability) to ensure researchers are providing meaningful and comparable results. Whilst some of these outcomes were highlighted through the RE-AIM MDIC, the poor level of reporting for the ‘adoption’ and ‘implementation’ dimensions suggest these measures are seldom investigated. Moreover, many of these outcomes need to be assessed at an individual level in order to assess the impact of implementation or scale-up efforts, yet most sports injury prevention research is currently conducted at a team or population level. Only once these eight implementation outcomes are evaluated and acted upon can we expect to see a notable public health benefit, or more specifically a reduction of injury risk in rugby.

All five RE-AIM dimensions are intended to be treated with equal importance and its developers have proposed that an intervention’s impact can be calculated by multiplying all dimensions (e. g. impact=R × E × A × I × M) [83]. Though this is a solid theoretical approach, it is possible that an intervention’s overall impact could be maximised by focusing on improving its ‘reach’. For some public health interventions, ‘reach’ could be millions of participants, whilst smaller changes are likely in the remaining dimensions (e. g. improving ‘adoption’ rates by 20%). It is imperative that all dimensions are addressed, but it may be that an initial focus on ‘reach’ would provide faster, broader improvements in injury risk reduction whereas other strategies might yield smaller gains over longer time frames. Moreover, the importance of certain dimensions may relate to the intervention type. To illustrate, protective equipment can be effective only if it is ‘adopted’ and users are fully adherent, whereas neuromuscular training programmes may be effective when completed two or three times per week rather than at every session [59] [63].

The suitability of the RE-AIM framework to assess public health interventions has been questioned, given that it fails to fully investigate contextual factors or use a mixed-methods approach to provide assessment [71]. Consequently, multiple adaptations have been made to the original five-dimension framework since its inception. Finch and Donaldson [84] developed the RE-AIM Sports Setting Matrix specifically for sports injury prevention interventions, assessing each dimension at multiple hierarchical levels from national governing bodies to end-users. The RE-AIM MDIC assesses the use of qualitative methods at each dimension, whilst expanding ‘adoption’ and ‘maintenance’ dimensions to investigate reporting at multiple user levels. The developers of RE-AIM have recently created an extension of the framework, called the Pragmatic Robust Implementation and Sustainability Model (PRISM), whereby internal (e. g. patient characteristics and perspectives) and external (e. g. environmental policies and guidelines) contexts are explored [85] [86]. The RE-AIM framework was selected for use in this review because it has been previously employed to assess injury prevention programmes across various sports [32], including rugby [30] [46]. This allows for direct comparisons between studies and reviews, whilst affirming that key knowledge gaps are prevalent throughout sports injury prevention research. Given the limitations of the RE-AIM framework, future research requires greater integration of implementation science principles and more sophisticated frameworks to shift the way researchers design and evaluate interventions to advance the injury prevention field.

The number of studies included in this review positively reflects the efforts of the rugby research community to reduce injury risk across all levels of the game. However, attention should now be directed towards evaluating implementation factors more broadly to address the identified knowledge gaps. This systematic review has shown that there has been consistent under-reporting of important criteria relating to the RE-AIM framework. To ensure routine information collection, implementation outcome measures should be incorporated into reporting checklists [80] and research frameworks [25] to aid with the development of interventions and study designs for better real-world impact. Journals should also encourage authors to include all relevant implementation information, such as delivery agent characteristics, in articles or supplementary content.

Limitations

Notably, the quality of the papers included in this review varied greatly and only seven level one studies were included, two of which scored ‘poorly’, whilst another two were ‘fair’ according to the Downs and Black criteria. Preference is often given to level one studies in evidence-based medicine due to their low risk of bias [40]; however, in this body of literature it appears that such studies are as susceptible to bias as those at “lower” evidence tiers, though the nature of that bias likely differs. For example, most included studies were observational cohorts or cross-sectional surveys, with low Downs and Black scores largely attributable to non-randomisation and lack of participant blinding. Yet, these studies scored well on external validity items, reflecting the trade-off between internal and external validity inherent in different study designs. It also highlights the potential strengths of designs that have traditionally been viewed as “inferior” when the focus is on implementation rather than intervention efficacy. Therefore, given that the focus of the present review was on the evaluation of implementation factors, rather than the effects of a particular intervention, the identified biases are unlikely to have directly affected the conclusions. The overall quality of the evidence does, however, speak to the challenges of implementing interventions that are supported by studies of varied designs, each with significant limitations.

Eight papers were unavailable for full-text review. Whilst it is not possible to say how these studies may have influenced the results, many were published prior to 1998 and likely written in a similar way to other studies of that era with a focus on ‘effectiveness’.

The lead author completed the quality assessment and data extraction process independently. Consequently, individual interpretation of each point may have led to systematic error when scoring. However, the inter-rater reliability of the checklist has been previously shown as ‘good’ [87]. Using a similar data extraction method as adopted here, the percentage agreement between two reviewers using the RE-AIM MDIC has been previously shown to be high (81–91%) [32]. Whilst the methodological quality of this review may be affected by the process being conducted independently, these findings support the use and reliability of the tools.


#
#

Conclusion

Research currently focuses heavily on the ‘effectiveness’ of interventions, with minimal acknowledgement of the ‘adoption’, ‘implementation’ and ‘maintenance’ of prevention strategies. There needs to be a greater focus on improving intervention uptake and scale-up factors in order to maximise injury prevention efforts. To aid this, there must be a shift in the sport injury prevention research community away from randomised controlled trials in favour of more pragmatic study designs with a greater emphasis on external validity, utilising reporting checklists and research frameworks drawn from implementation science disciplines. There also needs to be a clearer differentiation in study design between implementation and scale-up, with concentrated effort at evaluating both steps in the injury prevention process. This approach may improve the speed that research translates into practice, whilst leading to more favourable individual and population-level outcomes.


#
#

Conflict of Interest

CB completed this review as part of his PhD which is fully funded by the RFU. JCB has been involved in ongoing evaluation of the BokSmart programme. CDM and KAS were involved in the development of Activate and continue to evaluate its effectiveness. KAS is the medical research lead for the RFU. KAS is editor at the International Journal of Sports Medicine.

Acknowledgements

The authors wish to acknowledge the help of Peter Bradley, health subject librarian at the University of Bath, for his help in selecting the databases used in this review.

Supplementary Material


Correspondence

Craig Barden
Department for Health, University of Bath Wessex House 6.02, Claverton Down
BA2 7AY Bath
United Kingdom of Great Britain and Northern Ireland   
Phone: 01225383617   

Publication History

Received: 19 February 2020

Accepted: 14 June 2020

Publication Date:
28 July 2020 (online)

© 2020. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany


Zoom Image
Fig. 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart for rugby injury prevention literature retrieval. ‘Ineligible rugby format’ category included all non-field-based and/or non-contact rugby.
Zoom Image
Fig. 2 Level of reporting per RE-AIM MDIC dimension (all papers pooled). REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimensions Item Checklist.
Zoom Image
Fig. 3 Level of reporting for each RE-AIM MDIC item (all papers pooled). REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimensions Item Checklist.