Evaluating the Implementation of Injury Prevention Strategies in Rugby Union and League: A Systematic Review using the RE-AIM Framework

Abstract

Rugby (union and league) has come under intense scrutiny due to its injury risk. Various interventions have been introduced to protect players from injury, with many deemed efficacious and advocated for use across various worldwide contexts. However, their implementation is less clear. The objective of this systematic review was to determine whether injury prevention interventions in rugby have evaluated their ‘reach’, ‘effectiveness’, ‘adoption’, ‘implementation’ and ‘maintenance’ as per the RE-AIM Multi-Dimension Item Checklist. Six electronic databases were searched in November 2019. Inclusion criteria included: English language, peer-reviewed journal article, original research, field-based rugby code, prospective intervention. Of the 4253 studies identified, 74 met the full inclusion criteria. Protective equipment, predominately mouthguards, was the intervention of interest in 44 studies. Other interventions included multimodal national injury prevention programmes, law changes and neuromuscular training programmes. ‘Effectiveness’ was the highest scoring RE-AIM dimension (55%), followed by ‘reach’ (26%). All other RE-AIM dimensions scored below 20%. Research currently focuses on determining intervention ‘effectiveness’. For injury prevention strategies to have their desired impact, there must be a shift to address all determinants associated with implementation. Consideration should be given to how this can be achieved by adopting specific reporting checklists, research frameworks and study designs.

Key words

rugby - injury - prevention - implementation - RE-AIM

Background

Rugby (union and league) is popular worldwide with millions of participants [1] [2]; however, the sport has recently come under scrutiny for its injury risk and resultant public health burden [3] [4]. A meta-analysis of time-loss match injuries in professional men’s rugby union reported an injury incidence rate of 81 per 1000 player-hours (/1000 h) [5]. In men’s community rugby union, the incidence is estimated at 43/1000 h [6], with similar rates in rugby league [7]. In the 2010 women’s Rugby World Cup, there was an injury incidence rate of 36/1000 h [8], whilst a systematic review of studies in youth rugby union and league had a pooled injury incidence of 27/1000 h [9]. Whether these rates are considered high, or indeed acceptable [4] [10] [11], it is in the interest of all stakeholders to make rugby as safe as possible for all players.

Various sports have successfully decreased injury risk through preventative measures such as protective equipment [12], neuromuscular training programmes [13], and law changes [14]. Whilst many of these interventions are well received and widely disseminated, most do not have their intended effect in the ‘real world’ owing to poor implementation. For example, the Nordic hamstring curl can reduce hamstring injuries by 51% [15], yet only 17% of professional football clubs use this exercise despite the high prevalence of hamstring injuries in the sport [16]. Similarly, the 11+ warm-up programme demonstrated a 25% injury reduction in youth and adult football [17]. However, there has been poor programme adoption worldwide despite the football world governing body developing and disseminating the programme [18] [19].

Implementation, or the uptake of research findings into practice [20], is complex and requires a thorough understanding of end-user behaviour along with environmental, social and delivery factors that may facilitate or hinder successful outcomes [21]. Unfortunately, evaluating this critical step is often overlooked in the sports injury prevention field. Instead, the focus is often upon developing efficacious interventions, and programmes may subsequently be promoted without recognition of the complexities of successful implementation [22]. If researchers and national governing bodies first sought to understand the contextual determinants of successful implementation, they may be empowered to develop specific strategies to effectively design and deliver interventions for widespread use. The term scale-up has been applied to this practice, and is defined as ‘enabling interventions to have maximum impact’ [23].

Numerous theories and frameworks have been developed to evaluate the impact of public health interventions [24]. RE-AIM (‘reach’, ‘effectiveness’, ‘adoption’, ‘implementation’, ‘maintenance’) is an evaluation framework [25] designed to assess outcomes and processes across different levels. Each dimension has a specific operational definition ([Table 1]), identified by ‘italics’ and should not be confused with similar terms used colloquially. The framework has been applied to physical activity, disease management, substance abuse [26], policy change [27], and the planning of behaviour change interventions [28]. More recently the framework has been used in sports injury prevention studies to assess the implementation of neuromuscular training programmes [29] and perceptions towards preventive interventions [30]. Within RE-AIM, an intervention’s overall impact is dependent on addressing all five dimensions [25]. For example, if an intervention has high ‘effectiveness’ but low ‘adoption’, it will have limited impact [31].

Table 1 Definition of the 5 dimensions of the RE-AIM framework [25] [83].

Dimension	Definition
‘Reach’ (R)	Proportion and representativeness of targeted population that participate in the intervention.
‘Effectiveness’ (E)	Intervention success or failure rate if implemented as per guidelines.
‘Adoption’ (A)	Proportion and representativeness of settings and delivery agents willing to adopt the intervention.
‘Implementation’ (I)	The extent to which the intervention is delivered as intended (fidelity, consistency, cost, time) in the real world.
‘Maintenance’ (M)	The extent to which the intervention is sustained over time at the institutional and individual level.

O’Brien and Finch [32] conducted a systematic review to evaluate the implementation of neuromuscular training programmes in team ball sports. They adopted the 31-item RE-AIM Model Dimensions Item Checklist (MDIC) to comprehensively assess the level of reporting for each dimension within the literature [33]. Large knowledge gaps existed regarding ‘adoption’ and ‘maintenance’ of interventions, with research focused heavily upon ‘effectiveness’ [32]. This is concerning given that interventions require ‘adoption’ and ‘maintenance’ to have their intended outcome [25]. These knowledge gaps are a call to action to evaluate and report determinants which influence implementation. Achieving this will enable stakeholders to adapt new or existing interventions and scale-up their implementation.

There has been a plethora of injury prevention studies in response to the reported injury rates in rugby. Yet, as promising interventions are rapidly being rolled out in individual rugby unions and around the world, it is unclear whether implementation efforts have been research-informed. Therefore, the objective of this systematic review was to assess whether implementation factors, categorised and appraised using the RE-AIM framework, have been evaluated in the rugby injury prevention literature across all levels of the game. Identification of implementation knowledge gaps around existing primary injury prevention interventions is critical in ensuring that researchers, policy makers, and governing bodies are able to address potential barriers to intervention scale-up.

Materials and Methods

Search strategy

This review was registered and conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [34]. A meta-analytic approach was not deemed appropriate given the review sought to evaluate implementation factors, rather than outcome effects, of injury prevention interventions. The review was conducted in accordance with the ethical standards of the International Journal of Sports Medicine [35].

In consultation with a health subject librarian, six electronic databases (PubMed, Embase, SPORTDiscus, PsycINFO, Web of Science, Scopus) were searched from their date of inception. Additional references were obtained through journal searches, exploring reference lists and contacting experts in this field. Four key search themes were identified (rugby, injury, prevention, RE-AIM) and expanded to maximise the number of retrieved articles. Where applicable, medical subject headings (MeSH) were used along with text words (full search strategy in online resource 1).

On completion of each database search, duplicates were removed and two pairs of reviewers screened titles and abstracts of the remaining articles. Reviewers screened the articles independently before reaching consensus within their pair (inter-rater reliability – Pair 1: 94% agreement, κ=0.56; Pair 2: 93% agreement, κ=0.56). If consensus could not be reached, a reviewer from the other pair made a final decision. For abstracts progressing to full text review, each pair reviewed half of the papers and through discussion reached consensus on final inclusion.

Eligibility criteria

Articles were included provided they were (1) in the English language, (2) peer-reviewed, (3) based on original research, (4) involved field-based contact rugby players (all codes), and (5) investigated a prospective primary prevention intervention. An intervention was defined as an intentional action or process designed to prevent or reduce injury risk in rugby (i. e. primary prevention). Studies were excluded if they assessed only injury risk factors (i. e. no intervention), reported duplicate data from another study, or did not report outcomes relating to the RE-AIM framework. Interventions focused on secondary or tertiary injury prevention were excluded.

Study design and quality appraisal

Studies were classified by their overall design according to the classification proposed by Curran et al [36]. Non-hybrid designs included either efficacy (explanatory trials conducted under ideal conditions), effectiveness (pragmatic trials conducted in ‘real-world’ settings) or full implementation studies [36] [37] based on their stated objectives. The primary aim of Type I hybrid effectiveness studies is to establish intervention effectiveness; the secondary aim is to understand implementation context. Co-primary aims of Type II hybrid effectiveness studies are to determine both effectiveness and utility of an intervention strategy with equal emphasis. Type III hybrid effectiveness studies aim to determine the utility of an implementation strategy with the secondary aim of assessing clinical outcomes [36]. The methodological quality of quantitative studies was assessed using a modified Downs and Black (DB) quality assessment tool (online resource 2) [38]. The score for each article was calculated to a maximum of 28 points and subsequently given a rating based on the total score: poor (≤14), fair (15–19), good (20–25), or excellent (26–28) [39]. Levels of evidence were categorised according to the guidelines of the Oxford Centre of Evidence-Based Medicine (online resource 3) [40]. Qualitative studies were assessed using the National Institute for Care and Excellence (NICE) Quality Appraisal Checklist for Qualitative Studies (online resource 4) [41]. The lead author conducted all quality appraisals and categorised all study designs.

Data extraction

On completion of full text screening, the lead author extracted the following data from each study: year of publication, participant demographics, country of investigation, intervention type, and intervention delivery agent (individuals who delivered intervention components). Playing age was categorised into youth (participants competing at under-21 and below) or senior (no age restriction). Playing standard was divided into amateur (youth, community, university, ‘semi-professional’) or professional. For the purpose of this review, named intervention programmes that were developed and endorsed by national governing bodies, typically including a variety of intervention strategies, were termed multimodal national injury prevention programmes.

Data pertaining to the RE-AIM framework were extracted using the 31-item RE-AIM MDIC (online resource 5) [33] which determined whether each item was addressed (answered ‘yes’, ‘no’, ‘inappropriate’, or ‘not applicable’). The level of reporting for each item or dimension was calculated as a proportion of ‘yes’ answers for eligible items (‘yes, ‘no’, ‘inappropriate’) [32]. A MDIC impact score was calculated for each study using the same method. For example, an article achieving 12 ‘yes’ answers, 10 ‘no’ answers, 8 ‘yes – inappropriate’ answers, and 1 ‘not applicable’ answer would achieve a score of ((12)/(12+10+8) × 100)=40%.

Results

Search strategy

Initial database searches yielded 4253 articles, with 3091 potentially relevant articles to be screened following duplicate removal ([Fig. 1]). Of these, 2766 were excluded during title and abstract screening, leaving 325 articles for full-text review. Two hundred and forty-three articles were excluded at this stage whilst 8 full texts could not be retrieved. A total of 74 studies were retained for synthesis.

Fig. 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart for rugby injury prevention literature retrieval. ‘Ineligible rugby format’ category included all non-field-based and/or non-contact rugby.

Study characteristics

The earliest published study was in 1977, with 34% of articles published between 2010–2019 ([Table 2]). Most studies were conducted in Australia (n=21), followed by New Zealand (n=17) and South Africa (n=12). Rugby union was the code of interest in 67 studies, six studies investigated rugby league and one study included both codes (online resource 6).

Table 2 Study level of evidence [40] and methodological quality rating [38].

OCEBM Model Level	Studies (n) per DB Score Rating (%)			Median DB Score (Range)	Mean EV Score	Mean IVScore	MDIC Dimensions Addressed	Mean MDIC Impact Score
	Good (20–25)	Fair (15–19)	Poor (≤14)
1 (n=7)	3 (4%)	2 (3%)	2 (3%)	19 (9–23)	1.7/3	7.9/13	57%	24%
2* (n=31)	6 (8%)	9 (12%)	13 (18%)	16 (11–21)	2.3/3	6.3/13	86%	19%
3 (n=33)	2 (3%)	17 (23%)	17 (23%)	15 (8–21)	2.2/3	5.8/13	100%	19%
4 (n=2)	-	-	2 (3%)	12 (12–12)	2.0/3	6.5/13	43%	16%
Total	11 (15%)	28 (38%)	34 (47%)	15 (8–23)	2.2/3	6.1/13	-	-

Note: OCEBM, Oxford Centre for Evidence-Based Medicine Model (online resource 3); DB, Downs and Black checklist; EV, external validity (DB items 11–13); IV, internal validity (DB items 14–26); MDIC, RE-AIM Model Dimension Items Checklist. *In the absence of specific guidelines, quasi-experimental studies were classified as level 2b evidence given their non-randomised design.

Protective equipment was the most frequently investigated intervention (n=44), followed by national injury prevention programmes (n=9), law changes (n=7), neuromuscular training programmes (n=7), and law enforcement (n=3). Other interventions included training load manipulation, stress management, injury prevention education, and a “multidisciplinary approach to reducing injury risk” in a professional team [42] (all n=1). Of the 44 equipment studies, mouthguards were the intervention of interest in 37 studies and headgear in 12 studies. A range of other protective equipment was investigated (all n=≤5).

Forty-one studies included only men, ten included men and women, and one study included only women. Participant gender in the remaining studies was either unreported (n=19) or deemed not applicable (n=3) given that researchers investigated general policies rather than targeted populations. Sixteen studies assessed youth rugby players, 12 assessed senior players, and 19 studies included both. One study did not describe participant age.

Efficacy/effectiveness were the most common study design (n=27), and there were 12 implementation studies. Type II hybrid designs (n=16) were more common than Type I (n=9) and type III hybrid designs (n=10). No delivery agents were identified in 36 cases, predominately in observational studies assessing players’ use of equipment. Referees (n=13) and coaches (n=11) were most frequently identified as delivery agents, followed by national governing bodies (n=9) and club sports science or medical staff (n=8). Other delivery agents included community-based dentists (n=6), individual rugby club staff (n=2), workshop instructors from governing bodies and research staff (both n=1), podiatrists (n=1) and parents (n=1).

Quality and level of evidence

Cross-sectional (Oxford level 3c) studies were the most frequent (n=31), followed by cohort studies (level 2b; n=27) and randomised controlled trials (level 1b; n=7). The median DB score was 15 (8–23) ([Table 2]). One study was qualitative in design, scoring ++ on the NICE quality assessment checklist for qualitative studies. The methodological quality score and level of evidence for each study can be found in online resource 6.

Reporting of RE-AIM dimensions

Most studies addressed at least one item pertaining to ‘effectiveness’ (96%) and 57% of studies reported information relating to ‘reach’. ‘Adoption-setting level’ details were reported in 41% of studies, but only 14% addressed ‘adoption-delivery agent level’. ‘Implementation’ was addressed in 31% of studies, with ‘maintenance’ (‘individual’ and ‘setting-level’) reported in 12% and 13% of studies respectively.

As per the RE-AIM MDIC, ‘effectiveness’ was the dimension with the highest level of reporting (55%), more than double the next nearest dimension (‘reach’, 26%). All remaining dimensions scored below 20% ([Fig. 2]).

Fig. 2 Level of reporting per RE-AIM MDIC dimension (all papers pooled). REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimensions Item Checklist.

Level of reporting of individual RE-AIM MDIC items

The level of reporting by individual RE-AIM MDIC item is presented in [Fig. 3]. Sixty-two studies reported a primary outcome measure (item 5), with 11 studies failing to report broader outcome measures (item 6). No studies reported the characteristics of intervention delivery agents (item 16) or the long-term attrition (>6 months) of participants (item 26). Only 11 studies provided information relating to adherence (item 18) and one study reported changes that were made to the intervention during delivery (item 19). Qualitative methods were rarely used or reported across dimensions (items 4, 9, 17, 22, 27, 31), with levels of reporting ranging from 3–7% for these items.

Fig. 3 Level of reporting for each RE-AIM MDIC item (all papers pooled). REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimensions Item Checklist.

Intervention types

Of the five intervention types found in more than one study, national injury prevention programme studies [30] [43] [44] [45] [46] [47] [48] [49] [50] had the greatest mean MDIC impact score, addressing all RE-AIM MDIC dimensions ([Table 3]). Law change studies [51] [52] [53] [54] [55] [56] [57] failed to report information relating to the ‘implementation’ and ‘adoption delivery-agent’ of such laws and also had the lowest mean DB score. Of protective equipment studies, 70% used ‘adoption’ as an outcome measure. However, these studies scored poorly on both ‘adoption’ dimensions because they omitted information relating to settings and delivery agents. ‘Maintenance’ of protective equipment use was not reported in any paper. Studies investigating NMT training programmes [58] [59] [60] [61] [62] [63] [64] had the highest median DB score but all failed to report programme ‘maintenance’.

Table 3 RE-AIM MDIC dimension score and quality assessment score per intervention type.

Intervention	REA	EFF	A-SL	A-DAL	IMP	M-IL	M-SL	Mean MDIC Impact Score	Median DB Score
Equipment (n=44)	23%	49%	13%	7%	13%	0%	0%	17%	15
National IPP (n=9)	33%	62%	28%	19%	13%	14%	22%	27%	16.5
Law Change (n=7)	17%	64%	75%	0%	0%	25%	29%	26%	14
NMT (n=7)	48%	56%	26%	4%	19%	0%	0%	22%	20
Law Enforcement (n=3)	10%	17%	22%	0%	33%	11%	44%	20%	18

Note: Mean level of reporting per RE-AIM MDIC dimension is reported by intervention type. NMT, neuromuscular training; IPP, injury prevention programme; REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimension Items Checklist; DB, Downs and Black Checklist.

National injury prevention programmes

Multimodal national injury prevention programmes were often addressed in multiple papers, each evaluating different outcome measures (e. g. not reporting duplicate data). BokSmart (South Africa) was the only programme to address all seven dimensions of the RE-AIM MDIC ([Table 4]), scoring on 19 out of the 31 items collectively across seven related papers [30] [43] [44] [45] [49] [55] [65]. RugbySmart (New Zealand) has been evaluated across five dimensions [48] [50] [54] [66], scoring on 12 questions. Three dimensions were addressed for the Rugby Injury Performance Project (RIPP; New Zealand) [66] [67] [68] [69] [70] and one paper presented data from both RugbySmart and RIPP [66]. Other national injury prevention programmes were identified, but related studies investigated only one specific intervention and were not classed as multimodal; for instance, the MayDay safety procedure for SmartRugby [46] [47] (Australia) and the Activate programme of Rugby Safe [59] [63] (England).

Table 4 Aspects of the RE-AIM framework addressed by national injury prevention programmes.

National IPP	REA	EFF	A-SL	A-DAL	IMP	M-IL	M-SL	Dimensions Addressed	MDIC Answered ‘Yes’
BokSmart	✓	✓	✓	✓	✓	✓	✓	100%	19/31
RugbySmart	✓	✓	✗	✗	✓	✓	✓	71%	12/31
RIPP	✓	✓	✗	✗	✓	✗	✗	43%	7/31

Note: IPP, injury prevention programme; RIPP, Rugby Injury Performance Project; REA, ‘Reach’; EFF, ‘Effectiveness’; A-SL, ‘Adoption – Setting level’; A-DAL, ‘Adoption – Delivery agent level’; IMP, ‘Implementation’; M-IL, ‘Maintenance – Individual level’; M-SL, ‘Maintenance – Setting level’; MDIC, RE-AIM Model Dimension Items Checklist.

Discussion

This is the first systematic review to evaluate the implementation of injury prevention strategies in rugby. Following PRISMA guidelines, 74 studies were identified and assessed using the 31-item RE-AIM MDIC. Studies routinely reported information regarding ‘effectiveness’, with little consideration given to reporting levels of ‘reach’, ‘adoption’, ‘implementation’, or ‘maintenance’.

It is perhaps not surprising that ‘effectiveness’ had the highest level of reporting. Systematic reviews conducted in a variety of contexts, including sports injury prevention [32] and behaviour change in public health [26] [71] reported similar findings. Research emphasis on ‘effectiveness’ as a primary outcome measure is not a recent trend, but notably the terms ‘efficacy’ and ‘effectiveness’ are often used interchangeably, despite different meanings [37]. Within the RE-AIM framework itself, both concepts are included in the ‘effectiveness’ domain, which has contributed to persistent assumptions that efficacy automatically leads to effectiveness. In 2011, Kessler and Glasgow [72] called for a 10-year moratorium on randomised controlled trials in healthcare, stating that efficacy studies were, problematically, becoming the only type of evidence considered by researchers and clinicians. This statement was reinforced by findings of a systematic review of sports injury prevention, where 420 efficacy studies were identified compared to just 32 effectiveness studies [73]. Whilst this paper is now dated and there have since been numerous calls for greater use of different study types to aid implementation [22] [74], it is evident the sports injury prevention field still has a long way to go in achieving this shift.

National injury prevention programmes had the greatest impact score of commonly used interventions, being the only intervention type for which all dimensions have been addressed. Boksmart evaluated the greatest number of items and dimensions, despite being the newest national injury prevention programme, whilst also being one of the only interventions supported by qualitative research. Law change studies also fared well despite 6 of the 7 studies failing to score on any ‘adoption’ items. It is plausible that because law changes are enforced across entire populations, reporting of participant characteristics or the delivery agents enforcing these laws are viewed as less important than ‘effectiveness’ outcomes. The same may also hold for the ‘implementation’ dimension, although results from law enforcement studies included in this review suggest tackle [65] [75] and ruck [76] laws may not be implemented as intended. There was no consideration of ‘maintenance’ of neuromuscular training programmes, but given three studies were published in 2017, there may be ongoing research into this dimension. It is also possible that the studies were stopped after efficacy was evaluated, which is a common limitation in the literature. Despite a disproportionate emphasis in the literature, protective equipment studies had the poorest RE-AIM MDIC score. Given the equivocal findings around protective equipment in rugby [77] [78], researchers should consider whether it is beneficial to direct efforts towards addressing implementation knowledge gaps in this context.

Whilst many randomised controlled trials report efficacious results, their highly controlled nature and failure to address the complexity of implementation [22] means that many interventions developed solely under this approach do not have their intended effect in the real world. This is in part associated with the poor external validity of these studies [79], reinforced by the results of the present review. Many studies failed to report the representativeness of the included settings, participants, and delivery agents. As a result, it would be challenging for readers and researchers to generalise the findings. To address this issue, Glasgow et al. [80] updated author guidelines, regardless of study type, to improve the reporting of external validity and aid readers when selecting interventions. However, this recommendation does not appear to have been routinely regarded in studies included in this review.

Interventions are often trialled in a controlled environment and then subsequently in a real-world setting [21]. However, this process is vastly time-consuming and in some healthcare fields it has been reported there is a 30-year lag between the discovery of an efficacious intervention and widespread adoption [81]. To accelerate this process, one approach could be to focus less upon the use of level one evidence, instead utilising pragmatic and/or qualitative study designs [33]. Curran et al. [36] have proposed the use of hybrid study designs with a dual focus of assessing effectiveness alongside implementation to expedite research into practice. Findings from the present review suggest that whilst hybrid designs are being utilised, there is still a bias towards establishing intervention efficacy. Furthermore, studies which were Type III hybrid designs or implementation studies commonly used surveys to assess barriers and facilitators rather than assessing true implementation or scale-up factors. Proctor et al. [82] have suggested that implementation research should focus on eight outcome measures (acceptability, adoption, appropriateness, implementation cost, feasibility, fidelity, penetration, and sustainability) to ensure researchers are providing meaningful and comparable results. Whilst some of these outcomes were highlighted through the RE-AIM MDIC, the poor level of reporting for the ‘adoption’ and ‘implementation’ dimensions suggest these measures are seldom investigated. Moreover, many of these outcomes need to be assessed at an individual level in order to assess the impact of implementation or scale-up efforts, yet most sports injury prevention research is currently conducted at a team or population level. Only once these eight implementation outcomes are evaluated and acted upon can we expect to see a notable public health benefit, or more specifically a reduction of injury risk in rugby.

All five RE-AIM dimensions are intended to be treated with equal importance and its developers have proposed that an intervention’s impact can be calculated by multiplying all dimensions (e. g. impact=R × E × A × I × M) [83]. Though this is a solid theoretical approach, it is possible that an intervention’s overall impact could be maximised by focusing on improving its ‘reach’. For some public health interventions, ‘reach’ could be millions of participants, whilst smaller changes are likely in the remaining dimensions (e. g. improving ‘adoption’ rates by 20%). It is imperative that all dimensions are addressed, but it may be that an initial focus on ‘reach’ would provide faster, broader improvements in injury risk reduction whereas other strategies might yield smaller gains over longer time frames. Moreover, the importance of certain dimensions may relate to the intervention type. To illustrate, protective equipment can be effective only if it is ‘adopted’ and users are fully adherent, whereas neuromuscular training programmes may be effective when completed two or three times per week rather than at every session [59] [63].

The suitability of the RE-AIM framework to assess public health interventions has been questioned, given that it fails to fully investigate contextual factors or use a mixed-methods approach to provide assessment [71]. Consequently, multiple adaptations have been made to the original five-dimension framework since its inception. Finch and Donaldson [84] developed the RE-AIM Sports Setting Matrix specifically for sports injury prevention interventions, assessing each dimension at multiple hierarchical levels from national governing bodies to end-users. The RE-AIM MDIC assesses the use of qualitative methods at each dimension, whilst expanding ‘adoption’ and ‘maintenance’ dimensions to investigate reporting at multiple user levels. The developers of RE-AIM have recently created an extension of the framework, called the Pragmatic Robust Implementation and Sustainability Model (PRISM), whereby internal (e. g. patient characteristics and perspectives) and external (e. g. environmental policies and guidelines) contexts are explored [85] [86]. The RE-AIM framework was selected for use in this review because it has been previously employed to assess injury prevention programmes across various sports [32], including rugby [30] [46]. This allows for direct comparisons between studies and reviews, whilst affirming that key knowledge gaps are prevalent throughout sports injury prevention research. Given the limitations of the RE-AIM framework, future research requires greater integration of implementation science principles and more sophisticated frameworks to shift the way researchers design and evaluate interventions to advance the injury prevention field.

The number of studies included in this review positively reflects the efforts of the rugby research community to reduce injury risk across all levels of the game. However, attention should now be directed towards evaluating implementation factors more broadly to address the identified knowledge gaps. This systematic review has shown that there has been consistent under-reporting of important criteria relating to the RE-AIM framework. To ensure routine information collection, implementation outcome measures should be incorporated into reporting checklists [80] and research frameworks [25] to aid with the development of interventions and study designs for better real-world impact. Journals should also encourage authors to include all relevant implementation information, such as delivery agent characteristics, in articles or supplementary content.

Limitations

Notably, the quality of the papers included in this review varied greatly and only seven level one studies were included, two of which scored ‘poorly’, whilst another two were ‘fair’ according to the Downs and Black criteria. Preference is often given to level one studies in evidence-based medicine due to their low risk of bias [40]; however, in this body of literature it appears that such studies are as susceptible to bias as those at “lower” evidence tiers, though the nature of that bias likely differs. For example, most included studies were observational cohorts or cross-sectional surveys, with low Downs and Black scores largely attributable to non-randomisation and lack of participant blinding. Yet, these studies scored well on external validity items, reflecting the trade-off between internal and external validity inherent in different study designs. It also highlights the potential strengths of designs that have traditionally been viewed as “inferior” when the focus is on implementation rather than intervention efficacy. Therefore, given that the focus of the present review was on the evaluation of implementation factors, rather than the effects of a particular intervention, the identified biases are unlikely to have directly affected the conclusions. The overall quality of the evidence does, however, speak to the challenges of implementing interventions that are supported by studies of varied designs, each with significant limitations.

Eight papers were unavailable for full-text review. Whilst it is not possible to say how these studies may have influenced the results, many were published prior to 1998 and likely written in a similar way to other studies of that era with a focus on ‘effectiveness’.

The lead author completed the quality assessment and data extraction process independently. Consequently, individual interpretation of each point may have led to systematic error when scoring. However, the inter-rater reliability of the checklist has been previously shown as ‘good’ [87]. Using a similar data extraction method as adopted here, the percentage agreement between two reviewers using the RE-AIM MDIC has been previously shown to be high (81–91%) [32]. Whilst the methodological quality of this review may be affected by the process being conducted independently, these findings support the use and reliability of the tools.

Conclusion

Research currently focuses heavily on the ‘effectiveness’ of interventions, with minimal acknowledgement of the ‘adoption’, ‘implementation’ and ‘maintenance’ of prevention strategies. There needs to be a greater focus on improving intervention uptake and scale-up factors in order to maximise injury prevention efforts. To aid this, there must be a shift in the sport injury prevention research community away from randomised controlled trials in favour of more pragmatic study designs with a greater emphasis on external validity, utilising reporting checklists and research frameworks drawn from implementation science disciplines. There also needs to be a clearer differentiation in study design between implementation and scale-up, with concentrated effort at evaluating both steps in the injury prevention process. This approach may improve the speed that research translates into practice, whilst leading to more favourable individual and population-level outcomes.

Conflict of Interest

CB completed this review as part of his PhD which is fully funded by the RFU. JCB has been involved in ongoing evaluation of the BokSmart programme. CDM and KAS were involved in the development of Activate and continue to evaluate its effectiveness. KAS is the medical research lead for the RFU. KAS is editor at the International Journal of Sports Medicine.

Acknowledgements

The authors wish to acknowledge the help of Peter Bradley, health subject librarian at the University of Bath, for his help in selecting the databases used in this review.

• References

• 1 World Rugby. Year in Review 2019. 2019. http://publications.worldrugby.org/yearinreview2019/en/1-1/ Accessed: Dec 2 2019
• 2 Rugby League International Federation. Strategic Plan 2018–2025. 2015. http://www.rlif.com/~media/docs/RLIF-Strategic%20Plan%202015-2025%2018pp%20LR.pdf Accessed: Jan 10 2020
• 3 Quarrie KL, Brooks JHM, Burger N. et al. Facts and values: on the acceptability of risks in children’s sport using the example of rugby - a narrative review. Br J Sports Med 2017; 51: 1134-1139
• 4 Tucker R, Raftery M, Verhagen E. Injury risk and a tackle ban in youth Rugby Union: reviewing the evidence and searching for targeted, effective interventions. A critical review. Br J Sports Med 2016; 50: 921-925
• 5 Williams S, Trewartha G, Kemp S. et al. A meta-analysis of injuries in senior men’s professional Rugby Union. Sports Med 2013; 43: 1043-1055
• 6 Yeomans C, Kenny IC, Cahalan R. et al. The incidence of injury in amateur male rugby union: A systematic review and meta-analysis. Sports Med 2018; 48: 837-848
• 7 King DA, Hume PA, Milburn PD. et al. Match and training injuries in rugby league: A review of published studies. Sports Med 2010; 40: 163-178
• 8 Taylor AE, Fuller CW, Molloy MG. Injury surveillance during the 2010 IRB Women’s Rugby World Cup. Br J Sports Med 2011; 45: 1243-1245
• 9 Freitag A, Kirkwood G, Scharer S. et al. Systematic review of rugby injuries in children and adolescents under 21 years. Br J Sports Med 2015; 49: 511-519
• 10 Fuller CW. Managing the risk of injury in sport. Clin J Sport Med 2007; 17: 182-187
• 11 Sports Collision Injury Collective. Open Letter: Preventing injuries in children playing school rugby 2016 http://www.sportcic.com/resources/Open%20Letter%20SportCIC%20March1st%202016.pdf Accessed: Aug 18 2019
• 12 Benson BW, Hamilton GM, Meeuwisse WH. et al. Is protective equipment useful in preventing concussion? A systematic review of the literature. Br J Sports Med 2009; 43 (Suppl. 01) i56-i67
• 13 Lauersen JB, Bertelsen DM, Andersen LB. The effectiveness of exercise interventions to prevent sports injuries: A systematic review and meta-analysis of randomised controlled trials. Br J Sports Med 2014; 48: 871-877
• 14 Emery C, MacPherson AK, Kang J. et al. Does changing policy to disallow body checking reduce the risk of concussion in 11 and 12-year-old ice hockey players?. Br J Sports Med 2013; 47: e1 DOI: 10.1136/bjsports-2012-092101.13.
• 15 Al Attar WS, Soomro N, Pappas E. et al. How effective are f-marc injury prevention programs for soccer players? A systematic review and meta-analysis. Sports Med 2016; 46: 205-217
• 16 Bahr R, Thorborg K, Ekstrand J. Evidence-based hamstring injury prevention is not adopted by the majority of Champions League or Norwegian Premier League football teams: The Nordic Hamstring survey. Br J Sports Med 2015; 49: 1466-1471
• 17 Thorborg K, Krommes KK, Esteve E. et al. Effect of specific exercise-based football injury prevention programmes on the overall injury rate in football: A systematic review and meta-analysis of the FIFA 11 and 11+ programmes. Br J Sports Med 2017; 51: 562-571
• 18 O'Brien J, Finch CF. Injury prevention exercise programmes in professional youth soccer: Understanding the perceptions of programme deliverers. BMJ Open Sport Exerc Med 2016; 2: e000075
• 19 Owoeye OB, Akinbo SR, Olawale OA. et al. Injury prevention in football: knowledge and behaviour of players and availability of medical care in a Nigerian youth football league. S Afr J Sports Med 2013; 25: 77-80
• 20 Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci 2006; 1 1. DOI: https://doi.org/10.1186/1748-5908-1-1.
• 21 Finch C. A new framework for research leading to sports injury prevention. J Sci Med Sport 2006; 9: 3-9 discussion 10
• 22 Hanson D. Research alone is not sufficient to prevent sports injury. Br J Sports Med 2014; 48: 682-684
• 23 Donaldson A, Finch CF. Applying implementation science to sports injury prevention. Br J Sports Med 2013; 47: 473-475
• 24 Tabak RG, Khoong EC, Chambers DA. et al. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med 2012; 43: 337-350
• 25 Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999; 89: 1322-1327
• 26 Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health 2013; 103: e38-e46
• 27 Jilcott S, Ammerman A, Sommers J. et al. Applying the RE-AIM framework to assess the public health impact of policy change. Ann Behav Med 2007; 34: 105-114
• 28 King DK, Glasgow RE, Leeman-Castillo B. Reaiming RE-AIM: using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. Am J Public Health 2010; 100: 2076-2084
• 29 Vriend I, Coehoorn I, Verhagen E. Implementation of an app-based neuromuscular training programme to prevent ankle sprains: A process evaluation using the RE-AIM Framework. Br J Sports Med 2015; 49: 484-488
• 30 Brown JC, Verhagen E, van Mechelen W. et al. Coaches' and referees' perceptions of the BokSmart injury prevention programme. Int J Sports Sci Coach 2016; 11: 637-647
• 31 Verhagen E, Finch CF. Setting our minds to implementation. Int J Sports Sci Coach 2011; 45: 1015-1016
• 32 O'Brien J, Finch CF. The implementation of musculoskeletal injury-prevention exercise programmes in team ball sports: A systematic review employing the RE-AIM framework. Sports Med 2014; 44: 1305-1318
• 33 Kessler RS PE, Glasgow RE, Klesges LM. et al. What does it mean to "employ" the RE-AIM model?. Eval Health Prof 2013; 36: 44-66
• 34 Shamseer L, Moher D, Clarke M. et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ 2015; 350: g7647
• 35 Harriss DJ, Macsween Atkinson G. Ethical standards in sport and exercise science research: 2020 update. BMJ 2019; 40: 813-817
• 36 Curran GM, Bauer M, Mittman B. et al. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012; 50: 217-226
• 37 Gartlehner G, Hansen RA, Nissman D. et al. A simple and valid tool distinguished efficacy from effectiveness studies. J Clin Epidemiol 2006; 59: 1040-1048
• 38 Downs SH. , N B The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998; 52: 377-384
• 39 Hooper P, Jutai JW, Strong G. et al. Age-related macular degeneration and low-vision rehabilitation: A systematic review. Can J Ophthalmol 2008; 43: 180-187
• 40 Centre for Evidence Based Medicine. Oxford Centre for Evidence-based Medicine – Levels of Evidence (March 2009). 2009 https://www.cebm.net/2009/06/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/ Accessed May 20 2019
• 41 National Institute for Health and Care Excellence. Methods for the development of NICE public health guidance (third edition): Appendix H Quality appraisal checklist – qualitative studies. 2012 https://www.nice.org.uk/process/pmg4/chapter/appendix-h-quality-appraisal-checklist-qualitative-studies Accessed May 17 2019
• 42 Tee JC, Bekker S, Collins R. et al. The efficacy of an iterative "sequence of prevention" approach to injury prevention by a multidisciplinary team in professional rugby union. J Sci Med Sport 2018; 21: 899-904
• 43 Brown JC, Gardner-Lubbe S, Lambert MI. et al. The BokSmart intervention programme is associated with improvements in injury prevention behaviours of rugby union players: An ecological cross-sectional study. Inj Prev 2015; 21: 173-178
• 44 Brown JC, Gardner-Lubbe S, Lambert MI. et al. Coach-directed education is associated with injury-prevention behaviour in players: an ecological cross-sectional study. Br J Sports Med 2018; 52: 989-993
• 45 Brown JC, Verhagen E, Knol D. et al. The effectiveness of the nationwide BokSmart rugby injury prevention program on catastrophic injury rates. Scand J Med Sci Sports 2016; 26: 221-225
• 46 Poulos RG, Donaldson A. Is sports safety policy being translated into practice: what can be learnt from the Australian rugby union Mayday procedure?. Br J Sports Med 2012; 46: 585-590
• 47 Poulos RG, Donaldson A. Improving the diffusion of safety initiatives in community sport. J Sci Med Sport 2015; 18: 139-144
• 48 Quarrie KL, Gianotti SM, Hopkins WG. et al. Effect of nationwide injury prevention programme on serious spinal injuries in New Zealand rugby union: ecological study. BMJ 2007; 334: 1150
• 49 Sewry N, Verhagen E, Lambert M. et al. Trends in time-loss injuries during the 2011-2016 South African Rugby Youth Weeks. BMJ 2018; 28: 2066-2073
• 50 Gianotti SM, Quarrie KL, Hume PA. Evaluation of RugbySmart: A rugby union community injury prevention programme. J Sci Med Sport 2009; 12: 371-375
• 51 Bohu Y, Julia M, Bagate C. et al. Declining incidence of catastrophic cervical spine injuries in French rugby: 1996–2006. Am J Sports Med 2009; 37: 319-323
• 52 Fuller CW, Raftery M, Readhead C. et al. Impact of the International Rugby Board's experimental law variations on the incidence and nature of match injuries in southern hemisphere professional rugby union. S Afr Med J 2009; 99: 232-237
• 53 Gabbett TJ. Influence of the limited interchange rule on injury rates in sub-elite Rugby League players. J Sci Med Sport 2005; 8: 111-115
• 54 Gianotti S, Hume PA, Hopkins WG. et al. Interim evaluation of the effect of a new scrum law on neck and back injuries in rugby union. Br J Sports Med 2008; 42: 427-430
• 55 Hendricks S, Lambert MI, Brown JC. et al. An evidence-driven approach to scrum law modifications in amateur rugby played in South Africa. Br J Sports Med 2014; 48: 1115-1119
• 56 Noakes TD, Jakoet I, Baalbergen E. An apparent reduction in the incidence and severity of spinal cord injuries in schoolboy rugby players in the Western Cape since 1990. S Afr Med J 1999; 89: 540-545
• 57 Reboursiere E, Bohu Y, Retiere D. et al. Impact of the national prevention policy and scrum law changes on the incidence of rugby-related catastrophic cervical spine injuries in French Rugby Union. Br J Sports Med 2018; 52: 674-677
• 58 Evans KL, Hughes J, Williams MD. Reduced severity of lumbo-pelvic-hip injuries in professional Rugby Union players following tailored preventative programmes. J Sci Med Sport 2018; 21: 274-279
• 59 Attwood MJ, Roberts SP, Trewartha G. et al. Efficacy of a movement control injury prevention programme in adult men’s community rugby union: a cluster randomised controlled trial. Br J Sports Med 2018; 52: 368-374
• 60 Brooks JH, Fuller CW, Kemp SP. et al. Incidence, risk, and prevention of hamstring muscle injuries in professional rugby union. Am J Sports Med 2006; 34: 1297-1306
• 61 Cooney M, Coleman F, Flynn A. A study of the knowledge that school rugby coaches have in the management and prevention of serious neck injury. Ir Med J 2000; 93: 171-174
• 62 Cusi MF, Juska-Butel CJ, Garlick D. et al. Lumbopelvic stability and injury profile in rugby union players. NZ J Sports Med 2001; 29: 14-18
• 63 Hislop MD, Stokes KA, Williams S. et al. Reducing musculoskeletal injury and concussion risk in schoolboy rugby players with a pre-activity movement control exercise programme: a cluster randomised controlled trial. NZ J Sport Med 2017; 51: 1140-1146
• 64 Naish R, Burnett A, Burrows S. et al. Can a specific neck strengthening program decrease cervical spine injuries in a men's professional rugby union team? A retrospective analysis. J Sports Sci Med 2013; 12: 542-550
• 65 Brown JC, Boucher SJ, Lambert M. et al. Non-sanctioning of illegal tackles in South African youth community rugby. J Sci Med Sport 2018; 21: 631-634
• 66 Quarrie KL, Gianotti SM, Chalmers DJ. et al. An evaluation of mouthguard requirements and dental injuries in New Zealand rugby union. Br J Sports Med 2005; 39: 650-651
• 67 Gerrard DF, Waller AE, Bird YN. The New Zealand Rugby Injury and Performance Project: II. Previous injury experience of a rugby-playing cohort. Br J Sports Med 1994; 28: 229-233
• 68 Marshall SW, Loomis DP, Waller AE. et al. Evaluation of protective equipment for prevention of injuries in rugby union. Int J Epidemiol 2005; 34: 113-118
• 69 Marshall SW, Waller AE, Dick RW. et al. An ecologic study of protective equipment and injury in two contact sports. Int J Epidemiol 2002; 31: 587-592
• 70 Marshall SW, Waller AE, Loomis DP. et al. Use of protective equipment in a cohort of rugby players. Med Sci Sports Exerc 2001; 33: 2131-2138
• 71 Harden SM, Gaglio B, Shoup JA. et al. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: A systematic review. Syst Rev 2015; 4: 155
• 72 RS K, Glasgow P. A proposal to speed translation of healthcare research into practice: dramatic change is needed. Am J Prev Med 2011; 40: 637-644
• 73 Klugl M, Shrier I, McBain K. et al. The prevention of sport injury: An analysis of 12 000 published manuscripts. Clin J Sport Med 2010; 20: 407-412
• 74 Finch CF. No longer lost in translation: the art and science of sports injury prevention implementation research. Br J Sports Med 2011; 45: 1253-1257
• 75 Fuller CW, Ashton T, Brooks JH. et al. Injury risks associated with tackling in rugby union. Br J Sports Med 2010; 44: 159-167
• 76 Kraak W, Bam J, Kruger S. et al. Sanctioning of illegal and dangerous ruck cleanouts during the 2018 Super Rugby Competition. Front Psychol 2019; 10: 803
• 77 Gardner AJ, Iverson GL, Williams WH. et al. A systematic review and meta-analysis of concussion in rugby union. Sports Med 2014; 44: 1717-1731
• 78 McCrory P. Do mouthguards prevent concussion?. Br J Sports Med 2001; 35: 81-82
• 79 Avellar SA, Thomas J, Kleinman R. et al. External validity: the next step for systematic reviews?. Eval Rev 2017; 41: 283-325
• 80 Glasgow P. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health 2003; 93: 1261-1267
• 81 Steckler A, McLeroy KR. The importance of external validity. Am J Public Health 2008; 98: 9-10
• 82 Proctor E, Silmere H, Raghavan R. et al. Outcomes for implemenation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38: 65-76
• 83 RE-AIM. Reach Effectiveness Adoption Implementation Maintenance. 2019. Accessed May 19 2019 www.re-aim.org/
• 84 Finch CF, Donaldson A. A sports setting matrix for understanding the implementation context for community sport. Br J Sports Med 2010; 44: 973-978
• 85 Glasgow RE, Harden SM, Gaglio B. et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019; 7: 64
• 86 Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf 2008; 34: 228-243
• 87 O'Connor SR, Tully MA, Ryan B. et al. Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study. BMC Res Notes 2015; 8: 224

Publikationsverlauf

Eingereicht: 19. Februar 2020

Angenommen: 14. Juni 2020

Artikel online veröffentlicht:
28. Juli 2020