Keywords
rotator cuff - methodology - systematic review
Introduction
Systematic reviews of randomized clinical trials present the highest level of scientific
evidence for clinical decision-making.[1]
[2] The aim of these studies is to clarify the divergences found in the literature,
so that it is possible to answer a specific question and synthesize the findings of
primary studies. A good methodological design of these studies is essential so that
more assertive interventions become possible.[1]
[2] Currently, ∼ 24 instruments are validated and used to determine the level of reliability
of scientific studies, such as Preferred reporting items for systematic reviews and meta-analyses (PRISMA) and Assessing the Methodological Quality of Systematic Reviews (AMSTAR).[3] The AMSTAR is a validated measurement tool that assesses the methodological quality
of systematic reviews.[3] It has 11 domains that evaluate the methods of construction of systematic reviews.[4]
[5]
The scientific literature indicates numerous forms of treatment for the various types
of injuries that affect the musculoskeletal system, such as rotator cuff dysfunctions.
Thus, it is important to select works with good foundation, high methodological rigor
and reliable sources of information for greater efficacy at the time of the therapeutic
approach.[6]
[7] Evidence indicates that systematic reviews using the methodology proposed by the
Cochrane collaboration have a greater methodological rigor when compared to studies
that do not adopt this methodology.[8]
[9] Thus, the aim of the present research was to evaluate the methodological quality
of Cochrane and non-Cochrane systematic reviews on the treatment of individuals with
rotator cuff dysfunctions and to compare, through AMSTAR, the quality of studies found
in the Cochrane, PubMed (Publisher: Medline), EMBASE and Qinsight databases
Methods
This is a descriptive and comparative cross-sectional investigative study.
Only systematic reviews were included, with Cochrane and non-Cochrane methodology,
involving any type of primary studies. The term "PICOS" (population or problem/intervention/control/outcome/study
design) was used as a basis. There was no restriction on the language and date of
publication of the studies.
Systematic reviews were included in which the research subjects were individuals>
16 years old, diagnosed with rotator cuff syndrome (impact syndrome, subacromial bursitis,
tendinosis or tendinopathy of cuff structures rotator, partial or total rupture of
rotator cuff and tendinosis structures or calcifying tendinopathy of rotator cuff
structures) regardless of the time of disease evolution, treated non surgically or
surgically. Only studies that did not have diagnostic confirmation of the disease
were excluded.
The search was performed by two authors using the official Medical Subject Heading
(MESH) terms– (rotatorcuff, shoulder impingement syndrome, shoulder joint) in the following databases: Cochrane Library, PubMed, EMBASE and Qinsight. There
were no restrictions on the language and date of publications during the search process.
Search strategies were translated for eachdatabase. The studies were selected by a
single evaluator respecting the inclusion criteria described. Initially, after searching
all databases, duplicates were removed, then the articles were analyzed based on their
titles and abstracts; when the information contained in these two items was not sufficient,
it was left for the analysis of the text in full. After these steps, with the final
number of studies selected, a download was performed for follow-up with the evaluations
with AMSTAR.
Two independent reviewers (Kriebel C. F. and Estevam J. A.) evaluated the methodological
quality of the systematic reviews selected using the AMSTAR as an instrument.[3]
[5] This tool has 11 domains that investigate the following aspects:
-
1) Presence of a previous protocol with ethical approval and predetermined research
objectives
-
2) Extraction and evaluation of the material performed by two independent evaluators
-
3) Comprehensive bibliographic search in at least two databases
-
4) Inclusion of gray literature as a criterion
-
5) Presence of list or references of included and excluded studies
-
6) Presence of list or references of the studies included in the research
-
7) Methodological evaluation of each included study and documentation of the results
obtained
-
8) Critical formulation of conclusions based on methodological analysis
-
9) Evaluation of the heterogeneity of each study
-
10) Evaluation of publication bias with the presence of available graphs or tests
-
11) Mention of the conflicts of interests of the publication and the studies included.
Each question has four answer options, which are: 1) Yes; 2) No; 3) Cannot answer
and 4) Does not apply. For the calculation of the final score, only the positive answers
(Yes) counted, assigning 1 point for each positive answer of the questionnaire.[3]
[5]
The information regarding the systematic reviews included in the present study was
recorded in Excel software (Microsoft Corporation, Redmond, WA, USA) for the creation
of the database. The collected material was analyzed in software R version 3.4.0 (RFoundation
for Statistical Computing, Vienna, Austria) Variance analysis with a fixed factor
and tukey fixed comparison method were used. A significance number was used for the
present study where p< 0.05. Excel software tools[6] we reapplied to measure some variables such as mean, standard deviation and percentage
of grades obtained by studies after evaluation with AMSTAR.
The present study was submitted to the research ethics committee of the Universidade
Federal de São Paulo, with approval on 07/16/2017 - CEUAxnumber5960140717
Results
A total of 76systematic reviews were analyzed in different databases, namely: 9 Cochrane
systematic reviews that used the methodology proposed by the Cochrane organization
and published in the Cochrane Library, 26 systematic reviews not indexed in the Cochrane
database but which described that their methods follow the Cochrane methodology, 6
reviews indexed in Embase, 26 reviews indexed in Pubmed and 10 reviews indexed in
Qinsite.
After statistical analysis, it was observed that the mean score for the 76 studies
was 6.1 (±2.1), with 9.1 (±0.9) being the mean of the studies with Cochrane methodology
and indexed in the Cochrane database, and 5.7 (±1.8) for the other studies.
[Figure 1] presents the analysis of the methodological quality of these systematic reviews,
the percentage of scores obtained by the studies from the analysis made with AMSTAR.
Fig. 1 Percentage of studies that received scores from the Amstar instrument evaluation.
Data on AMSTAR domains with the lowest scores among all papers are shown in [Figure 2].
Fig. 2 Items with the lowest overall score according to the Amstar instrument.
[Figure 3] shows the average of the scores obtained by the studies after applying the AMSTAR
instrument.
Fig. 3 Average profiles of the studies, according to the final score obtained by the Amstar
instrument.
To compare the quality of systematic reviews in relation to the final scores, the
variance analysis model with a fixed factor and tukey multiple comparison method were
used. These data are described in [Table 1].
Table 1
|
Comparisons
|
Descriptive level
|
|
RNI
|
Cochrane
|
0.001
|
|
Embase
|
Cochrane
|
0.014
|
|
Pubmed
|
Cochrane
|
0.001
|
|
Qinsite
|
Cochrane
|
0.001
|
|
Embase
|
RNI
|
0.999
|
|
Pubmed
|
RNI
|
0.494
|
|
Qinsite
|
RNI
|
0.672
|
|
Pubmed
|
Embase
|
0.943
|
|
Qinsite
|
Embase
|
0.943
|
|
Qinsite
|
Pubmed
|
0.999
|
Discussion
Systematic reviews evaluating randomized clinical trials have the best level of scientific
evidence. They are research models that offer individuals a synthesis of data, in
order to base, theoretically, clinical practices, and guide the construction of new
projects; for this, it is important that this material is clear, explicit and reproducible;
in addition, to add value to the instrument, primary studies must have high methodological
rigor. Only in this way will clinical decision-making be possible based on scientific
evidence.[10]
[11]
The present study evaluated 76 systematic reviews on the treatment for rotator cuff
dysfunctions, selected in different databases. There are consistent data that demonstrate
a higher methodological quality of studies using the methodology proposed by the Cochrane
collaboration for systematic reviews in comparison with studies that do not follow
this methodology, a finding that corroborates a study that compared, using this same
instrument, the quality of Cochrane and non-Cochrane systematic reviews related to
health interventions.[12]
Some challenges were encountered when evaluating systematic reviews. It is noted that
non-Cochrane methodology reviews have limited information, insufficient or missing
data, which makes it difficult for AMSTAR to determine points, such as quality analysis
of all included primary studies, design of the writing based on an initial protocol,
publication bias, list of papers that were excluded from the systematic review, a
comprehensive search in the literature and analysis of conflicts of interests of publications.
In the scientific literature there are studies that stimulate discussions about the
domains of AMSTAR.[10]
[11] The description of each item is thorough and allows the reader to understand the
importance of analyzing each topic.
Item 11 of the AMSTAR checklist refers to the declaration of conflict of interests;
the authors must clearly and objectively expose any source of funding or support for
the research; for scoring, reports on all possible sources of funding should be required
because they minimize a potential influence and judgments. In this regard, all the
studies evaluated in the present study brought inconsistent information, thus it is
suggested a better clarification of this item to allow deeper evaluations. The second
item with the lowest score refers to the requirement of an initial research protocol,
which would help to delimit the studies that were included and excluded and the quality
of these. This domain is important because it identifies possible deviations in the
protocol, reducing the risks of publication bias. All studies with Cochrane methodology
for systematic reviews use an a priori protocol, but the non-Cochrane studies did not specify the use of this in their reviews,
compromising the result of the evaluation. Increasing the search for unpublished literature,
published or not, enriches the work. Many authors did not include in their reviews
the gray literature (type of unconventional publication); some publications emphasize
that more comprehensive studies allow a general analysis on a given theme.[10]
In general, the reviews with Cochrane methodology positively meet the criteria established
in AMSTAR, so the quality of these reviews is significantly better compared with works
extracted from Pubmed, Qinsite and Embase databases. Similar conclusions were published
when evaluating the methodological rigor of Cochrane and non-Cochrane systematic reviews
on oral health treatments.[13]
Evidence-based clinical decision-making necessarily depends on the quality of reviews
and a high methodological rigor of these publications.[14] In thepresentstudy, it is noted that many systematic reviews do not meet the established
methodological standards, thus producing unreliable evidence. The eligibility criteria
should be clearly defined, and the methodology should include data on the study population,
details of interventions, evaluation methods and the challenges encountered in implementation,
so that reproduction and clinical applicability of the conducts are possible.
Study Limitations
The present study used the first version of the Amstar instrument (2007), because
the selection and analysis of the studies were published before the publication of
Amstar 2.
Conclusion
Based on the results of the present study, it is possible to conclude that systematic
reviews with Cochrane methodology have a better scientific quality than the ones with
non-Cochrane methodologies. It is necessary to increase the rigor in scientific publications,
so that evidence-based clinical practices are better conducted. Further studies are
needed to stimulate discussion about the quality of studies published in various databases.
