Key words
bias - exercise - scientific evidence - human performance - physiological data
Introduction
Ergogenic aids, defined as substances, devices, or procedures that have the potential
to enhance muscle size, neuromuscular function, or performance recovery [1] are frequently investigated by sports
scientists to improve athletes’ performance. Despite the increased number of
publications in this field, the quality of these studies may be compromised by
methodological limitations and the potential for publication bias, which tends to
favor positive results. These issues raise concerns about the reliability of
reported effects of ergogenic aids on athletic performance and emphasize the
importance of rigorous and transparent research practices in sports science.
Researchers often reference positive studies to generate new hypotheses, but it is
crucial to acknowledge the potential influence of publication bias on this
approach.
Scientific research aims to provide accurate and reliable measures of observed
outcomes. To achieve this, it is essential to conduct research with a good
methodology that is honest and free of bias [2]. Bias can be defined as the lack of internal validity in data
collection and data analysis, and a tendency to interpret outcomes in a target
population, causing equivocated conclusions [2]
[3]. In fact, different types of
bias exist in the literature, and they are typically classified based on the
direction of change they produce in a given parameter. For example, bias in data
collection occurs when the sample of studies is not randomized and the individual
inclusion is manipulated into the study groups.
Data analysis bias occurs when researchers create data that never existed, eliminate
data that contradict the study hypothesis, use inappropriate statistical analysis,
or perform multiple analyses until they obtain a result with a statistically
significant difference [2]
[4]. Interpretation bias is another type of
bias, which occurs when researchers disregard the original outcomes of the study and
interpret them based on their preconceived beliefs, hypotheses, and conflicts of
interest [2]
[5].
There is also the publication bias, which deals with journals more likely to publish
studies that report positive outcomes than studies that report negative outcomes
[2]
[4]. The dissemination of publications with this type of bias can create
the misconception that negative or statistically insignificant results are due to
negligent conduction, methodological failures, or inadequate experimental design
[6]
[7]. Publication bias poses a significant threat to the reliability of
scientific studies and can result in various consequences for society. These
consequences include the utilization of ineffective tools [8] and failure in replicating experiments [2].
Therefore, we aimed to evaluate whether publications on ergogenic aids are affected
by such bias, with a tendency for journals to publish studies presenting positive
results. We also aimed to assess the consistency between the actual results and
conclusions of publications on ergogenic aids in the field of sports science. Our
hypothesis was that editors and journals have a tendency to accept studies
demonstrating ergogenic benefits, resulting in a higher impact for the journal due
to multiple citations.
Materials and Methods
Two reviewers selected 87 journals between January and December 2020. Of the selected
journals, 85 were from the journal of citation reports (JCR) sports sciences list
and two were included due to their high number of publications related to human
performance. After this selection, the scope of each journal was observed, with the
remaining journals having met the following inclusion criteria: 1) have within their
scope the terms “sports medicine”, “sports science”,
“sports medicine and exercise”, “sports and exercise
medicine”, “sports and physiology”, “exercise and
physiology”, “exercise science and performance”; and 2)
journals not belonging to the sports sciences area if they met the inclusion
criteria and contributed to the area of sports sciences. The following non-inclusion
criteria were adopted: 1) publication of review articles exclusively; 2) no direct
relation to sport and performance; and 3) journals dedicated to medical treatment
or
rehabilitation.
After selecting the journals, we read the titles and abstracts of all published
articles between January and December 2021 to identify the studies that investigated
the effect of ergogenic aids. For the inclusion of the studies, the following
criteria were adopted: 1) to have been published in 2020; 2) to investigate the
effects of at least one ergogenic resource on physical performance; 3) only studies
with humans; and 4) having one or two variables directly or indirectly related to
physical performance. For the selection of studies, the following non-inclusion
criteria were adopted: 1) combination of ergogenic aids with any type of training;
2) evaluation of cognitive performance exclusively; and 3) no full access to the
article.
The reported outcomes of the articles were classified as positive, negative, or
neutral based on their identification in the conclusion and or title. The outcomes
of the variables used in the articles were identified in the actual results and
classified as positive, neutral, or negative based on whether a significant
increase, no change, or decrease was found between the experimental group and either
the placebo or control group.
Statistical analysis
Statistical analysis was performed using the software IBM SPSS (Version 23; IBM
Corp., Armonk, NY, USA). The chi-square test of independence was conducted to
verify association between variables outcomes and reported outcomes of the
articles identified in the conclusion and/or title. When a significant
association was found, the z-test adjusted by Bonferroni’s method was
used [9].
In addition, when necessary, the adjusted residual was observed to verify a
significant difference between count and expected count. Values below
–1.96 or above 1.96 were considered significant [10]. Lastly, Fisher’s exact test,
conducted when a cell has an expected count less than five, was used to verify
the association between reported outcomes of the articles and impact factor of
the journals, and between reported outcomes of the articles and publication
fees.
To perform the impact factor analysis, the normality of the data was checked and
we decided to use the median (=3.0 scores) to determine two groups of
journals: scores above 3.0 and below 3.0. The level significance adopted was
p<0.05.
Results
Of the 87 journals initially selected, only 20 met the inclusion and non-inclusion
criteria, resulting in a total of 219 studies. However, after a thorough screening
process, only 150 studies from 18 journals ([Table
1]) were deemed eligible for analysis ([Fig. 1]).
Fig. 1 Flow diagram of search process.
Table 1 Characteristics of selected journals.
|
Journals
|
Impact factor
|
Publication fees*
|
Number of issues/year
|
Total articles
|
|
A
|
12.68
|
No
|
24
|
333
|
|
B
|
5.20
|
No
|
6
|
90
|
|
C
|
5.07
|
No
|
24
|
53
|
|
D
|
4.03
|
No
|
6
|
333
|
|
E
|
3.61
|
No
|
1
|
193
|
|
F
|
3.53
|
Yes
|
12
|
220
|
|
G
|
3.37
|
Yes
|
1
|
55
|
|
H
|
3.25
|
No
|
12
|
245
|
|
I
|
3.04
|
Yes
|
12
|
330
|
|
J
|
2.97
|
No
|
12
|
430
|
|
K
|
2.85
|
Yes
|
24
|
209
|
|
L
|
2.78
|
No
|
10
|
149
|
|
M
|
2.74
|
Yes
|
12
|
202
|
|
N
|
2.60
|
No
|
24
|
303
|
|
O
|
2.58
|
No
|
12
|
249
|
|
P
|
2.56
|
No
|
14
|
112
|
|
Q
|
2.52
|
No
|
15
|
197
|
|
R
|
2.00
|
Yes
|
4
|
57
|
*or submission fees.
In our analysis of 150 studies from 18 journals, we identified 449 variables related
to physical performance. Of these variables, 181 (40.3%) showed
statistically significant improvements in performance, 18 (4%) showed
impairment, and 250 (56.7%) did not present significant changes ([Fig. 2]).
Fig. 2 Total number of articles and variables with positive, negative,
and neutral outcomes. The articles were classified based on their conclusion
and or title, while the variables used by the articles themselves were
classified based on their statistical result. a=significant
difference compared to articles.
In each study, the reported outcomes were identified in the conclusion and in the
title. Of the outcomes observed, 114 studies (76%) reported positive
outcomes, 4 studies (2.7%) reported negative outcomes, and 32 studies
(21.3%) reported neutral outcomes ([Fig.
2]). The chi-square test of independence showed a significant association
between reported outcomes of the articles and reported outcomes of the variables
(X2
(2)=57.803; p<0.001), detecting a
significant difference between them in the positive and neutral outcomes but not in
the negative outcomes ([Fig. 2]).
For the positive outcomes, the expected count of the reported outcomes of the
articles was significantly lower than that found (observed), while the expected
count of the variables outcomes was significantly higher than that found (observed).
On the other hand, for the neutral outcomes, the expected count of the reported
outcomes of the articles was significantly higher than that found (observed), while
for the variables outcomes it was significantly lower than that found (observed).
For the negative outcomes, no significant difference was found ([Table 2]).
Table 2 Count, expected count, and adjusted residual of the
reported outcomes of the variables and scientific
articles.
|
Outcomes
|
|
Articles
|
Variables
|
Total
|
|
Positive
|
Count
|
114
|
181
|
295
|
|
Expected Count
|
73.9
|
221.1
|
295.0
|
|
Adjusted Residual
|
7.6
|
–7.6
|
–
|
|
Negative
|
Count
|
4
|
18
|
22
|
|
Expected Count
|
5.5
|
16.5
|
22.0
|
|
Adjusted Residual
|
–0.8
|
0.8
|
–
|
|
Neutral
|
Count
|
32
|
250
|
282
|
|
Expected Count
|
70.6
|
211.4
|
282.0
|
|
Adjusted Residual
|
–7.3
|
7.3
|
–
|
Adjusted residual higher than 1.96 or less than –1.96 means a
significant difference between count and expected count.
Fisher’s exact test showed no significant association between reported
outcomes of the articles and either the impact factor
(X2
(2)=3.365; p=0.159) or publication fees
(X2
(2)=2.718; p=0.271) ([Table 3]).
Table 3 Association between reported outcomes of the articles
and impact factor and publication fees.
|
Reported outcomes of the articles
|
|
|
Impact factor
|
Positive
|
Negative
|
Neutral
|
Total
|
|
Above 3.0
|
39
|
3
|
14
|
56
|
|
Below 3.0
|
75
|
1
|
18
|
94
|
|
Total
|
114
|
4
|
32
|
150
|
|
Reported outcomes of the articles
|
|
|
Publication fees*
|
Positive
|
Negative
|
Neutral
|
Total
|
|
Yes
|
40
|
3
|
13
|
56
|
|
No
|
74
|
1
|
19
|
94
|
|
Total
|
114
|
4
|
32
|
150
|
*or submission fees.
Discussion
This study evaluated whether scientific publications on ergogenic aids and exercise
performance are affected by bias and also assessed the consistency between the
actual results and titles/conclusions presented. The main results confirm
our hypothesis, indicating a tendency for journals to publish studies with positive
results while neglecting those with negative results. Additionally, our analysis
revealed inconsistencies between the reported outcomes in the titles and conclusions
and the actual performance variable outcomes. Specifically, we observed a
predominance of non-significant outcomes in the performance variables, suggesting
a
potential discrepancy between the reported conclusions and the actual study
outcomes.
We observed an inconsistency between reported positive outcomes (76%) and
statistically significant variable outcomes (40.3%). Furthermore, a very
small rate of studies reported a negative outcome (4%). This inconsistency
can hinder the formation of knowledge about the effects of ergogenic aids, confusing
the interpretation of the reported outcomes. This confusion can lead to errors in
the application of ergogenic aids in the field of physical performance, such as
athletes using aids that may not be effective or safe, or coaches and trainers
making training and performance decisions based on incomplete or biased
information.
It is important to highlight that such errors could have serious consequences, not
only for the athletes’ performance but also for their health and well-being
[11]. Therefore, it is essential to
address these issues and ensure that the scientific evidence on ergogenic aids is
reliable and unbiased to guide the safe and effective use of these aids in sports
science.
A similar study to ours analyzed the effects of antidepressant medications by
comparing the results of published research in scientific journals with the results
of work in the United States Food and Drug Administration (FDA) database [12]. According to their analysis, the published
literature showed that 94%of trials investigating antidepressants reported
significant positive effects. This percentage differs from the FDA database, which
found significant positive effects in only 51%of the trials.
Several factors can contribute to an increase in results that do not show statistical
significance. These include variations in individual and temporal physiological
responses, a lack of clear protocols that elicit responses, and inherent measurement
errors in the testing process [13]
[14]. Non-positive results, on the other hand,
can contribute to the progress of research by providing evidence for alternative
hypotheses and directing attention towards key variables for improved
performance.
Our analysis of [Table 3] suggests that there
is no significant relationship between the reported outcomes of the articles and
either the impact factor or publication fees. This finding is notable, as previous
research has suggested that journals with high impact factors may be more likely to
publish articles reporting positive results [15]. However, our study found no statistically significant difference in
the proportion of positive, negative, or neutral outcomes between articles published
in high-impact and low-impact journals, or between those that required payment of
publication fees and those that did not.
While there may be other factors at play, such as sample size or study design, that
can influence the outcomes reported in scientific articles, our analysis indicates
that impact factor and publication fees alone are not reliable indicators of the
reported outcomes of research studies involving ergogenic aids and exercise
performance.
The findings of this study show that journals tend to prioritize studies with
positive results, often neglecting those with negative outcomes. Furthermore, our
analysis indicates inconsistencies between reported outcomes in titles and
conclusions and actual performance variable outcomes, with a higher prevalence of
non-significant results in performance variables. These observations suggest a
potential discrepancy between the reported conclusions and actual study outcomes.
Future research should aim to reduce bias and encourage the publication of studies
with both positive and negative results to improve the reliability of scientific
evidence on ergogenic aids and exercise performance.
Bibliographical Record
Moacir Marocolo, Gustavo R. Mota, Alex Batista Rodrigues, Roberto C. de Matos Leite,
Rodrigo Hohl, Rodney Coelho da Paixão, Hiago L. R. Souza, Anderson Meireles, Rhai
Arriel. Unveiling Bias: Examining the Influence of Positive Results on
Ergogenic Aids in Published Sports Science Studies. Sports Med Int Open 2024; 08:
a21816798.
DOI: 10.1055/a-2181-6798
