Information Sources and Decision-Making in Neurosurgery: Results of a Survey of Members of the Brazilian Neurosurgery Society

Abstract

Introduction In all surgical disciplines, including neurosurgery, there are questions about the level of evidence supporting surgical practices and the mechanisms and adequacy of knowledge translation.

Objectives To assess the perception of Brazilian neurosurgeons of information sources and decision-making mechanisms related to their medical practices.

Methods An online questionnaire was sent to the 2,400 members of the Brazilian Neurosurgical Society.

Results A total of 32% of the neurosurgeons completed the questionnaire, 53% had more than 10 years experience, 67% had worked in public hospitals, 34% had performed spine surgeries, and 30% had performed brain tumor surgeries. The therapeutic decisions were based mostly on internship learning (54%) and personal professional experience (52%). The most common information sources were scientific abstracts (53%) and the Internet (47%). A total of 89% believed that evidence-based medicine was relevant, 93% believed protocols or guidelines were necessary, and 74% subscribed to a medical journal. Nonetheless, only 43% had protocols implemented in their services, 93% highly valued a surgeon's personal experience, and 63% showed little familiarity with the interpretation of scientific concepts in the literature. Among the respondents, 83% were willing to try an innovative treatment alternative if it was shown to improve clinical outcomes and reduce severe complications.

Conclusions The disparity in the responses highlights the need to implement recommendations that improve decision-making mechanisms.

Resumo

Introdução Em todas as disciplinas cirúrgicas, incluindo a neurocirurgia, existem questões sobre o nível de evidência que apoia as práticas cirúrgicas e os mecanismos e adequação da translação do conhecimento.

Objetivos Avaliar a percepção de fontes de informação e mecanismos de tomada de decisão dos neurocirurgiões brasileiros em relação às práticas médicas.

Métodos Um questionário on-line foi enviado aos 2.400 membros da Sociedade Brasileira de Neurocirurgia.

Resultados Um total de 32% dos neurocirurgiões preencheram o questionário, 53% tinham mais de 10 anos de experiência, 67% trabalharam em hospitais públicos, 34% realizaram cirurgia de coluna, e 30%, de cérebro. As decisões terapêuticas basearam-se principalmente no aprendizado de estágio (54%) e na experiência profissional pessoal (52%). As fontes de informação mais comuns foram resumos científicos (53%) e a Internet (47%). Um total de 89% acreditava que a medicina baseada em evidências era relevante, 93% acreditavam que protocolos ou diretrizes eram necessários, e 74% tinham assinaturas de uma revista médica. No entanto, apenas 43% apresentaram protocolos implementados em seus serviços, 93% valorizaram a experiência pessoal de um cirurgião, e 63% mostraram pouca familiaridade com a interpretação de conceitos científicos na literatura. Entre os respondentes, 83% estavam dispostos a tentar uma alternativa de tratamento inovador se este demonstrasse melhorar os resultados clínicos e reduzir as complicações graves.

Conclusões A disparidade nas respostas destaca a necessidade de implementar recomendações que melhorem os mecanismos de tomada de decisão.

Keywords

neurosurgery - evidence-based medicine - source of information - decision-making

Palavras-chave

neurocirurgia - medicina baseada em evidências - fonte de informação - tomada de decisões

Introduction

Neurosurgical practices, similarly to the practices of other medical specialties, depend on the diffusion, acceptance, and establishment of specific technical procedures and clinical management protocols. This is undertaken through the generation of scientific data, ideally obtained through randomized controlled trials (RCTs) or, in their absence, based on other information sources with the highest possible level of evidence.[1]

Although an increasing number of clinical trials are reported in the scientific literature, there are insufficient quality RCTs in surgical specialties providing a high level of evidence upon which to base clinical practice. This may imply that surgeons tend to be conservative regarding their own practices, only subtly modifying a procedure already performed with apparent success.[2] [3] [4] [5]

Thus, from a more objective and analytical perspective, there is a great lack of evidence, that is, scientific demonstration, for many surgical and neurosurgical procedures. Furthermore, in the literature, too many authors, common procedures, and personal opinions are published as constituting scientific proof, which is clearly against good scientific practice.[6] Therefore, the quality of published neurosurgical clinical trials is a cause of concern, as are the difficulties regarding adequate knowledge translation or the assessment of the scientific of proposed treatments in neurosurgery.[7] [8] [9] [10] [11] [12]

To approach this topic, a survey of surgeons who are members of the Brazilian Neurosurgery Society (SBN, in the Portuguese acronym) was conducted to assess the transmission of scientific knowledge. Therefore, the primary objective of this study was to assess the perception of the neurosurgeons of the information sources and decision-making mechanisms related to their own medical practices. The secondary objectives were to characterize the importance given to several sources of knowledge, to identify the willingness of neurosurgeons to change their current practices and the factors involved in this decision, to characterize their perception of scientific trials, and to identify differences in the transmission of neurosurgical knowledge among the different groups of neurosurgeons.

Materials and Methods

In September 2015, using a cross-sectional observational design, a questionnaire was sent to the 2,400 members of the SBN. The questionnaire was specifically designed for the present study and was distributed through SurveyMonkey (SurveyMonkey, San Mateo, CA, US) tool. It contained 15 questions divided into 5 sections: characterization of the participants; perception of the research in neurosurgery and the decision-making process; the way knowledge is obtained and transmitted; how neurosurgeons handle new therapeutic alternatives; and analysis of ethical considerations in the conception and implementation of clinical trials. The SBN sent a link to the online survey by email to all its members. Reminder emails were sent 3 times within 15 days. During these two weeks, a link and a request to fill out the questionnaire were also available on the SBN website (see the complete questionnaire as supplemental material).

The descriptive analysis of the response frequencies and the comparison between subgroups of duration and place of practice was performed using confidence intervals in comparative analyses and the Pearson chi-squared test for the correlations. For cases with 20% or more of the observations with a response frequency lower than 5, the Fisher exact test was used. Statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS, IBM Corp., Armonk, NY, US). version 20.0.

Responses were collected anonymously, jointly analyzed, and only complete responses for each question were considered. The ethical committee of Faculdade de Medicina de Lisboa and the board of directors of the SBN were consulted for approval and saw no objections, and deferred the need for formal informed consent.

Results

The response rate was 32% (769) of the questionnaires sent. Of these, 87.5% (660) had answers to all 15 questions, and 22.5% (109) had partial answers.

More than half of the respondents (53.3%) reported having over 10 years of clinical practice. Among those with less than 10 years of practice, 15.6% were residents, 18.24% had less than 5 years of practice, and 12.7%, between 5 and 10 years.

Over 2/3 (67.8%) of the neurosurgeons performed activities in public hospitals, and 37.4% worked exclusively in the public sector. Of the total number of surgeons, 30.4% worked equally between public and private practices, and only 13.2% worked exclusively in the private sector.

The most frequent subspecialties of the respondents were spine (34.1%), followed by brain tumors, 29.3%. Other surgical subdivisions did not present high response rates: vascular (7.8%), pediatrics (5.6%), and functional (3.3%).

Source of Information and Research Perception for the Decision-Making Process

In therapeutic decision-making, neurosurgeons reported valuing especially the education they had received during their specialty residency (54.3%), as well as their personal professional experience (51.9%), while consultation of information in the literature, protocols, and academic experiences were reported by less than 30%. Concerning the sources of information used weekly, 52.7% read scientific article abstracts; 46.8% consulted information available on the Internet, 46.5% held rounds with colleagues, and 45% consulted textbooks ([Table 1]).

Table 1 Frequency of consultation of sources of information

Response options	Daily	Weekly	Monthly	1 or 2 times in the past year	None in the past year
Reading scientific article abstracts	17.6%	52.7%	23.4%	5.2%	1.1%
Consultation of information available on the web	34.1%	46.7%	15.5%	2.7%	0.9%
Discussions of therapies or management with neurosurgeon colleagues	33.2%	46.5%	12.3%	5.1%	2.0%
Consulting textbooks	19.1%	44.9%	27.1%	7.7%	1.1%
Participation in courses or workshops	1.0%	1.7%	17.4%	69.9%	9.9%
Participation in national symposiums and congresses	1.4%	0.6%	6.5%	78.8%	12.7%
Participation in international congresses	1.0%	0.3%	1.7%	44.5%	52.5%

Note: Lines by decreasing order for weekly responses, p < 0.005

Regarding the perception of the neurosurgeons of information related to their clinical practice, 87.9% believed evidence-based medicine (EBM) to be relevant or highly relevant, and 93.3% also considered the personal experience of the surgeon relevant or highly relevant ([Table 2]). Of all respondents, 92.9% believed the implementation of protocols for clinical decision-making to be relevant or highly relevant, and 43.1% have protocols implemented in their neurosurgical services. Furthermore, 73.6% of the surgeons subscribed to a medical journal.

Table 2 Importance of the surgeons' professional experience and EBM in neurosurgery

Response options	Relevant	Very relevant	Little relevant	Irrelevant
Personal Experience	51.0%	42.0%	6.7%	0.0%
EBM	50.0%	37.9%	11.3%	0.8%

Abbreviation: EBM, evidence-based medicine.

Note: p ≤ 0.005.

Disposition to Change Current Clinical Practice

The majority (82.6%) of the respondents showed a personal willingness to modify their current clinical practice if the goal was the improvement of long-term outcomes (74.7%) and a reduction in severe complications (62.1%). All other reasons elicited less than 40% of responses. Those who stated that they were “not willing” to change their current practice justified this answer with the beliefs that there is a lack of scientific evidence to support new procedures, and that the learning curve of the use of a new technique with which they are not familiar is too steep; both reasons obtained 48.2% of responses each.

Perception of Clinical Studies

When requested to identify the most important types of scientific studies, the respondents mainly valued cohort studies (63.2%), followed by case-control studies (52.8%), observational studies (49.1%) and clinical trials (43.1%). The following criteria were used to classify scientific articles as being excellent: levels and degrees of evidence (76.1%), rigorousness of the statistical analysis (56.1%), the existence of a control group (52.3%), and the names of authors/institutions (37% and 34.1% respectively). The number of authors did not influence the attributed value of the article.

Regarding controlled studies, 65.5% perceived that randomization improves the quality of controlled trials, and 61.1% stated being aware that such studies are not common in neurosurgery. The respondents reported an intermediate level of familiarity with academic concepts ([Table 3]), with the percentage varying between 40.7% and 51% for all concepts presented (clinical guidance norms, guidelines, clinical cases, evidence-based medicine, therapeutic protocols, case series, evidence levels, systematic reviews, grades of recommendation, meta-analyses).

Table 3 Frequency of factors that influence the therapeutic decision-making in neurosurgery

Response options	Always or almost always	Most of the times	Somewhat frequently	Never or almost never
Education received during the specialty internship	54.3%	33.1%	11.7%	0.8%
Personal professional experience	51.9%	35.9%	11.3%	1.3%
Recent medical literature (published in the previous year)	34.7%	42.1%	20.6%	2.5%
Protocols or international guidelines	34.0%	35.8%	24.2%	6.3%
Medical visits	30.4%	28.5%	26.6%	14.5%
Protocols or guidelines/clinical guidance norms	27.1%	36.3%	28.7%	7.7%
Education received during the medical course	24.9%	24.3%	30.8%	18.0%
Continued education (courses and congresses)	24.0%	33.7%	37.4%	4.8%
Protocols from the place they work	23.6%	31.1%	26.6%	18.7%
Opinions of neurosurgeon colleagues	12.8%	25.5%	57.2%	4.4%

Abbreviation: EBM, evidence-based medicine.

Note: Lines in decreasing order for the responses always or almost always; p ≤ 0.05 and confidence interval: 3.5–3.7

Comparison of Duration of Practice

When comparing groups by duration of clinical experience (≤ 10 years versus > 10 years), those with up to 10 years of clinical experience favor EBM, the learning received during residency, and consultation of recently published literature and guidelines, while those with over 10 years of clinical experience prefer personal opinions, case discussions with colleagues, and attendance of international congresses. There were no statistical differences in responses by practice location (public versus private).

Discussion

Results from this cross-sectional survey indicate that Brazilian neurosurgeons valued especially the education they received during their specialty residency and their personal professional experience, in detriment of consulting data in the literature, protocols, or academic experiences of others.

Evidence-based medicine was valued in the clinical practice, as was personal clinical experience, and there is a willingness to modify current clinical practices to improve long-term clinical outcomes and reduce complications. The more experienced the surgeons, the more likely they were to rely on their clinical experience, and the less likely they were to rely on the scientific literature.[5] [13]

Neurosurgeons inadequately stratified and validated the different types of clinical studies, although there is a preference for scientific articles with high scientific evidence, and randomization is considered to improve the quality of controlled studies.

When subdivided by duration of clinical experience, the younger neurosurgeons prefer EBM, while their seniors prefer personal experience. The 32% response rate was quite superior to other online medical questionnaires.[14] [15]

The learning resources of the respondents with more than 10 years of clinical practice were mostly based, over the previous year, on therapeutic conduct discussions with colleagues, while the younger respondents were more likely to refer to the published literature. These data corroborate previous studies that point out that not only does learning undergo a historic evolution of its role and means of acquisition, but also that the development of competencies to search and use information to produce scientific knowledge may generate practical changes.[16]

For those with the most experience, the individual professional experience and the opinion of the colleagues of the neurosurgeon are considered a priority in the neurosurgeon's day-to-day therapeutic decision-making, while the younger neurosurgeons prefer the education received during residency, recently published literature, and guidelines.[16] In the neurosurgical sphere,[17] there is a great emphasis and a tendency to prioritize the development of psychomotor faculties that guarantee adequate performance of surgical techniques, leaving the cognitive aspects in the background, such as the investment in obtaining scientific knowledge to back up decision-making with logical rationales. Nonetheless, an interest in scientific investment to back up the decision-making process and the neurosurgical practice may be considered by some authors as an ethical professional attitude.[18]

The concept of EBM was relevant to 87.9% of the respondents, while professional experience was important to 93.3%. This apparently irreconcilable dichotomy, valuing both professional experience and EBM, is not foreign to other authors. They perceive that the development of a surgeon as the one who seeks better evidence for decision-making is long and needs investment, requiring search in research databases and libraries with up-to-date material, as well as contact with centers of excellence and time spent with specialists to master the instruments of the evidence-based surgery practice.[16] [19] [20]

Within this context, there is a tendency in the literature to consider the professional with many years of experience in surgery as scientifically outdated and inclined to make decisions based on empiricism and the outcomes of their own practice.[20] [21]

The adoption of EBM includes the potential to improve professional qualifications through the development of competencies, to contribute to fostering research, and to improve the use of diagnostic methods and the objectivity of treatments. With that, the prognostic perspectives and the life expectancy will likely improve, the cost of treatment may be reduced, and an improvement in the quality of life will ensue as a natural consequence of this process.[5] [17] [19] [22] [23] [24]

Although the introduction of clinical guidelines is positive in the sense of facilitating the review of the vast available evidence, which the great majority of the respondents consider important, the actual transference of this importance to the organization by the implementation of these guidelines is still not optimal in a significant number of services.[25] [26]

The majority of the respondents appear to have enough knowledge to orient themselves and seek relevant scientific studies. Nevertheless, they do not seem familiar with the classic concepts of study subtypes. Additionally, most surgeons did not have a clear idea of concepts such as the prevalence of controlled studies, the advantage of their random character and of randomization itself, or about the scientific levels of neurosurgery publications. Therefore, the relevance of enhancing scientific knowledge must be pointed out, especially regarding RCTs, due to their importance.[27] [28]

Some authors have analyzed the difficulties found in surgical clinical research standards, and indicate many problems in performing RCTs in surgery: the structural, cultural, and psychological resistance to the use of randomization, the variability inherent to surgery that requires a precise definition of the interventions, a strict monitoring of the quality, the surgery learning curves that pose difficulties to the time and execution of randomized trials regarding new techniques, the comparison of surgical and non-surgical treatments, and, lastly, the rare, urgent and life-threatening situations that result in difficulties in the recruitment, consent and randomization.[2] [3] [6] [11] [27] [29] [30] [31] [32]

Furthermore, it must be pointed out that the inadequate stratification of research studies by the respondents must serve as a warning for the need of a greater clarification of the true understanding of the neurosurgeons of the different types of studies, especially since they mentioned subscribing to at least one scientific journal.

Within the concepts presented, making better choices regarding health and healthcare requires the best possible evidence. Therefore, as rich and different digital data sources become broadly available for research, analytical tools grow in power and sophistication.[2] [3] [18] [30] [33]

Research and health communities now have the opportunity to quickly and efficiently generate the scientific evidence necessary to support improved decision-making in health and healthcare, without reducing the importance of the opinions and qualitative information of the specialists as a complementary source of knowledge. Therefore, it is considered an opportunity to use qualitative methods to complement high quality quantitative data within a more focused approach.[2] [3] [18] [30] [33] [34] [35]

Therefore, surgical research must consider daily clinical surgery and surgical translational research issues, introducing new techniques and laboratory results in assisting the patient, and require clinical surgeons with competency in research. Consequently, it is necessary to allocate major efforts in developing and maintaining high-quality surgical investigations in academic surgical departments, including individual career-advisory programs and clinical trial centers aiming at attractive academic positions in surgery, and the promotion of translational research, as a benefit to patient care.[12] [36] [37]

The present study is limited by the fact that only answers from a part of the SNB members were analyzed, and the fact that those who answered might belong to subgroups of members who are more motivated or more familiar with website platforms. However, the members of the SBN presented a questionnaire response rate superior to that of other similar studies,[14] [15] and were divided into 2 similar groups with more or less than 10 years of neurosurgical experience, with most working in the public service, and mainly performing spine or brain tumor surgeries. The concentration of respondents in the early years of their career adequately reflects the age distribution of specialists in a country in which medical education in neurosurgery has increasingly progressed over the past 60 years.[13] The concentration in public hospitals reflects their association with treatment resources to treat patients with greater complexity, while the preference for spine pathologies reflects the normal distribution[5] [9] [13] [16] [18] of neurosurgical subspecialties. Nonetheless, it is not impossible that the respondents may overestimate their use of resources in order to unconsciously provide a positive view of themselves.

Although we have included a large sample of the Brazilian neurosurgery field, the extrapolation of the results must be considered with caution prior to the comparison of the results with other national neurosurgeon samples or with other Brazilian surgeons.

Conclusion

The members of the SBN who answered the questionnaire did so with an above-average response rate, with the majority working in public settings, especially performing spine or brain tumor surgeries. The results differ by duration of experience.

The most preferred information sources are weekly reading of scientific articles abstracts, discussing conducts with colleagues, and consulting textbooks. Here, the older surgeons prefer therapeutic discussions with colleagues, and the younger surgeons prefer consulting the literature.

The least experienced neurosurgeons privilege the education received during their medical residency when making therapeutic decisions, the recent medical literature and national and international guidelines, while those more experienced prefer to rely on their individual professional experience.

Neurosurgeons stated willingness to try innovative treatment alternatives, especially when taking into account the improvement in outcomes and reduction in severe complications. When changing practice was not considered, it was due to the lack of scientific evidence or to the risks of using unfamiliar techniques.

The vast majority of the respondents attributed great relevance both to EBM and the personal experience of the surgeon. They considered neurosurgical protocols as being very important, although less than half of the respondents have protocols in place at their respective hospitals. They also mentioned having little familiarity with the interpretation of scientific concepts in the literature, despite identifying articles as being excellent due to their evidence level, highly strict statistical analysis and the existence of a control group.

All this demonstrates the need to implement recommendations to improve decision-making mechanisms. The SBN or another representing authority could eventually consider undertaking this responsibility.

No conflict of interest has been declared by the author(s).

• References

• 1 Lee K. The Philosophical Foundations of Modern Medicine. London: Polgrave Macmillan; 2012
• 2 Ziewacz JE, McGirt MJ, Chewning Jr SJ. Adverse events in neurosurgery and their relationship to quality improvement. Neurosurg Clin N Am 2015; 26 (02) 157-165 , vii
• 3 Solomon MJ, Laxamana A, Devore L, McLeod RS. Randomized controlled trials in surgery. Surgery 1994; 115 (06) 707-712
• 4 https://pt.surveymonkey.com/r/Preview/?sm=AZYhti6WuHOf_2FDJxGGyyXFmhOSKoFw2yoZDLJ3mQdUG6S2a_2FbQFn3zo5oLM9RSrz
• 5 Medeiros LR, Stein A. Medicina baseada em evidências e análise de decisão na clínica cirúrgica. Revista AMRIGS 2001; 45 (1,2): 45-50
• 6 Barker II FG. Editorial: Randomized clinical trials and neurosurgery. J Neurosurg 2016; 124 (02) 552-556 , discussion 556–557
• 7 Weinstein JN, Lurie JD, Olson PR, Bronner KK, Fisher ES, Morgan TS. United States' trends and regional variations in lumbar spine surgery: 1992-2003. Spine 2006; 31 (23) 2707-2714
• 8 Agazzi E, Faye J. The Problem of the Unity of Science. London: World Scientific; 2001
• 9 Dantas AK. Avaliação do aprendizado em técnica cirúrgica empregando três estratégias de ensino. [Thesis]. São Paulo: Faculdade de Odontologia da Universidade de São Paulo; 2010
• 10 Heros RC. Randomized clinical trials. J Neurosurg 2011; 114 (02) 277-278 , discussion 278–279
• 11 Schöller K, Licht S, Tonn JC, Uhl E. Randomized controlled trials in neurosurgery–how good are we?. Acta Neurochir (Wien) 2009; 151 (05) 519-527 , discussion 527
• 12 Vranos G, Tatsioni A, Polyzoidis K, Ioannidis JP. Randomized trials of neurosurgical interventions: a systematic appraisal. Neurosurgery 2004; 55 (01) 18-25 , discussion 25–26
• 13 Gusmão SS, Souza JG. História da Neurocirurgia no Brasil. São Paulo: Sociedade de Neurocirurgia do Brasil; 2008
• 14 Kongsved SM, Basnov M, Holm-Christensen K, Hjollund NH. Response rate and completeness of questionnaires: a randomized study of Internet versus paper-and-pencil versions. J Med Internet Res 2007; 9 (03) e25
• 15 VanGeest JB, Johnson TP, Welch VL. Methodologies for improving response rates in surveys of physicians: a systematic review. Eval Health Prof 2007; 30 (04) 303-321
• 16 Gasque KCGD. O papel da experiência na aprendizagem: perspectivas na busca e no uso da informação. Transinformacao 2008; 20 (02) 149-158
• 17 Traynelis VC. The geometry of education: patterns for growth. Clin Neurosurg 2005; 52: 1-5
• 18 Isolan GR. A construção do conhecimento pelo jovem neurocirurgião: ética, ciência e a importância do treinamento em laboratório de microcirurgia. J Bras Neurocirurg 2009; 20 (03) 314-334
• 19 Gomes MM. Medicina baseada em evidências: princípios e práticas. Rio de Janeiro: Reichmann & Affonso; 2001: 1-13
• 20 Schanaider A. Cirurgia baseada em evidências: modismo ou necessidade?. Acta Cir Bras 2002; 17 (01) 71-74
• 21 Black N. Evidence-based surgery: A passing fad?. World J Surg 1999; 23 (08) 789-793
• 22 Gomes LF. Educação Médica Contínua em MGF. Rev Port Clin Geral 2003; 19: 89
• 23 O'Brien BJ, Heyland D, Richardson WS, Levine M, Drummond MF. Users' guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. B. What are the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA 1997; 277 (22) 1802-1806
• 24 Sauerland S, Lefering R, Neugebauer EA. The pros and cons of evidence-based surgery. Langenbecks Arch Surg 1999; 384 (05) 423-431
• 25 Santos P, Martins C, Sá L, Hespanhol A, Couto L. Motives for requesting an electrocardiogram in primary health care. Cien Saude Colet 2015; 20 (05) 1549-1554
• 26 Santos P, Nazaré I, Martins C, Sá L, Couto L, Hespanhol A. As Normas de Orientação Clínica em Portugal e os Valores dos Doentes. Acta Med Port 2015; 28 (06) 754-759
• 27 Altman DG. Melhor relato de ensaios controlados randomizados: a declaração CONSORT. BMJ 1996; 313: 570-571
• 28 Begg C, Cho M, Eastwood S. , et al. Melhorar a qualidade da notificação de ensaios controlados aleatórios: a declaração CONSORT. JAMA 1996; 276: 637-639
• 29 Can OS, Yilmaz AA, Hasdogan M. , et al. Has the quality of abstracts for randomised controlled trials improved since the release of Consolidated Standards of Reporting Trial guideline for abstract reporting? A survey of four high-profile anaesthesia journals. Eur J Anaesthesiol 2011; 28 (07) 485-492
• 30 Cândido DNC, Barbosa FT. Qualidade dos ensaios clínicos aleatórios em Neurocirurgia publicados no Brasil. Arq Bras Neurocir 2009; 28 (02) 43-47
• 31 Mansouri A, Cooper B, Shin SM, Kondziolka D. Randomized controlled trials and neurosurgery: the ideal fit or should alternative methodologies be considered?. J Neurosurg 2016; 124 (02) 558-568
• 32 Ioannidis JP, Haidich AB, Pappa M. , et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA 2001; 286 (07) 821-830
• 33 Haines SJ. Randomized clinical trials in neurosurgery. Neurosurgery 1983; 12 (03) 259-264
• 34 Califf RM, Robb MA, Bindman AB. , et al. Transforming Evidence Generation to Support Health and Health Care Decisions. N Engl J Med 2016; 375 (24) 2395-2400
• 35 Vollmar B. [Research as attractiveness parameter for young surgeons]. Chirurg 2012; 83 (04) 319-322
• 36 Gittes GK. The surgeon-scientist in a new biomedical research era. Surgery 2006; 140 (02) 123-131
• 37 Menger MD, Schilling MK, Schäfers HJ, Pohlemann T, Laschke MW. How to ensure the survival of the surgeon-scientist? The Homburg Program. Langenbecks Arch Surg 2012; 397 (04) 619-622