Summary
Objectives:
Comparison of classification methods using data of one clinical study. The tuning
of hyperparameters is assessed as part of the methods by nested-loop cross-validation.
Methods:
We assess the ability of 18 statistical and machine learning classifiers to detect
glaucoma. The training data set is one case-control study consisting of confocal scanning
laser ophthalmoscopy measurement values from 98 glaucoma patients and 98 healthy controls.
We compare bootstrap estimates of the classification error by the Wilcoxon signed
rank test and box-plots of a bootstrap distribution of the estimate.
Results:
The comparison of out-of-bag bootstrap estimators of classification errors is assessed
by Spearman’s rank correlation, Wilcoxon signed rank tests and box-plots of a bootstrap
distribution of the estimate. The classification methods random forests 15.4%, support
vector machines 15.9%, bundling 16.3% to 17.8%, and penalized discriminant analysis
16.8% show the best results.
Conclusions:
Using nested-loop cross-validation we account for the tuning of hyperparameters and
demonstrate the assessment of different classifiers. We recommend a block design of
the bootstrap simulation to allow a statistical assessment of the bootstrap estimates
of the misclassification error. The results depend on the data of the clinical study
and the given size of the bootstrap sample.
Keywords
Classification - confocal scanning laser ophthalmoscopy - Glaucoma - R - machine learning
by small data sets - parameter tuning
