Methods Inf Med 2017; 56(02): 95-111
DOI: 10.3414/ME16-02-0013
Schattauer GmbH

Combined Vision and Wearable Sensors-based System for Movement Analysis in Rehabilitation

Sofija Spasojević
1   School of Electrical Engineering, University of Belgrade, Belgrade, Serbia
2   Mihailo Pupin Institute, University of Belgrade, Belgrade, Serbia
3   Institute for Systems and Robotics, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal
Tihomir V. Ilić
4   Department of Neurophysiology, Medical Faculty of Military Medical Academy, University of Defense, Belgrade, Serbia
Slađan Milanović
5   Institute for Medical Research, Department of Neurophysiology, University of Belgrade, Serbia
Veljko Potkonjak
1   School of Electrical Engineering, University of Belgrade, Belgrade, Serbia
Aleksandar Rodić
2   Mihailo Pupin Institute, University of Belgrade, Belgrade, Serbia
José Santos-Victor
3   Institute for Systems and Robotics, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal
› Author Affiliations
Funding: This work is partially funded by the Ministry of Education, Science and Technology Development of the Republic of Serbia under the contracts TR-35003, III-44008, III-44004 and #ON175012. This work was partially funded by the EU Project POETICON++ and the Portuguese FCT Project [UID/EEA/50009/2013]. The work is complementary supported by the Alexander von Humboldt project ..Emotionally Intelligent Robots – EIrobots”, Contract no. 3.4-IP-DEU/112623.
Further Information

Publication History

received: 06 March 2016

accepted: 22 October 2016

Publication Date:
25 January 2018 (online)


Background: Traditional rehabilitation sessions are often a slow, tedious, disempowering and non-motivational process, supported by clinical assessment tools, i.e. evaluation scales that are prone to subjective rating and imprecise interpretation of patient’s performance. Poor patient motivation and insufficient accuracy are thus critical factors that can be improved by new sensing/processing technologies.

Objectives: We aim to develop a portable and affordable system, suitable for home rehabilitation, which combines vision-based and wearable sensors. We introduce a novel approach for examining and characterizing the rehabilitation movements, using quantitative descriptors. We propose new Movement Performance Indicators (MPIs) that are extracted directly from sensor data and quantify the symmetry, velocity, and acceleration of the movement of different body/hand parts, and that can potentially be used by therapists for diagnosis and progress assessment.

Methods: First, a set of rehabilitation exercises is defined, with the supervision of neurologists and therapists for the specific case of Parkinson’s disease. It comprises full-body movements measured with a Kinect device and fine hand movements, acquired with a data glove. Then, the sensor data is used to compute 25 Movement Performance Indicators, to assist the diagnosis and progress monitoring (assessing the disease stage) in Parkinson’s disease. A kinematic hand model is developed for data verification and as an additional resource for extracting supplementary movement information.

Results: Our results show that the proposed Movement Performance Indicators are relevant for the Parkinson’s disease assessment. This is further confirmed by correlation of the proposed indicators with clinical tapping test and UPDRS clinical scale. Classification results showed the potential of these indicators to discriminate between the patients and controls, as well as between the stages that characterize the evolution of the disease.

Conclusions: The proposed sensor system, along with the developed approach for rehabilitation movement analysis have a significant potential to support and advance traditional rehabilitation therapy. The main impact of our work is two-fold: (i) the proposition of an approach for supporting the therapists during the diagnosis and monitoring evaluations by reducing subjectivity and imprecision, and (ii) offering the possibility of the system to be used at home for rehabilitation exercises in between sessions with doctors and therapists.

  • References

  • 1 Jankovic J. Parkinson’s disease: clinical features and diagnosis. J Neurol, Neurosurgery and Psychiatry. 2008; 79 (04) 368-376.
  • 2 Goetz C, Poewe W, Rascol O. et al. Movement Disorder Society Task Force report on the Hoehn and Yahr staging scale: Status and Recommendations. Mov Disord. 2004; 19 (09) 1020-1028.
  • 3 Goetz C, Tilley B, Shaftman S. et al. Movement Disorder Society-Sponsored Revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS): Scale Presentation and Clinimetric Testing Results. Mov Disord. 2008; 23 (15) 2129-2170.
  • 4 Stamford JA, Schmidt PN, Friedl KE. What Engineering Technology Could Do for Quality of Life in Parkinson’s Disease: A Review of Current Needs and Opportunities. IEEE J Biomed Health Inform. 2015; 19 (06) 1862-1872.
  • 5 Zhou H, Hu H. Human motion tracking for rehabilitation – A survey. Biomed Signal Process Control. 2008; 3 (01) 1-18.
  • 6 Parisi F, Ferrari G, Giuberti M. et al. Body-Sensor-Network-Based Kinematic Characterization and Comparative Outlook of UPDRS Scoring in Leg Agility, Sit-to-Stand, and Gait Tasks in Parkinson’s Disease. IEEE J Biomed Health Inform. 2015; 19 (06) 1777-1793.
  • 7 Patel S, Park H, Bonato P. et al. A review of wearable sensors and systems with application in rehabilitation. J Neuroeng Rehabil. 2012; 9: 21.
  • 8 Gonzalez-Jorge H, Riveiro B, Vazquez-Fernandez E. et al. Metrological evaluation of Microsoft Kinect and Asus Xtion sensors. Measurement. 2013; 46 (06) 1800-1806.
  • 9 Gonçalves AR, Gouveia ER, Cameirão MS. et. al. Automating senior fitness testing through gesture detection with depth sensors. IET International Conference on Technologies for Active and Assisted Living (TechAAL). 2015. Nov. 5; London, UK.:
  • 10 Antón D, Goñi A, Illarramendi A. Exercise Recognition for Kinect-based Telerehabilitation. Methods Inf Med. 2015; 54 (02) 145-155.
  • 11 Khoshelham K, Elberink S. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors. 2012; 12 (02) 1437-1454.
  • 12 Clark R, Pu Y, Fortina K. et al. Validity of the Microsoft Kinect for assessment of postural control. Gait and Posture. 2012; 36 (03) 372-377.
  • 13 Chang C, Lange B, Zhang M. et al. Towards Pervasive Physical Rehabilitation Using Microsoft Kinect. 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth). 2012. May 21–24; San Diego, CA.: 159-162.
  • 14 Chang Y, Han W, Tsai Y. A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy. Res Dev Disabil. 2013; 34 (11) 3654-3659.
  • 15 Chang Y, Chen S, Huang J. A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Res Dev Disabil. 2011; 32 (06) 2566-2570.
  • 16 Gama A, Chaves T, Figueiredo L. et al. Guidance and Movement Correction Based on Therapeutics Movements for Motor Rehabilitation Support Systems. 14th Symposium on Virtual and Augmented Reality (SVR); 2012 May 28–31. Rio de Janiero, Brasil.: IEEE; 2012: 191-200.
  • 17 Calin A, Cantea A, Dascalu A. et al. Mira – Upper Limb Rehabilitation System using Microsoft Kinect. Studia Universitatis Babes-Bolyai, Informatica. 2011; 56 (04) 63-74.
  • 18 Galna B, Barry G, Jackson D. et al. Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson’s disease. Gait and Posture. 2014; 39 (04) 1062-1068.
  • 19 Galna B, Jackson D, Schofield G. et al. Retraining function in people with Parkinson’s disease using the Microsoft Kinect: game design and pilot testing. J Neuroeng Rehabil. 2014: 11-60.
  • 20 Albiol-Pérez S, Lozano-Quilis J, Gil-Gómez H. et al. Virtual rehabilitation system for people with Parkinson’s disease. 9th international conference on disability, virtual reality and associated technologies (ICDVRAT). 2012. Sept. 10–12; Laval, France.: 423-426.
  • 21 Spasojevic S, Santos-Victor J, Ilic T. et al. A Vision-based System for Movement Analysis in Medical Applications: the example of Parkinson’s Disease. Nalpantidis L, Krüger V, Eklundh JO, Gasteratos A. Proceedings of the 10th International Conference on Computer Vision Systems – Volume 9163 (ICVS 2015).. New York: Springer; 2015: 424-434.
  • 22 Okuno R, Yokoe M, Akazawa K. et al. Finger Taps Movement Acceleration Measurement System for Quantitative Diagnosis of Parkinson’s disease. 28th Annual International Conference of the Engineering in Medicine and Biology Society (EMBS). 2006. Aug. 30 – Sept. 3; New York, NY.: IEEE; 2006: 6623-6626.
  • 23 Smeja M, Foerster F, Fuchs G. et al. 24-h Assessment of tremor activity and posture in Parkinson’s disease by multi-channel accelerometry. J Psycho-physiol. 1999; 13 (04) 245-256.
  • 24 LeMoyne R, Coroian C, Mastroianni T. Quantification of Parkinson’s disease characteristics using wireless accelerometers. International Conference on Complex Medical Engineering (ICME). 2009. April 9–11; Tempe, AZ.: IEEE; 2009: 1-5.
  • 25 Salarian A, Russmann H, Wider C. et al. Quantification of tremor and bradykinesia in Parkinson’s disease using a novel ambulatory monitoring system. IEEE Trans Biomed Eng. 2007; 54 (02) 313-322.
  • 26 Burkhard PR, Langston JW, Tetrud JW. Voluntarily simulated tremor in normal subjects. Clin Neurophysiol. 2002; 32 (02) 119-126.
  • 27 Kandori A, Yokoe M, Sakoda S. et al. Quantitative magnetic detection of finger movements in patients with Parkinson’s disease. Neurosci Res. 2004; 49 (02) 253-260.
  • 28 Shima K, Tsuji T, Kan E. et al. Measurement and Evaluation of Finger Tapping Movements Using Magnetic Sensors. 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS). 2008 Aug. 20–25; Vancouver, BC.: IEEE; 2008: 5628-5631.
  • 29 Shima K, Tsuji T, Kandori A. et al. Measurement and Evaluation of Finger Tapping Movements Using Log-linearized Gaussian Mixture Networks. Sensors. 2009; 9 (03) 2187-2201.
  • 30 Niazmand K, Tonn K, Kalaras A. et al. Quantitative Evaluation of Parkinson’s Disease using sensor based smart Glove. 24th International Symposium on Computer-Based Medical Systems (CBMS). 2011 June 27–30; Bristol, UK.: IEEE; 2011: 1-8.
  • 31 Prochazka A, Bennett DJ, Stephens MJ. et al. Measurement of Rigidity in Parkinson’s Disease. Mov Disord. 1997; 12 (01) 24-32.
  • 32 Hanson M, Powell H, Frysinger R. et al. Teager Energy Assessment of Tremor Severity in Clinical Application of Wearable Inertial Sensors. IEEE/ NIH Life Science Systems and Applications Workshop (LISA). 2007 Nov. 8–9; Bethesda, MD.: IEEE; 2007: 136-139.
  • 33 Iacono RP, Lonser RR, Maeda G. et al. Chronic Anterior Pallidal Stimulation for Parkinson’s Disease. Acta Neurochir. 1995; 137 1-2 106-112.
  • 34 Su Y, Geng D, Allen CR. et al. Three-Dimensional Motion System (“Data-Gloves”): Application for Parkinson’s disease and Essential Tremor. IEEE International Workshop on Virtual and Intelligent Measurement Systems (VIMS). 2001 May 19–20; Budapest, Hungary.: IEEE; 2001: 28-33.
  • 35 Su Y, Allen CR, Geng D. et al. 3-D Motion System (“Data-Gloves”): Application for Parkinson’s disease. IEEE Trans Instrum Meas. 2003; 52 (03) 662-674.
  • 36 Morrow K, Docan C, Burdea G. et al. Low-cost Virtual Rehabilitation of the Hand for Patients Post-Stroke. International IEEE Workshop on Virtual Rehabilitation 2006. New York, NY.: IEEE; 2006: 6-10.
  • 37 Lange B, Koenig S, McConnell E. et al. Interactive game-based rehabilitation using the Microsoft Kinect. IEEE Virtual Reality Short Papers and Posters (VRW); 2012 March 4–8. Costa Mesa, CA.: IEEE; 2012: 171-172.
  • 38 Cho CW, Chao WH, Lin SH. et al. A vision-based analysis system for gait recognition in patients with Parkinson’s disease. Expert Syst Appl. 2009; 36 (03) 7033-7039.
  • 39 Lum P, Burgar C, Shor P. et al. Robot-Assisted Movement Training Compared With Conventional Therapy Techniques for the Rehabilitation of Upper-Limb Motor Function After Stroke. Arch Phys Med Rehabil. 2002; 83 (07) 952-959.
  • 40 Vaisman L, Dipietro L, Krebs H. A Comparative Analysis of Speed Profile Models for Wrist Pointing Movements. IEEE Trans Neural Syst Rehabil Eng. 2013; 21 (05) 756-766.
  • 41 Plamondon R. A kinematic theory of rapid human movements. Part I. Movement representation and generation. Biol Cybern. 1995; 72 (04) 295-307.
  • 42 Gribble P, Ostry D. Origins of the Power Law Relations Between Movement Velocity and Curvature: Modeling the Effects of Muscle Mechanics and Limb Dynamics. J Neurophysiol. 1996; 76 (05) 2853-2860.
  • 43 Bullock D, Grossberg S. Adaptive neural networks for control of movement trajectories invariant under speed and force rescaling. Hum Mov Sci. 1991; 10 (01) 3-53.
  • 44 Mirkov D, Milanovic S, Ilic D. et al. Symmetry of Discrete and Oscillatory Elbow Movements: Does it Depend on Torque That the Agonist and Antagonist Muscle can Exert?. Motor Control. 2002; 6 (03) 271-281.
  • 45 Available from:
  • 46 Maetzler W, Domingos J, Srulijes K. et al. Quantitative Wearable Sensors for Objective Assessment of Parkinson’s Disease. Mov Disord. 2013; 28 (12) 1628-1637.
  • 47 Keus S, Bloem B, Hendriks E. et al. Evidence-Based Analysis of Physical Therapy in Parkinson’s Disease with Recommendations for Practice and Research. Mov Disord. 2007; 22 (04) 451-460.
  • 48 Pötter-Nerger M, Wenzelburger R, Deuschl G. et al. Impact of Subthalamic Stimulation and Medication on Proximal and Distal Bradykinesia in Parkinson’s Disease. Eur Neurol. 2009; 62 (02) 114-119.
  • 49 Okuno R, Yokoe M, Fukawa K. et al. Measurement system of finger-tapping contact force for quantitative diagnosis of Parkinson’s disease. 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS); 2007 Aug. 22–26. Lyon, France.: IEEE; 2007: 1354-1357.
  • 50 Available from:
  • 51 Spong MW, Hutchinson S, Vidyasagar M. Robot Modeling and Control.. John Wiley & Sons; 2006
  • 52 Field A. Discovering statistics using SPSS.. 3rd ed. London: Sage; 2009
  • 53 Fisher RA. The use of multiple measurements in taxonomic problems. Annals of Eugenics. 1936; 7 (02) 179-188.