Forensic Odontology: Automatic Identification of Persons Comparing Antemortem and Postmortem Panoramic Radiographs Using Computer Vision

Abstract

Purpose In forensic odontology the comparison between antemortem and postmortem panoramic radiographs (PRs) is a reliable method for person identification. The purpose of this study was to improve and automate identification of unknown people by comparison between antemortem and postmortem PR using computer vision.

Materials and Methods The study includes 43 467 PRs from 24 545 patients (46 % females/54 % males). All PRs were filtered and evaluated with Matlab R2014b including the toolboxes image processing and computer vision system. The matching process used the SURF feature to find the corresponding points between two PRs (unknown person and database entry) out of the whole database.

Results From 40 randomly selected persons, 34 persons (85 %) could be reliably identified by corresponding PR matching points between an already existing scan in the database and the most recent PR. The systematic matching yielded a maximum of 259 points for a successful identification between two different PRs of the same person and a maximum of 12 corresponding matching points for other non-identical persons in the database. Hence 12 matching points are the threshold for reliable assignment.

Conclusion Operating with an automatic PR system and computer vision could be a successful and reliable tool for identification purposes. The applied method distinguishes itself by virtue of its fast and reliable identification of persons by PR. This Identification method is suitable even if dental characteristics were removed or added in the past. The system seems to be robust for large amounts of data.

Key Points: 

• Computer vision allows an automated antemortem and postmortem comparison of panoramic radiographs (PRs) for person identification.

• The present method is able to find identical matching partners among huge datasets (big data) in a short computing time.

• The identification method is suitable even if dental characteristics were removed or added.

Citation Format

• Heinrich A, Güttler F, Wendt S et al. Forensic Odontology: Automatic Identification of Persons Comparing Antemortem and Postmortem Panoramic Radiographs Using Computer Vision. Fortschr Röntgenstr 2018; 190: 1152 – 1158

Zusammenfassung

Ziel In der forensischen Odontologie ist der Vergleich von Ante-mortem und Post-mortem Panorama-Röntgenaufnahmen (PR) eine zuverlässige Methode zur Personenidentifizierung. Das Ziel ist die automatische Identifizierung von unbekannten Personen mithilfe eines Vergleichs von Ante-mortem und Post-mortem PR unter Anwendung der Computer Vision.

Material und Methoden Die Studie umfasste 43 467 PR von 24 545 Personen (46 % Frauen/ 54 % Männer). Alle PR wurden mit Matlab R2014b, einschließlich der Toolboxen image processing und computer vision system, gefiltert und ausgewertet. Der Identifizierungsprozess verwendet die SURF-Funktion, um Übereinstimmungspunkte zwischen zwei PR (unbekannte Person und Datenbankeintrag) zu finden.

Ergebnisse Insgesamt konnten 34 von 40 Personen (85 %) aufgrund einer hohen Zahl von Übereinstimmungspunkten zwischen dem aktuellsten PR und einem Datenbankeintrag eindeutig identifiziert werden. Die maximale Zahl an Übereinstimmungen zwischen zwei PR betrug 259 Punkte für dieselbe Person und nur 12 Punkte für zwei verschiedene Individuen. Daher waren 12 Übereinstimmungspunkte die Schwelle für eine eindeutige Identifizierung.

Schlussfolgerung Die Anwendung eines automatischen, Computer-Vision-gestützten PR-Systems kann ein erfolgreiches und zuverlässiges Werkzeug zur Identifikation sein. Die angewandte Methode zeichnet sich durch eine schnelle und zuverlässige Identifizierung von Personen anhand von PR aus. Diese Identifikationsmethode ist für große Datenmengen robust und auch dann erfolgreich, wenn in der Vergangenheit Zahnmerkmale entfernt oder hinzugekommen sind.

Kernaussagen: 

• Zur Personenidentifizierung ist die Computer Vision für einen automatisierten Vergleich zwischen Ante-mortem und Post-mortem Panorama-Röntgenaufnahmen (PR) geeignet.

• Die beschriebene Methode ermöglicht die Identifizierung einer unbekannten Person durch Abgleich mit großen Datensätzen (Big Data) in kurzer Rechenzeit.

• Die Identifizierungsmethode ist auch dann geeignet, wenn Zahnmerkmale entfernt oder hinzugefügt wurden.

Key words

forensic odontology - panoramic radiographs - identification - computer vision - big data - postmortem orthopantomogram

Introduction

In forensic odontology the comparison between antemortem and postmortem panoramic radiographs (PRs) is a reliable method for person identification [1] [2] [3] [4] [5]. Dentition and dentures are individual and relatively resistant to decay. These structures are quite stable with respect to postmortal changes like decomposition, but not trauma. Furthermore, PRs are registered and documented by dental practices and hospitals, which makes it easy for the examiner to access the necessary reference material. However, it is very difficult and extremely time-consuming to find the right reference material if the person’s identity is unknown.

Computer vision can be helpful to automate and advance tasks that the human visual system can do [6]. It concerns the automatic extraction, analysis and understanding of useful information from a single image or a sequence of images [6]. Computer vision is already used to identify persons, e. g. for face recognition and biometrics [7]. Person identification by PR has high potential for reliable and quick analysis processes. This field of science is closely related to applied forensic medicine, in particular to identification purposes in case of mass disaster [4]. Globalization and migration result in an increasing amount of data that cannot be compared manually in an appropriate way. Automated PR systems are able to facilitate large-scale analyses.

There are various publications about methods for person identification using PRs [8] [9] [10] [11] [12] [13] [14] [15] [16]. Most of the currently described methods for identifying individuals from PRs use only small databases (less than 100 persons). For example, Nomir et al. [8] [9] [10], Lin et al. [11] and Zhou et al. [12] present systems for tooth separation, tooth shape description and matching for bite-wing images extracted by PR. They used iterative and/or adaptive thresholding and integral projection for tooth separation. The databases included 93 to 187 antemortem bite-wing images. Ølberg et al. [13] combined methods for the extraction of tooth and dental work contours from PR and used the Hausdorff-distance measure for ranking persons. The database included 67 persons. Jain et al. [14] used a semi-automatic method to extract shapes of the teeth from PRs, and find the affine transform that best fits the shapes. The database included 130 antemortem images. Hofer et al. [15] used a snake algorithm to improve the contours of dental work. The matching was performed with the Levenshtein distance of a dental code with the position and size of the dental work and the distance to neighboring dental work. The database included 68 PRs. The potential of an automatic person identification tool with the help of PRs has not yet been sufficiently studied.

The purpose of this study is to test an automatic person identification tool by comparing antemortem and postmortem PRs using computer vision. A database with multiple entries of dental characteristics from about 24 500 intra-vitam persons was used to test the developed software system.

Materials and Methods

The study includes 43 467 PR datasets of 24 545 persons, acquired between October 2006 and July 2016. The datasets with the RIS service description “panoramic radiograph” were exported from the local hospital PACS. The participants’ age distribution during acquisition was as follows: 5 % < 18 years, 30 % 18 to 35 years, 25 % 36 to 53 years, 28 % 54 to 71 years and 12 % > 71 years. All PRs were filtered and evaluated on a computer (Processor Intel Core i3^®-4160, Intel, Santa Clara, California, USA) with Matlab R2014b (MathWorks^®, Natick, Massachusetts, USA) including the toolboxes image processing and computer vision system.

For dental characteristics extraction, the following image processing steps were performed for all PRs. First, the PRs were scaled to a size of 2440 × 1280 pixels and then cropped to 1728 × 833 pixels to remove the overexposed edges of the image. Afterwards, a 3 × 3 Sobel filter [17] was used for eight directional masks (0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°). Sobel is a neighborhood-based gradient operator and performs a 2 D spatial gradient measurement on an image. For an orientation of 0°, the direction of maximum contrast from black to white runs from left to right on the image and can be extended to include all eight compass directions (rotate it in 45° increments counterclockwise). The largest intensities of all eight direction images were used to create an image with emphasizing edges. Finally, the Speeded Up Robust Features (SURF) algorithm [18] was applied to find blob features. A blob is a region of an image in which properties, e. g. brightness or color, are approximately constant. The length of the SURF feature vector was 64. The descriptors from a region around each point of interest were extracted. The pixels represent and match features specified by a single-point location. The results were saved in a dental characteristic database and assigned to a unique person ID.

The matching process is based on the SURF feature to find unique corresponding points between two PRs (unknown person and database entry) that are rotated and scaled with respect to each other. The function performs a forward-backward match to keep the best result. The number of found matching points is an indicator for identification. Afterwards, the random sample consensus (RANSAC) algorithm [19] was used to exclude outliers. This allows the definition of a minimum necessary number of matching points (threshold), for which an identification is unique with a positive predictive value (PPV) of one.

For evaluation, 40 persons were randomly selected from the database. The selection criteria were based on the following requirements: The selected person had at least one other reference PR in the database, and eight persons per age category (age younger than 18 yrs, 18 ≤ 35 yrs, 36 ≤ 53 yrs, 54 ≤ 71 yrs and older than 71 yrs) were selected. For each person, the most recent PR acquisition was used for the matching process with the database. One selected PR was compared with the remaining 43 466 PRs in the database. In addition, 10 individual datasets without another reference PR were randomly selected (two per age category), to examine the possibility of false-positive results. The signal processing time for each person identification was measured.

Results

The 40 randomly selected persons were between 9 and 83 years old. The average number of matching points was 44.38 ± 51.18 (median 28.07, maximum 259) for images of the same person. For non-identical individuals, the number was significantly lower at 3.73 ± 0.17 matching points (median 3.77, maximum 12). The applied method yields unambiguous identification for 34 out of 40 (85 %) individuals ([Fig. 1]) with a PPV of one and a minimum necessary number of 13 matching points. For the six persons with ambiguous identification, there were 2 (ID 6, 17 years old), 4 (ID 32, 65 years old), 75 (ID 40, 80 years old), 190 (ID 7, 9 years old) or 209 (ID 8, 14 years old) other possible individuals with the same or more matching points (maximum 10 matching points). In these cases the PPV varied between 0.33 and 0.005. Filtering of potential individuals was not possible for ID 16 (compare with [Fig. 1]). In this example, the PR contains only seven teeth without dentures. For all 10 test persons without a reference in the database, the maximum number of matching points was 10 and on average 3.72 ± 0.17. The signal processing lasted 138 ± 30 minutes per test person.

Fig. 1 Maximum number of matching points for two PRs of the same person and different individuals for all 40 randomly selected test persons. The threshold for the minimum number of matching points for unique identification is shown in gray.

Abb. 1 Maximale Zahl der Übereinstimmungspunkte für zwei PR derselben Person und verschiedener Individuen für alle 40 zufällig ausgewählten Testpersonen. Die Mindestzahl von Übereinstimmungspunkten für eine eindeutige Identifizierung ist in grau dargestellt.

[Fig. 2] shows the results for PR comparisons for each of the 40 randomly selected individuals related to time between acquisitions and the number of matching points. The time between acquisitions was on average 2.31 ± 2.14 years for cases with a successful identification. A successful identification was even possible when the time between acquisitions of two PRs was over 8.5 years ([Fig. 3a], compare with [Fig. 2] ID 12) or when parts of the dentures ([Fig. 3b], compare with [Fig. 2] ID 37) or braces ([Fig. 3c], compare with [Fig. 2] ID 13) were removed.

Fig. 2 Overall results for PR comparison for all 40 randomly selected individuals of the five age bands. Relation of time between one PR and the reference PR of the same person is shown. The size of the bubble represents to the number of matching points.

Abb. 2 Gesamtergebnisse für den PR-Vergleich von 40 zufällig ausgewählten Personen der fünf Altersgruppen. Die Zeit zwischen einem PR und Referenz-PR derselben Person ist dargestellt. Die Größe der Blase entspricht der Zahl der gefundenen Übereinstimmungen.

Fig. 3 PR examples of three persons a–c and the related matching points for successful person identification – graphics with timestamp in the left column with red marked characteristics and their identified counterparts with green marked characteristics in the right column.

Abb. 3 PR-Beispiele von drei Personen a–c und die zugehörigen Übereinstimmungspunkte für eine erfolgreiche Personenidentifikation – Grafiken mit Zeitstempel, in der linken Spalte mit rot markierten Merkmalen und deren identifizierte Gegenstücke mit grün markierten Merkmalen in der rechten Spalte.

The application of the developed PR comparison system leads to the following results in case of variable conditions. PR acquisitions with a closed mouth complicate feature extraction ([Fig. 4a], compare with [Fig. 2] ID 40). Likewise, the results show that persons with just a few or no characteristics (dental fillings, implants) do not differ significantly from each other ([Fig. 4b], compare with [Fig. 2] ID 8). Therefore, the detection rate for younger test persons is lower compared to older individuals ([Fig. 2], [Table 1]). Furthermore, the matching process can be complicated when large changes of dentition appear. [Fig. 2] ID 11 (22 years old) shows an example for major PR changes due to a denture after an accident.

Fig. 4 Examples for failed person identification with non-standardized PR acquisition (mouth open/closed, a or containing a small number of characteristics b – graphics with timestamp and colored characteristics.

Abb. 4 Beispiele für eine fehlgeschlagene Personenidentifikation mit nicht standardisierter PR-Akquisition (Mund offen/geschlossen a oder mit einer kleinen Zahl von Merkmalen b – Grafiken mit Zeitstempel und farbigen Merkmalen.

Table 1 Summary of the detection rate for the identification process and explanation of possible influencing variables (n = 40).
Tab. 1 Zusammenfassung der Erkennungsrate für den Identifikationsprozess und Erläuterung möglicher Einflussgrößen (n = 40)

person age (years)	detection rate (count)	reasons for low matching points	reasons for well matching points
< 18	5/8	very little/no dental fillings large changes (tooth growth) between images	oblique/distinctive teeth with little fillings
18 – 35	7/8	only 7 teeth without a filling or implants
36 – 53	8/8
54 – 72	7/8	only 9 teeth with more/larger filling between images	no teeth, only 12 pins with artificial attachment
> 72	7/8	many fillings but mouth open and closed between images
all	34/40	large changes of dentures between images, radiograph overexposed, only one (unfavorable) image in the database	comparable small/large fillings and dentures (braces, implants) between images

Discussion

The experimental setup was designed to identify an unknown person based on their PR using an automatic algorithm system. The proposed tool is able to filter large databases with many entries of possibly matching partners. A threshold for unambiguous identification could be found. In case of more than one positive result (in dependency on the matching points), the small group of potential identical persons can be checked manually to increase identification certainty. Generally, qualified personnel remains essential in forensic odontology for individual assessment of an identity.

The recent literature [8] [9] [10] [11] [12] [13] [14] [15] uses different approaches and algorithms for person identification. The cited studies applied them only to small databases. Their algorithms are possibly insufficient for PR detection within large databases. The mismatched results were mainly due to inaccurate contours caused by poor image quality [8] [9] [10] [11] [12] [13] or because the shapes of the teeth changed due to artificial prosthesis, tooth growth, and tooth extraction [8] [9] [10] [12] [13]. Further reasons were a different viewing angle on bitewing images [12] or a translation offset in the image [13]. In [14] [15] the segmentation result had to be corrected manually, this method is not suitable for large databases. In comparison to the cited references and methods, the combined algorithms in our developed software application show a higher flexibility for the matching process, which works reliably even with large databases. Antemortem and postmortem radiographs of other body regions, e. g. the skull, pelvis and lumbar spine have previously been used for person identification [20] and should also be analyzed with the developed software. This is based on the fact that not just dental characteristics but also principal anatomical information can be used for the matching process.

There was no false-positive result in our study. 12 matching points were the threshold for unique assignment. In detail, the key factor to obtain those results is the combination of robust algorithms. The Sobel-Feldman operator [17], as an edge detection algorithm, calculates a rather inaccurate approximation of the image gradient, but the result is still of sufficient quality for person identification by PR. The SURF algorithm from Herbert Bay [18] allows fast and robust recognition of dental characteristics. The descriptor provides unique and robust description of an image feature. Furthermore, the descriptor is invariant against scaling, rotation, illumination change, image noise, and to a certain extent, perspective distortion. The RANSAC algorithm [19] is a valuable iterative method to estimate parameters of a mathematical model from a set of observed data that contains outliers. The matching process can be successful without using the RANSAC algorithm, but false-positive results are conceivable, because there is no suitable threshold level related to the number of required matching points.

The relatively short computing time of 138 minutes for 43 466 PR comparisons per person will allow the application on larger datasets. The PR matching processing software is not optimized, yet. The signal processing time can be reduced if the algorithms are optimized. For example, the approximate age of a searched person can be preselected. In addition, a faster computer processing unit can reduce the signal processing time drastically.

The main limitation of this basically robust software system is that persons with just a few teeth or no characteristics (dental fillings, implants) do not differ significantly from each other, because tooth shapes are not always sufficient for the matching process. The matching process is based on small image details. Very large changes of dental work can complicate identification, too. Furthermore, the unsuccessful identifications can possibly be a result by the lack of image quality [3] [16]. Dental characteristics could be insufficiently extractable in an overexposed radiograph. In particular, the use of automated PR evaluation requires an open mouth to capture all mandibular and maxillary teeth with good quality. This study contains PRs of a university hospital, which also includes complicated cases. To enhance the database, additional pseudonymous datasets can possibly be included from dental practices and other hospitals.

The acquisition of a postmortem PR requires technical and methodological prerequisites, which are often not given in forensic medicine. Therefore, cooperation between forensic medicine and radiology is recommended. The acquisition process can be simplified with the application of postmortem computed tomography data and a multiplanar reformation (MPR) to create a synthesized PR [16] [21]. However, synthesized MPR images are slightly blurred and show minimal geometric distortion as well as reduced superimposition of oral structures [16] [21]. Furthermore, there are artifacts if metal implants are imaged. These points can disturb the matching process and need to be solved.

In conclusion, this study used a multiple algorithm software tool for person identification based on PRs and yielded robust identification results for individuals, even for cases where dental characteristics changed over time. Quick identification through a large dataset of PRs creates a foundation for further research and development.

No conflict of interest has been declared by the author(s).

• References

• 1 Wood R. Forensic aspects of maxillofacial radiology. Forensic Sci Int 2006; 159: S47-S55
• 2 Fuhrmann A, Schreiner U, Lockemann U. et al. Identifikation unbekannter Toter durch odontologische Untersuchungen. Rechtsmedizin 2001; 11: 37-41
• 3 Du Chesne A, Benthaus S, Teige K. et al. Post-mortem orthopantomography–an aid in screening for identification purposes. Int J Legal Med 2000; 113: 63-69
• 4 Dettmeyer RB, Schütz HF, Verhoff MA. Identifizierung. In: Rechtsmedizin. Berlin, Heidelberg: Springer; 2014: 249-256 . doi:10.1007/978-3-642-55022-5_16
• 5 Kirchhoff S, Fischer F, Lindemaier G. et al. Vergleichende Untersuchungen mit Dental-Computertomographie, Orthopantomographie und visuellem Zahnstatus in der forensischen Odontologie. Fortschr Röntgenstr 2008; 180: VO_211_212 . doi:10.1055/s-2008-1073502
• 6 Sonka M, Hlavac V, Boyle R. Image processing, analysis, and machine vision: Cengage Learning. 2014
• 7 Chen CH. Handbook of pattern recognition and computer vision: World Scientific. 2015
• 8 Nomir O, Abdel-Mottaleb M. A system for human identification from X-ray dental radiographs. Pattern Recognit 2005; 38: 1295-1305
• 9 Nomir O, Abdel-Mottaleb M. Hierarchical contour matching for dental X-ray radiographs. Pattern Recognit 2008; 41: 130-138
• 10 Nomir O, Abdel-Mottaleb M. Fusion of matching algorithms for human identification using dental X-ray radiographs. IEEE Trans Inf Forensics Security 2008; 3: 223-233
• 11 Lin PL, Lai YH, Huang PW. Dental biometrics: Human identification based on teeth and dental works in bitewing radiographs. Pattern Recognit 2012; 45: 934-946
• 12 Zhou J, Abdel-Mottaleb M. A content-based system for human identification based on bitewing dental X-ray images. Pattern Recognit 2005; 38: 2132-2142
• 13 Ølberg JV, Goodwin M. Automated Dental Identification with Lowest Cost Path-Based Teeth and Jaw Separation. Scandinavian Journal of Forensic Science 2016; 22: 44-56
• 14 Jain AK, Chen H. Matching of dental X-ray images for human identification. Pattern Recognit 2004; 37: 1519-1532
• 15 Hofer M, Marana AN. Dental biometrics: human identification based on dental work information. In: Computer Graphics and Image Processing. SIBGRAPI 2007 – XX Brazilian Symposium on: IEEE; 2007: 281-286
• 16 Forrest AS. Collection and recording of radiological information for forensic purposes. Aust Dent J 2012; 57: 24-32
• 17 Burger W, Burge MJ. Digitale Bildverarbeitung: Eine Algorithmische Einführung Mit Java. Springer-Verlag; 2009
• 18 Bay H, Ess A, Tuytelaars T. et al. Speeded-Up Robust Features (SURF). Comput Vis Image Underst 2008; 110: 346-359 . doi:http://dx.doi.org/10.1016/j.cviu.2007.09.014
• 19 Fischler MA, Bolles RC. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 1981; 24: 381-395
• 20 Riepert T, Schweden F, Schild H. et al. Identifizierung unbekannter Leichen durch Röntgenbildvergleich. Fortschr Röntgenstr 1995; 162: 241-245 . doi:10.1055/s-2007-1015873
• 21 Tohnak S, Mehnert A, Mahoney M. et al. Synthesizing dental radiographs for human identification. J Dent Res 2007; 86: 1057-1062