Author: Ester Gonz´ alez Sosa
(Ingeniero de Telecomunicaci´ on, ULPGC)
UNIVERSIDAD AUT ´ONOMA DE MADRID ESCUELA POLIT´ECNICA SUPERIOR
DEPARTAMENTO DE TECNOLOG´IA ELECTR ´ONICA Y DE LAS COMUNICACIONES
Face and Body Biometrics in the Wild:
Advances in the
Visible Spectrum and Beyond
–TESIS DOCTORAL–
BIOMETR´IA FACIAL Y CORPORAL EN ENTORNOS NO CONTROLADOS:
VISIBLE Y OTROS ESPECTROS
A Thesis submitted for the degree of:
Doctor of Philosophy
Madrid, June 2017
Colophon
This book was typeset by the author using LATEX2e. The main body of the text was set using a 11-points Computer Modern Roman font. All graphics and images were included formatted as Encapsuled Postscript (TMAdobe Systems Incorporated). The final postscript output was converted to Portable Document Format (PDF) and printed.
Copyright© 2017 by Ester Gonzalez Sosa. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the author. Universidad Autonoma de Madrid has several rights in order to reproduce and distribute electronically this document.
This Thesis was printed with the financial support from EPS-UAM and the Biometric Data Pattern Analytics Lab (BiDA) Lab.
contact: [email protected]
Department: Tecnolog´ıa Electr´onica y de las Comunicaciones Escuela Polit´ecnica Superior
Universidad Aut´onoma de Madrid (UAM), SPAIN PhD Thesis: Face and Body Biometrics in the Wild: Advances in the
Visible Spectrum and Beyond Author: Ester Gonz´alez Sosa
Ingeniero de Telecomunicaci´on
(Universidad de Las Palmas de Gran Canaria) Advisors: Rub´en Vera Rodr´ıguez
Doctor Ingeniero de Telecomunicaci´on (Swansea University)
Universidad Aut´onoma de Madrid, SPAIN Juli´an Fi´errez Aguilar
Doctor Ingeniero de Telecomunicaci´on (Universidad Polit´ecnica de Madrid) Universidad Aut´onoma de Madrid, SPAIN
Year: 2017
Committee: President: Javier Ortega Garc´ıa
Universidad Aut´onoma de Madrid, SPAIN
Secretary: ´Alvaro Garc´ıa Mart´ın Universidad Aut´onoma de Madrid, SPAIN
Vocal 1: Mark S. Nixon
University of Southampton, UNITED KINGDOM
Vocal 2: Miguel ´Angel Ferrer
Universidad de las Palmas de Gran Canaria, SPAIN
Vocal 3: Hugo Proen¸ca
Universidade da Beira Interior, PORTUGAL
The research described in this Thesis was carried out within the Biometric Data Pattern Analytics
The author was awarded with a PhD scholarship from Universidad Aut´onoma de Madrid between 2013 and 2017 which supported the research summarized in this
Dissertation.
The author was awarded with a COST STSM - Short-Term Scientific Mission from the European project COST 2101 Biometric For Identity Documents and Smart Cards, which supported his research stay carried out at EURECOM, Biot, France
from April 2015 to July 2015.
The author was awarded with a travel and fees grant to the 10th International Summer School on Biometrics provided by the European Commission, IAPR,
MORPHO and the IEEE Biometrics Council in June 2013.
Abstract
T
he evolution of biometrics is an on-going process.Nowadays, biometric systems can achieve satisfactory performance in many applications, especially in controlled scenarios. When it comes to less controlled scenarios, these systems are affected by plenty of different variability sources that greatly hinder their performance. In the particular case of face recognition systems, these variability sources are related to changes in pose, illumination, expression, occlusions, sub- ject to camera distance, low resolution, and so forth.As a way of mitigating the negative impact of these challenging conditions, this Thesis is focused on the search of additional identity clues to enhance the performance of hard biometric systems. In the context of unconstrained scenarios where the only hard biometric trait is the face, we propose the use of facial soft biometrics to improve the performance of face recognition systems. Later, motivated by the fact that faces and bodies are equally salient and available in unconstrained scenarios at a distance, we propose the use of body-based biometrics to alleviate the impact of challenging conditions over face biometrics.
The approach proposed in this Thesis aims to leverage characteristic body static informa- tion from single-shot images. This method is also beneficial when combining body information with face, as they both can be extracted from the same frame, avoiding the need for camera synchronization.
The last motivation for this Thesis is the existence of millimeter waves (mmWs) scanners deployed in international airports scanning the full human body. To date, the only purpose of these scanners has been concealed weapon detection. Motivated by the interesting properties of mmWs related to transparency of clothes, this PhD Thesis aims to shed some light into the utility of shape and texture information extracted from millimeter images for person recognition purposes.
This Dissertation in Part I first summarizes and classifies the most relevant works related to the Thesis. The experimental chapters are then divided into two parts, the first being related to biometric systems developed in the visible spectrum while the latter part is concerned to biometric systems deployed beyond the visible spectrum, concretely in the millimeter wave range.
The first experimental part (Part II of the Dissertation) starts giving some insights regard- ing face recognition systems under challenging conditions. In the first place, we analyze face recognition systems under occlusions using a facial region-based approach, with the idea of using information exclusively based on non-occluded facial regions. Thereafter, the problem of face recognition under surveillance scenarios is investigated within the International Conference on Biometrics 2016 - Face Recognition in the Wild Competition using the QUIS-CAMPI dataset.
This benchmark truly encompasses all the singularities of surveillance scenarios including pose, illumination, occlusions, and expression, among others.
ranging from far scenarios with full body exposure to close scenarios in which only the head and upper torso are visible. The utility of multimodal approaches combining face and body-based biometrics are also assessed.
In the second experimental part (Part III of the Dissertation), we proceed to study the potential use of millimeter images for person recognition, given the fact that mmW scanners already acquire the full body signature. In the first case, we study whether shape information are discriminative enough for person recognition. This is motivated by the fact that mmW waves have the property to pass through clothes, and therefore shape information extracted from mmW images may be more robust to clothing variations than visible images. To this aim, we analyze different shape-based feature approaches.
For the same transparency property, mmW imaging can also be exploited to extract texture from the human body. Thereby, we also study whether mmW texture information is convenient for person recognition. Concretely, we conduct experiments based on hand-crafted and deep learning with different mmW body parts, namely: face, torso and whole body. Different multi- modal and multi-algorithmic fusion schemes are investigated, including the fusion of shape and texture information.
The research work described in this Dissertation has led to novel contributions which include:
i) state-of-the-art results in a challenging and unconstrained face recognition competition, ii) empirical evidence of performance improvement while fusing soft biometrics with face biometrics in unconstrained scenarios, iii) the proposal of multimodal approaches combining facial and body information that surpasses the individual performance attained with face in two concrete applications: gender estimation and person recognition, iv) assessment of body-shape feature approaches to perform person recognition using millimeter images, and v) evaluation of texture- based information from millimeter images for person recognition. Besides, the research work completed throughout the Thesis includes the generation of various literature reviews and the generation of new biometric resources.
A la memoria de mis seres queridos.
Lord, Im not praying for miracles and visions, I’m only asking for strength for my days.
Teach me the art of small steps.
Make me clever and resourceful, so that I can find important discoveries and experiences among the diversity of days.
Help me use my time better.
Present me with the sense to be able to judge whether something is important or not.
I pray for the power of discipline and moderation, not only to run throughout my life, but also to live my days reasonably, and observe unexpected pleasures and heights.
Save me from the naive belief that everything in life has to go smoothly.
Give me the sober recognition that difficulties, failures, fiascos, and setbacks are given to us by life itself to make us grow and mature.
Send me the right person at the right moment, who will have enough courage and love to utter the truth!
I know that many problems solve themselves, so please teach me patience.
You know how much we need friendship.
Make me worthy of this nicest, hardest, riskiest and most fragile gift of life.
Give me enough imagination to be able to share with someone a little bit of warmth, in the right place, at the right time, with words or with silence.
Spare me the fear of missing out on life.
Do not give me the things I desire, but the things I need.
Teach me the art of small steps!
Antoine de Saint-Exupery
Acknowledgements
“If You Want To Go Fast, Go Alone. If You Want To Go Far, Go Together”
(African Proverb)
Even though my name appears on the front page of this Dissertation, the merit of this work is far from being entirely mine. I would like to thank to my co-advisors Dr. Ruben Vera and Prof. Julian Fierrez, both of whom played a key role in the development of this thesis. Dr. Vera and Prof. Fierrez have managed to guide me smartly throughout my PhD studies for almost five years. To Ruben for his frankness, willingness to listen and your daily compromises with regard to my work. Also for calming me down, downplaying at certain times and helping me to make proper decisions. To Julian: for betting on me to have the best possible outcomes, providing me with flexibility and a great working environment, inspiring me with you expertise in the field and rigor sense, and especially for paving the way for my various research internships.
I would also like to express my gratitute to Prof. Javier Ortega-Garcia, for giving me the opportunity to join the BiDA-Lab, back in September 2012. I would not be writing these acknowledgements if it were not for him selecting me for this prestigious PhD opportunity. I would also like to recognize the confidence that he placed on my shoulders for certain tasks, which enabled me discover other parts of my professional life as well as some aspects from myself.
It would be unfair not to mention Prof. Miguel Angel Ferrer Ballester, my MSc Thesis co-advisor, with whom I had the opportunity to embark on my research during the late years of my MSc. studies, I am sure that my leap from Gran Canaria to Madrid would have not been so easy, if he would have not had faith in me, e.g. encouraging me to write research papers even though I was not sure what a research paper was about. I really appreciate your altruistic support in me throughout my PhD, assisting me each time I return home and especially during the final stages of the Thesis. I would also like to mention Dr. Ivan Perez and Dr. Felix Tobajas, from my undergraduate studies, for their valuable tips, which motivated me to pursue a scientific career.
The journey of my PhD degree has been accompanied by many excellent professionals and colleagues from the ATVS (now BiDA and AUDIAS), who have contributed in such a way that my stay at UAM had a special flair. Thanks again to Dr. Ruben Vera, this time for being a close friend to me, teaching me so many things and also for sharing the good moments outside the lab. Dr. Aythami Morales, my former MSc Thesis co-advisor and ATVS colleague in Madrid, thanks for always being there, willing to help and to listen to and for the unexpected surprises of Irene and Diana, who made working days all the more pleasant. Ruben Tolosana, first student, then PhD-brother, thanks for your support, your positivity and the serious-funny conversations shared during the late evenings in the lab. To Alejandro Acien (I must admit your sun clock deserves a higher grade) and Javier Hernandez-Ortega (nice working experience, the fruits are still pending to come).
Following their dreams outside UAM there are also some former ATVS members to whom I am also grateful: Dr. Pedro Tome (for your warm welcome during my early days at ATVS and for your scientific support during my beginnings), Dr. Marta Gomez (for introducing me to couscous and passing on me your your enthusiasm for travelling), Dr. Ram Krish (the first Indian person that I ever met, glad to have met you and shared with you meaningful conversations with you); Dr. Ruifang Wang (cheerful Chinese lady), and Javier Eslava (ICB-2013 hard-working mate). My thanks also to Miriam Moreno (first mmW
the rest of URSI-2016 crew (especially Dr. Marcos Escudero and Pablo Sanchez), also to all the students that have given me the opportunity to teach and so find one of my roles in life, especially Beatriz Cid Fernandez and Pablo Vicente and lastly but not least, to Manual Vazquez for your important assistance during my first teaching experience.
If I had to highlight the best part of doing a PhD, it would be without a doubt, the opportunity to have visited foreign institutions or to have attended international conferences all over the world, which made me realize how small I am and how many things in this world that are yet to be explored. The opportunity to live abroad, discovering new working environments, and especially to meet new people and places, has enabled me to easily forget the sour moments during my PhD.
The first research internship took place back in 2015 in EURECOM (France) with Dr. Jean-Luc Dugelay. I wish to thank him for allowing me to work under his supervision for more than three months, while making me feel at home. I would also like to especially mention Dr. Antitza Dantcheva and my luck of working with her and learning for her experience. Thanks Antitza for all your support and encouragement from the very moment that we first met each other. I would also like to take this time to remember all the colleagues and friends that made my stay in Cote d’Azur truly unforgettable. The coffee breaks, volleyball matches, tips and parties shared with Valeria, Chiara, Natacha, Grigori, Neslihan, Rajeev, Jonas, Leela and the “The Fantastic Three”: Jose, Giovanni (Zipi and Zape), and Robin (South African-Indian gentleman). Jose, the Spanish guy in love with Lanzarote that started helping me even before knowing me, you know that I truly appreciate you. Giovanni, “bueno bueno”, the craziest and funniest Italian guy I have ever met (Les Etoiles filantes, “calma calma”, PARTYYYY). Robin (Thomas), it was a pleasure to find out that we have more in common than we first had thought and to share some exciting sightseeing moments at Cote d’Azur and Madrid together (also best English proof reader).
My second research stay was a short one and took place at TNO, in February 2016, in the Netherlands.
Overall, I would like to thank TNO for providing me temporary access to valuable data for my research.
To Richard Den Hollander, in particular, for helping me to feel comfortable despite me being the only non-Dutch person working there. This stay allowed me to spend more time with my friends Maria Theresa and Paolo, to whom I felt in debt for being close to me during those weeks.
My last research stay would have not occurred if it had not been for my mentor at the International Mentoring Program from the IMFAHE Foundation: Joaquin Lopez-Herraiz. You gave me the last push that I needed to do my last research internship in USA, apart from providing me with some great advice.
Thanks to him I flew “across the pond” to Rutgers University (New Jersey), under the supervision of Prof. Vishal M. Patel. Despite being extremely busy, he always found time to reassure everything was fine, either face-to-face or remotely. Certainly this stay was to help my PhD. Thanks for supporting me in all senses and encouraging me to do my best in each and every stage. This adventure also allowed
me to meet fantastic people from all over the world: Mahdi (many thanks for everything) and Kazem (Iranians), Pramudita (Sri Lanka, nice philosophical chats), Vishwa (my pseudo supervisor), Atul (who explained Sufi to me), Parneet and the chinese men: Xing (who was very funny and helpful), He (clever man), Lidan (I thank God you were there), Pooyang and Shaogang. Each one of you really made a difference during my research stay. Funny moments (some of them very rainy) and meals, both at the lab and outside. I also have good memories from the moments shared at home with Takashi (Now I am in love with Joe Hisaishi’s music, 2016 USA Elections evening, I have learnt so much from you) Suichi (cute Japanese) and Gonzalo (Long life to Azores). I would also like to thank my European friends, who were always ready to conquer the East Coast: Max, Hendrik, Anna, and Caroline. Lastly, Familia Velez (bendicion!) who deserves my immense gratitude for hosting me as a member of their family and providing me with all that I needed (Rebe). I now finally know what a Thanksgiving dinner looks like.
Time to Spanish!
A todas mis compis de piso: Isa Mar´ın, Mar´ıa del Mar, Luc´ıa, Amparo, Federica, Glenny y en especial a mis casi hermanas Teresa (y Kike) y Mariana, ambas un regalo inesperado que me estaba esperando en Madrid. A Mar´ıa Romero, por tu capacidad de escucha, comprensi´on, de dar ´animos y por ser una tierra firme donde pisar durante todo este tiempo y a todos los niveles. Gracias a Mar´ıa Diaz, ´Angela, Almu, Marta y Tere Sancho, por ayudarme a ver siempre la otra cara de la moneda y ser la compa˜n´ıa de muchos momentos divertidos. A Gema, ´unica en su especie, un placer tenerte en mi vida. A mi prima Raquel, por contribuir a echar un poco de menos a la familia. A mis compa˜neras de fatiga: Susana y Lourdes, qu´e bien sienta ser comprendida. Muy en especial gracias a cada una de las personas de la parroquia de San Jos´e que me han ayudado a relativizar mis altibajos (porque desde puntos de vista m´as elevados, los obst´aculos parecen m´as pequenos) y ense˜narme a disfrutar de cada d´ıa. Gracias a Marina y a tus padres por pensar en m´ı. Gracias Ancora (en especial Fran, Marta y Encarna) por ense˜narme que al final siempre sale el sol. A la OSPAL, y en especial a Lorena y a Cristina, por permitirme ser parte de la magia de la m´usica.
A mis telecos-canarios, la mayor´ıa de ellos afincados en Madrid: Cris (and Cream, siempre al tanto de m´ı), Jes´us (intruso), Dani y Raquel (¿Para cu´ando nos aventuramos en un tren?), Omar (No me cansar´e de repetir de que deber´ıas estar trabajando en Google, gracias por tu aporte en esta tesis), Cassandra (la mejor Teleco que queda por Gran Canaria) e Himar (¿Pensabas que me iba a olvidar de ti?).
Voy terminado. Mis amigos de siempre. Isra y Laura (y Mateo y Mar´ıa), Miriam (y Jes´us), David (y D´ebora), Carmen, Alis (e Irene), Pablo Timple, El´ıas y Luc´ıa, Noem´ı, Miguel, Pablo RH, Jes´us RH, Priscila, Dani y familia Campos-Garc´ıa. A pesar de que ya no pasamos tanto tiempo juntos, el lugar que ocupa cada uno en mi coraz´on es irremplazable. Gracias Omy por tu comprensi´on y tu paciencia conmigo y muy en especial a Dr. Mois´es D´ıaz (alg´un d´ıa profesor) por escucharme, entenderme, apoyarme y animarme en mis d´ıas m´as grises, y por creer en m´ı a veces m´as que yo (no te perdonar´e nunca las cucas de Canc´un). Gracias tambi´en al resto de personas especiales que siguen disfrutando de la vida por Ba˜naderos y sus alrededores, que siempre me reciben con una sonrisa, y de las que no merezo su atenci´on (Chencha, Estrella, Jose y Tere, Cheli, etc.). De la provincia chicharrera no puedo olvidarme de Misael, familia Gonz´alez-Mart´ın y de Sara (tu letras, yo n´umeros, pero a´un as´ı nos entendemos a la perfecci´on).
Por ´ultimo, quiero agradecer en especial a toda mi familia todo el apoyo que me han mostrado. A mis abuelos Mar´ıa Dolores, Santiago y Teresita, muy en especial, por alegrarme mis ma˜nanas diariamente, por escucharme y hacer el esfuerzo de entender porqu´e una chica de 28 a˜nos todav´ıa sigue estudiando y va al “colegio”. A todos mis t´ıos y primos, que me demuestran que 2000 km no son suficientes para olvidarse de m´ı. Gracias Marisa por tu apoyo cercano y tu ejemplo de servicio. Gracias Chago por tus escapadas varias a Madrid (que p´erdida!) haciendo turismo gastron´omico, y a Fede y Merci (turismo
todo lo que soy. Por no cortar mis alas aunque eso signifique estar lejos. Gracias pap´a por ser mi primer mentor, siempre un paso por delante m´ıa, encestando siempre con las palabra justas y que de verdad que yo necesitaba escuchar. Gracias mam´a por tu ejemplo de eficacia y organizaci´on, por gastar tu tiempo en que yo aprovechara mejor el m´ıo, por hacer el esfuerzo de entenderme (aunque para ti la carrera m´as bonita del mundo sea magisterio infantil), por tu paciencia y muy en especial por tu apoyo en los ´ultimos momentos de la tesis (“El sol siempre brilla encima de las nubes”). Les quiero.
And the best . . . is yet to come!
Ester
Mayo 2017
Glossary
AUC: Area Under the Curve.
BBGE: Body based Gender Estimation.
BDI: Body Dynamic Information.
BSI: Body Static Information.
CCA: Canonical Correlation Analysis.
CC: Contour Coordinates.
CCTV: Close-circuit Television.
CMC: Cumulative Match Curve.
CNN: Convolutional Neural Networks.
COTS: Comercial off The Shelf System.
CWD: Concealed Weapon Detection.
DCT: Discrete Cosine Transform.
DTW: Dyamic Time Warping.
ED: Euclidean Distance.
EER: Equal Error Rate.
FAR: False Acceptance Rate.
FBGE: Fase based Gender Estimation.
FD: Fourier Descriptors.
FRR: False Rejection Rate.
FTA: Failure To Acquire.
GEI: Gait Energy Image.
GEV: Gait Energy Volume.
GBU: The Good, Bad, and Ugly database.
GROUPS: The Images of Groups database.
HD: Hamming Distance.
LGPHS: Local Gabor Binary Pattern Histogram Sequence.
LDA: Linear Discriminant Analysis.
LFW: Labeled Faces in the Wild database.
LWIR: Long Wave Infrared.
MHD: Modified Hausdorff Distance.
mmW: Millimeter Waves.
MWIR: Medium Wave Infrared.
NIR: Near Infrared.
NN: Neural Network.
PaSC: Point and Shoot Challenge.
PCA: Principal Component Analysis.
PETA: PEdesTrian Attribute database.
PTZ: Pan-Tilt-Zoom.
RCP: Row and Column Profiles.
ROC: Receiver Operating Curve.
SIFT: Scale Invariant Feature Transform.
SC: Shape Contexts descriptor.
SFFS: Sequential Floating Forward Selection.
SVM: Support Vector Machines.
smW: Submillimeter Waves.
SWIR: Short Wave Infrared.
TNR: True Negative Rate.
TPR: True Positive Rate.
VIS: Visible Spectrum.
VLWIR: Very Long Wave Infrared.
XR: X-ray.
Contents
Abstract IX
Acknowledgements XIII
Glossary XVII
List of Figures XXIII
List of Tables XXVI
I Problem Statement and Contributions 1
1. Introduction 3
1.1. Biometric Systems . . . 4
1.1.1. Modalities and Applications of Biometric Systems . . . 4
1.2. Challenges in Biometric Recognition . . . 6
1.3. Body Information . . . 10
1.4. Motivation of the Thesis . . . 11
1.5. The Thesis and Main Contributions . . . 12
1.6. Outline of the Dissertation . . . 13
1.7. Detailed Research Contributions . . . 14
2. Related Works 19 2.1. Face Recognition in the Wild . . . 19
2.1.1. Face Recognition under Occlusions . . . 20
2.1.2. Face Recognition in Surveillance Scenarios . . . 23
2.2. Soft Biometrics . . . 25
2.2.1. Soft Biometrics for Person Recognition . . . 27
2.3. Gender Estimation . . . 30
2.3.1. Gender from Face . . . 31
2.3.2. Gender from Body . . . 32
2.3.3. Multimodal Approaches for Gender Estimation . . . 34
2.7. Chapter Summary and Conclusions . . . 46
3. Performance Evaluation of Biometric Systems 47 3.1. Performance of Biometric Systems . . . 47
3.2. Databases . . . 50
3.2.1. Face Databases . . . 50
3.2.2. Full Body Databases . . . 52
3.2.3. Millimeter Databases . . . 55
3.3. Chapter Summary and Conclusions . . . 58
II Visible Spectrum 61 4. Face Recognition 63 4.1. Face Recognition under Occlusion . . . 63
4.1.1. System Description . . . 64
4.1.2. Experimental Protocol . . . 66
4.1.3. Results . . . 67
4.2. Face Recognition in the Wild . . . 70
4.2.1. Hand-crafted Approach . . . 71
4.2.2. Deep Learning Approach . . . 73
4.2.3. Experimental Protocol . . . 75
4.2.4. Results . . . 75
4.3. Chapter Summary and Conclusions . . . 80
5. Facial Soft Biometrics for Person Recognition 83 5.1. Manual Annotation of Soft Biometrics . . . 83
5.2. Automatic Estimation of Soft Biometrics . . . 86
5.2.1. Performance of Automatic Extraction of Soft Biometrics . . . 86
5.3. Biometric Systems . . . 87
5.3.1. Soft Biometrics Verification System . . . 87
5.3.2. Face Verification Systems . . . 88
5.4. Experimental Protocol . . . 90
5.5. Results . . . 90
CONTENTS
5.5.1. Bag of Soft biometrics . . . 90
5.5.2. Combining Hard and Soft Biometrics . . . 92
5.6. Chapter Summary and Conclusions . . . 97
6. Body Information for Biometrics 99 6.1. Gender through Body Information . . . 99
6.1.1. Face-based Gender Estimation Algorithm . . . 100
6.1.2. Body-based Gender Estimation Algorithm . . . 101
6.1.3. Fusion . . . 104
6.1.4. Database and Experimental Protocol . . . 104
6.1.5. Results . . . 107
6.2. Person Recognition through Body Information . . . 110
6.2.1. Face-based Person Recognition System . . . 111
6.2.2. Body-based Person Recognition System . . . 111
6.2.3. Fusion . . . 113
6.2.4. Database and Experimental Protocol . . . 113
6.2.5. Results . . . 114
6.3. Chapter Summary and Conclusions . . . 117
III Beyond the Visible Spectrum 121 7. mmW Person Recognition through Shape 123 7.1. Shape-based Biometric Person Recognition . . . 123
7.2. System Description . . . 124
7.2.1. Preprocessing . . . 124
7.2.2. Body Shape-based Descriptors . . . 126
7.2.3. Matching . . . 128
7.3. Experimental Protocols . . . 129
7.3.1. BioGiga . . . 129
7.3.2. mmW TNO . . . 130
7.4. Results . . . 132
7.4.1. Results with BioGiga . . . 132
7.4.2. Results with mmW TNO . . . 135
7.5. Chapter Summary and Conclusions . . . 144
8. mmW Person Recognition through Texture 147 8.1. System Description . . . 147
8.1.1. Preprocessing: mmW Body Parts . . . 147
8.1.2. Feature Approaches . . . 147
8.1.3. Matching . . . 152
8.4. Discussion: The Future of mmW Biometrics . . . 163 8.4.1. Active mmW Imaging for Person Recognition . . . 164 8.4.2. Legal Issues with Body Scanners . . . 164 8.5. Chapter Summary and Conclusions . . . 165
IV Conclusions 167
9. Conclusions and Future Work 169
9.1. Conclusions . . . 170 9.2. Future Work . . . 173
A. Resumen Extendido de la Tesis 175
A.1. Resumen . . . 175 A.2. Conclusiones . . . 177 A.3. L´ıneas de Trabajo Futuro . . . 181
List of Figures
1.1. Biometric recognition in the wild. . . 7 1.2. Body Dynamic Information and Body Static Information. . . 10 1.3. Dependence among chapters. . . 15 2.1. Disguised examples from the LFW database. . . 21 2.2. Examples of London riots and Boston marathon forensic scenarios. . . 21 2.3. Failed recognition cases of deep learning approaches under occlusions. . . 22 2.4. Experimental framework of the International Conference on Biometrics 2016 -
Face Recognition in the Wild Competition, extracted from [Neves and Proen¸ca, 2016]. . . 23 2.5. Examples from the QUIS-CAMPI dataset. . . 24 2.6. Anthropometric method of Alphonse Bertillon. . . 26 2.7. Main applications of soft biometrics for recognition. . . 28 2.8. Ranges of the electromagnetic spectrum. . . 38 2.9. Benefits of millimeter waves. . . 41 2.10. Fog visibility with millimeter waves. . . 42 2.11. Transfer learning. . . 43 2.12. Alexnet and VGG-face convolutional neural networks architectures. . . 46 3.1. Diagram of the two processes involved in a biometric recognition system: enroll-
ment and recognition. . . 48 3.2. Performance evaluation of biometric systems. . . 49 3.3. ARFace database. . . 50 3.4. Illustrative examples from the Labeled Faces in the Wild dataset. . . 51 3.5. QUIS-CAMPI dataset. . . 51 3.6. The Multibiometric Tunnel Database: subset of male and female images corre-
sponding to the far distance-setting analyzed . . . 53 3.7. Three representative distance scenarios from The Multibiometric Tunnel Database 53 3.8. Illustrative examples from the MIT Pedestrian database. . . 54 3.9. BioGiga database. . . 56 3.10. The mmW TNO database. . . 57
4.6. Procedure to obtain the final similarity score between a test image and all training images from a particular subject following the LBP facial region approach. . . . 72 4.7. Feature map activations from the first convolutional layer of the fine-tuned CNN
architecture of mug-shot and probe images for two different subjects. . . 74 4.8. Proposed systems for the ICB-RW competition. . . 76 4.9. Frontalization examples. . . 78 4.10. Rank-1 examples. . . 79 4.11. Rank-2 examples. . . 79 4.12. Occlusion assessment. . . 80 5.1. Correlation of soft biometrics. . . 85 5.2. Verification system based on a bag of soft biometrics. . . 91 5.3. Relative improvement of soft biometrics over hard-biometrics. . . 94 5.4. Qualitative results of the fusion between manual soft biometrics and face biometrics. 96 6.1. General diagram of the multimodal approach proposed for gender estimation using
face and body static information. . . 100 6.2. Body shape-based gender estimation algorithm. . . 102 6.3. Distances settings employed for the evaluation of the multimodal gender estima-
tion approach based on face and body static information. . . 105 6.4. Overall accuracy of hand-crafted features (left) and deep learning (right) for all
distance scenarios using the Multi-Biometric Tunnel database. . . 106 6.5. Multimodal person verification system. . . 110 6.6. VR in function of sum weights. . . 116 7.1. Shape information for biometrics. . . 124 7.2. Contour extraction of BioGiga images. . . 125 7.3. Contour Coordinates. . . 126 7.4. Fourier descriptors. . . 126 7.5. Shape contexts descriptor. . . 127 7.6. Row and column profiles. . . 128 7.7. Active mmW image. . . 131 7.8. Verification results for person recognition using shape information from BioGiga. 132 7.9. Verification results following the protocol 3:1 and the BioGiga database. . . 134
LIST OF FIGURES
7.10. Computational time. . . 135 7.11. Influence of contour resolution with BioGiga. . . 136 7.12. Performance of Shape Contexts with respect to the number of angular bins, radial
bins and its dimensionality. . . 137 7.13. Impact of feature dimensionality. . . 137 7.14. Verification results following the frontal protocol and the mmW TNO database. . 139 7.15. Verification results following the cross-pose protocol and the mmW TNO database.139 7.16. Recognition errors. . . 140 7.17. Identification results following the frontal protocol and the mmW TNO database. 142 7.18. Identification results following the cross-pose protocol and the mmW TNO database.142 7.19. Automatic segmentation. . . 144 8.1. Millimeter body parts. . . 148 8.2. Hand-crafted features. . . 150 8.3. Learned features. . . 151 8.4. CNN level fusion. . . 154 8.5. ROC curves drawn for mmW face, mmW torso and mmW whole body. . . 156 8.6. CMC curves drawn for mmW face, mmW torso and mmW whole body. . . 158 8.7. Can texture approaches complement each other to boost recognition? . . . 159
List of Tables
1.1. Example applications of biometrics. . . 6 2.1. Public available datasets focused on facial occlusions. . . 20 2.2. Taxonomy of soft biometrics. . . 26 2.3. Overview of related works of gender estimation based on body static information. 32 2.4. Overview of related works of gender estimation based on multimodal approaches. 34 2.5. Regions beyond the visible spectrum commonly used for biometric applications. . 39 2.6. Alexnet and VGG-face configurations. . . 45 3.1. Dimensionality statistics of face images from the QUIS-CAMPI dataset. . . 52 3.2. Millimeter wave databases used for person recognition purposes. . . 58 4.1. Development and evaluation sets from the ARFace database. . . 67 4.2. Performance of individual regions and fusion schemes with Face++ and Multiscale
LBP system. . . 67 4.3. Performance of Face++, Multiscale LBP and LBP systems. . . 69 4.4. Facial regions discriminability. . . 77 4.5. Results from the ICB-RW competition, with 9 participants. . . 81 5.1. Soft biometrics extracted from the LFW database. . . 84 5.2. Facial attributes extracted from the LFW database. . . 84 5.3. Estimated soft biometrics. . . 86 5.4. Automatic extraction of soft biometrics. . . 88 5.5. Evaluation of age accuracy. . . 89 5.6. Person verification with bag of soft biometrics using manual labels. . . 91 5.7. Fusion of manual soft biometrics with hard biometrics. . . 93 5.8. Fusion of estimated soft biometrics with hard biometrics. . . 97 6.1. Performance of the Viola-and-Jones face detection algorithm for the Multi-Biometric
database. . . 104 6.2. Hand-crafted approach for all distance scenarios using the Multi-Biometric Tunnel
database. . . 107
7.2. Impact of clutter. . . 141 7.3. Impact of automatic segmentation. . . 144 7.4. Performance of body regions. . . 145 8.1. Impact of pose variations over mmW torsos for verification and identification modes.157 8.2. Multi-algorithm fusion in identification mode. . . 160 8.3. Multi-algorithm fusion in verification. . . 160 8.4. Multimodal fusion schemes: identification. . . 161 8.5. Multimodal fusion schemes: verification. . . 161 8.6. Shape and texture fusion for verification. . . 163 8.7. Shape and texture fusion for identification. . . 164
Part I
Problem Statement and
Contributions
Chapter 1
Introduction
I
s Biometrics a mature technologyable to provide robust solutions to all scenarios? Cer- tainly not. When good quality samples are acquired from cooperative users under controlled conditions, the task of person recognition in terms of performance can be considered as a solved problem from the current state-of-the-art biometric systems. However, there are still a number of challenging issues to deal with in many scenarios and applications. This PhD Thesis aims to alleviate the impact of all these nuisance conditions by the exploration of: i) soft or body-based biometric information, and ii) ranges of the electromagnetic spectrum beyond the visible, such as millimeter, to enhance person recognition in particular applications.Biometric recognition, or simply biometrics is a technological area, whose aim is to discrim- inate automatically between subjects in a reliable way and according to some target application based on one or more signals derived from physical or behavioral traits, such as face, fingerprint, iris, voice, hand, signature, etc. [Jain et al., 2016] These personal traits are commonly denoted as biometric modalities or also as biometrics. Nowadays, due to the expansion of the networked society, the ability to reliably and automatically identify individuals in real-time is a fundamen- tal requirement in many applications. Establishing the identity of individuals is recognized as fundamental not only in numerous governmental, legal or forensic operations, but also in a large number of civilian applications such as financial transactions and computer security.
The difficulties associated with person identification and individualization were already high- lighted by the pioneers of forensic sciences. Alphonse Bertillon developed in the eighteenth century an anthropometric identification approach, based on the measure of physical charac- teristics of a person [Bertillon, 1893]. Automatic person authentication has been a subject of study for more than forty years [Atal, 1976; Kanade, 1973], although it has not been until the last decade when biometric research has been established as an specific research area. This is evidenced by recent reference texts [Jain et al., 2016, 2011b; Ratha and Govindaraju, 2008; Ross et al., 2006; Tistareli et al., 2009], specific conferences [Boult et al., 2014; Fierrez et al., 2013;
Hoque et al., 2017; Singh et al., 2016], common benchmark tools and evaluations [Beveridge et al., 2013b; Kemelmacher-Shlizerman et al., 2016; Neves and Proen¸ca, 2016; Phillips et al., 2000; Phillips, 2006; Phillips et al., 2011, 2009a,b; Przybocki and Martin, 2004; Yeung et al.,
plications and biometric traits. We also outline the main factors biometric systems in the wild need to cope with, and introduce the idea of body static information as additional information potentially useful for different biometric applications. We finish the chapter by stating the The- sis, giving an outline of the Dissertation, and summarizing the research contributions originated from this work.
Although no special background is required for this chapter, the reader will benefit from introductory readings in biometrics such as [Jain et al., 2008, 2006, 2004c, 2011b]. A deeper reference is [Jain et al., 2016].
1.1. Biometric Systems
Traditionally, a user could be identified through something known only by the user, such as a password, or something owned exclusively by him/her, for instance: a card. The main inconvenience of these methods relies on the high facility of password/card missapropiation, the need to remember multiple passwords and maintain multiple authentication tokens. A biometric system aims to establish an identity based on who you are or what you produce, rather than by what you possess or what you know. Biometric systems can be regarded as a specific application of pattern recognition. Biometrics may provide efficient and reliable solutions to recognize individuals. First research on Biometrics date back more than forty years with popular automatic person authentication works by [Atal, 1976] and [Kanade, 1973]. The development of automated biometric systems based on traits such as voice [Pruzansky, 1963], face [Bledsoe and Chan, 1965], and signature [Mauceri, 1965] can be traced back even to the 1960s.
1.1.1. Modalities and Applications of Biometric Systems
Biometric traits can be classified into physiological or behavioral. Physiological biometrics include those traits describing what the person is such as face, fingerprint, iris, hand geom- etry, palmprint, ear, retina, sclera, periocular region, vascular patterns, DNA, electrocardio- graph (ECG) and electroencephalograph (EEG). Conversely, Behavioral biometrics incorpo- rate information regarding what the person does or the way humans behave such as the way they talk (speech); sign/write (signature/handwriting); walk (gait or footsteps); type keyboards (keystroking) or use the mouse (mouse dynamics), among others.
In theory, any human characteristic can be used as a biometric identifier as long as it satisfies these general requirements:
1.1 Biometric Systems
Universality: which indicates to what extent a biometric is present in the world popula- tion.
Uniqueness or Distinctiveness: which means that the trait has to be unique for every single person, or at least discriminative enough to distinguish between subjects.
Permanence: which entails that the biometric trait should have a compact representation permanent or invariant over a sufficiently large period of time.
Collectability: which refers this biometric trait has to be easily measured quantitatively.
Apart from these aforementioned requirements, the following practical criteria are also de- sired:
Performance: which involves the efficiency, accuracy, speed, robustness and resource requirements of particular implementations based on the biometric trait.
Acceptability: which refers to whether people are willing to use the biometric trait for authentication purposes and under which conditions.
Circumvention: which reflects the difficulty to fool a system based on a given biometric trait by fraudulent methods.
Cost: which refers to all costs that would be necessary to introduce the system in a real-world scenario.
Proportionality: which refers to the trade-off between the amount of privacy you give to the system and the services you have in return. For instance: it is not logical to compromise your fingerprint information to have access to the gym.
The aforementioned biometric traits may also be denoted as hard biometrics, as they are sufficiently discriminative for distinguishing subjects. In the literature, some researches have proposed also the use of some complementary information about the individual that is human understandable, such as gender, age or ethnicity information, which are insufficient to determine the identity of a subject independently, but provides useful information when using some of them, or when combined with some hard biometric traits. This complementary information is commonly referred in the literature as soft biometrics.
Table 1.1 describes different applications in which biometrics are found to be useful. Nowa- days we find biometric applications in a variety of scenarios from the more controlled to the uncontrolled ones, from forensic or law enforcement to commercial or personal uses. Most de- ployed biometric systems are mainly based on iris, fingerprint and face. This is mainly due to the availability of large evaluation databases and the support given to the most popular biometric traits through technology evaluations conducted by the American National Institute of Stan- dards and Technology, serving as an excellent resource to benchmark the current recognition
Legal documents e-passport (F and FP ), Spanish ID card (F, and FP )
Access control Walt Disney World Park Attraction (HG), Airplane Boarding Pass (F ) Law enforcement forensic scenarios (AFIS, FP, PP, DNA)
surveillance scenarios (F ) Commercial applications Siri (SP ), Google Assistant (SP )
E-learning Coursera authentication (KS )
E-health Biometric-based mobile vaccine registry in Benin, Africa (FP ) Personal devices security Mobile unlocking (F, FP ), Fraud Detection
Continuous Active Authentication (F and contextual attributes) Payments or Banking Apple pay (FP ), Loopay (FP ), Alipay (FP and F ), BBVA (FP ).
performance of various biometric systems. The remainder biometric traits have either reached only commercial applications or are emerging traits that need more technological maturity and acceptance before deployment.
Due to the diverse nature of biometric applications, no single biometric trait is likely to be optimal and satisfy the requirements of all applications. In fact, the choice of a biometric trait for a particular application usually depends on the degree to which the general requirements and practical criteria are satisfied. Although system performance is one of the most important practical criteria, other issues such as acceptability or proportionality in the application context must be also considered during the selection of biometric trait. Designing a biometric system not only requires knowledge of biometric technology, but also a good understanding of application requirements and issues related to human factors, ergonomics, and environmental variables.
When there is no single biometric trait optimal that satisfy all requirements for a particular application, one common solution is to use multibiometric systems, which combine different biometric traits to attain the desired level of performance or any other criteria [Ross et al., 2006].
1.2. Challenges in Biometric Recognition
Biometrics has reached a level of maturity when good quality samples are acquired from cooperative users under controlled conditions. In this regard, biometrics may be seen as a solved problem and can support reliable practical applications. Despite this progress, a number of challenging issues continue to inhibit the full potential of biometrics to automatically recognize humans. Regardless of the specific source of variability, the main goal of the biometric research
1.2 Challenges in Biometric Recognition
CONTROLLED
UNCONTROLLED
AGING
EXPRESS
ION LOW RESOL
UT ION SPOOFING A
TTACKS ILL
UMINA TION
SENS OR INT
EROPER
ABILITY
DIST ANCE
OCCL USION
POSE
Figure 1.1: Biometric recognition in the wild is affected by different sources: sensor interoperability, aging, expression, illumination, pose, distance, low-resolution and spoofing.
community focuses towards robustness, which in a simple way may be seen as minimizing the intraclass variability and maximizing the interclass variability.
Fig 1.1 summarizes the different sources of variabilities that currently affect biometric recog- nition. The reader should bear in mind that although these nuisance factors may be related to any biometric trait, the following discussion will have more emphasis on face and body biometrics in uncontrolled scenarios.
Illumination: Real environments are usually characterized by variable lighting conditions (natural vs. artificial, indoor vs. outdoors) that together with non-uniform background, shad- ows, underexposure, and overexposure in images, hinder greatly person recognition performance.
Indeed, this illumination problem leads in some cases to have intra class differences larger than inter class differences. Although most preprocessing methods, mainly based on histogram equal- ization techniques, perform almost perfectly when handling well-controlled lighting variations, they are still very poor in handling uncontrolled illumination variations.
Occlusions: Uncontrolled and uncooperative scenarios lead to situations in which the target may be occluded by other objects or people in the scene or because it’s wearing accessories like caps, sunglasses, balaclava or make-up, among others. Aside from being unintentionally or
as they lead to non frontal images (left/right or up/down). The main different approaches to tackle pose variations are grouped into pose-robust feature extraction approaches, multiview subspace learning approaches, face synthesis approaches, and hybrid approaches [Ding and Tao, 2016].
Expression: Variations in the person’s expression may happen more frequently when there is no cooperation from the users. Natural expressions result from the stretching or flexing the underlying muscles and, consequently, the facial skin is deformed in a non linear manner.
Physical expressions of yawn, grief, laughter or annoyance, among others may greatly change the face appearance and therefore degrade face recognition performance. One common approach to handle expression variations is based on matching on local approaches [Shan et al., 2009].
Aging: refers to changes in a biometric trait or the corresponding template over a time span, which can potentially impact the performance of a biometric system. There are two types of aging: trait aging and template aging. While trait aging refers to the biological change of a biometric sample over a person’s lifetime, template aging involves changes in a person’s biometric template. Trait aging does not necessarily imply template aging. Choosing time- invariant features can mitigate the impact of trait aging over template aging. Concerning face and body, time may change the person’s facial structure, appearance and body proportions.
Sensor Interoperability: The acquisition process introduces variations in the biometric data of an individual. Traditional research on biometric systems has worked under the assumption that gallery and probe images are acquired with the same device. Improvement in sensors has mitigated the intra class variations caused by sensor limitations (noise, blur, etc.) to a large extent. However, techniques to compensate variations derived from the involvement of different sensors within biometric systems remains an open problem in the context of biometrics [Ross and Jain, 2004]. For instance in the context of face recognition in forensic or surveillance scenarios, there can be mug-shot images, CCTV or PTZ images.
Spoofing Attacks: Spoofing is the act of cheating the system by disguising the identity to prevent being identified or to fake someone’s identity to gain an illegitimate access as a valid user. It has become a critical requirement, especially in unsupervised applications where full
1.2 Challenges in Biometric Recognition
monitorization is almost impossible1. Due to lack of secrecy, which is the idea that everybody can easily obtain face information from a distance or via Internet or social networks, it is relatively simple to forge attacks for biometric systems. There are even more sophisticated techniques based on 3D facial masks 2. Checking for signs of liveness through blood pulse or eye blinking are some of the proposed techniques in the literature to detect this emerging spoofing attacks [Galbally et al., 2014; Marcel et al., 2014].
Distance and Low Resolution: Some specific scenarios require performing recognition at large distances. In these scenarios at a distance, common biometric traits such as fingerprint or iris are useless or much more difficult to acquire. Closed-circuit television devices or pan-tilt- zoom are the ordinary cameras deployed in surveillance scenarios. Available biometric traits at stand-off distances comprises mainly face, body, gait and soft biometrics. Unlike the mugshot scenario, images extracted from surveillance cameras are acquired passively, in the sense that they just record video without focusing in any particular target and at low spatial resolution.
Besides, since subjects are not expected to be cooperative, they also introduce quality degrada- tion factors such as large camera-subject distance, pose-deviation, speed at which the subject is moving, difference in background context, illumination variations and occlusion caused by other objects or people in the scene, among others.
Performing recognition in forensic scenarios is still more challenging, as it entails the com- parison CCTV or PTZ images or even high-quality images, known as mugh-shot, with CCTV or PTZ images, usually with lower resolution and impaired by plenty of variations.
The search of robustness is focused on improving the different modules of the biometric recog- nition pipeline: from the acquisition to the feature extraction and the classification/matching modules. Regarding the acquisition systems, efforts have been carried out to improve the quality of the acquired samples, introducing 3D information or depth information. In what concerns machine learning, the most recent milestone has been deep learning technology [LeCun et al., 2015].
To remedy the weaknesses of hard-based biometric systems in very challenging scenarios, the use of multimodal approaches may enhance the task of recognition given some assumptions.
Among the different multimodal approaches proposed in the literature, we foresee that in un- constrained scenarios in which the only hard biometric modality available is the face, the use of facial soft biometrics may plan a role in the overall performance improvement.
In other cases, the robustness may come from the choice of an alternative spectrum range different from the visible. For instance, the use of near infrared helps to better extract iris texture information [Daugman, 2004], to achieve illumination invariance with face or other biometric traits or even to allow recognition in the darkness [Li et al., 2007a]. Likewise, thermal radiation
1http://travel.cnn.com/hong-kong/visit/hong-kong-airport-security-fooled-these-hyper-real-silicon-masks- 743923/
2http://www.thatsmyface.com/and http://real-f.jp/
Waist-to-hip ratio = 1.0 Shoulder-to-hip ratio = 1.5
Figure 1.2: Body Dynamic Information and Body Static Information that can be retrieved from surveillance scenarios.
was proposed in the literature as an effective liveness detection method, leveraging the property that those images are created based on the temperature pattern of the object [Pan et al., 2008].
Millimeter imaging (mmW) is an emerging imaging technology that can potentially provide benefits for the task of person recognition. Its particular benefits rely on its ability to pass through clothes and other materials, allowing to be more invariant to clothes variation and clutter.
1.3. Body Information
The demand for human identification at stand-off distances has gained considerable attrac- tion, particularly due to the need for covertly recognizing non cooperative individuals in uncon- strained environments (such as surveillance scenarios). Still, it is an unsolved problem. In those scenarios, traditional and most reliable biometric traits such as fingerprint or iris cannot be easily acquired. Although face is available in surveillance scenarios or scenarios at a distance, it normally suffers from low resolution, illumination or occlusions, among other variability sources, that greatly hinder the feasibility of face-based person recognition. In some cases, it might not be even possible to detect the face, and therefore not being able to apply face recognition.
An issue that has been disregarded to a certain degree is the utilization of body information for recognition. However, in unconstrained scenarios at a distance both face and body informa- tion may be available. Body information may be retrieved in a static or a dynamic manner, see Fig. 1.2. Body Static Information (BSI) contains information regarding texture, appearance or shape from a particular frame or single-shot image. Body Dynamic Information (BDI) collects body information throughout time. One clear example of a biometric trait that utilize BDI is gait [Nixon, 2016], which commonly extracts BDI from lateral views of the subject.
Body information may be retrieved using background subtraction techniques [Brutzer et al., 2011]. This means that even in cases in which face detection fails, it may be still possible to
1.4 Motivation of the Thesis
perform person recognition through body information (subject to good performance of tracking and background substraction algorithms).
Body information has been widely explored in the literature mainly through biometric sys- tems based on gait [Makihara et al., 2015; Nixon et al., 2010], and scarcely studied using Body Static Information [Kakadiaris et al., 2016]. Even if Body Dynamic Information is richer than static, it is also more difficult to obtain. For instance, capturing a full gait cycle (a sequence of several images) without occlusions from persons, obstacles, etc. is challenging in practical scenarios and less likely to be obtained than capturing information from a single frame. This motivates us to analyse more in depth the utility of BSI.
Besides, the exploration of body information in other ranges of the spectrum has been not covered to date. As a novelty, this Dissertation explores the potential use of millimeter wave images for the task of person recognition exploring body shape and texture information.
1.4. Motivation of the Thesis
Provided that person recognition performance attained with primary biometric systems is still an open problem in uncontrained scenarios, this Thesis is focused on the search of addi- tional identity clues to enhance the performance of hard biometric systems. In the context of unconstrained scenarios where the only hard biometric trait is the face, we propose the use of facial soft biometrics to improve the performance of face recognition systems. Later, motivated by the fact that faces and bodies are equally salient and available in unconstrained scenarios at a distance, we propose the use of body-based biometrics to alleviate the impact of challenging conditions over face biometrics. The research carried out in this area has been mainly motivated by the following five observations:
The first observation comes from the fact that the task of person recognition in unconstrained scenarios is still unsolved. Indeed, there exist different worldwide initiatives that are currently trying to assist the development of person recognition algorithms. IARPA’s Janus program1
“aims to dramatically improve the current performance of face recognition tools by fusing the rich spatial, temporal, and contextual information available from the multiple views captured by today’s media in the wild”. Likewise, Odin program2 “focuses to develop biometric presentation attack detection technologies to ensure biometric security systems can detect when someone is attempting to disguise their biometric identity”. Focusing in the Labeled Faces in the Wild face recognition benchmark (LFW), in which latest results are almost perfect, one may think that the problem of face recognition is already solved. However, there are still factors such as performing recognition at large scale with aging across distances and low resolution, among others, which are not addressed in this dataset. As an answer to these ongoing problems, the MegaFace Benchmark has been proposed recently in the literature [Kemelmacher-Shlizerman et al., 2016]. Concretely, in comparison to the 13K images from 5K subjects of LFW, this new
1https://www.iarpa.gov/index.php/research-programs/janus
2https://www.iarpa.gov/index.php/research-programs/odin
The third observation that has motivated this Thesis is that body information is normally available in unconstrained scenarios at a distance, such as surveillance scenarios. This informa- tion may appear completely in the most favorable case or partially if it is occluded. However, there is little effort in the literature exploring the utility of body information as a avenue to foster recognition in challenging scenarios.
The fourth observation is strongly related to the second one. As described in Section 1.3, body information may be used in a static or dynamic way. The mainstream approach in the literature has been focused on using dynamic information, mainly based on gait biometrics.
However, there are some concerns regarding the doability of transferring gait-based biometric systems into industry. Indeed, although there are plenty of commercial biometric systems based on face, fingerprint and iris, to the best of our knowledge, there is no available commercial software based on gait biometrics. However, Body Static Information could be integrated more easily in commercial software comprising face information. Rather than only using the face information contained in a frame, Body Static Information present in the same frame might be incorporated into the biometric system framework.
The last observation relies on the existence of millimeter scanners deployed in international airports scanning the full human body. To date, the only purpose of these scanners is concealed weapon detection. Further exploration of mmW images for person recognition purposes is diffi- cult to find in the literature. The availability of a full body signature in deployed scanners along with some benefits of millimeter waves also motivates us for this Thesis.
1.5. The Thesis and Main Contributions
The Thesis developed in this Dissertation can be stated as follows:
Person Recognition in challenging scenarios can be improved by using body static information in the visible spectrum and beyond, not commonly used in biometrics, such as body shape, body texture or soft biometrics. Depending on the scenario, such body information may be a very useful source of information for person recognition, generating soft biometrics or complementing hard biometrics like face.
The approach we follow to develop this PhD Thesis is based in two steps: i) performance analysis of hard biometric systems under some challenging conditions, and ii) exploration of
1.6 Outline of the Dissertation
methodologies to use facial soft biometrics or Body Static Information to foster person recogni- tion.
The main contributions in these two steps are summarized as follow:
Methodological and algorithmic contributions on how to employ body static information for recognition or soft biometric estimation, providing experimental evidences in different regions of the spectrum and analyzing the impact of distance.
Experimental contributions based on the discrimination capabilites of soft biometrics esti- mated manually or automatically and the benefits of using them for recognition purposes as bag of soft biometrics or in combination with hard biometric systems.
1.6. Outline of the Dissertation
The dissertation is structured according to a traditional complex type with background theory, practical methods, and experimental studies in which the methods are applied [Paltridge, 2002].
The Dissertation is divided into four parts. Part I introduces the problem statement and the contributions originated from this Dissertation. Then, there are two experimental parts: Part II describes the experimental works conducted in the visible spectrum, while Part III addresses the experimental works carried out beyond the visible spectrum. The Dissertation concludes with Part IV. The chapter structure is as follows:
Part I: Problem Statement and Contributions
• Chapter 1 introduces current challenges in biometric recognition and then gives the motivation, outline and contributions of this PhD Thesis.
• Chapter 2 summarizes related works which have motivated this Thesis.
• Chapter 3 introduces the biometric databases used in this Dissertation for face recog- nition, body-based recognition, soft biometrics prediction and soft biometrics recog- nition.
Part II: Visible Spectrum
• Chapter 4 studies the problem of face recognition under two particular challenging conditions: i) face recognition under occlusions, and ii) face recognition in surveil- lance scenarios.
• Chapter 5 explores a set of facial soft biometrics for recognition in unconstrained scenarios. This study is conducted analyzing both manual and automatic soft bio- metrics. Then, the recognition performance of this set soft biometrics in an exclusive manner or complementing a hard biometric system is studied experimentally.
• Chapter 7 studies person recognition using shape information from millimeter wave images.
• Chapter 8 studies person recognition using texture information from millimeter wave images and discusses several multimodal and multi-algorithmic fusion methodologies.
Part IV: Conclusions
• Chapter 9 concludes the Dissertation summarizing the main results obtained and outlining future research lines.
The dependence among the chapters is illustrated in Fig. 1.3. For example, before reading any of the experimental Chapters 4, 5, 6, 7, and 8 (shaded in Fig. 1.3), one should read first Chapters 1, 2 and 3. Before Chapter 7 it is recommended to read Chapter 6.
The methods developed in this PhD Thesis are strongly based on popular approaches from the pattern recognition literature. The reader is referred to standard texts for a background on the topic [Duda et al., 2001; Theodoridis and Koutroumbas, 2008]. Chapters 4, 5, 6, 7, and 8 assume a knowledge of the fundamentals of image processing [Gonzalez and Woods, 2006], and computer vision [Bigun, 2006].
1.7. Detailed Research Contributions
The research contributions of this PhD Thesis are the following (some publications appear in several items of the list, journal publications are in bold):
NOVEL METHODS.
1. Multimodal approach based on body static information and face for gender recogni- tion.
• E. Gonzalez-Sosa, A. Dantcheva, R. Vera-Rodriguez, JL Dugelay, F. Bremond and J. Fierrez. “Image- based Gender Estimation from Body and Face across Distances’, in Proc. of IAPR Int. Conf. on Pattern Recognition, 2016, Cancun (MEXICO).
2. Multimodal approach based on body static information and face for person recogni- tion.
• J. Hernandez-Ortega, E. Gonzalez-Sosa, A. Morales-Moreno, R. Vera-Rodriguez and J. Fierrez. “Person Recognition at a Distance: Improving Face Recognition Through Body Static Information’, Technical Report.
1.7 Detailed Research Contributions
Chapter 1
"Introduction"
Chapter 2
"Related Works"
Chapter 3
"Performance Evaluation"
PART I: PROBLEM STATEMENT AND CONTRIBUTIONS
Chapter 4
"Face Recognition"
Chapter 5
"Body Information"
Chapter 6
"Soft Biometrics for Recognition"
PART II: VISIBLE SPECTRUM
Chapter 7
"mmW Person Recognition through Shape"
Chapter 8
"mmW Person Recognition through Texture"
PART III: BEYOND THE VISIBLE SPECTRUM
Chapter 9
"Conclusions"
PART IV: CONCLUSIONS Preceeding block is recommended Preceeding block is required
Experimental Chapters
Introduction, Related Works, Materials and Conclusions
Figure 1.3: Dependence among chapters.
NEW BIOMETRIC SYSTEMS.
1. State of the art results in the ICB-RW 2016 competition.
• E. Gonzalez-Sosa, R. Vera-Rodriguez, J. Fierrez and J. Ortega-Garcia. “Exploring Facial Regions in Unconstrained Scenarios: Experience on ICB-RW’, IEEE Intelligent Systems, 2017.
NEW BIOMETRIC DATA.
1. Groundtruth of soft biometrics and facial attribute information from the Labeled Faces in the Wild Database.
• E. Gonzalez-Sosa, R. Vera-Rodriguez, J. Hernandez-Ortega and J. Fierrez. “Facial Soft Biometrics for Unconstrained Person Recognition’, Technical Report.
NEW EXPERIMENTAL STUDIES
