Team members:
Kellen Evoy Margaret Xie Nathan Faber-Good
Diana Burek Harvey Ho
Tommas Meesussen Tobias Moktar Dr. Rachel Jiang
Dr. Ed Sykes
Objective method for the diagnosis of Chronic pain
Objectives
Establish PSPI prediction functionality with OpenFace software tools
Research on 2D/3D facial expressions (FEs) analysis and deceptive detection (fake vs. real pain)
Research micro-expression detection & relevance
Research verbal deception (fake vs. real pain)
Research on thermal FEs analysis
Pain Detection
P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, S. Chew, and
I. Matthews, “Painful monitoring: Automatic pain monitoring using
the unbc-mcmaster shoulder pain expression archive database,” Image
and Vision Computing, vol. 30, no. 3, pp. 197–205, 2012.
Action Units for pain detection
P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, S. Chew, and
I. Matthews, “Painful monitoring: Automatic pain monitoring using
the unbc-mcmaster shoulder pain expression archive database,” Image
and Vision Computing, vol. 30, no. 3, pp. 197–205, 2012.
OpenFace 2.0
Facial Behavior Analysis Toolkit
excellent AU prediction results with OpenFace 2.0
OpenFace 2.0 is a framework that implements modern facial behavior analysis algorithms including: facial landmark detection, head pose tracking, eye gaze and facial action unit recognition.
T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, “Openface
2.0: Facial behavior analysis toolkit,” in International Conference on
Automatic Face & Gesture Recognition. IEEE, 2018, pp. 59–66.
OpenFace 2.0
Initial PSPI Prediction Process
calculate the PSPI for each image, and compare against ground-truth
PSPI [ PSPI = AU4 + max(AU6, AU7) + max(AU9, AU10) + AU43 (omitted) ]
Investigating Results
an image of ground-truth 14 PSPI and compare it to what OpenFace classified it.
OpenFace predicted PSPI.
Analyzing New Results
Micro Expressions
Last 1/25 to 1/5 of a second
Caused by conflicting signals from extrapyramidal (involuntary) and pyramidal (voluntary) motor systems.
Occur when one’s voluntary or posed emotion is something other than their true emotion.
Because of this, Micro Expressions can be key indicators of deception.
ME (Micro Expression) Detection
Detection involves two unique problems:
Spotting:
Being able to tell whether a micro expression has happened
Recognizing:
Given a spotted micro expression, being able to tell what emotion is contained in the micro expression
Li et al and Borza et al both propose solutions for ME detection algorithms:
With Borza et al achieving an 86.95% true positive rate on the CASME2 dataset
Why we used PSPI
The labels in this dataset are complex, most consisting of multiple values
PSPI is a single metric describing pain
on a 16-point scale
ME Detections: Current Technologies
Local Binary Pattern Eulerian Video Magnification
https://ieeexplore-ieee-org.library.sheridanc.on.ca/document/8117030
http://people.csail.mit.edu/mrub/papers/vidmag.pdf
3D Processing & Analysis
• Overview of Next Phase
• Discuss Considering 3D Analysis
• Literature Highlights
• Exploring a Identify/recreate current state-of-the-art baseline system
• Establish an experiment protocol for gathering 3D data (Kinect DK)
3D/depth image analysis (literature
review)
Why consider 3D?
• 3D analysis has become a popular research topic, providing many potential benefits:
• With the face being a 3D object, many communicative signals involve changes in depth and head rotation that can’t be identified from 2D capture [1].
• Improved estimation of facial shape from 2D images [2].
• Motion capture capabilities [2].
• While it may be possible to create an adequate system for binary classification from 2D images/videos (fake vs. real pain), pain quantification from 2D video remains difficult [3].
• In 2014, Bartlett et al determined a computer vision system distinguishes faked vs. real pain
better than humans, with an 85% success rate [4].
Literature Review – General
Automatic Recognition methods Supporting Pain Assessment: A Survey (2019)
• Werner et al provide a very in-depth survey paper highlighting key research in automatic pain recognition [5].
• Identifies current pain recognition approaches and relevant research for each;
• Camera-based (employed by 70% of automated pain research)
• Contact-sensor
• Audio
• Multimodal (full-body, 3D/depth, other modality fusions)
• Identifies a wide variety of processing strategies (features, models).
• Provides extensive information for 11 datasets.
• Also suggests researchers “find a more specific and sensitive objective observational measure than
PSPI” in the future.
Literature Review – 3D
Pain Intensity Estimation From Mobile Video Using 2D and 3D Facial Keypoints (2020, Preprint)
• Lee et al investigate the usefulness of smartphone video (2D & 3D) for pain intensity estimation [7].
• “Across all of the tasks, we consistently see the 3D features outperforming the 2D features. This is promising since these 3D features are a unique component of our collected dataset and it is hoped that they can provide additional
capability in detecting pain in facial expressions.”
Literature Review – 3D dataset
• 2020, Ye et al introduce SIAT-3DFE: A High- Resolution 3D Facial Expression Dataset of 500 subjects.
• 8000 3D facial expression models
• 32,000 texture images (16,000 high-
fidelity 2D texture images)
3D Processing Overview
• No obvious OpenFace 2.0 Equivalent for 3D data
• Solutions for face-detection, emotion-identification, etc exist but are all custom pipelines
Localization Registration Feature Extraction
Emotion/Pain Detection Iterative
Closest Point Curvature
Features Motion
Vectors
CNN Classification
(Happy)
Depth maps
• Similar to an image, but instead of storing a unique color each pixel, it stores a depth value for each pixel.
• Given its format, some existing 2D algorithms can be applied to depth
maps and yield greater accuracy (Xu yan et al.)
3D models
• Actual polygons of the face
• Algorithms for face detection, registration, etc are unique to 3D
• Typically requires minimum 2 cameras to generate a 3D model
• A depth map can also be used to generate a 3D model
References
• [1] X. Zhang et al., "A high-resolution spontaneous 3D dynamic facial expression database," 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, 2013, pp. 1-6, doi: 10.1109/FG.2013.6553788.
• [2] D. Cosker, E. Krumhuber and A. Hilton, "A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling," 2011 International Conference on Computer Vision, Barcelona, 2011, pp. 2296-2303, doi: 10.1109/ICCV.2011.6126510.
• [3] J. Egede, T. Olugbade, C. Wang, S. Song, N. Berthouze, M. Valstar, A. Williams, H. Meng, M. Aung, and N. Lane. “Emopain challenge 2020:
Multimodal pain evaluation from facial and bodily expressions,” IEEE International Conference on Automatic Face and Gesture Recognition Workshop, 2020
• [4] Bartlett MS, Littlewort GC, Frank MG, Lee K. "Automatic decoding of facial movements reveals deceptive pain expressions". Curr Biol.
2014;24(7):738-743. doi:10.1016/j.cub.2014.02.009
• [5] Werner, Philipp & Lopez-Martinez, Daniel & Walter, Steffen & Al-Hamadi, Ayoub & Gruss, Sascha & Picard, Rosalind. (2019). Automatic Recognition Methods Supporting Pain Assessment: A Survey. IEEE Transactions on Affective Computing. 1-1. 10.1109/TAFFC.2019.2946774.
• [6] R. Irani et al., “Spatiotemporal Analysis of RGB-D-T Facial Images for Multi-Modal Pain Level Recognition,” in IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2015.
• [7] M. Lee, J. S. E. Lee, L. Kennedy, A. Girgensohn, L. Wilcox, C. W. Tan, and B. L. Sng, “Pain Intensity Estimation from Mobile Video Using 2D and 3D Facial Keypoints,” 17-Jun-2020. [Online]. Available: https://deepai.org/publication/pain-intensity-estimation-from-mobile-video-using-2d- and-3d-facial-keypoints. [Accessed: 26-Aug-2020].
• [8] Y. Ye, Z. Song, J. Guo and Y. Qiao, "SIAT-3DFE: A High-Resolution 3D Facial Expression Dataset," in IEEE Access, vol. 8, pp. 48205-48211, 2020, doi: 10.1109/ACCESS.2020.2979518.
