Unmasking Deepfakes: Behavioral Science Meets Data Science

Introduction

We live in an era where the phrase "seeing is believing" is no longer valid. The progress in AI-generated video and audio, known as deepfakes, has led to a situation where synthetic media appears and sounds just like the real thing. With implications ranging from disinformation campaigns to fraud, the stakes have reached unprecedented levels.

Can behavioral science techniques — the same ones used in interpersonal interviews and credibility assessments — help us detect deepfakes more reliably than machine learning alone?

Why Behavioral Cues Still Matter

Traditional deception detection looks at human signals like:

• Facial micro-expressions (fleeting, involuntary cues)

• Blink rate and eye gaze (timing, direction, synchrony)

• Head movement patterns (natural vs. mechanical)

• Verbal/linguistic inconsistencies (missing detail, unusual phrasing)

  
  

Deepfakes add complexity by eliminating or creating cues. A generated video can flawlessly mimic eye contact or a smile. Nonetheless, my research revealed that subtle inconsistencies persist, particularly when analyzing multiple behavioral channels simultaneously.

Research Approach

• Tools: Python, MediaPipe, OpenCV, FFmpeg, plus custom scripts for blink/gaze/head pose tracking

• Methods Applied:

  • Facial Action Coding System (FACS) for micro-expression detection/annotation

  • SCAN/LSISCAN for linguistic analysis

  • Frame-by-frame anomaly detection for blink/gaze/head angle shifts

  • Error Level Analysis (ELA) to spot compression artifacts

This video was artificially generated by instructing AI to envision and then animate in a photo-realistic manner what it perceives to be the "most attractive woman".

A synthetically generated photo-realistic animation of an AI generating "the most attractive woman" circa 2025

When a moving subject was created, I applied my code to detect facial landmarks and calculate the head's yaw, pitch, and roll. In addition, my code monitored the eye gaze's roll and pitch, as well as measured the blink rate and duration.

Applying the code: Facial landmark, gaze, and blink detection/rate tracking with an overlayed facial mesh

Key Findings

1. Micro-expressions Survive AI Smoothing

   • Deepfakes struggle with ultra-brief facial cues (<0.5 seconds). A fake smile often “lingers” too long or misses eye involvement

2. Blink & Gaze Patterns Are Telltale

   • Many synthetic videos show under-blinking, unnatural gaze locks, or robotic shifts in head position (often referred to as Uncanny Valley)

3. Language Betrays Synthetic Sources

   • Generated text/speech is often coherent but sometimes lacks sensory richness or shows unusual repetition

4. Fusion > Isolation

   • No single method was perfect. But combining visual + linguistic + forensic cues raised detection accuracy significantly

These graphs were created by running my code on a synthetically generated video, which purportedly produced moving subjects that accurately represented various emotions, though these claims are likely overstated. Despite this, the graphs served as a proof of concept, demonstrating that my code could effectively detect, derive, and quantify emotion from the video.

Real-World Applications

• Counterintelligence & Insider Threat: Spot synthetic personas, manipulated interviews, or influence attempts

• Legal & Forensic Use: Authenticate digital evidence before it enters court

• Media & Corporate Security: Identify disinformation before it spreads virally

• Training & Awareness: Equip analysts and executives with practical skills to question synthetic cues

  
  

Code Contributions

During my research, I created a collection of open-source pipelines, employing a range of tools and techniques:

• Blink/Gaze/Head Pose Detector (Python + MediaPipe)

• Microexpression Annotation Tool aligned to FACS

• Linguistic Analyzer (SCAN/LSISCAN applied to transcripts)

• Dashboard Visualizations (3D head-pose timelines, multimodal overlays)

Looking Ahead

The arms race between synthetic generation and detection will only intensify. My next steps are:

• Expanding datasets to include cross-cultural facial behavior

• Integrating behavioral pipelines with deepfake detection AI models

• Building real-time dashboards (VR/AR compatible) for analyst use

  
  

Conclusion

Behavioral science isn’t obsolete in the age of AI — it’s the missing piece. By combining credibility analysis techniques with modern data science, we can still reveal the subtle signals of synthetic deception.

Deepfakes exploit what we believe. Behavioral analysis helps us protect what we trust.

References

• Agarwal, S., Farid, H., Gu, Y., He, M., Nagano, K., & Li, H. (2020). Protecting World Leaders Against Deep Fakes. CVPR.

• Chesney, R., & Citron, D. (2019). Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security. California Law Review.

• Dolhansky, B. et al. (2020). The DeepFake Detection Challenge Dataset. arXiv.

• Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System. Consulting Psychologists Press.

• Ekman, P. (2002). Emotions Revealed. Holt.

• Farid, H. (2009). Image Forgery Detection: A Survey. IEEE Signal Processing Magazine.

• Farid, H. (2019). Photo Forensics. MIT Press.

• Jack, R. E., Caldara, R., & Schyns, P. G. (2012). Internal representations reveal cultural diversity in facial expressions of emotion. J Exp Psychol Gen.

• Khalid, H. et al. (2021). FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset. arXiv.

• Li, Y., Chang, M.-C., & Lyu, S. (2018). In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking. IEEE WIFS.

• Li, Y. et al. (2020). Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics. CVPR.

• NIST (2020). Media Forensics Challenge (MFC).

• Rössler, A. et al. (2019). FaceForensics++: Learning to Detect Manipulated Facial Images. ICCV.

• Sapir, A. (2016). SCAN: Scientific Content Analysis. LSI.

• Stern, J. A., Walrath, L. C., & Goldstein, R. (1984). The endogenous eyeblink. Psychophysiology.

• U.S. ODNI (2021). Foreign Malign Influence Themes and Narratives.

• Vrij, A. (2008). Detecting Lies and Deceit: Pitfalls and Opportunities. Wiley.

• Zhou, P. et al. (2021). Two-Stream Neural Networks for Tampered Face Detection. CVPR.