/ Machine Learning

Head pose estimation by TensorFlow and OpenCV

Head pose estimation by TensorFlow and OpenCV

Head pose estimation

This repo shows how to estimate human head pose from videos using TensorFlow and OpenCV.

Head-pose-estimationv

Head-pose-estimation

Dependence

  • TensorFlow 1.4
  • OpenCV 3.3
  • Python 3

The code is tested under Ubuntu 16.04.

How it works

There are three major steps:

  1. Face detection. A face detector is adopted to provide a face box containing a human face. Then the face box is expanded and transformed to a square to suit the needs of later steps.

  2. Facial landmark detection. A custom trained facial landmark detector based on TensorFlow is responsible for output 68 facial landmarks.

  3. Pose estimation. Once we got the 68 facial landmarks, a mutual PnP algorithms is adopted to calculate the pose.

Miscellaneous

  • The marks is detected frame by frame, which result in small variance between adjacent frames. This makes the pose unstaible. A Kalman filter is used to solve this problem, you can draw the original pose to observe the difference.

  • The 3D model of face comes from OpenFace, you can find the original file here.

GitHub