Python tools for 3D face: 3DMM, Mesh processing(transform, camera, light, render), 3D face representations.
This project implements some basic functions related to 3D faces.
You can use this to process mesh data, generate 3D faces from morphable model, reconstruct 3D face with a single image and key points as inputs, render faces with difference lightings(for more, please see examples).
In the beginning, I wrote this project for learning 3D face reconstruction and for personal research use, so all the codes are written in python(numpy). However, some functions(eg. rasterization) can not use vectorization to optimize, writing them in python is too slow to use, then I choose to write these core parts in c++(without any other big libraries, such as opencv, eigen) and compile them with Cython for python use.
So the final version is very lightweight and fast.
In addition, the numpy version is also retained, considering that beginners can focus on algorithms themselves in python and researches can modify and verify their ideas quickly. I also try my best to add references/formulas in each function, so that you can learn basic knowledge and understand the codes.
# Since triangle mesh is the most popular representation of 3D face, # the main part is mesh processing. mesh/ # written in python and c++ | cython/ # c++ files, use cython to compile | io.py # read & write obj | vis.py # plot mesh | transform.py # transform mesh & estimate matrix | light.py # add light & estimate light(to do) | render.py # obj to image using rasterization render mesh_numpy/ # the same with mesh/, with each part written in numpy # slow but easy to learn and modify # 3DMM is one of the most popular methods to generate & reconstruct 3D face. morphable_model/ | morphable_model.py # morphable model class: generate & fit | fit.py # estimate shape&expression parameters. 3dmm fitting. | load.py # load 3dmm data
left: random example generated by 3dmm
right: fitting face with 3dmm using 68 key points
fix camera position & use orthographic projection. (often used in reconstruction)
then transform face object: scale, change pitch angle, change yaw angle, change roll angle
fix obj position & use perspective projection(fovy=30). (simulating real views)
then move camera position and rotate camera: from far to near, down & up, left & right, rotate camera
single point light: from left to right, from up to down, from near to far
render different attributes in image pixels.
: depth, pncc, uv coordinates
render different attributes in uv coordinates.
: colors(texture map), position(2d facial image & corresponding position map)
Python 2 or Python 3
- skimage (for reading&writing image)
- scipy (for loading mat)
- matplotlib (for show)
- Cython (for compiling c++ file)
Clone the repository
git clone https://github.com/YadiraF/face3d cd face3d
Compile c++ files to .so for python use (ignore if you use numpy version)
cd face3d/mesh/cython python setup.py build_ext -i
Prepare BFM Data (ignore if you don't use 3dmm)
(examples use cython version, you can change
mesh_numpyto use numpy version)
cd examples python 1_pipeline.py
For beginners who want to continue researches on 3D faces, I strongly recommend you first run examples according to the order, then view the codes in mesh_numpy and read the comments written in the beginning in each file. Hope this helps!
Moreover, I am new in computer graphics, so it would be great appreciated if you could point out some of my wrong expressions. Thanks!