The following is a research project initially inspired by CLIPMatrix. Following the release of CLIP-Actor and AvatarCLIP I’ve decided to just release my code since my work was quite similar. The main difference is that this uses meshes which are much faster and less memory-intensive, additionally I explore multiple independent optimizable parameters


This project is for research purposes only


This codebase uses SMPL-X Models and the smplx pip library. By using this code you agree to the SMPL-X Model License and smplx License


This code base relies on nvdiffmodeling and in turn nvdiffrast.


Get started by testing out the features through the collabs

Create a character from just a text prompt Change an expression with text prompt Pose with a description (⚠️ WIP) Create fantastic creatures like CLIPMatrix


git clone --recurse-submodules [email protected]:NasirKhalid24/CLIPMesh-SMPLX.gitcd CLIPMesh-SMPLX# or manually unzip so the .npz files are in the folder as shown belowunzip /models/smplx/ -d /models/smplx   pip install virtualenvvirtualenv ENVsource ENV/bin/activatepip install --upgrade pippip install -r requirements.txt
CLIPMesh-SMPLX│└───models│   └───smplx│       │   SMPLX_NEUTRAL.npz│       │   ....│       │   ....

To replicate the demos you can use the configs provided.

# For a single character generationpython --path=configs/single.yaml# For a single expression generationpython --path=configs/expression.yaml# For a single pose generationpython --path=configs/pose.yaml# For a CLIPMatrix type generationpython --path=configs/pose.yaml# For something else use paramspython \--optim   body expression texture normal \--options face full back \--epochs  1000 \--gpu     0 \--face_text "a thin 3D render of the face of a James Bond" \--full_text "a thin 3D render of the James Bond" \--back_text "a thin 3D render of the back of James Bond" \--debug_log true--log_int 250


View Github