CLIPMesh-SMPLX
The following is a research project initially inspired by CLIPMatrix. Following the release of CLIP-Actor and AvatarCLIP I’ve decided to just release my code since my work was quite similar. The main difference is that this uses meshes which are much faster and less memory-intensive, additionally I explore multiple independent optimizable parameters
License
This project is for research purposes only
SMPL-X
This codebase uses SMPL-X Models and the smplx pip library. By using this code you agree to the SMPL-X Model License and smplx License
Renderer
This code base relies on nvdiffmodeling and in turn nvdiffrast.
Demos
Get started by testing out the features through the collabs
 |
 |
 |
 |
|---|---|---|---|
| Create a character from just a text prompt | Change an expression with text prompt | Pose with a description (⚠️ WIP) | Create fantastic creatures like CLIPMatrix |
Setup
git clone --recurse-submodules [email protected]:NasirKhalid24/CLIPMesh-SMPLX.gitcd CLIPMesh-SMPLX# or manually unzip so the .npz files are in the folder as shown belowunzip /models/smplx/smplx_npz.zip -d /models/smplx pip install virtualenvvirtualenv ENVsource ENV/bin/activatepip install --upgrade pippip install -r requirements.txtCLIPMesh-SMPLX│└───models│ └───smplx│ │ SMPLX_NEUTRAL.npz│ │ ....│ │ ....To replicate the demos you can use the configs provided.
# For a single character generationpython main.py --path=configs/single.yaml# For a single expression generationpython main.py --path=configs/expression.yaml# For a single pose generationpython main.py --path=configs/pose.yaml# For a CLIPMatrix type generationpython main.py --path=configs/pose.yaml# For something else use paramspython main.py \--optim body expression texture normal \--options face full back \--epochs 1000 \--gpu 0 \--face_text "a thin 3D render of the face of a James Bond" \--full_text "a thin 3D render of the James Bond" \--back_text "a thin 3D render of the back of James Bond" \--debug_log true--log_int 250
