Rembg Virtual Greenscreen Edition (Dr. Tim Scarfe)
Rembg Virtual Greenscreen Edition is a tool to create a green screen matte for videos
- Take any video file and convert it to an alpha matte to apply for a virtual green screen
- It runs end-to-end non-interactively
- You need ffmpeg installed and on your path
- There is also a powershell script
./remove-bg.ps1which will do the job in a manual way i.e. first create frames, then run the
rembg -p ...command and then run
ffmpegto create the matte movie. This was my first approach to solve this problem but then I migrated onto just making a new version of rembg.
If you have any ideas for speeding this up further, please let us know. We have tried quite a few things at this stage and are a bit stuck on how to proceed from here. See some of the "evolution" in the Whimsical notes.
pip install rembg-greenscreen greenscreen -g "path/video.mp4"
Experimental parallel green screen version;
greenscreen --parallelgreenscreen "path/video.mp4" --workernodes 3 --gpubatchsize 5
The command above will produce a
video.matte.mp4 in the same folder, also works with
avi extensions. Uses ffmpeg under the hood to stream and re-encode the frames into a grayscale matte video.
Be careful with the default parameters, my 11GB GPU is already pretty much maxed with 3 instances of the NN with 5 image gpu batches in forward pass.
You can see how much free GPU ram you have with
- Don't use VBR videos, it will run forever -- use Handbrake to convert them to CFR
- https://github.com/NathanUA/U-2-Net (thanks to these folks for making the semantic segmentation model and publishing online)