VMD Audio/Text control with natural language
This repository is a proof of principle for performing Molecular Dynamics analysis, in this case with the program VMD, via natural language commands.
It contains code to generate VMD commands from natural language queries given as text commands. It also contains code to control VMD via audio commands contained in a .wav recording, and via a live speak-to-text interface.
Converting text queries into VMD commands requires an OpenAI API key that allows access to the GPT-3 model. Live speak-to-text requies a google cloud speech-to-text authentication
Here is an example in text mode
- Installation can be performed with
pip. We recommend doing this inside of a virtual environment.
pip install [email protected]+git://github.com/whitead/marvis.git
- Transcription of live audio requires the additional installation of the pyaudio package (in conda, this can be achieved by
conda install pyaudio).
Use of the two APIs above requires setting environment variables. In
BASH, one would do
export GOOGLE_APPLICATION_CREDENTIALS=path_to_google_api_license_file.json export OPENAI_API_KEY=sh-....
After installation, the commands
marvis-text will be added to your path.
To convert text commands to VMD commands, use the
marvis-text. For example,
marvis-text --mock brings up an interactive prompt:
$ marvis-text --mock ========================== ==== MARVIS IS READY ===== ========================== marvis-text> rotate the protein around the z axis by 45 degrees Mock Sending: ";rotate z by 45 1" (VMD Command)|rotate z by 45 1 marvis-text>
To interface with VMD in live mode, use
--vmd flag can be used to specify the path to VMD, which by default is
In the simplest case, simply type
--safe_mode can be used to ask for confirmation of voice to text before sending the command to OpenAI
marvis on a Mac, where VMD is in the applications folder, one might want to create a vmd.sh script such as:
#!/bin/bash /Applications/VMD\ 1.9.4a51-x86_64-Rev9.app/Contents/vmd/vmd_MACOSXX86_64 $*
And then run:
$ marvis --vmd ./vmd.sh