FairyTailor: Multimodal Generative Framework for Storytelling
Human-in-the-loop visual story co-creation.
Users can create a cohesive children's story by weaving generated texts and retrieved images with their input.
With co-creation, writers contribute their creative thinking, while generative models contribute to their constant workflow.
FairyTailor adds another modality and modifies the text generation process to help producing a coherent and creative story.
After cloning the repository:
Client (Vue 2.6)
Install and check that the client compiles:
cd client npm i npm run build
Install and activate the environment (conda provided):
conda env create -f environment.yml conda activate MultiModalStory
Install environment globally in the directory:
pip install -e . pip install git+https://github.com/openai/CLIP.git
After installation run:
python -m spacy download en_core_web_sm
In python terminal:
nltk.download('wordnet') nltk.download('sentiwordnet') nltk.download('averaged_perceptron_tagger')
Large Data Management (dvc)
Our large data files are stored on IBM's Cloud Object Storage, and to pull data files from that platform you will use a special, read-only
dvc pull -f
Which will pull:
- backend/outputs (five preset stories)
- backend/story_generator/downloaded (transformers)
- client/public/unsplash25k (styled images)
Running the framework during developemnt
cd client npm run devw
Backend (with server auto reload):
uvicorn backend.server:app --reload --reload-dir backend
Open the uvicorn server
localhost:8000 in your web browser
New huggingface transformer
- Place the transformer in
- Update the current model path by changing the constant
New images folder
- Replace the folder
- Update the current path by changing the constant
- Add functions to the backend endpoint at
client/src/js/api/mainApi.jsto call the backend endpoint from the client.
- Update the corresponding user components in