Interactive Attention Visualization

A small example of an interactive visualization for attention values as being used by transformer language models like GPT2 and BERT.

by Hendrik Strobelt and Sebastian Gehrmann for the SIDN IAP class at MIT, Jan 2020



  • Install Anaconda or Miniconda
  • run conda env create -f environment.yml to create a new environment called attnvis

Running the demo

  • activate conda environment: conda activate attnvis
  • run server: python
  • visit http://localhost:8888/

Structure             -- contains the interface to the pytorch/huggingface backend          -- defines a REST interface for the calls

client/*           -- contains all client files
client/index.html  -- main file including JS code
client/styles.css  -- all CSS styles defined in here
client/tools.js    -- helper functions

Libraries used