Hopular: Modern Hopfield Networks for Tabular Data

Bernhard Schäfl1, Lukas Gruber1, Angela Bitto-Nemling1, 2, Sepp Hochreiter1, 2

1 ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria 2 Institute of Advanced Research in Artificial Intelligence (IARAI)

Detailed blog post on this paper at this link.

.

Requirements

The software was developed and tested on the following 64-bit operating systems:

  • Rocky Linux 8.5 (Green Obsidian)
  • macOS 12.4 (Monterey)

As the development environment, Python 3.8.3 in combination with PyTorch Lightning 1.4.9 was used. More details on how to install PyTorch Lightning are available on the official project page.

Installation

The recommended way to install the software is to use pip/pip3:

$ pip3 install git+https://github.com/ml-jku/hopular

Usage

Hopular has two modes of operation:

  • list for displaying various information.
  • optim for optimizing Hopular using specified hyperparameters.

More information regarding the operation modes is accessible via the -h flag (or, alternatively, by --help).

$ hopular -h
$ hopular <mode> -h

To display all available datasets, the --datasets flag has to be specified in the list mode.

$ hopular list --datasets

Optimizing a Hopular model using the default hyperparameters is achieved by specifying the corresponding dataset in the optim mode.

$ hopular optim --dataset <dataset_name>

Examples

To optimize a Hopular model on the GlassIdentificationDataset using the default hyperparameters, only the dataset name itself needs to be specified. More details on the default values are available in the console interface implementation.

$ hopular optim --dataset "GlassIdentificationDataset"

Optimizing a smaller Hopular model on the GlassIdentificationDataset utilizing only 4 modern Hopfield networks, 2 iterative refinement blocks, and a scaling factor of 10 is achieved by manually specifying the respective hyperparameters.

$ hopular optim --dataset "GlassIdentificationDataset" --num_heads 4 --num_blocks 2 --scaling_factor 10

Disclaimer

The datasets, which are part of this repository, are publicly available and may be licensed differently. Hence, the LICENSE of this repository does not apply to them. More details on the origin of the datasets are available in the accompanying paper.

License

This repository is MIT-style licensed (see LICENSE), except where noted otherwise.

GitHub

View Github