makemore
makemore is the most accessible way of tinkering with a GPT.
The one-file script makemore.py takes one text file as input, where each line is assumed to be one training thing, and generates more things like it. For example, we can feed it a database of names, and then use it to generate new cool baby name ideas that sound name-like, but are not already existing names. Or if we feed it a database of company names then we can generate new ideas for a name of a company. Or we can just feed it valid scrabble words and generate english-like babble.
Under the hood, the script trains a (character-level) Transformer, identical to the one that powers GPT and friends.
This is not meant to be a heavyweight library with switches and knobs. It’s one hackable file of ~500 lines of code. PyTorch is the only requirement. Go nuts.
Usage
The included names.txt dataset, as an example, has the most common 32K names takes from ssa.gov for the year 2018. It looks like:
emma
olivia
ava
isabella
sophia
charlotte
...
Let’s point the script at it:
$ python makemore.py -i names.txt -o names
Training progress and logs and model will all be saved to the working directory names. The default model is a super tiny 200K param transformer; Many more training configurations are available – see the argparse and read the code. Training does not require any special hardware, it runs on my Macbook Air and will run on anything else, but if you have a GPU then training will fly. As training progresses the script will print some samples throughout. However, if you’d like to sample manually, you can use the --sample-only flag, e.g. in a separate terminal do:
$ python makemore.py -i names.txt -o names --sample-only
This will load the best model so far and print more samples on demand. Here are some unique baby names that get eventually generated from current default settings (test logprob of ~1.92):
dontell
khylum
camatena
aeriline
najlah
sherrith
ryel
irmi
taislee
mortaz
akarli
maxfelynn
biolett
zendy
laisa
halliliana
goralynn
brodynn
romima
chiyomin
loghlyn
melichae
mahmed
irot
helicha
besdy
ebokun
lucianno
Have fun!
License
MIT
