This repo includes an official code release for the CodeGen models, as presented in the paper:

Title: A Conversational Paradigm for Program Synthesis

Authors: Erik Nijkamp*, Bo Pang*, Hiroaki Hayashi*, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong (* indicates equal contribution)

The current version releases the sampling code, while the detailed training code will be released soon.


git clone
cd CodeGen

wget -P checkpoints && tar -xvf checkpoints/codegen-350M-mono.tar.gz -C checkpoints/
wget -P checkpoints && tar -xvf checkpoints/codegen-2B-mono.tar.gz -C checkpoints/
wget -P checkpoints && tar -xvf checkpoints/codegen-6B-mono.tar.gz -C checkpoints/
wget -P checkpoints && tar -xvf checkpoints/codegen-16B-mono.tar.gz -C checkpoints/

python3.8 -m venv .venv
source .venv/bin/activate
pip3 install --upgrade pip setuptools
pip3 install -r requirements.txt
python3 -m jaxformer.hf.sample --model codegen-350M-mono --context "def hello_world():"

Released Models

We release models of various sizes trained on various datasets. The models are named in the following format:


model-size has 4 options 350M, 2B, 6B, 16B.

data has 3 options nl, multi, mono. nl models are randomly initialized and trained on the Pile, a 825.18 GB English text corpous. multi models are initialized from nl models and then trained on a corpus with code data of multiple programming languages. mono models are initialized from multi models and then trained on a corpus with Python code.

The model names can be provided to the --model flag for See a sample usage above in Setup.


If you find our code or paper useful, please cite the paper:

  title={A Conversational Paradigm for Program Synthesis},
  author={Nijkamp, Erik and Pang, Bo and Hayashi, Hiroaki and Tu, Lifu and Wang, Huan and Zhou, Yingbo and Savarese, Silvio and Xiong, Caiming},
  journal={arXiv preprint},


Our code is BSD-3 licensed. See LICENSE.txt for details.


View Github