BMInf

BMInf (Big Model Inference) is a low-resource inference package for large-scale pretrained language models (PLMs). It has following features:

• Hardware Friendly. BMInf supports running models with more than 10 billion parameters on a single NVIDIA GTX 1060 GPU in its minimum requirements. Running with better GPUs leads to better performance. In cases where the GPU memory supports the large model inference (such as V100 or A100), BMInf still has a significant performance improvement over the existing PyTorch implementation.
• Open. The parameters of models are open. Users can access large models locally with their own machines without applying or accessing an online API.
• Comprehensive Ability.  BMInf supports generative model CPM1 [1], general language model CPM2.1 [2], and dialogue model EVA [3]. The abilities of these models cover text completion, text generation, and dialogue generation.
• Upgraded Model. Based on CPM2 [2], the newly upgraded model CPM2.1 is currently supported. Based on continual learning, the text generation ability of CPM2.1 is greatly improved compared to CPM2.
• Convenient Deployment. Using BMInf, it will be fast and convenient to develop interesting downstream applications.

Demo

Documentation

Our documentation provides more information about the package.

Install

From pip: pip install bminf

From source code: download the package and run python setup.py install

From docker: docker run -it --gpus 1 -v $HOME/.cache/bigmodels:/root/.cache/bigmodels --rm openbmb/bminf python3 examples/fill_blank.py

Here we list the minimum and recommended configurations for running BMInf.

Quick Start

Here we provide a simple script for using BMInf.

Firstly, import a model from the model base (e.g. CPM1, CPM2, EVA).

import bminf
cpm2 = bminf.models.CPM2()

Then define the text and use the <span> token to denote the blank to fill in.

text = "北京环球度假区相关负责人介绍，北京环球影城指定单日门票将采用<span>制度，即推出淡季日、平季日、旺季日和特定日门票。<span>价格为418元，<span>价格为528元，<span>价格为638元，<span>价格为<span>元。北京环球度假区将提供90天滚动价格日历，以方便游客提前规划行程。"

Use the fill_blank function to obtain the results and replace <span> tokens with the results.

for result in cpm2.fill_blank(text, 
    top_p=1.0,
    top_n=10, 
    temperature=0.9,
    frequency_penalty=0,
    presence_penalty=0
):
    value = result["text"]
    text = text.replace("<span>", "\033[0;32m" + value + "\033[0m", 1)
print(text)

Finally, you can get the predicted text. For more examples, go to the examples folder.

Supported Models

BMInf currently supports these models:

CPM2.1. CPM2.1 is an upgraded version of CPM2 [1], which is a general Chinese pre-trained language model with 11 billion parameters. Based on CPM2, CPM2.1 introduces a generative pre-training task and was trained via the continual learning paradigm. In experiments, CPM2.1 has a better generation ability than CPM2.

CPM1. CPM1 [2] is a generative Chinese pre-trained language model with 2.6 billion parameters. The architecture of CPM1 is similar to GPT [4] and it can be used in various NLP tasks such as conversation, essay generation, cloze test, and language understanding.

EVA. EVA [3] is a Chinese pre-trained dialogue model with 2.8 billion parameters. EVA performs well on many dialogue tasks, especially in the multi-turn interaction of human-bot conversations.

Besides these models, we are now working on adding more PLMs especially large-scale PLMs. We welcome every contributor to add their models to this project by proposing an issue.

Performances

Here we report the speeds of CPM2 encoder and decoder we have tested on different platforms. You can also run benchmark/cpm2/encoder.py and benchmark/cpm2/decoder.py to test the speed on your machine!

GitHub - OpenBMB/BMInf: Low-resource Inference Package for Large Pretrained Language Models (PLMs)

Low-resource Inference Package for Large Pretrained Language Models (PLMs) - GitHub - OpenBMB/BMInf: Low-resource Inference Package for Large Pretrained Language Models (PLMs)

GitHubOpenBMB