An open-source toolkit built on top of PyTorch and is developed

Glyce

Glyce is an open-source toolkit built on top of PyTorch and is developed by Shannon.AI.

What is Glyce ?

Glyce is a Chinese char representation based on Chinese glyph information. Glyce Chinese char embeddings are composed by two parts: (1) glyph-embeddings and (2) char-ID embeddings. The two parts are combined using concatenation, a highway network or a fully connected layer. Glyce word embeddings are also composed by two parts: (1) word glyph-embeddings and (2) word-ID embeddings. Glyph word embedding is the output of forwarding glyph embeddings for each char in the word through the max-pooling layer. Then glyph word embedding combine word-ID embedding using concatenation to get glyce word embedding.

Here are the some highlights in glyce:

1. Utilize Chinese Logographic Information

Glyce utilizes useful logographic information by encoding images of historical and contemporary scripts, along with the scripts of different writing styles.

2. Combine Glyce with Chinese Pre-trained BERT Model

We combine Glyce with Pre-trained Chinese BERT model and adopt specific layer to downstream tasks. The Glyce-BERT model outperforms BERT and sets new SOTA results for tagging (NER, CWS, POS), sentence pair classification, single sentence classification tasks.

3. Propose Tianzige-CNN(田字格) to Model Chinese Char

We propose the Tianzige-CNN(田字格) structure, which is tailored to Chinese character modeling. Tianzige-CNN(田字格) tackle the issue of small number of Chinese characters and small sacle of image compared to Imagenet.

4. Auxiliary Task Performs As a Regularizer

During training process, image-classification loss performs as an auxiliary training objective with the purpose of preventing overfitting and promiting the model's ability to generalize.

Experiment Results

1. Sequence Labeling Tasks

Named Entity Recognition (NER)

MSRA(Levow, 2006), OntoNotes 4.0(Weischedel et al., 2011), Resume(Zhang et al., 2018).

Chinese Part-Of-Speech Tagging (POS)

CTB5, CTB6, CTB9 and UD1

Chinese Word Segmentation (CWS)

PKU, CITYU, MSR and AS.

2. Sentence Pair Classification

Dataset Description

The BQ corpus, XNLI, LCQMC, NLPCC-DBQA

3.Single Sentence Classification

Dataset Description

ChnSentiCorp, Fudan and Ifeng.

4. Chinese Semantic Role Labeling

Dataset Description

CoNLL-2009

Model Performance

5. Chinese Dependency Parsing

Dataset Description

Chinese Penn TreeBank 5.1. Dataset splits follows (Dozat and Manning, 2016).

Model Performance

Requirements

• Python Version >= 3.6
• GPU, Use NVIDIA TITAN Xp with 12G RAM
• Chinese scripts could be found in Google Drive. Please refer to the description and download scripts files to glyce/glyce/fonts/.

NOTE: Some experimental results are obtained by training on multi-GPU machines. May use DIFFERENT PyTorch versions refer to previous open-source SOTA models. Experiment environment for Glyce-BERT is Python 3.6 and PyTorch 1.10.

Installation

# Clone glyce 
git clone [email protected]:ShannonAI/glyce.git
cd glyce 
python3.6 setup.py develop 

# Install package dependency 
pip install -r requirements.txt

Quick start of Glyce

Usage of Glyce Char/Word Embedding

import torch 
import torch.nn as nn 
from torch.nn import CrossEntropyLoss 


from glyce import GlyceConfig 
from glyce import CharGlyceEmbedding, WordGlyceEmbedding 


# load glyce hyper-params
glyce_config = GlyceConfig()


# if input ids is char-level
glyce_embedding = CharGlyceEmbedding(glyce_config)
# elif input ids is word-level 
glyce_embedding = WordGlyceEmbedding(glyce_config)


# forward input_ids into embedding layer 
glyce_embedding, glyph_loss = glyce_embedding(input_ids)


# utilize image classifiation loss as the auxiliary loss 
glyph_decay = 0.1 
glyph_ratio = 0.01 # the proportion of image classification loss to total loss 
current_epoch = 3 


# compute loss 
loss_fct = CrossEntropyLoss()
task_loss = loss_fct(logits, labels)
total_loss = task_loss * (1 - glyce_ratio) + glyph_ratio * glyph_decay ** (current_epoch+1) * glyph_loss 

Usage of Glyce-BERT

1. Preparation

• Download and unzip BERT-Base, Chinese pretrained model.

• Install PyTorch pretrained bert by pip as follows:

pip install pytorch-pretrained-bert

• Convert TF checkpoint into PyTorch

export BERT_BASE_DIR=/path/to/bert/chinese_L-12_H-768_A-12

pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch \
$BERT_BASE_DIR/bert_model.ckpt \
$BERT_BASE_DIR/bert_config.json \
$BERT_BASE_DIR/pytorch_model.bin

2. Train/Dev/Test DataFormat

• Sequence Labeling Task

模型的输入: [CLS] 我 爱 北 京 天 安 门
模型的输出: ‘我爱北京天安门' 对应的命名实体标签 O  O  B-LOC M-LOC M-LOC M-LOC E-LOC 

Model Input: [CLS] I like Bei Jing Tian An Men
Model Output: labels corresponds to 'I like BeiJing Tian An men' are O  O  B-LOC M-LOC M-LOC M-LOC E-LOC 

• Sentence Pair Classification

模型的输入: [CLS] 我 爱 北 京 天 安 门 [SEP] 北 京 欢 迎 您
模型的输出: [CLS] 位置对应的输出,假如是语意匹配任务则为 0,代表两句话的语意不相同

Model Input: [CLS] I like Bei Jing Tian An Men [SEP] Bei Jing Wel ##come you 
Model Output: the output in [CLS] position. 0 if task sign is semantic matching. It means that the semantic of two sentences are different. 

• Single Sentence Classification

模型的输入: [CLS] 我 爱 北 京 天 安 门
模型的输出: [CLS] 位置对应的输出,假如是情感分类任务则为1,代表情感为积极

Model Input: [CLS] I like Bei Jing Tian An Men. 
Model Output: the output in [CLS] position. 1 if task is sentiment analysis. It means the sentiment polarity is positive. 

3. Start Train and Evaluate Glyce-BERT

• scritps/*_bert.sh are the commands we used to finetune BERT.
• scripts/*_glyce_bert.sh are the commands we used to obtained the results of Glyce-BERT.
• scripts/ctb5_binaffine.sh is the command that we used to reimplement PREVIOUS SOTA result on CTB5 for dependency parsing.
• scripts/ctb5_glyce_binaffine.sh is the command that we used to obtain the SOTA result on CTB5 for dependency parsing.

For example, training command of Glyce-BERT for sentence pair dataset BQ is included in scripts/glyce_bert/bq_glyce_bert.sh.
Start train and evaluate Glyce-BERT on BQ by,

bash scripts/glyce_bert/bq_glyce_bert.sh

Notes:

• repo_path refer to work directory of glyce.
• config_path refer to the path of configuration file.
• bert_model refer to the the directory of the pre-trained Chinese BERT model.
• task_name refer to the task signiature.
• data_dir and output_dir refer to the directories of the "raw data" and "intermediate checkpoints" respectively.

GitHub