/ Machine Learning

Pre-training of Deep Bidirectional Transformers for Language Understanding

Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Simple implementation of BERT (TensorFlow 2.0 style) https://arxiv.org/abs/1810.04805

ToDo

  • mecab tokenizer
  • Multiprocessing (for preprocessing NSP data in parallel)

Requirements

pip install sentencePiece

Future Work

  • GPT2

GitHub