Short Text Mining in Python
This package shorttext is a Python package that facilitates supervised and unsupervised learning for short text categorization. Due to the sparseness of words and the lack of information carried in the short texts themselves, an intermediate representation of the texts and documents are needed before they are put into any classification algorithm. In this package, it facilitates various types of these representations, including topic modeling and word-embedding algorithms.
Since release 1.5.2, it runs on Python 3.9. Since release 1.5.0, support for Python 3.6 was decommissioned. Since release 1.2.4, it runs on Python 3.8. Since release 1.2.3, support for Python 3.5 was decommissioned. Since release 1.1.7, support for Python 2.7 was decommissioned. Since release 1.0.8, it runs on Python 3.7 with 'TensorFlow' being the backend for keras. Since release 1.0.7, it runs on Python 3.7 as well, but the backend for keras cannot be TensorFlow. Since release 1.0.0, shorttext runs on Python 2.7, 3.5, and 3.6.
- example data provided (including subject keywords and NIH RePORT);
- text preprocessing;
- pre-trained word-embedding support;
gensimtopic models (LDA, LSI, Random Projections) and autoencoder;
- topic model representation supported for supervised learning using
- cosine distance classification;
- neural network classification (including ConvNet, and C-LSTM);
- maximum entropy classification;
- metrics of phrases differences, including soft Jaccard score (using Damerau-Levenshtein distance), and Word Mover's distance (WMD);
- character-level sequence-to-sequence (seq2seq) learning;
- spell correction;
- API for word-embedding algorithm for one-time loading; and
- Sentence encodings and similarities based on BERT.
Documentation and tutorials for
shorttext can be found here: http://shorttext.rtfd.io/.
To install it, in a console, use
>>> pip install -U shorttext
or, if you want the most recent development version on Github, type
>>> pip install -U git+https://github.com/stephenhky/[email protected]
Developers are advised to make sure
Keras >=2 be installed. Users are advised to install the backend
Tensorflow (preferred) or
Theano in advance. It is desirable if
Cython has been previously installed too.
See installation guide for more details.