Aspect-Based-Sentiment-Analysis: Transformer & Explainable ML

Aspect Based Sentiment Analysis

The task is to classify the sentiment of potentially long texts for several aspects. In building this package, we focus on two things. Firstly, the package works as a service. It can be freely adjusted and extended to your needs. It is standalone and scalable. You just benefit from the fine-tuned State of the Art models. Secondly, we wish to explain model decisions, so you can infer how much predictions are reliable. We desire to provide the robust and stable ML package.

There are over 100 repositories on GitHub around this problem 1 2 3 4 5 6 7 8 9 . All of them are hard to commercialize and reuse open-sourced research projects. Their purpose is to turn the evaluation score up. It is hard to go through the entire process and reproduce results from scratch. The pre/post-processing is divided into stages that makes it hard to use. Last but not least, there is no/little effort to understand model reasoning. We try to clean this excellent research up. Please give a star if you like the project. This is important to keep this project alive.

Quick Start

To start, use the fine-tuned model.
The load function downloads a model into the package directory,
sets up the Tensorflow model and returns the ready-to-use pipeline.
The pipeline nlp wraps the model and keeps non-differential pre/post-processing needed
to make a prediction and to interpret results.
Please take a look at the details here.

import aspect_based_sentiment_analysis as absa

nlp = absa.load()
text = ("We are great fans of Slack, but we wish the subscriptions "
        "were more accessible to small startups.")

slack, price = nlp(text, aspects=['slack', 'price'])
assert price.sentiment == absa.Sentiment.negative
assert slack.sentiment == absa.Sentiment.positive

Now, we wish to infer how much predictions are reliable.
In our task, we are curious about two things at most.
Firstly, we want to be sure that the model connects the correct
word or words with the aspect. If the model does it wrong, the sentiment
concerns the different entity. Secondly, even if the model recognized
the aspect correctly, we need to understand the model reasoning better.
To do so, we wish to discovers patterns, a weighted sequence
of words, and their approximated impact to the prediction. We want to
avoid a situation wherein a single word or weird word combination
triggers the model.

# Verify the model decision
html = absa.probing.explain(slack)
display(html)

Here, we have two things.
Firstly, we see the model's definition of the "slack" aspect.
The model pays attention to the word "slack" correctly.
Nonetheless, we need to understand that
the sentiment rather concerns the whole collocation "great fans of slack".
Secondly, we go through patterns that impact on a prediction.
We can cluster patterns into two groups.
The patterns in the first group (marked as green) support a decision
(in this case, push towards the positive sentiment).
Basically, they represent different combinations of weighted words: great, fans, slack.
The patterns in the second group (marked as red) disagree with a decision.
Interestingly, the model recognizes the complex structure like "wish ... more".
Please note that this analysis is a rough approximation.
Take a look at the details here.

Ready-to-Use Models

In the table, we present the state of the art results on the most common evaluation dataset
(SemEval 2014 Task 4 SubTask 2, details here).
The project assumption is to use the published architectures
(even if I was tempted to do my own).
We recommend bert-ada for its simplicity (default).
Take a look at the our model implementation details here.

There are two available models for the restaurant and laptop domains.
The hyper-parameters optimization with the explanation how to train a model is here.
You can easily reproduce our evaluations.
Look at the performance tests here.

Installation

You can use the pip:

pip install aspect-based-sentiment-analysis

Otherwise, clone the code and create the new environment via
conda:

git clone [email protected]:ScalaConsultants/Aspect-Based-Sentiment-Analysis.git
conda env create -f=environment.yml
conda activate Aspect-Based-Sentiment-Analysis

Further Research

Even the task is narrow and well-defined, there is still massive work to do.
Below we present our few open research issues.
We encourage you to help us to improve this package.

• Provide a concrete confidence measure.
• Build the separated model which correlates patterns and linguistics dependencies.
• Process several aspects at once.
• Adapt models across different domains.
• Proper model calibrations (neutral).
• Robust evaluations (adversarial attacks).
• Distill the model (compress the tuned model).
• More interactive visualization tools.

References

How to use BERT for the Aspect-Based Sentiment Analysis:

• Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence (NAACL 2019)
  [code][paper]
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis (NAACL 2019)
  [code][paper]
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis
  [code][paper]

Introduction to the BERT interpretability:

• Are Sixteen Heads Really Better than One?
  [code][paper]
• A Primer in BERTology: What we know about how BERT works
  [paper]
• What Does BERT Look At? An Analysis of BERT's Attention
  [code][paper]
• Visualizing and Measuring the Geometry of BERT
  [code][paper]
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
  [paper]
• Adversarial Training for Aspect-Based Sentiment Analysis with BERT
  [paper]
• Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT
  [paper]
• exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models
  [code][paper]
• Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings
  [code][paper]
• Attention is not Explanation
  [code][paper]
• Attention is not not Explanation
  [code][paper][blog post]
• Hierarchical interpretations for neural network predictions
  [code][paper]
• Analysis Methods in Neural NLP
  [code][paper]
• Visualization for Sequential Neural Networks with Attention
  [code]
• NeuroX: Toolkit for finding and analyzing important neurons in neural networks
  [code][paper]

The State of the Art results:

• A Multi-task Learning Model for Chinese-oriented Aspect Polarity Classification and Aspect Term Extraction
  [code][paper]
• Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification
  [code][paper]
• Adversarial Training for Aspect-Based Sentiment Analysis with BERT
  [code][paper]

Other interesting:

• Multi-Dimensional Explanation of Ratings from Reviews
  [paper]
• Extracting Syntactic Trees from Transformer Encoder Self-Attentions
  [paper]
• Master Thesis: Transfer and Multitask Learning for Aspect-Based Sentiment Analysis Using the Google Transformer Architecture
  [code]
• Create interactive textual heat maps for Jupiter notebooks
  [code]
• A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc
  [code]
• More you can find here.

GitHub