LIREx: Augmenting Language Inference with Relevant Explanation

I am interested in artificial intelligence. Last year I took several free courses for it on EDX.org such as the professional certificate in data science and CS50 introduction to artificial intelligence. It really is an unlimited topic and I stumbled across this great site that has academic papers in all subjects. A quick search of artificial intelligence on the site brings up new artificial posted daily. I want to push myself since my lucky accident and have decided to read and write about one of these papers as often as I can. I will try to sum up the main points of this paper as best I can. I will try to link to some resources I used to try to understand. Most of the following is selections from the paper that I felt summed up the main points.

Today’s paper is LIREx: Augmenting Language Inference with Relevant Explanation by Xinyan Zhao and V.G.Vinod Vydiswaran from the University of Michigan.

Abstract

Natural language explanations (NLEs) are a special form of data annotation in which annotators identify rationales (most significant text tokens) when assigning labels to data instances, and write out explanations for the labels in natural language based on the rationales. NLEs have been shown to capture human reasoning better, but not as beneficial for natural language inference (NLI). In this paper, we analyze two primary flaws in the way NLEs are currently used to train explanation generators for language inference tasks. We find that the explanation generators do not take into account the variability inherent in human explanation of labels, and that the current explanation generation models generate spurious explanations. To overcome these limitations, we propose a novel framework, LIREx, that incorporates both a rationale- enabled explanation generator and an instance selector to select only relevant, plausible NLEs to augment NLI models. When evaluated on the standardized SNLI data set, LIREx achieved an accuracy of 91.87%, an improvement of 0.32 over the baseline and matching the best-reported performance on the data set. It also achieves significantly better performance than previous studies when transferred to the out-of- domain MultiNLI data set. Qualitative analysis shows that LIREx generates flexible, faithful, and relevant NLEs that allow the model to be more robust to spurious explanations. The code is available at https://github.com/zhaoxy92/LIREx.

Definitions of Acronyms

Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data.

Natural language inference (NPI) is the task of determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”.

Natural language explanations (NLEs) are a special form of data annotation in which annotators identify rationales (most significant text tokens) when assigning labels to data instances, and write out explanations for the labels in natural language based on the rationales.

Language Inference with Relevant Explanation (LIREx) a novel framework that incorporates both a rationale-enabled explanation generator and an instance selector to select only relevant, plausible NLEs to augment NLI models.

Stanford Natural Language Inference (SNLI) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailmentcontradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE).

 MultiGenre NLI (MultiNLI) Corpus: Like SNLI (and just as big), but with a more diverse variety of text styles and topics.

Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. It can not only process single data points (such as images), but also entire sequences of data (such as speech or video).

Bidirectional Encoder Representations from Transformers (BERT) : an AI language model that Google now applies to search results and featured snippets,  a new technique for NLP pre-training 

RoBERTa: A Robustly Optimized BERT Pretraining Approach : iterates on BERT’s pretraining procedure, including training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and dynamically changing the masking pattern applied to the training data.

Generative Pretrained Transformer 2 (GPT-2) is a pre-trained language model which we can use for various NLP tasks, such as: Text generation. Language translation. Building question-answering systems, and so on

NILE : Natural Language Inference with Faithful Natural Language Explanations – a novel NLI method which utilizes auto-generated label-specific NL explanations to produce labels along with its faithful explanation

Summery

Natural Language Explanations are a special type of annotation for data that identify rationales or most significant text tokens for a data instance. An example would be the analysis of the premise : A man wearing a red uniform and helmet stands on his motorbike, with the hypothesis : A man sitting in a car would generate the explanation : A man cannot be both standing and sitting, and : A motorbike is not a car.

They have been suggested to potentially improve the performance and interpretability of deep learning-based models – i.e. ei- ther augmenting model performance by incorporating NLEs as additional contextual features, or explaining model decisions by training an explanation generator.

The major issue with these are that:

(1) there is a lack of rationale in NLE generation: Current approaches for explanation generation produce only one specific explanation for each data instance. However, these approaches ignore the variability in human reasoning and alternative explanations.

(2) Inclusion of spurious explanations: This happens because while deep learning-based text generators are powerful enough to generate readable sentences, they often lack commonsense reasoning ability

The proposed solution to these problems is:

Language Inference with Relevant Explanations (LIREx). LIREx augments the NLI model with relevant, plausible NLEs produced and selected by a rationale-enabled explanation generator and an instance selector.

Augmenting NLI with Relevant Explanations

Given a premise-hypothesis (P-H) pair, a label-aware rationalizer predicts rationales by taking as input a triplet (P, H, x; x ∈ {entail, neutral, contradict}) and outputs a rationalized P-H pair, (P, Hx). Next, the NLE generator generates expla- nations (Ex) for each rationalized P-H pair. Then, the expla- nations are combined with the original P-H pair as input to the instance selector and inference model to predict the final label.

Instance Selector and Inference

When a P-H pair and the generated explanations are fed into an inference model, the model benefits from the addition of the explanations when they are correct. On the other hand, incorrect explanations lead to large uncertainty during the inference process. So, we first select a single plausible explanation for the final inference. To achieve this, we develop a simple strategy assuming that when the labels are correct, the NLEs generated based on the corresponding enabled rationales are the correct explanations. This allows us to only estimate which NLE is generated by the gold label-enabled rationale. To further simplify this task, if we assume that the gold label-enabled explanation is more likely to be plausible than the other two explanations, we could identify the gold label-enabled explanation by accurately predicting the correct label for the standard NLI task. In other words, a good prediction on the NLE selection task can be achieved by just training a standard NLI classification model.

Conclusion

In this work they identified two flaws in the current strategy of using NLEs for the NLI task. To overcome these limitations, they proposed a novel framework, called LIREx, that incorporates both a rationale-enabled explanation generator and an instance selector to augment NLI models with only relevant, plausible NLEs.

I hope you found this as interesting as I did. This was my first attempt at this but I hope to continue to read more papers and try my best to create a summery like today. I find that trying to explain what you learn is a good way to solidify what you learn.

Leave a Reply

Scroll to Top