Small forewarning, this is still much more of a research codebase than a
library. No support is provided.
If you use this code for your research, please cite the paper.
Introduction
ActiveQA is an agent that transforms questions online in order to find the best
answers. The agent consists of a Tensorflow model that reformulates questions
and an Answer Selection model. It interacts with an environment that contains
a question-answering system. The agent queries the environment with variants
of a question and calculates a score for the answer against the original
question. The model is trained end-to-end using reinforcement learning.
This version addresses the SearchQA
question-answering task, and the environment consists of the Bi-directional
Attention Flow (BiDAF) model of
Seo et al. (2017).
Setup
Dependencies
We require tensorflow and many other supporting libraries. Tensorflow should be
installed separately following the docs. To install the other dependencies use
pip install -r requirements.txt
Note: We only ran this code with Python 2, so Python 3 is not officially
supported.
Data
Download the source dataset from SearchQA,
GloVe, and NLTK corpus and save
them in $HOME/data.
export DATA_DIR=$HOME/data
mkdir $DATA_DIR
Download
Download the SearchQA dataset (~600 MB) for training, testing, and validation
here: https://drive.google.com/open?id=1OxRhw81g7amW3aBd_iu2By5THysgr2uv
<Download the dataset to $DATA_DIR/SearchQA.zip>
unzip $DATA_DIR/SearchQA.zip -d $DATA_DIR
Note that Python2 and Python3 handle Unicode differently and hence the
preprocessing output differs. For converting the SearchQA format to SQuAD format
either version can be used; use Python3 for other datasets.
gRPC
We need to compile the gRPC interface for the Environment Server.
chmod +x compile_protos.sh; ./compile_protos.sh
Run Environment Server
The training requires running the environment gRPC
server, which receives queries from the ActiveQA agent and sends back one
response per query.
The checkpoint of a BiDAF model trained on SearchQA is already provided in
data/bidaf, so you don't have to train one yourself. However, if you want to
reproduce our training, clone the
BiDAF repository and run
We first train reformulator from a model pretrained on UN and Paralex datasets.
It should take a week on a single P100 GPU to reach ~42 F1 score on SearchQA's
dev set.
Note: if you don't want to wait a week of training, you can download this
checkpoint of the reformulator
trained on SearchQA, with dev set F1 score of 42.5. Note that this is not
the exact model analyzed in the paper, but one with equivalent performance.
Selector Training
After training the reformulator, we can now train the selector. It should take
2-3 days on a single P100 GPU to reach ~47.5 F1 score on SearchQA's dev set.
Note: If you don't want to wait 2-3 days for the training to finish, you can
download a checkpoint of the selector.
The checkpoint is trained on SearchQA, achieving an F1 score of ~47.5 on the dev
set.
References
This repository relies on the work of the following repositories:
@inproceedings{buck18,
author = {Christian Buck and
Jannis Bulian and
Massimiliano Ciaramita and
Andrea Gesmundo and
Neil Houlsby and
Wojciech Gajewski and
Wei Wang},
title = {Ask the Right Questions: Active Question Reformulation with Reinforcement
Learning},
booktitle = {Sixth International Conference on Learning Representations (ICLR)},
year = {2018},
month = {May},
address = {Vancouver, Canada},
url = {https://openreview.net/forum?id=S1CChZ-CZ},
}
Ecosystem Role
Standard MoltPulse indexed agent.
Embed Badge
Show off your Pulse Score in your GitHub README to build trust and rank higher.