Skip to content

henriblancke/neural-chat

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Neural Chatbot (seq2seq)

Resources

  1. Sequence to Sequence Learning with Neural Networks
  2. A Neural Conversational Model
  3. A Diversity-Promoting Objective Function for Neural Conversation Models
  4. A Persona-Based Neural Conversation Model

Getting started

Install the requirements

pip install -r requirements.txt

Download train and test data

python s3.py --retrieve

Start training the model

python run.py

Test the trained model

python run.py --test

Save model checkpoints to S3

python s3.py --save

Using beam search

A lot of the research building upon seq2seq to make an AI chatbot uses some approach of beam search. One of the aims of this project was to implement beam search. It is nonetheless still in an experimental phase. The beam search approach in this project is based on the work of Nikita Kitaev.

To use beam search, either set the beam_search flag in the variables.py file to True or run the run.py script with --beam_search=True flag.

Data input format

The input data is split into a training and testing dataset. Both the training and testing dataset contain Q&A or conversational turns. The input file contains a question followed by its response or answer. For simplicity we forget the context of the conversation when looking at conversational turns, there are ways to model conversational context.

Parameters

Tuning the model parameters

Most of the models parameters can be customized in the variables.py file using TensorFlow flags. All flag arguments are optional since reasonable default values are provided. More info below on tuning parameters.

Training parameters:

Flag Description
--learning_rate LEARNING_RATE Learning rate.
--learning_rate_decay_factor LEARNING_RATE_DECAY_FACTOR Learning rate decays by this much.
--max_gradient_norm MAX_GRADIENT_NORM Clip gradients to this norm.
--steps_per_checkpoint STEPS_PER_CHECKPOINT How many training steps to do per checkpoint.

Model architecture:

Flag Description
--batch_size BATCH_SIZE Batch size to use during training.
--size SIZE Size of each model layer.
--num_layers NUM_LAYERS Number of layers in the model.
--vocab_size VOCAB_SIZE Vocabulary size.
--model_type MODEL_TYPE Seq2Seq model type: 'embedding_attention' or 'embedding'
--buckets BUCKETS Implement the model with buckets
--nobuckets
--max_sentence_length MAX_SENTENCE_LENGTH Maximum sentence length for model WITHOUT buckets

Data parameters:

Flag Description
--max_train_data_size MAX_TRAIN_DATA_SIZE Limit on the size of training data (0: no limit).

Directories:

Flag Description
--data_dir DATA_DIR Data directory.
--train_dir TRAIN_DIR Training directory.
--log_dir LOG_DIR Logging directory.

Roadmap

  • Add Tensorboard
  • Add more advanced text pre-processing (see preprocess.py)
  • Add option to use pre-trained embeddings
  • Implement Diversity promotion