Loading multiple models into memory at the same time

[j6mes]

Hi (making this a github issue by Daniil's request)

I've been working on making an interactive version of this system so that it can be queried via a web API. I'm having some issues with the pipeline as the sentence ensemble currently only loads one model into memory at a time which means that querying my API takes about 20 minutes to swap between all the models.

I've been working in this file: https://github.com/j6mes/fever-athene-system/blob/master/src/athene/system.py

Is it possible to load all the models into memory at once? I get issues with tensorflow variable scoping which makes it difficult to do: I was thinking of doing something very simple with the following changes:

Original

    selection_model = SentenceESIM(h_max_length=sargs.c_max_length, s_max_length=sargs.s_max_length, learning_rate=sargs.learning_rate,
                       batch_size=sargs.batch_size, num_epoch=sargs.num_epoch, model_store_dir=sargs.sentence_model,
                       embedding=sentence_loader.embed, word_dict=sentence_loader.word_dict, dropout_rate=sargs.dropout_rate,
                       num_units=sargs.num_lstm_units, share_rnn=False, activation=tf.nn.tanh)

Changed

    selection_models = [SentenceESIM(h_max_length=sargs.c_max_length, s_max_length=sargs.s_max_length, learning_rate=sargs.learning_rate,
                       batch_size=sargs.batch_size, num_epoch=sargs.num_epoch, model_store_dir=sargs.sentence_model,
                       embedding=sentence_loader.embed, word_dict=sentence_loader.word_dict, dropout_rate=sargs.dropout_rate,
                       num_units=sargs.num_lstm_units, share_rnn=False, activation=tf.nn.tanh)] * args.num_model

    for i, model in enumerate(selection_models):
            logger.info("Restore sentence model {}".format(i))
            model_store_path = os.path.join(args.sentence_model, "model{}".format(i + 1))
            selection_model.restore_model(os.path.join(model_store_path, "best_model.ckpt"))