Skip to content

Source code for Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling Strategy

License

Notifications You must be signed in to change notification settings

WingsBrokenAngel/Semantics-AssistedVideoCaptioning

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

55 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling Strategy

Github Watchers GitHub stars GitHub forks License

Table of Contents

  1. Description
  2. Dependencies
  3. Manual
  4. Data
  5. Results
    1. Comparison on Youtube2Text
    2. Comparison on MSR-VTT
  6. Results and Data for the Final Version of the Paper
  7. Citation

Description

This repo contains the code of Semantics-Assisted Video Captioning Model, based on the paper "Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling Strategy". It is under review at Frontiers in Robotics and AI.

We propose three ways to improve the video captioning model. First of all, we utilize both spatial features and dynamic spatio-temporal features as inputs for semantic detection network in order to generate meaningful semantic features for videos. Then, we propose a scheduled sampling strategy which gradually transfers the training phase from a teacher guiding manner towards a more self teaching manner. At last, the ordinary logarithm probability loss function is leveraged by sentence length so that short sentence inclination is alleviated. Our model achieves state-of-the-art results on the Youtube2Text dataset and is competitive with the state-of-the-art models on the MSR-VTT dataset.

The overall structure of our model looks like this overall structure. Here is some captions generated by our model. captions


If you need a newer and more powerful model, please refer to Delving-Deeper-into-the-Decoder-for-Video-Captioning.


Dependencies

  • Python3.6
  • TensorFlow 1.13
  • NumPy
  • sklearn
  • pycocoevalcap(Python3)

Manual

  1. Make sure you have installed all the required packages.
  2. Download pycocoevalcap and put it along with msrvttt, msvd, tagging folders.
  3. Download files in the Data section.
  4. cd path_to_directory_of_model; mkdir saves
  5. run_model.sh is used for training models and test_model.sh is used for testing models. Specify the GPU you want to use by modifying CUDA_VISIBLE_DEVICES value. Specify the needed data paths by modifying corpus, ecores, tag and ref values. The words will be sampled by argmax strategy if argmax is 1 and they will be sampled by multinomial strategy if argmax is 0. name is the name which you give to the model. test refers to the path of the saved model which is to be tested. Do not give a parameter to test if you want to train a model.
  6. After completing the configuration of the bash file, then bash run_model.sh for training, bash test_model.sh for testing.

Results

Comparison on Youtube2Text

Model B-4 C M R Overall
LSTM-E 45.3 31.0
h-RNN 49.9 65.8 32.6
aLSTMs 50.8 74.8 33.3
SCN 51.1 77.7 33.5
MTVC 54.5 92.4 36.0 72.8 0.9198
ECO 53.5 85.8 35.0
SibNet 54.2 88.2 34.8 71.7 0.8969
Our Model 61.8 103.0 37.8 76.8 1.0000

Comparison on MSR-VTT

Model B-4 C M R Overall
v2t_navigator 40.8 44.8 28.2 60.9 0.9325
Aalto 39.8 45.7 26.9 59.8 0.9157
VideoLAB 39.1 44.1 27.7 60.6 0.9140
MTVC 40.8 47.1 28.8 60.2 0.9459
CIDEnt-RL 40.5 51.7 28.4 61.4 0.9678
SibNet 40.9 47.5 27.5 60.2 0.9374
HACA 43.4 49.7 29.5 61.8 0.9856
TAMoE 42.2 48.9 29.4 62.0 0.9749
Our Model 43.8 51.4 28.9 62.4 0.9935

Data

MSVD

  • MSVD tag index2word and word2index mappings(ExternalRepo)
    • We use the same word-index mapping in semantic tag to the code in this link.

MSRVTT

  • MSR-VTT Dataset:

    • train_val_test_annotation.zip (GoogleDrive)
      • SHA-256: ce2d97dd82d03e018c6f9ee69c96eb784397d1c83f734fdb8c17aafa5e27da31
    • msr-vtt-v1.part1.rar (GoogleDrive)
      • SHA-256: 3445e0d1bffda3739110dfcf14182b63222731af8a4d7153f0ac09dbec39a0d3
    • msr-vtt-v1.part2.rar (GoogleDrive)
      • SHA-256: b550997526272ab68a42f1bd93315aa2bbb521c71f33d0cb922fbbfb86f15aae
    • msr-vtt-v1.part3.rar (GoogleDrive)
      • SHA-256: debbd0e535e77d9927ffb375299c08990519e22ba7dac542b464b70d440ef515
  • Data and Models for both MSVD and MSR-VTT

    • data.zip
    • SHA-256: fadd721eaa0f13aff7c3505e4784a003514c33ffa5a934a9dcf13955285df11f

ECO

  • Source Code: GitHub.
  • ECO_full_kinetics.caffemodel (GoogleDrive)
    • MD5 31ed18d5eadfd59cb65b7dcdadc310b4
    • SHA-1 b749384d2dac102b8035965566e3030fce465c20

Results and Data for the Final Version of the Paper (Updating)

Results

  1. MSVD Results MSVD Results

  2. MSR-VTT Results MSR-VTT Results

Data and Models

GoogleDrive

  • SHA256: d2a731794ef1bc90c9ccd6c7fe5e92fa7ad104f9e9188ac751c984b23d3a939b

Citation

@ARTICLE{10.3389/frobt.2020.475767,
    AUTHOR={Chen, Haoran and Lin, Ke and Maye, Alexander and Li, Jianmin and Hu, Xiaolin},   
    TITLE={A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling},      
    JOURNAL={Frontiers in Robotics and AI},      
    VOLUME={7},      
    PAGES={129},     
    YEAR={2020},      
    URL={https://www.frontiersin.org/article/10.3389/frobt.2020.475767},       
    DOI={10.3389/frobt.2020.475767},      
    ISSN={2296-9144},   
    ABSTRACT={Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.}
}

About

Source code for Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling Strategy

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published