MM-Locate-News: Multimodal Focus Location Estimation in News

This is the official GitHub page for the paper (link here):

Golsa Tahmasebzadeh, Eric Müller-Budack, Sherzod Hakimov, Ralph Ewerth: "MM-Locate-News: Multimodal Focus Location Estimation in News". In: 29th International Conference on Multimedia Modeling (MMM), Bergen, Springer, 2023, 204–216.

Installation

git clone https://github.com/golsa-tahmasebzadeh/mm-locate-news.git
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Download Data

Download the data from here and put in the root directory.

Download Checkpoints

For mm-locate-news dataset download the trained models from here and extract in experiments/snapshots. For BreakingNews dataset download the trained models from here and extract in breakingnews/experiments/snapshots.

Evaluation

To evaluate the models based on mm-locate-news data:

python evaluate.py --model_name <MODELNAME> --test_check_point <CHECKPOINT>

To evaluate the models based on Breakingnews data:

python breakingnews/bn_evaluate.py --model_name <MODELNAME> --test_check_point <CHECKPOINT>

To evaluate Cliff-clavin:

python Cliff-clavin/evaluate_cliff.py

To evaluate Mordecai:

python Mordecai/evaluate_mordecai.py

To evaluate ISN:

python ISN/evaluate_ISN.py

Training

To train the models based on mm-locate-news dataset:

python train.py \
--model_name <MODELNAME> \
--resume <CHECKPOINT> \
--freeze_image <[True, False]> \
--freeze_text <[True, False]>

To train the models based on BreakingNews dataset:

python breakingnews/bn_train.py \
--model_name <MODELNAME> \
--resume <CHECKPOINT> 

Inference

To get the output predictions for an input image and text pair download this and put it in inference folder.

python -m spacy download en_core_web_sm
python inference/predict.py --test_image_path <IMAGEPATH> --text_input_path <TEXTPATH>

Citation

@inproceedings{DBLP:conf/mmm/TahmasebzadehMHE23,
  author       = {Golsa Tahmasebzadeh and
                  Eric M{\"{u}}ller{-}Budack and
                  Sherzod Hakimov and
                  Ralph Ewerth},
  title        = {MM-Locate-News: Multimodal Focus Location Estimation in News},
  booktitle    = {MultiMedia Modeling - 29th International Conference, {MMM} 2023, Bergen,
                  Norway, January 9-12, 2023, Proceedings, Part {I}},
  series       = {Lecture Notes in Computer Science},
  volume       = {13833},
  pages        = {204--216},
  publisher    = {Springer},
  year         = {2023},
  url          = {https://doi.org/10.1007/978-3-031-27077-2\_16},
  doi          = {10.1007/978-3-031-27077-2\_16}
}