Semantic Encoding of Clarify Visual Concepts + Fasttext Embeddings in ECoG

Repository contains code for the results described in the paper: Berezutskaya, J., Freudenburg, Z. V., Ambrogioni, L., Güçlü, U., van Gerven, M. A., & Ramsey, N. F. (2020). Cortical network responses map onto data-driven features that capture visual semantics of movie fragments. Scientific reports, 10(1), 1-21.

Content

The repository contains code for

• extraction of the visual concepts using Clarifai
• extraction of the Fasttext word embeddings per frame based on the visual concepts
• interpretation and visualization of the extracted concepts
• linear regression models to fit the extracted concepts to the ECoG responses
• affinity propagation clustering of the beta-weights of the regression for intepretation of the neural tuning profiles
• control linear models using low-level visual feature sets and binary labels of the visual concepts

Dependencies

• Scikit-learn
• Numpy
• Scipy
• Clarifai API
• Statsmodels

Originally written in Python 2.7, Anaconda release

Citation

If this code has been helpful to you, please cite the related paper:

@article{berezutskaya2020cortical,
  title={Cortical network responses map onto data-driven features that capture visual semantics of movie fragments},
  author={Berezutskaya, Julia and Freudenburg, Zachary V and Ambrogioni, Luca and G{\"u}{\c{c}}l{\"u}, Umut and van Gerven, Marcel AJ and Ramsey, Nick F},
  journal={Scientific reports},
  volume={10},
  number={1},
  pages={1--21},
  year={2020},
  publisher={Nature Publishing Group}
}

Visual summary