Skip to content
/ VEC Public

Visual and Embodied Concepts evaluation benchmark

Notifications You must be signed in to change notification settings

TobiasLee/VEC

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 

Repository files navigation

VEC

Visual and Embodied Concepts evaluation benchmark. EMNLP 2023 Main Conference.

For shape, material and color task, there is an object and corresponding two options for the given property, e.g., chair is made of wood instead of jade.

For mass, temperature, hardness, size and height task, the pair is consists of two items and a relation label, e.g., red lego brick is more light-weight than a hammer

You can easily load the dataset from Huggingface Datasets API:

from datasets import load_dataset


data = {}

for task in ['color', 'size', 'shape', 'height', 'material', 'mass', 'temperature', 'hardness']:
    data[task] = load_dataset("tobiaslee/VEC", task)


print(data['material']['test'][0])
# instance
# {'obj': 'chair', 'positive': 'wood', 'negative': 'jade', 'relation': 'material'}
# meaning:  chair is made of 

print(data['mass']['test'][0])
# {'obj1': 'red lego brick', 'obj2': 'hammer', 'relation': 'mass', 'label': 0} 
# meaning:  red logo brick is more light-weight than hammer.
# label=0 indicates`<`  while label=1 indicates `>` 

Citation

If you found this benchmark, please kindly cite our paper:

@article{li2023vec,
  title={Can Language Models Understand Physical Concepts?},
  author={Li, Lei and Xu, Jingjing and Dong, Qingxiu and Zheng, Ce and Liu, Qi and Kong, Lingpeng and Sun, Xu},
  journal={arXiv preprint arXiv:2305.14057},
  year={2023}
}

About

Visual and Embodied Concepts evaluation benchmark

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published