Question about data processing in pretrain data.

[JJJYmmm]

Why the following actions were taken. Is there anything special about cc12m I missed?

OFA/data/pretrain_data/unify_dataset.py

Lines 321 to 323 in a36b91c

	if type == 'caption' and dataset_name == 'cc12m':
	target_item[:2] = self.src_dict.pad()
	target_item[-1] = self.eos_item

Looking forward to your reply.

[JJJYmmm]

Another question.
When computing the loss AdjustLabelSmoothedCrossEntropyCriterion, sample_patch_num is added into the model input(sample[0], which I think is correspond to sample_v1, the vision-language data)

OFA/criterions/label_smoothed_cross_entropy.py

Lines 177 to 178 in a36b91c

	if self.sample_patch_num > 0:
	sample[0]['net_input']['sample_patch_num'] = self.sample_patch_num

It seems that sample_patch_num can select fixed number of image features. So why it's just used in VL data?

OFA/models/ofa/unify_transformer.py

Lines 759 to 769 in a36b91c

	if sample_patch_num is not None:
	patch_orders = [
	random.sample(range(image_num_patches), k=sample_patch_num)
	for _ in range(patch_images.size(0))
	]
	patch_orders = torch.LongTensor(patch_orders).to(device)
	image_embed = image_embed.gather(1, patch_orders.unsqueeze(
	2).expand(-1, -1, image_embed.size(2)))
	image_num_patches = sample_patch_num
	image_padding_mask = image_padding_mask.gather(1, patch_orders)
	image_position_ids = image_position_ids.gather(1, patch_orders)