Also, the Target path is ignored.

I did a bit of digging for this.

When a AskBidRequest is received by a compute node, we run bidding for the job. The interesting thing is when computing the disk usage in DiskUsageCalculator, we do not take into consideration the parsedResources.

Ideally the totalDiskRequirements (obtained by computing the size from StorageProvider) should match the parsedUsage resources. If they are not the same the compute node should ideally reject the job request.

@wdbaruni what do you think ?

I'd recommend exploring the StorageProvider implementation used for fetching URL storage sources: https://github.com/bacalhau-project/bacalhau/blob/main/pkg/storage/url/urldownload/storage.go#L79.

It's worth noting that the size of the content may not always be readily available or accurately reported by the server - as the comment in the above link suggests.

One way we can (try to) mitigate this is by aborting downloads that exceed the content length specified by the server.

Indeed completely agree to you.

However, what I was pointing to was that there are no checks overall. Lets say you replace the urldownload type to s3 . Now even in that case the job would pass right ? As ideally there should be a comparison of actual input size and one provided in the job spec.

I may be wrong about this. But i don't see any such checks so. If you feel there are such checks could you please point me to it ? Also for s3 we are successfully fetching the size in all scenarios.

func (s *StorageProvider) GetVolumeSize(ctx context.Context, volume models.InputSource) (uint64, error) {
	ctx, cancel := context.WithTimeout(ctx, config.GetVolumeSizeRequestTimeout())
	defer cancel()

	source, err := s3helper.DecodeSourceSpec(volume.Source)
	if err != nil {
		return 0, err
	}

	client := s.clientProvider.GetClient(source.Endpoint, source.Region)
	objects, err := s.explodeKey(ctx, client, source)
	if err != nil {
		return 0, err
	}
	size := uint64(0)
	for _, object := range objects {
		size += uint64(object.size)
	}
	return size, nil
}

Thank you for clarifying offline. As we discussed. The checks are happening via the interface ShouldBidBasedOnUsage. So the problem at hand is to update the urldownload storage provider, to return the size of the download, which may not be readily available.

Will update once i have an approach for it.