Flexible inputs in template repository for Algorithm containers

[kiranvaidhya]

I was looking at the code for creating an algorithm container. The Algorithm class has a _load_input_image function which is constrained to load only one input image. Perhaps we could add a new function that accepts flexible inputs?

evalutils/evalutils/evalutils.py

Lines 201 to 217 in dbfabbd

	def _load_input_image(self, *, case) -> Tuple[SimpleITK.Image, Path]:
	input_image_file_path = case["path"]
	input_image_file_loader = self._file_loaders["input_image"]
	if not isinstance(input_image_file_loader, ImageLoader):
	raise RuntimeError(
	"The used FileLoader was not of subclass ImageLoader"
	)
	# Load the image for this case
	input_image = input_image_file_loader.load_image(input_image_file_path)
	# Check that it is the expected image
	if input_image_file_loader.hash_image(input_image) != case["hash"]:
	raise RuntimeError("Image hashes do not match")
	return input_image, input_image_file_path

I need an update for my own algorithm container, but I can also imagine that other high-profile challenges will need evalutils to handle flexible inputs for algorithm containers. @jmsmkn, @silvandeleemput if you have any specific ideas for this, I can make a pull request for this feature.

[silvandeleemput]

To do this we probably need to rework/rethink how the Algorithm class works. It also relates to #246, where we would like to adhere to the input/output algorithm interfaces here: https://grand-challenge.org/algorithms/interfaces/ .

It might be a good idea to allow a user to specify expected interfaces for the inputs and outputs, so we know where to look for the inputs and where to put the respective outputs.

e.g.:

evalutils init algorithm testregistration
What kind of algorithm is this? (Classification, Segmentation, Detection): S
Number of inputs? [1]: 2
For a list of available interfaces see: https://grand-challenge.org/algorithms/interfaces/
Interface for first input? ["Generic Medical Image"]: Fixed Image
Interface for second input? ["Generic Overlay"]: Moving Image
Number of outputs? [1]: 1
Interface for first output? ["Generic Medical Image"]: Warped Image
All the usual...

This way the Algorithm class will know where to look/expect input files, it will know what default validation to use. Also, outputs will be written to default locations.

We probably lose the ability to process multiple input files in a single run, but for grand-challenge this should be alright.

[jmsmkn]

Rather than asking all the questions we can expose the interface on the grand challenge side, ask the user for their API key, and then say, "What algorithm/challenge" do you want to create an algorithm for? Get all the interfaces for that algorithm, then create the template.

We should reduce the number of options they have to fill out as most of them are unused.

You can process multiple inputs by changing the base from /input/ to /input/<job_uuid>/ and /output/ to /output/<output_uuid>/, then everything else unchanged. We plan on adding support for that in the near future.

[silvandeleemput]

Rather than asking all the questions we can expose the interface on the grand challenge side, ask the user for their API key, and then say, "What algorithm/challenge" do you want to create an algorithm for? Get all the interfaces for that algorithm, then create the template.

I see, so you would like people to use grand-challenge to configuring/designing the interfaces to the algorithm/evaluation and subsequently query the interfaces for that algorithm (using gc-api) inside evalutils.

We should reduce the number of options they have to fill out as most of them are unused.

That's true, are you referring to all the algorithm-related settings, that are currently unused? E.g. required-number of CPUs etc... Should we make a separate issue for this?

You can process multiple inputs by changing the base from /input/ to /input/<job_uuid>/ and /output/ to /output/<output_uuid>/, then everything else unchanged. We plan on adding support for that in the near future.

That's great! For now, we can move forward to support the single case I guess, and later add support for multiple inputs.

[jmsmkn]

No need for a separate issue, can be tackled as part of a clean up here when they're redefined.