Skip to content

Organizer_Codalab competition YAML definition language

Jump to bottom
Adrien Pavão edited this page Mar 17, 2023 · 20 revisions

This page describes all the attributes in the Codalab competition definition language, using YAML. This is used to create configuration files in Codalab competition bundles. If in doubt about YAML, consult the reference card.

Section 1: Competition properties:

  • title: < String between double quotes>
  • description: < String between double quotes>
  • image: <File name, .jpg or .png; competition logo> Why is it called logo in the editor?
  • has_registration: <True/False. If true, organizer must approve participants>
  • force_submission_to_leaderboard: <True/False. If false, participants must submit manually>
  • disallow_leaderboard_modifying: <True/False. If true, submissions cannot be changed>
  • enable_detailed_results: <True/False. True if html file written>
  • end_date: <YYYY-MM-DD. No end date if left blank>
  • admin_names: < Comma separated list of Codalab IDs of co-organizers> Why is it called admins in the editor?
  • html:
    overview: overview.html             # Detailed competition description.
    evaluation: evaluation.html         # Instructions and evaluation metrics.
    terms: terms_and_conditions.html    # Rules and prizes.
    data: data.html                     # Dataset description.
    p1: additionalPage1.html            # Additional pages (optional).
    p2: additionalPage2.html
    p3: additionalPage3.html

Some properties of competitions cannot be specified in the YAML file. However, they are available from the editor:

  • Queue: Queue selected to send execution jobs. New in v1.5.
  • Publicly Available: Checked if the competition is published.
  • Enable medical image viewer: No longer in use in v1.5. Only for medical competitions.
  • Show datasets from yaml: No longer in use in v1.5. Remains for backward compatibility.
  • Reward: Amount of competition prize.
  • Allow teams: Checked if profile-declared team name is used.
  • Enable per submission metadata: If checked, when submitting an entry, competitors will be asked for Team name, Method name, Method description, etc.
  • Allow sharing of public submissions: If checked, participants will be able to share their submissions with others.
  • Enable forum: If checked, participants will be able to make posts to the forum.
  • Anonymous leaderboard: If checked, participants' names will not appear in the leaderboard.
  • Enable Competition level teams: If checked, per competition team registration will be required for participation. New in v1.5.
  • Organizers need to approve the new team: If checked, the organizers will need to approve the teams. New in v1.5.

Section 2: Phase properties

  • phasenumber: < Phase number>
  • label: <String. Label used to identify the phase>
  • description: <String. Short phase description>
  • color: <Tab colors, one of: white, orange, yellow, green, blue, purple>
  • start_date: <YYYY-MM-DD, Phase start (ends when new phase begins)>
  • max_submissions: < Total number of submissions per phase allowed>
  • max_submissions_per_day: < Maximum number of submissions per day>
  • is_scoring_only: <True/False. If true (default) no code submission> (Results Scoring Only is the name in the editor)
  • execution_time_limit: < Execution time limit (in seconds)>
  • public_data: <Zip file. Data downloadable by participants (in Files page)> New in v1.5.
  • input_data: <Zip file. Data fed to submission. No need if is_scoring_only == True>
  • ingestion_program: . New in v1.5.
  • scoring_program: < Zip file. Scoring program>
  • reference_data: < Zip file. Truth values>
  • starting_kit: <Zip file. Sample code downloadable by participants (in Files page)>. New in v1.5.
  • auto_migration: <True /False. First submission = last one of previous phase.>
  • phase_never_ends: <True /False. If checked, the phase does not end at the beginning of the next one.>
  • force_best_submission_to_leaderboard: <True /False. If True, if submission beats old score, put submission on leaderboard>

Some phase properties cannot be specified in the YAML file. However, they are available from the editor:

  • Leaderboard Mode: < Default or Hide Results>. The mode "Hide Results" makes the leaderboard results invisible to non organizers ("private leaderboard").

WIP Storage Features:

  • Delete all submissions except latest and or best: <True /False (false by default). If checked, when a participant submits a successful submission, all submissions from the same phase, except the highest scored, latest, and leaderboard entry will be deleted. This happens regardless of if the new submission beats the old highest scored. This will not be triggered by failed submissions only successful submissions. Successful submissions are prioritized over failed submissions, so it is possible for a user to have a newer failed submission get deleted in favor of keeping an older successful submission. There should always be at least 3 submissions kept (the new submission, the latest, and best scored), but 4 may be kept in some cases such as having a leaderboard entry that is lower scored than a previous submission with a higher score than the leaderboard entry (or they are equal in score), where it will keep all 4. (New submission, latest, best scored, and the leaderboard entry.) Marked with caution, so it is not accidentally enabled causing unwanted deletions.>

  • Max submission size in megabytes: <Int 0-10000, default of 0. When zero this feature is disabled. When not 0, the uploaded submission's zip file size will be checked. If it over the limit specified, the submission will be rejected with an error message detailing the file's size and the maximum allowed size.>

  • Max megabyte usage for each participant.: <Int 0-500000, default of 0. When zero this feature is disabled. When not 0, the uploaded submission's zip file size will be checked against the participant's current storage use (The total storage use of this participant for the competition.). If their sum is over the limit specified, the submission will be rejected with an error message detailing the participant's use after the submission and the value of this limit. Enabling on a competition retroactively may have undesired results. (IE: Participants needing to delete submissions to free up space for more submissions)>

Section 3: Leaderboard properties

This section allows you to configure the leaderboard. Don't panic! For memory:

Alias indicators:
    '&'  : Anchor property.
    '*'  : Alias indicator.

This is a rather generic example of leaderboard configuration:

leaderboards:                 # This indicates the beginning of the leaderboard description.
    Results: &RESULTS         # The leaderboard table anchor is &RESULTS.         
        label: RESULTS        # The table is called "RESULT".
        rank: 1
columns:                      # Now comes the list of columns:
    set1_score:               # This column will display the results of set1_score.
        leaderboard: *RESULTS # It will show up in the table referred to as &RESULTS.
        label: Set 1          # The column will be named "Set 1" in the leaderboard.
        numeric_format: 4     # There will be 4 decimals displayed.
        rank: 2               # This will be the second column
        sorting: desc         # The column will have descending order ("asc" - for ascending order)
    set2_score:
        leaderboard: *RESULTS
        label: Set 2
        numeric_format: 4
        rank: 3
    ave_score:                # Ranking by average rank
        leaderboard: *RESULTS
        label: < Rank >      
        numeric_format: 4
        rank: 1
        computed:             # This column will show the average rank of all score columns.
            operation: Avg
            fields: set1_score, set2_score
    ExecutionTime:            # This column will show the execution time.
        leaderboard: *RESULTS
        label: ExecutionTime 
        numeric_format: 2
        rank: 4

When a scoring program runs, it must return a file scores.txt. For the example above that file should contain something like:

set1_score: 0.1234
set2_score: -2.4321
ExecutionTime: 0.5

The scoring program may optionally also return a file scores.html containing detailed results (in free format), which will be linked from the leaderboard in the column "Detailed Results". This will only work if "enable_detailed_results" is set to True (see Section 1). The Detailed Results Page provides more details on this feature.

Clone this wiki locally