Close

Welcome to the Midas server dedicated by Kitware to hosting automated grand challenges. Info on the Midas Platform with user documentation available here.

The intention of hosting automated grand challenges here is to make the scoring and ranking open and transparent. The Midas Platform, the Challenge plugin for hosting grand challenges on Midas, and the Covalic metric executables used for scoring results are all open source and freely available. Bug reports and comments are welcome via email at midas@public.kitware.com.

Click on the link below to participate in the challenge or view results when they become available:

ISBI 2013 Challenge on Prostate Segmentation

MICCAI BRATS 2012 Challenge on Multimodal Brain Tumor Segmentation

General information on automated grand challenges

Grand challenges are a way to compare algorithms on a common dataset. The challenge will consist of one or more phases, the phases can either occur offsite (all participants are not in the same location) or onsite (all participants are located together, such as at a conference or meeting). A phase of a challenge can either be for competitors to see only their own scores on a dataset (and how their scores change as they make changes to their methods), or it can have a global dashboard where competitors are ranked against each other based on some scoring and ranking metrics.

A common structure is to have a training phase with a dataset for competitors to test their algorithms against, and then a testing phase with a separate dataset and a global dashboard to rank competitors against each other. There are many available metrics to score results, similarly there are many methods to rank competitors based on their metric scores. Each challenge can decide on their own specific metrics and ranking schemes.

At the high level, the way automated grand challenge hosting works is that there is a set of input data cases with a ground truth for each case. The ground truth may be kept private from the competitors or shared with them. Competitors will submit a result for each case in the input data. The Midas Grand Challenge system will run the submitted result case against the ground truth for that case, calculating whatever metrics have been decided. These scoring results are then presented to the competitors. If the challenge is in a phase that involves a global dashboard, the rankings are calculated for each of the competitors and a global dashboard with the rankings is displayed.

Available Scoring Metrics

  • Average distance of boundary surfaces
  • 95th percentile Hausdorff distance of boundary surfaces
  • Dice overlap
  • Cohen's kappa
  • Sensitivity
  • Specificity
  • Positive Predictive Value

Participating in a Grand Challenge

Each grand challenge will be associated with one (or possibly many) Communities in Midas. Identify the specific community for your Grand Challenge, and then look to that Community's info tab for more specific information. The following instructions should apply generally to all grand challenges.

Submitting Results

One time steps

  1. Register a user with Midas
  2. Login with your new user
  3. Once you have a login, navigate to the correct community for your challenge
  4. Click on Join the Community

Steps for scoring submissions

  1. Navigate to your submissions folder, which will be created for you under your My folders area in a subfolder specific to your challenge
  2. Click the Upload here link on the right, under the ACTIONS menu
  3. Browse for or drag-and-drop your results files for the correct phase and dataset, then upload the results
  4. Click on the Score Submissions button on the top of the page
  5. Select the challenge and the phase of the challenge that you wish to have results scored for on the Submission Scoring wizard, as long as you have at least one input that can be scored (has a file name convention match), click on the Score Results button.
  6. You will now be taken to a dashboard that will display your scores as they are calculated. 

You may also upload a zip file containing your submissions.

  1. Upload a zip file (should be a flat structure with only files at the top level) of your submission files to the correct submissions folder.
  2. The zip will be created as an item in the folder.
  3. Click the "view" action on the item.
  4. Click the "Extract archive" action on the item view (and delete the original).
  5. The individual files that were in the zip will now appear as items in the folder.

In order to submit results, click on the Join The Community button for the correct community, this will create some specifically named data folders in your user's space.  If these folders have not been created, navigate to the community page, click on the leave the community link, and then join the community again.  The folders that will have been created are in the My folders link to the left (which leads to your Midas user's space). You will have to upload your results in the appropriate folder for the phase of the challenge.  These folders are created with permissions such that you can read/edit/delete the files there, and moderators of the challenge can view these files, but other competitors will not be able to read/edit/delete these files.

Naming Conventions for Submitted Files

You will need to match your results to the subject via a naming convention.  For example, in the BRATS-1 Training Dataset, Truth Folder, you can see there is a file corresponding to subject BRATS_HG0001 called BRATS_HG0001_truth.mha.  The input data for this subject BRATS_HG0001 is in the BRATS-1 Images subfolder.  To submit a result for this subject, for the training data, you would navigate to your Midas user's submission folder for the BRATS-1 training phase, and upload your results file (which should be named BRATS_HG001_result.mha) in that folder.

Scoring Results

Once you are satisfied with your current submission, you can have your submissions scored.  Whether you can resubmit results later for further scoring or score improvement depends on the setup of that phase of the challenge. Check your challenge for details.

Results folder validation will then occur, and will report to you which of your results files in the Submission Folder will be matched up with subject truths from the Truth folder.  This will give you an opportunity to correct the names of any of your submission files, re-upload them to your submission folder, and re-validate them.  As long as there is at least one match between your submitted results and the subjects in the Truth folder, you can score your results by clicking on the Score Results button.  After clicking this button, you will be taken to a dashboard display of your current scoring.  The scoring will be processed via a batch system and will take some time.  As the results come in, your scoring dashboard will be updated. You will receive an email at the email address you registered with when your scoring is complete.

Viewing your Previous Scored Results

To see a listing of previous runs, click on My folders at the left, then click on the My challenge scores tab.  You will see a listing of previous scoring runs, clicking on the timestamp of any of those will take you to the detailed scoring for that results submission.

FAQ and Support

If your question is not answered in the FAQ, you can look in the info tab of the Community associated with your challenge for a support email specific to that challenge. For questions in general about Midas hosting of grand challenges, please email midas@public.kitware.com.

Q). 

What is the difference between Community space and My folders?

A). 

Midas creates hierarchies of data specific to a particular task or group of people under the name Community. This data is intended to be shared among those with permissions to see it. For individual users, Midas creates a My folders area, accessible via a link at the left, where individual users can store their own folders and items under whatever permission scheme they think appropriate.

Q).

The Score Submissions wizard is not finding my folder by default.

A).

It is possible you have deleted your submission folder or that it was not created correctly. If you leave the community associated with your challenge and re-join it, you will have the proper folders created in your My folders area, then you can upload your results to these newly created folders. If you have deleted these folders, do not try to manually recreate them with the same names, as this will prevent automated creation of folders with the same names in the same locations when you rejoin the community.

Q).

How can I open the mha file format?

A).

The mha file format can be opened with a variety of Open Source tools, including 3D Slicer. It can also be opened programmatically using VTK or ITK.

Q).

Can I have my results separated into header and data files?

A).

No, you must combine the header and data of each case into a single file.