Our initiative aims to fill existing gaps in fully automated MS lesion segmentation by providing participants with an extensively annotated MRI dataset derived from a heterogeneous cohort of MS patients. Our dataset contains series of data of about 90 patients acquired at multiple timepoints. These timepoint vary from 1 to 5 per patient. Thus, we have approximately 150 MRI data series (timepoints) in total. Each timepoint consists of three different scan modalities: T1-w, T2-w and FLAIR. This dataset was meticulously preprocessed and annotated on FLAIR sequences by our experts, encompassing T1-w and T2-w sequences for comprehensive lesion characterization. A notable strength of our initiative lies in the substantial number of patients and annotated scans, surpassing publicly available datasets traditionally used for MS lesion segmentation. Furthermore, our dataset is distinctively advantageous as it is representative of real-world scenarios, featuring authentic MS patients, and is acquired "in everyday practice", reflecting a heterogeneous acquisition environment without constraints. The goal is to build an algorithm capable of generating an automatic segmentation of MS lesions for an input MRI data series (T1-w, T2-w and FLAIR) not seen by the model under training. It is planned to provide a competition website/platform that will be used for the duration of the competition, which will remain active thereafter. The platform will contain a private area for participants; in this area participants can download the training/validation/test set, upload 1-2 page summary describing the methodology and results (segmentation masks obtained on the test set). In this competition, only fully automatic methods allowed. All methods may provide an automatic segmentation of the MS lesions. The data used in the training phase is limited to that provided by the challenge in order to compare methods in the same environment, so no additional data are allowed. Since this is the first challenge, it is not possible to estimate the exact number of participants. Given the recent interest in the context of multiple sclerosis lesion segmentation, we are confident that a significant number of groups will participate in the challenge (about 10). Furthermore, we plan to launch the first edition of the MSLesSeg competition at ICPR 2024, with subsequent editions planned as an annual tradition. We plan to ask each participant to produce a 1-2 page summary detailing their technical solutions to the challenge. Participants who do not comply with this request by the deadline will be excluded from the final rankings and awards. Teams with the highest scores in the challenge ranking will be reviewed by the organizing committee and will receive an electronic certificate of successful participation. The methods presented by these teams will also be described in the challenge report paper. The organizers of the challenge will provide the results of a baseline methods. The organizing committee will publish a paper describing the competition process and results (including rankings) of the challenge. The authors of the best selected works will be invited to submit their contribution to a special issue of a valuable Journal. Each participant may upload (during the upload time) a maximum of 3 result files; the best one will be considered for the participant's ranking. Each participant must upload also a 1-2 page summary describing the methodology before the deadline. After the deadline, the score will be calculated only for participant which upload the 1-2 page summary and at least one result file. All results will be shown publicly on the competition website, which will be updated during the open training round and announced on the final round phases of the closed test round of the task.

The dataset will be public. All the details about the dataset are available on the site used for the challenge. The dataset contains MRI scans of patients acquired at different timepoints and labelled scans and has received the approval of the ethics committee of the hospitals where the scans were acquired. The dataset and the annotations will be under CC BY 4.0 license. The competition cohort consists of patients diagnosed with multiple sclerosis who were clinically scanned with MRI brain acquisition protocols at different hospital centers in the city of Catania, Italy. All the data were acquired in the last 3 years, during brain MRI scans. The data were acquired in different hospital centers with different 1.5 Tesla MRI scanners (about 3), with different MRI protocols, then annotated by two expert raters and validated by an expert specialized in neurology and multiple sclerosis. The training, validation and test sets will be selected randomly from the dataset, to ensure that the distributions of multiple sclerosis patients conform to the distributions of a real-world scenario. All data provided during the challenge will be described on the challenge site. Each case consists of T1-w, T2-w and FLAIR-w scans (extracted from whole brain MRI) of a patient, acquired at several time-point, and annotations of the labelled MS lesions from the FLAIRs for each time-point. All the annotation were produced manually by two know-context operators and validated by a multiple sclerosis expert neurology. Both the annotators had professional experience on the task, while the neurologist which validated the segmentation masks had many years of experience in multiple sclerosis MRI evaluation. MS lesion annotations of the dataset were performed using Jim 9, a commercial semi-automated segmentation tool, the link to which can be found here: https://www.xinapse.com/j-im-9-software/ The process used to determine the annotations is as follows: 1. Each MRI was reviewed by an experienced neurologist specialising in multiple sclerosis. The neurologist identified and located the MS lesions using his expertise in MRI interpretation. 2. Two know-context operators segmented each identified lesion in the FLAIR scan, using T1- and T2- weighted scans for complete lesion characterisation. (MS lesions typically appear hyperintense in T2- weighted and FLAIR scans and hypointense in T1-weighted scans). The labels were generated using the tool mentioned above. 3. The neurologist verified the accuracy of the segmentations by visual inspection of the produced labels together with the corresponding MRI scans, manually correcting any discrepancies if necessary. Steps 2 and 3 were iterated until the neurologist was satisfied with the quality of the generated labels. During the manual labelling process, we consider a possible source of segmentation error resulting mainly from the existence of small MS lesions in MRI scans. Patients with multiple sclerosis often have numerous small lesions, usually less than 3 mm, so that the precise identification of these small lesions becomes difficult. To reduce potential inaccuracies, a validation process was implemented during the final data verification phase, systematically on all segmentation masks. Lesions that could not be distinctly identified on two consecutive slices were subsequently excluded from the segmentation masks. Errors in the segmentation process can also depend on inter- and intra-operator variability. To address this issue, the masks labeled by annotators underwent validation by the expert neurologist specializing in multiple sclerosis. We did an initial pre-processing on the MRIs: the original DICOM file was anonymised and converted to Nifti. It was then registered with the standard MNI-152 template. Finally, skull-stripping was performed, in order to extract only the brain tissue. All the annotations were produced after the registration of MRI with MNI-152 template from the FLAIR sequences, where the lesions appear hyperintense relative to the white matter. T1- and T2-weighted scans were used to confirm the presence of the lesions.

Participants will develop their methods using the training/validation dataset provided in the first phase. At the end of the first phase, they will evaluate the results of their method on the provided test set, which will be released in the second phase. Subsequently, participants will have to submit the results obtained on the test set (the predicted segmentation masks) and produce a 1-2 page summary describing their proposed method. The submission instruction will be available after the approval of the competition. Participants' methods will be evaluated based on their accuracy in segmenting MS lesions, measuring how well the predicted lesions from the proposed method overlap with the real lesions in the ground-truth mask, using a metric based on overlap. This overlap-based metric will be used to rank the proposed methods. The evaluation metric used will be the Mean Dice Score (DSC), which is a commonly used metric for evaluating the performance of binary segmentation. It is particularly useful in medical imaging for tasks such as lesion segmentation, where precise object delineation is critical. The Dice score value is between 0 and 1, where 0 means that there is no agreement between the predicted mask and the ground-truth mask, while a value of 1 means that the predicted mask and the ground-truth mask are completely overlapping. Specifically, we will calculate the Dice score for each segmentation mask generated by their proposed method (one mask for each data series in the test set) using the following formula:

where ‘A’ denotes the predicted segmentation mask and ‘B’ the ground-truth mask. Here, the notation denote the cardinality of each set of voxel. The final performance ranking will be determined by the overall Mean Dice Score calculated across all data series in the test set.