Challenge Overview

This year the BioASQ challenge will comprise the following tasks. Participants may choose to participate in any or all of the tasks and their subtasks. Participant registration for any of the tasks will open here.

BioASQ Task 13b: Biomedical Semantic QA (involves IR, QA, summarization)

Task 13b will use benchmark datasets containing training and test biomedical questions, in English, along with gold standard (reference) answers. The participants will have to respond to each test question with relevant articles (in English, from designated article repositories), relevant snippets (from the relevant articles), exact answers (e.g., named entities in the case of factoid questions) and 'ideal' answers (English paragraph-sized summaries). More than 5,000 training questions (that were used as dry-run or test questions in previous year) are already available, along with their gold standard answers (relevant  articles, snippets, exact answers, summaries). About 300 new test questions will be used this year. All the questions are constructed by biomedical experts from around Europe. 

The test dataset of Task 13b will be released in batches, each containing approximately 100 questions. The task will start on March, 2025. Separate winners will be announced for each batch. Participation in the task can be partial; for example, it is acceptable to participate in only some of the batches, to return only relevant articles (and no article snippets), or to return only exact answers (or only `ideal' answers). System responses will be evaluated both automatically and manually. Read more...

BioASQ Task Synergy 13: Biomedical Semantic QA For developing issues

Task Synergy will use benchmark datasets of test biomedical questions for developing issues, such as COVID-19, in English. The participants will have to respond to each test question with relevant articles (in English, from designated article repositories), relevant snippets (from the relevant articles), exact answers (e.g., named entities in the case of factoid questions) and 'ideal' answers (English paragraph-sized summaries). No special training questions are available for this task, but expert feedback will be incrementally provided instead, based on participant responses for each round. Using this feedback, the participants can improve their systems and provide better answers for persisting and/or new questions. Meanwhile, the participants may also train their systems using the training dataset from previous versions of task Synergy and of task b as well, both available at the BioASQ Participants Area . All the questions are constructed and assessed by biomedical experts from around Europe. Participation in the task can be partial, i.e. participants may enter the task in any of the rounds.  

The Synergy Task will run in four rounds, starting with an initial set of questions for developing issues, such as COVID-19, in the first round on January, 2025. The questions will persist in later rounds until fully answered. In addition, new versions of the questions or new questions may also added in later rounds. Separate winners will be announced for each round. Participation in the task can be partial; for example, it is acceptable to participate in only some of the rounds, to return only relevant articles (or only article snippets), or to return only exact answers (or only `ideal' answers). System responses will be manually assessed and feedback for the responses will be provided at the end of each round. Read more...

BioASQ Task MultiClinSum: Multilingual Clinical Summarization

Task MultiClinSum will rely on a corpus of manually selected full clinical case reports and their corresponding clinical case report summaries derived from case report publications written in the previously mentioned languages. For evaluation proposes, automatically generated summaries will be compared against manually generated summaries generated by the original authors, exploring Rouge-2 scores and BERTScore for evaluation assessment. You can join anytime from March, 2025 onwards. 

The BioASQ Task MultiClinSum is co-ogranized with the Barcelona Supercomputing Center. Read more...

BioASQ Task BioNNE-L: Nested NER in Russian and English

The BioNNE-L shared task focuses on NLP challenges in entity linking, also known as medical concept normalization (MCN), for English and Russian languages. The goal is to map biomedical entity mentions to a comprehensive set of medical concept names and their concept unique identifiers (CUIs) from the UMLS. You can join anytime from Feb, 2025 onwards. 

The train/dev datasets include annotated mentions of disorders, anatomical structures, and chemicals. We design our data to account for a complex structure of nested entity mentions and the partial nature of medical terminology. Participants are allowed to train any model architecture on any publicly available data to achieve the best performance.

The BioASQ Task BioNNE-L is co-ogranized with the Kazan Federal University. Read more...

BioASQ Task ELCardioCC: Clinical Coding of Greek Cardiology Discharge Letters

The Task ELCardioCC concerns i) the assignment of cardiology-related ICD-10 codes to discharge letters from Greek hospitals, ii) the extraction of the specific mentions of ICD-10 codes from the discharge letters. We will use evaluation metrics, such as micro and macro F-measure for subtask (i) and token F-measure for subtask (ii). You can join anytime from Feb, 2025 onwards. 

The dataset for the ELCardioCC task includes 500 cardiology discharge letters in Greek, annotated by at least two experts at the document level with ICD-10 codes, as well as at the corresponding mention level with the ICD-10 code mentions.

The BioASQ Task ELCardioCC is co-ogranized with the Aristotle University of Thessaloniki. Read more...

BioASQ Task GutBrainIE: Gut-Brain interplay Information Extraction

The Task GutBrainIE aims to foster the development of Information Extraction (IE) systems that support experts by automatically extracting and linking knowledge from scientific literature, facilitating the understanding of gut-brain interplay and its role in neurological diseases. The task is divided into two subtasks. In the first subtask, participants are provided with PubMed abstracts discussing the gut-brain interplay and asked to extract named entities about the gut-brain interplay from PubMed abstracts to link them to the corresponding concepts in a reference ontology. In the second subtask, the participants are asked to identify binary relations – i.e., presence/absence – between any pair of entities they extract within an abstract. The submitted runs are evaluated based on Precision, Recall, and F1 measures for each subtask using gold annotations created by domain experts. You can join anytime from Feb, 2025 onwards. 

The dataset for the GutBrainIE task includes circa 1,000 PubMed abstracts annotated by experts with entity mentions, corresponding concepts in the reference ontology, and binary relations.

The BioASQ Task GutBrainIE is co-ogranized with the University of Padua. Read more...