Replication Challenge

Replication Challenge is an exciting opportunity to get in touch with researchers from prominent research institutions and work together on a replication study of a selected work.

The activity is designated for early stage researchers (PhD students, research-oriented master students or other research enthusiasts) and focuses on topics falling within the scope of the DisAI project, namely:

• multilingual language technologies
• multimodal natural language processing
• trustworthy artificial intelligence

while preferably covering (but not limited to) the domain of:

• disinformation combating 

Each involved participant will be assigned a mentor from the leading partners. Together they will select a scientific paper (related to project topics) and replicate the research described therein. This way, the early-stage researchers will get better acquainted with the state-of-the-art in their field of study and gain practical research experience. The involved participants will have 3 months to replicate the selected research work. 

Each researcher will prepare a final report on the achieved results and submit it according to the instructions below (please scroll down for more info about the submission process).

The activity will culminate with a 1-day workshop, where the participants will present the results of their work. In the replication study, the participants and their mentors are encouraged to extend the replicated work with some novel interesting ideas, so their results can be published to the scientific community as well.

To participate in the replication study, please apply according to the instructions below. Approximately 10 early stage researchers will be selected after balancing the interest and mentors capacity.

Mentors and Research Works to Replicate

Note that the list of papers for replication is not exhaustive. You can propose your own paper when applying for the Replication Challenge, which can be considered by the mentors (UPDATED).

• Dr. Simon Ostermann
  Lab Manager, Senior Researcher, Group Lead “Data and Resources”, DFKI:
  • ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts 
  • Parameter-Efficient Transfer Learning for NLP
  • SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
  • The Power of Scale for Parameter-Efficient Prompt Tuning
  • other
• Dr. Stefanos Papadopoulos, Dr. George Karantaidis
  AI Researchers at Information Technologies Institute, CERTH:
  • BCMF: A bidirectional cross-modal fusion model for fake news detection
  • Self-Supervised Distilled Learning for Multi-modal Misinformation Identification
  • INO at Factify 2: Structure Coherence based Multi-Modal Fact Verification
  • Team Triple-Check at Factify 2: Parameter-Efficient Large Foundation Models with Feature Representations for Multi-Modal Fact Verification
  • SpotFake: A Multi-modal Framework for Fake News Detection
  • End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models
  • other
• Dr. Matúš Pikuliak
  Senior researcher, NLP Team, KInIT:
  • Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models
  • CONCRETE: Improving Cross-lingual Fact-checking with Cross-lingual Retrieval
  • other
• Assoc. prof. Michal Gregor
  Expert researcher, NLP Team, KInIT:
  • Scaling sentence embeddings with large language models
  • MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks

Application

Fill in this form and indicate your interests. We will contact you with the decision, eventually containing information about mentor and research work assignment.

Results submission

After finalising the replication study, participants are obliged to submit a report summarising the achieved results.  

Submit the report by the submission date of the run you are involved in by email to replication@disai.eu using the template.

After submission, be prepared to receive feedback and instructions to provide a camera ready version.

Schedule

• October 4th, November 28th – Call for Participation
• Run I
  • October 4th – October 20th – Application and ESR-Mentor Matching
  • November 2nd – January 31th February 7th – Replication Study Realisation
  • January 31th February 7th – Report submission
  • April 15th 9:00 – Review, Camera-ready submission
• Run II
  • December 11th – April 11th – Replication Study Realisation
  • April 11th – Report submission
  • April 18th 12:00 – Review, Camera-ready submission
• April 22nd – Workshop in Bratislava (see below)

The Reproducibility and Replicability Workshop

The participants of the replication challenge will present their findings at the 1-day workshop that will be organised on April 22nd 2024 in Bratislava as an hybrid event.

Schedule (Monday, April 22nd):

10:00-10:15	Welcome Welcome from organisers
10:15-11:15	Rafael Tolosana Calasanz & Andrea Hrčková: Reproducibility in AI (AI4Europe)
11:15-12:15	Lunch
12:15-13.45	Replication Studies: Part I PhD students: Viliam Balara, Róbert Belanec, Ivana Beňová, Ivan Vykopal
13:45-14:15	Coffee Break
14:15-16:15	Replication Studies: Part II Master students: Samuel Revúcky, Matej Jurčák, Marek Kajan, Kristína Sásiková
16:15-16:30	Goodbye