How to participate

Getting started

NewsREEL runs as campaign-style evaluation activity of the Conferences and Labs of the Evaluation Forum (CLEF), a non-profit association that supports benchmarking information access systems. On this page, we outline what you have to do in order to participate in this year’s campaign.

  1. Register via the CLEF registration system. You can sign up for each task individually, but for maximum benefit from the complementarity of the tasks, we suggest signing up for both tasks. Registration is free. Register as soon as possible so you can get started now. The system is open until 21 April 2017.

  2. Once you have registered, you will receive an email from us with further information on how to get started. Below, slides of a tutorial that we gave at ACM RecSys’15 in Vienna should give you a general overview of NewsREEL.

    • If you registered for NewsREEL Live, please have a look at this tutorial in which we describe how to register with the Open Recommendation Platform (ORP), the platform that handles all recommendation requests in the online scenario. Participating this task requires running a recommendation server. This server may be hosted by the respective participant. plista offers virtual machines for all participants who have no server at their disposal. There exist already 3 SDKs which handle the communication with ORP. We suggest to use one of these in order to focus on the recommendation algorithms. The SDKs are provided in PHP, Java, and Node.js.

    • If you registered for NewsREEL Replay, we will provide you the access information to download a dataset consisting of content updates, requests for articles and users’ clicks on recommended articles. NewsREEL Replay uses the framework Idomaar to replay this dataset as a stream of recommendation requests. Idomaar is an open source framework and can be downloaded from Github. Please have a look at the description of the dataset to get a better understanding of its nature. We also prepared a short tutorial on how to get started with this task.

  3. Once you downloaded the dataset and Idomaar (NewsREEL Replay) and/or created an account on ORP (NewsREEL Live), you can already start benchmarking the performance of your algorithms. Note that we see CLEF NewsREEL as a challenge in which we aim to identify the best algorithms to recommend news articles. For NewsREEL Live, we therefore determined a few testing periods during which we record the performance of all registered teams. We are currently still in the process of determining what might be the best time slots for this. Please keep an eye on the important dates on the right hand side of the website where we will announce the test period dates. Don’t be afraid if you think that your algorithms will not be among the best performing runs. The most important thing is not to win but to take part!

Small print

Registration and participation in NewsREEL is free. However, we would love to learn from you and your experience in participating in the campaign. Therefore, we ask you to describe your experiments in form of working notes, and present your results at the CLEF 2017 conference which will take place from 11-14 September 2017 in Dublin, Ireland. If you anticipate issues registering for the conference please contact us immediately. Since we received various questions about this, we would like to highlight that although the working notes will be published online, the notes do not count as full peer-reviewed publications.This means that you can still publish your results at other venues such as ACM RecSys or related conferences and journals. The working notes should be submitted via easychair by 26 May 2017. Working notes instructions and guidelines will be made available soon. In order to get an impression of what would be expected, you can have a look at the working notes that have been published in the CEUR proceedings in 2014, 2015, and 2016, respectively.

Please note that by participating in CLEF NewsREEL, you automatically accept a usage agreement with plista. The data set can be used for research purposes. Distribution and commercial use are prohibited.