Overview

Commercial providers of information access systems (such as Amazon or Google) usually evaluate the performance of their algorithms by observing how large numbers of their customers interact with different instances of their services. Unfortunately, due to the lack of access to large-scale systems, university-based research is struggling to catch up with this large-scale evaluation methodology. NewsREEL, short for News Recommendation Evaluation Lab, aims to bridge this “evaluation gap” between Academia and Industry.

NewsREEL addresses the following information access task:

Whenever a visitor of an online news portal reads a news article on their side, the task is to recommend other news articles that the user might be interested in.

NewsREEL offers two kinds of resources to evaluate news recommendation algorithms. First, we provide a large-scale dataset based on recorded events on a set of publishing platforms. Second, we facilitate access to the Open Recommendation Platform (ORP) developed and operated by plista. ORP lets researchers experience authentic conditions as business experience when delivering content to users.

By providing this service for millions of users, the recommendation scenario requires solutions to significant research challenges, such as processing information in real-time, handling the vast amount of data, and providing suitable recommendations. By providing access to the infrastructure of a company, we offer professionals and students the opportunity to develop skills that are in high demand in industry, while at the same time allowing them to familiarize themselves with the academic practice of evaluation of information access systems.

In 2018, NewsREEL has become part of the MediaEval benchmark. We supply a dataset featuring content including images and texts. Participants ought to learn a model to predict how popular an article will become.

In addition, we encourage researchers in the fields of recommender systems, news analytics, and their intersection to submit their findings to the INRA workshop held in conjunction with CIKM 2018. Please, make sure to submit your contribution until 1 August, 2018.

Please follow @NewsRecSys for further updates.