WEPIR 2018: Workshop on Evaluation of Personalisation in Information Retrieval

To be held in conjunction with the ACM SIGIR Conference on Human Information Interaction and Retrieval,
CHIIR 2018, March 11-15, 2018 in New Brunswick, New Jersey, USA.

WEPIR 2018 Tentative Agenda

Thursday, March 15, 2018

9:00-17:00 , SC & I Building, Room: TBC

9:00 - 9:15

Workshop Introduction
(Welcome, purpose, goals, agenda, Introduction of participants)
(15 Minutes)

9:15 - 9:55

Keynote Talk
(30 Minutes + 10 Minutes Q&A)

Ben Carterette, University of Delaware, USA
Title: Offline Evaluation of Personalization Using Logged Data

10:00 - 10:30

Coffee Break

10:30 - 11:10

Paper Presentations I (Each Presentation 15 minutes plus 5 minutes Q&A)

Sanna Kumpulainen, Hugo Huurdeman and Heikki Keskustalo
Personalization needs extension towards task stages in collaborative research work tasks

Jacek Gwizdka
Neuro-physiological data as a source of evaluation metrics for personalized IR

11:10 - 12:30

Panel on Evaluation of Personalized IR systems

Panel Members:
Michael Cole, LexisNexis, USA
Noriko Kando, National Institute of Informatics, Japan
Paul Thomas, Microsoft Australia

12:30 - 13:30


13:30 - 14:30

Paper Presentations II (Each Presentation 15 minutes plus 5 minutes Q&A)

Sanda Erdelez and Isa Jahnke
Personalized Systems and Illusion of Serendipity: A Sociotechnical Lens

Sherrie Hall, Rachel Champoux, Sara Garver, Caroline Harriott and Krysta Chauncey
Designing the user experience of a co-adaptive data analytics interface in response to user trait

Gareth Jones, Gabriella Pasi, Andrea Angiolillo and Camilla Sanvitto
A Proposed Method for Laboratory-Based Evaluation of Personalised Information Retrieval

14:30 - 15:00

Breakout Session 1 - Group breakout on defined topic followed by reporting back with group discussion

15:00 - 15:30

Coffee Break

15:30 - 16:30

Breakout Session 2 - Group breakout on defined topic followed by reporting back with group discussion

16:30 - 17:00

Wrap-up, Summary, Future Plans

Extended submission deadline: Midnight Friday 16th of February, 2018

Call for Papers

The purpose of the WEPIR 2018 workshop is to bring together researchers from different backgrounds, interested in advancing the evaluation of personalisation in information retrieval. The workshop focus is on the development of a common understanding of the challenges, requirements and practical limitations of meaningful evaluation of personalisation in information retrieval. The planned outcome of the workshop is the proposal of methodologies to support evaluation of personalised information retrieval from both the perspectives of the user experience in interactive search settings, and of user models for personalised information retrieval and their algorithmic incorporation in the search process.

Since the focus of the workshop is primarily on exchange of ideas and development of new research activities, only short paper contributions will be sought in the form of 2 to 4 page papers in the standard ACM conference format.

Topics for contributions include but not be limited to:

  • Task design for evaluation of personalised information retrieval.
  • Test collections for personalised information retrieval.
  • Evaluation metrics for peronsalised information retrieval.
  • Protocols for evaluation of interactive personalised information retrieval.
  • User modeling for personalised information retrieval.
  • Search algorithms for personalised information retrieval

Submitted contributions will be selected for inclusion in the workshop on the basis of reviews by the programme committee.

Workshop format:

The topic of evaluation of the incorporation of personalisation within search applications and algorithms and their impact on user engagement and experience of search is currently underexplored within the information retrieval community. This is particularly the case from the perspective of comparative evaluation of interactive and algorithmic elements of personalised search systems, and the representation and exploitation of user models. To reflect this, the workshop will focus on establishing and exploring the principles working towards the outcome of a proposed framework. The workshop will have the following elements:

  • Invited talks: focusing on user-centered interactive issues, and on evaluation of the algorithmic component of search.
  • Presentations of papers submitted in response to an open call for research and position papers. Papers will be presented as short oral and poster format. The number of oral presentations will be limited to allow for the maximum of time for interactive activities, with other papers being presented as posters.
  • Working groups focusing on relevant topics for the evaluation of personalised information retrieval, including experimental protocols, test collection development, evaluation metrics.
  • Consolidation session: integration of the activities of the working groups, and proposal of agreed framework or frameworks for the evaluation of personalised information retrieval

Paper Submissions

The workshop is accepting short papers from 2 to 4 pages (including references) describing approaches or ideas/challenges on the topics of the workshop. Submissions don't need to be anonymized

Submissions should be in ACM Standard SIGCONF format. LaTeX and Word templates are available at (http://www.acm.org/publications/proceedings­-template).

Papers should be submitted in pdf format through the EasyChair system (https://easychair.org/conferences/?conf=wepir2018) no later than midnight Sunday 11th of February, 2018 (AoE). Submissions will be reviewed by members of the workshop program committee. Accepted papers will be included in the extended CHIIR 2018 Proceedings and will be available via the ACM Digital Library. In addition, the WEPIR workshop proceedings will be indexed with CEUR. Authors of select papers may be invited to contribute to a journal publication which describes the outcomes of the workshop.

Important Dates :

Anywhere on Earth Time Zone

Midnight Friday 16th of February 2018: Deadline for paper submission

Midnight Sunday 11th of February 2018: Deadline for paper submission

Midnight Thursday 22nd of February 2018: Notification to authors

Midnight Thursday 1st of March 2018: Camera-ready paper due

Midnight Thursday, 15th of March 2018: WEPIR Workshop at CHIIR

Further Information

Further information is available by emailing the workshop organizers at wepir@adaptcentre.ie.

Workshop Organizers

Gareth J. F. Jones, ADAPT Centre, Dublin City University, Ireland

Nicholas J. Belkin, Rutgers University, USA

Séamus Lawless, ADAPT Centre, Trinity College Dublin, Ireland

Gabriella Pasi, University of Milano-Bicocca, Italy