SIGIR 2009 Workshop on The Future of IR Evaluation

23 July 2009
Workshop Homepage Program Call for Participation Organizers


Evaluation is at the core of information retrieval: virtually all progress owes directly or indirectly to test collections built within the so-called Cranfield paradigm. However, in recent years, IR researchers are routinely pursuing tasks outside the traditional paradigm, by taking a broader view on tasks, users, and context. There is a fast moving evolution in content from traditional static text to diverse forms of dynamic, collaborative, and multilingual information sources. Also industry is embracing "operational" evaluation based on the analysis of endless streams of queries and clicks.

The questions we expect to address can be succinctly summarized as to make IR evaluation more "realistic." There is however no consensus on what then "real" IR is, or should be, and various directions have been proposed:

The workshop brings together all stake-holders, ranging from those with novel evaluation needs, such as a PhD candidate pursuing a new IR-related problem, to senior IR evaluation experts. Desired outcomes are insight into how to make IR evaluation more "realistic," and at least one concrete idea for a retrieval track or task (at CLEF, INEX, NTCIR, TREC) that would not have happened otherwise.


Workshop at SIGIR (23 July 2009)
Program and proceedings online (14 July 2009)
Notification, 20 short papers accepted (3 July 2009)
33 submissions received (16 June 2009)
Deadline extended to June 15 (16 May 2009)
Call for papers published (20 March 2009)
Website online (10 March 2009)
Workshop proposal accepted by ACM SIGIR (6 March 2009)


The workshop is part of SIGIR 2009 in Boston.


June 15, 2009Deadline for Paper Submissions
 Prepare your 2 page PDF using the ACM format
Submit online using EasyChair
July 2, 2009Notification of Acceptance
 Details of accepted papers published online
July 8, 2009Deadline for Camera Ready Copies
July 23, 2009SIGIR 2009 Workshop on the Future of IR Evaluation


This workshop will be held as part of the 32st Annual International ACM SIGIR Conference on Research & Development on Information Retrieval, Boston, 2009. Information on Boston can be found in the Wikipedia.