Gettysburg College Computer Science Department Colloquium Series, Spring 2018
Using Multi-Arm Bandits to Build Information Retrieval Test Collections
Distinguished ACM Speaker and Computer Scientist at the National Institute of Standards and Technology
Thursday, April 26, 11:30 a.m.
Glatfelter Hall, room 301
Test collections are a vital piece of the research infrastructure for information retrieval. Constructing fair, reusable test collections for data sets large enough to be of interest is challenging because of the number of human relevance assessments required. Various approaches for minimizing the number of judgments required have been proposed including a suite of methods based on multi-arm bandit optimization techniques. Simulation of the bandit methods on existing test collections demonstrates the methods can build high-quality collections using many fewer judgments than the traditional "pooling" method requires, but they had not heretofore been used to construct a new collection. TREC 2017 provided the opportunity to build a collection de novo using a bandit technique. Subsequent analysis of the process shows that bandit techniques have a dependency on the overall number of relevant documents: when judgment budgets are small relative to the (unknown) number of relevant documents, the resulting collection may be both not reusable and unfair.
Ellen Voorhees is the project manager of the Text REtreival Conference (TREC) at NIST. This international workshop defines methodology and builds infrastructure required for large-scale evaluation of technologies for processing natural language text and searching diverse media types. Her research focuses on developing and validating appropriate evaluation schemes to measure system effectiveness in these areas