Gettysburg College Computer Science Department Colloquium Series, Fall 2017
Measuring Search Effectiveness
Distinguished ACM Speaker and Computer Scientist at the National Institute of Standards and Technology
Thursday, October 19, 11:30 a.m.
Glatfelter Hall, room 301
In contrast to most areas of computer science research, information retrieval research has a rich tradition of experimentation dating back to the 1960s. This experimental methodology involves comparing retrieval system effectiveness using test collections, a type of benchmark task. Because test collections were introduced in a series of experiments run at the College of Aeronautics, Cranfield, England, UK, the methodology has become known as `Cranfield'.
In 1981, Karen Sparck Jones edited a collection of papers on information retrieval evaluation. The volume was dedicated to Cyril Cleverdon, principal architect of the Cranfield experiments, and it was published to reflect the state of IR experimentation 20 years after Cranfield. In the introduction to the volume, Sparck Jones observes "It is arguable that our current understanding of information processing is like that of sixteenth century herbalists: it embodies some observation and insight, but lacks detailed analysis and supporting theory."
Ten years after the book was published, the National Institute of Standards and Technology began the TREC project that had the explicit goal of facilitating IR research through building test collections and standardizing evaluation procedures. Today, another twenty-five years later, search is routinely used in ways unimagined in 1981, and the number of IR investigations is likewise well beyond anything anticipated then. Nonetheless, fifty-odd years after its introduction, Cranfield remains the dominant experimental paradigm in the field. This talk will describe how the Cranfield methodology has evolved and how it is used today to evaluate search effectiveness.
Ellen Voorhees is a Computer Scientist at the National Institute of Standards and Technology (NIST). Her primary responsibility at NIST is to manage the Text REtrieval Conference (TREC) project. She received a B.Sc. in computer science from the Pennsylvania State University, and M.Sc. and Ph.D. degrees in computer science from Cornell University. Prior to joining NIST she was a Senior Member of Technical Staff at Siemens Corporate Research in Princeton, NJ where her work on intelligent agents applied to information access resulted in three patents.