Expected Results
From ATIRE
This page documents some of the expected results when searching with the ATIRE search engine. The documented commands assume that the default filenames are used and all binaries from the project are in the PATH. The exact output may not match due to parallel indexing, but should match to 3 or 4 decimal points.
Contents |
Wall Street Journal
The Wall Street Journal topics and assessments are contained within the evaluation folder of the ATIRE search engine repository. These are not necessarily the best results possible, but are used to verify that changes made to the search engine have not affected the results
atire -a evaluation/WSJ.qrels -q evaluation/51-100.titles
Expected output:
Processed 50 topics: MAP: 0.221737
The following was generated 2014-02-11 using ATIRE changeset 1847:118b625a87ce on Mac OS X 10.9.1 .
WSJ
| Ranking Function | MAP |
|---|---|
| -RBM25:0.9:0.4 | 0.222400 |
| -Rlmds:2500 | 0.224130 |
| -RDPH | 0.214867 |
| -RDFRee | 0.195040 |
Wikipedia 2009
The Wikipedia 2009 queries and assessments are available from the INEX website.
The queries first need to be run through the topic extractor found in the evaluation directory like so:
INEXtopics_to_ANTquery -clean 2009-topics.xml > 2009.topics INEXtopics_to_ANTquery -clean 2010-topics.xml > 2010.topics
Then after removing topics that have no assessments:
atire -a inex2009.qrels -q 2009.topics atire -a inex2010-article.qrels -q 2010.topics
Expected output:
Processed 68 topics: MAP: 0.344935 Processed 52 topics: MAP: 0.351517
TREC Collections
See TREC Results
Copyright © 2011 ATIRE.ORG. ALL Rights Reserved