wiki:main_page

NTCIR10 RITE2

RITE-2 is an evaluation-based workshop aiming to recognize entailment, paraphrase, and contradiction between sentences, which is a common problem shared widely among researchers of natural language processing and information access. By evaluating participanting systems using large scale test collections of Japanese and Chinese (simplified and traditional), we expect to obtain new knowledge, promote development of data and tools, and greatly advance the field of recognizing textual entailment.

Changes from NTCIR-9 RITE

  • Entrance Exam Subtask (Japanese): While the setting of this subtask in the NTCIR-9 RITE was similar to the Binary Class (BC) task, in the NTCIR-10 RITE-2, t1 is not explicitly given. If sentences that entail given t2 are found in large-scale texts (Wikipedia or textbooks), then t2 is considered as true. The percentage of correct answers for multiple-choice questions will be used as the evaluation measure as well as the accuracy of entailment relation recognition. The process of answering will become more human-like as compared to RITE-1. (The BC-like data will also be provided as a part of BC subtask data.)
  • Addition of unit-tests (Japanese): Research focusing on single linguistic phenomena in recognizing textual entailment has been considered difficult because the task usually requires various types of linguistic and semantic analyses. In order to support such research, we provide a dataset obtained from a subset of the BC data in which semantic relations are broken down into a set of single linguistic phenomena.
  • Providing results of basic linguistic analyses such as dependency parsing and predicate-argument structure analysis, and a generic entailment recognition tool which can be modified easily.

RITE-2 SUBTASKS

Four subtasks will be offered in RITE-2.

  • Binary Class (BC) Subtask (Japanese and Chinese): Given a text pair <t1, t2>, a system automatically identifies whether the text t1 entails (infers) the hypothesis t2 or not. For a part of the BC data, we will provide a unit test data in which entailment relations have been broken down into a set of single linguistic phenomena.
  • Multi Class (MC) Subtask (Japanese and Chinese): Unlike the BC Subtask, given a text pair <t1, t2>, a system automatically detects (forward / bi-directional) entailment or no entailment contradiction / independence).
  • Entrance Exam Subtask (Japanese): In this subtask, t1 is not explicitly given, and t2 is obtained from the actual university entrance exam. By refereeing to knowledge such as Wikipedia and textbooks, a system finds appropriate texts to judge entailment relations between the knowledge and a given t2. This subtask attempts to emulate human。ヌs process of answering the questions of entrance exams as the task of recognizing textual entailment.
  • RITE4QA Subtask (Chinese): Same as the BC task in terms of input and output, but the contribution of recognizing textual entailment technologies is evaluated in a practical scenario, answer filtering in question answering.

RITE-2 ORGANIZERS

  • Hiroshi Kanayama (IBM Research-Tokyo)
  • Noriko Kando (National Institute of Informatics)
  • Tomohide Shibata (Kyoto University)
  • Hideki Shima (Carnegie Mellon University, USA)
  • Koichi Takeda (IBM Research-Tokyo)
  • Junta Mizuno (Tohoku University)
  • Teruko Mitamura (Carnegie Mellon University, USA)
  • Yusuke Miyao ((National Institute of Informatics)
  • Yotaro Watanabe (Tohoku University)
  • Shuming Shi (Microsoft Research Asia, PRC)
  • Cheng-Wei Lee (National Taiwan Ocean University, Taiwan)
  • Chuan-Jie Lin (Academia Sinica, Taiwan)