This page describes the instructions of the RITE-2 formal run, so please carefully read.

Formal run period is between January 9th (00:00 UTC-12) and January 16th (23:59 UTC-12). Please note that the submission page will be closed at January 16 (23:59 UTC-12).

The following are the names of subtask or dataset in the RITE-2 formal run. When you submit the formal run results, please check the corresponding items in the submission page.

Japanese subtasks (JA)

• BC
• MC
• ExamBC
• ExamSearch
• UnitTest

Simplified Chinese subtasks (CS)

Traditional Chinese datasets (CT)

For each test dataset, you can submit up to three runs.

In terms of the data format for the formal run, please carefully read Submission Format, and check the format of your result files using Evaluation Tool. Note that your submission is rejected if your submitted file is not accepted by the evaluation tool.

Please stop developing your system after distributing the formal run data. (a minimal modification related to I/O or a bug fix are allowed.) Especially, please do not develop resources by hand after looking into the formal run data. The following are some cases judged as manual intervention or not manual intervention.

• manual intervention
  • creation of inference rules from sentences in the formal run data by hand
  • use of crowd sourcing platforms such as Mechanical Turk, use of Yahoo Answer to develop inference rules or entailment pairs of words
• not manual intervention
  • automatically preprocessing sentences in the formal run data using linguistic analyzers
  • automatically expanding the meaning of words in sentences in the formal run data from the Web

Please mail to the RITE-2 mailing list (ntcir10-rite2 at googlegroups.com) if you have any question about this point.

When making formal run results of a subtask, please do not use the dataset of the other subtasks. For example, in making the formal run of Exam Search, using the formal run data of Exam BC is prohibited.

You can upload the results for the formal run from the following site (EasyChair). The system description paper will be also submitted from this site. All of the results for the formal run tasks you participated should be uploaded together.

formal run submission page:

https://www.easychair.org/conferences/?conf=ntcir10rite2

In RITE-2, we decided to use EasyChair for submitting both formal run results and system description papers. When submitting formal run results, please specify the zipped result file in the part “PDF”. For submitting your formal run results, please carefully read How to submit your formal run results and How to re-submit your formal run results. Please mail to the RITE-2 mailing list (ntcir10-rite2 at googlegroups.com) if you have any questions.

You can submit up to three runs for each subtask. Please zip all of the results together with a system description. The filename of your formal run data should be RITE2-(TeamID)-(Lang)-(SubtaskName)-(RunNumber).txt.

• TeamID: your team ID (e.g. TOHOK)
• Lang: the language of the subtask (JA, CS or CT)
• SubtaskName: the subtask name (e.g. BC)
• RunNumber: 01, 02 or 03

Let us consider a case where TeamID is TOHOK and participating tasks are JA-BC and JA-ExamBC. Then the files you should prepare are as follows.

+--TOHOK
|  +--RITE2-TOHOK-description.txt
|  +--RITE2-TOHOK-JA-BC-01.txt
|  +--RITE2-TOHOK-JA-BC-02.txt
|  +--RITE2-TOHOK-JA-BC-03.txt
|  +--RITE2-TOHOK-JA-ExamBC-01.txt
|  +--RITE2-TOHOK-JA-ExamBC-02.txt
|  +--RITE2-TOHOK-JA-ExamBC-03.txt

After zipping all of the files as TOHOK.zip, please send the zipped file from the EasyChair site. For more details, see submit your results or re-submit your results. In the ExamSearch subtask, if you intend to submit the search results of t1, please set the filename as RITE2-(TeamID)-(Lang)-(SubtaskName)-(RunNumber)-ID.txt. For example, if you send three runs together with their search results of t1, then the files will be as follows.

+--TOHOK	 
|  +--RITE2-TOHOK-description.txt	 
|  +--RITE2-TOHOK-JA-ExamSearch-01.txt	 
|  +--RITE2-TOHOK-JA-ExamSearch-01-ID.txt	 
|  +--RITE2-TOHOK-JA-ExamSearch-02.txt	 
|  +--RITE2-TOHOK-JA-ExamSearch-02-ID.txt	 
|  +--RITE2-TOHOK-JA-ExamSearch-03.txt	 
|  +--RITE2-TOHOK-JA-ExamSearch-03-ID.txt

In the description file, please write detailed description of your system used in the runs for each subtasks you participated. The description must follow the template. The information you wrote will be used to write the RITE-2 overview paper.

More precisely, for each question, only applicable items should be left．

TOHOK-JA-BC-01
system: Extracted a set of morphemes from t1 and t2, and decide on the label based on the percentage of overlap between the sets of morphemes.
questionnaire:
1. what approach your system can be categorized into?
* statistical (e.g. machine learning)

2. fundamental approach of your system (you can choose more than one) 
* overlap
...

TOHOK-JA-BC-02
system: Extracted a set of morphemes from t1 and t2, and decide on the label based on the percentage of overlap between the sets of morphemes.
questionnaire:
1. what approach your system can be categorized into?
* statistical (e.g. machine learning)

2. fundamental approach of your system (you can choose more than one) 
* overlap
...

Each result of the formal run will be evaluated by Evaluation Tool. Please carefully read Submission Format and check the format of the files for submission using the Evaluation Tool.

In the period between January 9th (00:00 UTC-12) and January 16th (23:59 UTC-12), you can submit the results as many time as you want. The submitted file at last will be registered as the official results.

This section describes a process of a new submission. If you intend to re-submit the results, please see How to re-submit your formal run results.

1. Login to EasyChair
   • https://www.easychair.org/conferences/?conf=ntcir10rite2
   • If you have no EasyChair account, please create an account from sign up for an account.
   • 
2. Move to the result submission form
   • By clicking New Submission at the upper left, then you can see the following page.
   • 
3. Input each item
   1. Please input the authors' names, mail-addresses, country names and affiliations. Since these information can be modified in paper submission, only inputting the representative author is allowed. Please specify one or more than one authors as “Corresponding author”.
      • 
   2. Please input TeamID in “Title” and a brief description of your system in “Abstract”.
      • 
   3. Please specify three keywords (but this information is not used anywhere).
      • 
   4. Please specify all of the subtasks you intend to attend.
      • 
   5. You will see the following error message if the form is incomplete.
      • 
4. Choosing the submission file
   • Please specify the zipped result file in the form (while the form is written as Upload Paper, please don't case). You do not need to upload your paper.
5. Submission
   • After you entered all of the items, click “Submit” button.
6. You will see the following page if the submission succeeded.
   • 

1. Login to EasyChair
   • https://www.easychair.org/conferences/?conf=ntcir10rite2
   • After the first submision, a link “Paper x” is added in the right side of “New Submission”.
   • 
2. Re-submission and modification of the information
   • By clicking “Paper x”, the page that includes information of submitted results will be opened. You can change paticipating subtasks or resubmit youre formal run results.
   • 
3. Re-submitting results
   • By clicking “Submit a new version”, you will see the following page. You can submit a new (zipped) result from this page. The file submitted at last will be registered as the official results.
   •