User Tools

Site Tools


wiki:results

This is the official RITE2 formal run results page. Some special runs are marked with the following colors:

  • RED names are baseline runs.
  • Gray names are unofficial runs.

Japanese (JA)

BC

TeamMacroF1AccuracyY-F1Y-Prec.Y-Rec.N-F1N-Prec.N-Rec.
DCUMT-JA-BC-0180.4981.6475.7684.9568.3685.2279.9591.24
WSD-JA-BC-0380.0880.6676.6877.6075.7883.4782.7884.18
SKL-JA-BC-0279.4679.8476.6674.5478.9182.2584.0780.51
BnO-JA-BC-0378.9379.3475.9574.2577.7381.9083.3380.51
WSD-JA-BC-0278.7779.6774.3878.9570.3183.1580.1086.44
WSD-JA-BC-0178.6179.5174.2378.6070.3182.9980.0586.16
SKL-JA-BC-0178.6178.8576.3371.9781.2580.8985.0577.12
BnO-JA-BC-0278.3179.0274.4076.2372.6682.2280.8783.62
BnO-JA-BC-0177.6178.3673.4975.6271.4881.7280.1683.33
KitAi-JA-BC-0177.1177.7073.4473.4473.4480.7980.7980.79
OKA1-JA-BC-0276.7177.0573.8870.7177.3479.5382.4276.84
JAIST-JA-BC-0276.4776.8973.3571.0675.7879.5981.6077.68
SKL-JA-BC-0376.4077.2172.0374.2769.9280.7779.1382.49
KitAi-JA-BC-0376.1676.7272.4871.9273.0579.8380.2979.38
JAIST-JA-BC-0175.5676.2371.5171.9471.0979.6179.2779.94
OKA1-JA-BC-0174.5974.5974.3064.5587.5074.8887.8365.25
KYOTO-JA-BC-0274.5075.5769.2873.3665.6379.7376.9082.77
IBM-JA-BC-0174.4974.9271.1968.7373.8377.7980.0075.71
IBM-JA-BC-0273.4073.7770.2667.0273.8376.5479.5773.73
JAIST-JA-BC-0373.0873.7768.7568.7568.7577.4077.4077.40
IBM-JA-BC-0372.9073.4469.0867.5470.7076.7278.0775.42
KitAi-JA-BC-0272.3572.4670.6363.9278.9174.0781.6367.80
ut12-JA-BC-0172.2373.9365.3673.8958.5979.1173.9685.03
OKA1-JA-BC-0371.7172.1368.2865.3671.4875.1577.8872.60
FLL-JA-BC-0367.9970.0059.9668.1653.5276.0270.9081.92
*TKDDI-JA-BC-0363.8369.0250.1377.2437.1177.5366.9492.09
TKDDI-JA-BC-0263.5568.6949.8776.0037.1177.2366.8091.53
*TKDDI-JA-BC-0263.5568.6949.8776.0037.1177.2366.8091.53
*TKDDI-JA-BC-0163.4568.6949.6076.4236.7277.2966.7491.81
TKDDI-JA-BC-0163.4568.6949.6076.4236.7277.2966.7491.81
FLL-JA-BC-0163.0668.3649.0875.6136.3377.0566.5391.53
Baseline-JA-BC-0162.5363.9355.2857.6353.1369.7867.9171.75
NTTD-JA-BC-0361.9062.3058.0354.4562.1165.7769.5062.43
*FLL-JA-BC-0561.0563.2851.7257.6946.8870.3766.1775.14
FLL-JA-BC-0259.7364.1046.4562.0937.1173.0064.7783.62
ut12-JA-BC-0359.0165.4142.8269.9130.8675.2164.3990.40
ut12-JA-BC-0257.8464.7540.7769.1628.9174.9163.8290.68
NTTD-JA-BC-0157.5964.1040.9766.0929.6974.2063.6488.98
*FLL-JA-BC-0655.6957.7046.2549.5543.3665.1462.4468.08
EHIME-JA-BC-0154.3459.3439.2252.6331.2569.4661.5779.66
*FLL-JA-BC-0452.5855.0841.7045.7938.2863.4760.1067.23
THK-JA-BC-0152.4053.2845.9244.6547.2758.8760.1857.63
NTTD-JA-BC-0250.3862.4625.8975.4715.6374.8661.2296.33
EHIME-JA-BC-0250.1451.4841.9642.1341.8058.3158.1558.47
JUNLP-JA-BC-0148.8349.0245.7241.3251.1751.9357.3447.46
EHIME-JA-BC-0348.0548.3644.0540.3948.4452.0556.4448.31
KYOTO-JA-BC-0346.4260.9818.4975.0010.5574.3560.1097.46
KYOTO-JA-BC-0141.9760.009.6392.865.0874.3259.2399.72

MC

TeamMacroF1AccuracyB-F1B-Prec.B-Rec.F-F1F-Prec.F-Rec.C-F1C-Prec.C-Rec.I-F1I-Prec.I-Rec.
SKL-JA-MC-0159.9669.5367.1872.1362.8676.4776.8576.1021.1525.5818.0375.0670.5480.19
SKL-JA-MC-0258.2568.6169.2977.1962.8674.9473.3676.5913.5916.6711.4875.1771.4979.25
SKL-JA-MC-0355.4568.0763.2465.1561.4373.8569.7078.549.2015.386.5675.5173.3377.83
WSD-JA-MC-0354.3969.5368.2959.5780.0075.2972.7378.050.000.000.0073.9970.5177.83
WSD-JA-MC-0254.1868.9868.8363.1075.7173.9570.6777.560.000.000.0073.9470.0478.30
WSD-JA-MC-0153.9868.8068.2959.5780.0074.7072.4877.070.000.000.0072.9369.3676.89
FLL-JA-MC-0153.6764.9669.0168.0670.0070.3560.5683.908.8242.864.9266.5071.3562.26
JAIST-JA-MC-0152.6066.9766.6764.8668.5774.5569.7980.000.000.000.0069.2065.6873.11
BnO-JA-MC-0252.4457.6658.9444.5387.1462.1873.0354.1520.3816.6726.2368.2778.5360.38
JAIST-JA-MC-0252.2765.3363.5160.2667.1471.8459.6890.245.9733.333.2867.7680.5258.49
BnO-JA-MC-0352.1165.6959.7454.7665.7172.9367.3679.515.8025.003.2869.9569.6370.28
BnO-JA-MC-0152.0366.4265.8259.0974.2972.2367.2378.050.000.000.0070.0568.4771.70
JAIST-JA-MC-0351.4865.3367.9260.6777.1471.1358.4990.730.000.000.0066.8683.6955.66
*FLL-JA-MC-0451.2764.2364.9459.5271.4368.8465.7872.203.1750.001.6468.1564.5672.17
KYOTO-JA-MC-0250.1264.7859.5549.0775.7169.9069.5770.240.000.000.0071.0167.8174.53
*FLL-JA-MC-0235.1244.7125.9336.8420.0049.5843.8257.0716.0042.869.8448.9847.1650.94
THK-JA-MC-0130.9849.0921.9575.0012.8660.7547.7783.4128.5752.1719.6743.6354.6136.32
Baseline-JA-MC-0126.6145.440.000.000.0056.1843.0180.985.4115.383.2844.8854.3638.21
EHIME-JA-MC-0325.8940.334.7614.292.8649.4639.2666.838.8242.864.9240.5144.3837.26
EHIME-JA-MC-0124.4728.1014.5811.4820.0037.9742.0134.6312.369.4018.0332.9541.4327.36
*FLL-JA-MC-0322.4734.4911.0115.388.5740.6434.5949.270.000.000.0038.2337.7938.68
EHIME-JA-MC-0221.9936.312.417.691.4343.9835.7857.073.0320.001.6438.5539.4137.74
JUNLP-JA-MC-0121.4222.6316.8312.8824.2924.4230.2220.4917.1712.4127.8727.2734.2922.64
KYOTO-JA-MC-0117.0440.330.000.000.008.2975.004.393.23100.001.6456.6439.5999.53

ExamBC

TeamMacroF1Acc.Correct Answer RatioY-F1Y-Prec.Y-Rec.N-F1N-Prec.N-Rec.
BnO-JA-ExamBC-0267.1570.3155.5656.9664.7150.8777.3472.7682.55
BnO-JA-ExamBC-0366.9768.7557.4159.3059.6558.9674.6474.3774.91
KDR-JA-ExamBC-0266.9068.7551.8559.0659.7658.3874.7374.1975.27
BnO-JA-ExamBC-0166.8669.8757.4156.8763.5751.4576.8472.7381.45
KDR-JA-ExamBC-0366.6468.3047.2259.2058.8659.5474.0974.3673.82
WSD-JA-ExamBC-0164.9067.8652.7854.7260.0050.2975.0971.6278.91
IBM-JA-ExamBC-0364.1864.5145.3760.7453.0271.1067.6276.8560.36
WSD-JA-ExamBC-0364.7167.6352.7854.5559.5950.2974.8771.5278.55
SKL-JA-ExamBC-0264.0465.6349.0756.5055.2557.8071.5972.6670.55
WSD-JA-ExamBC-0263.9667.6351.8552.4660.6146.2475.4770.5781.09
KDR-JA-ExamBC-0163.3164.5149.0756.6853.6160.1269.9472.8367.27
SKL-JA-ExamBC-0161.6567.6329.6346.4964.2936.4276.8068.5787.27
SKL-JA-ExamBC-0360.4763.1742.5950.1552.5347.9870.8068.9772.73
KitAi-JA-ExamBC-0159.8463.1736.1148.2852.7444.5171.4068.2174.91
IBM-JA-ExamBC-0259.3361.8346.2949.2650.6147.9869.4168.3170.55
KitAi-JA-ExamBC-0359.0561.3845.3749.2750.0048.5568.8368.2169.45
JAIST-JA-ExamBC-0259.0463.3941.6745.7053.4939.8872.3967.4078.18
JAIST-JA-ExamBC-0358.6564.9642.5942.4958.0033.5374.8066.9584.73
IBM-JA-ExamBC-0158.5361.6144.4447.2450.3344.5169.8267.4672.36
JAIST-JA-ExamBC-0157.5563.1740.7442.1153.5734.6873.0066.3781.09
KitAi-JA-ExamBC-0257.1658.7139.8149.0446.8451.4565.2967.4463.27
KYOTO-JA-ExamBC-0256.8262.0543.5241.7851.2635.2671.8565.9678.91
NTTD-JA-ExamBC-0255.5755.5834.2654.8845.1569.9456.2671.1146.55
Baseline-JA-ExamBC-0154.7756.4732.4145.9844.1547.9863.5565.3861.82
NTTD-JA-ExamBC-0353.1254.0234.2646.6342.2552.0259.6164.6855.27
NTTD-JA-ExamBC-0152.0258.9331.4833.8144.7627.1770.2363.2778.91
JUNLP-JA-ExamBC-0150.4650.8930.5645.8139.9153.7655.1062.7949.09
*TKDDI-JA-ExamBC-0349.0862.5028.7022.9455.5614.4575.2263.2892.73
TKDDI-JA-ExamBC-0148.6262.2826.8522.1254.5513.8775.1163.1292.73
TKDDI-JA-ExamBC-0248.6262.2826.8522.1254.5513.8775.1163.1292.73
THK-JA-ExamBC-0143.7762.2826.8511.5261.116.3676.0362.3397.45
KYOTO-JA-ExamBC-0338.5761.3821.301.1450.000.5876.0161.4399.64
KYOTO-JA-ExamBC-0137.8660.9420.370.000.000.0075.7361.2199.27

ExamSearch

TeamMacroF1Acc.Correct Answer RatioSearch Prec.Search Rec.Y-F1Y-Prec.Y-Rec.N-F1N-Prec.N-Rec.
*KDR-JA-ExamSearch-0258.1264.5132.41NA NA 41.7657.0032.9574.4866.6784.36
*KDR-JA-ExamSearch-0157.5963.8433.33NA NA 41.3055.3432.9573.8766.3883.27
*KDR-JA-ExamSearch-0357.3963.1734.26NA NA 41.7053.6434.1073.0866.2781.45
NTTD-JA-ExamSearch-0155.0258.0425.9317.6343.4143.3745.2841.6266.6765.0568.36
*BnO-JA-ExamSearch-0254.7756.4731.4826.5613.0845.9844.1547.9863.5565.3861.82
*BnO-JA-ExamSearch-0152.4554.9126.8522.3210.9941.6241.6241.6263.2763.2763.27
*BnO-JA-ExamSearch-0351.7851.7931.48NA NA 51.5742.1266.4752.0066.8642.55
NTTD-JA-ExamSearch-0249.1549.3325.9317.6343.4152.2141.0671.6846.0866.4435.27
KYOTO-JA-ExamSearch-0146.5762.9528.7047.14 3.6217.0062.969.8376.1562.9596.36
KYOTO-JA-ExamSearch-0245.4162.2826.8539.29 2.4215.0857.698.6775.7562.5696.00

UnitTest

TeamMacroF1AccuracyY-F1Y-Prec.Y-Rec.N-F1N-Prec.N-Rec.
*FLL-JA-UnitTest-0177.7790.8794.8494.3995.2860.7162.9658.62
*FLL-JA-UnitTest-0376.9891.2995.1393.6196.7058.8268.1851.72
JAIST-JA-UnitTest-0274.5289.2193.8793.8793.8755.1755.1755.17
*TKDDI-JA-UnitTest-0374.0085.8991.5896.3587.2656.4144.9075.86
*TKDDI-JA-UnitTest-0173.5185.4891.3296.3486.7955.7044.0075.86
*TKDDI-JA-UnitTest-0273.5185.4891.3296.3486.7955.7044.0075.86
TKDDI-JA-UnitTest-0173.5185.4891.3296.3486.7955.7044.0075.86
TKDDI-JA-UnitTest-0273.5185.4891.3296.3486.7955.7044.0075.86
BnO-JA-UnitTest-0173.4485.8991.6395.8887.7455.2644.6872.41
NTTD-JA-UnitTest-0268.9884.2390.7393.9487.7447.2239.5358.62
BnO-JA-UnitTest-0368.7880.0887.5697.1379.7250.0035.8282.76
JAIST-JA-UnitTest-0167.3679.6787.4096.0580.1947.3134.3875.86
NTTD-JA-UnitTest-0161.9974.2783.6095.1874.5340.3828.0072.41
THK-JA-UnitTest-0153.2671.3782.3589.9475.9424.1817.7437.93
Baseline-JA-UnitTest-0151.7086.3192.5888.4197.1710.8125.006.90
*FLL-JA-UnitTest-0251.3577.5987.0888.3585.8515.6314.2917.24
NTTD-JA-UnitTest-0347.9153.9465.6395.5050.0030.1918.4682.76
BnO-JA-UnitTest-0246.8087.9793.6087.97100.000.000.000.00
KYOTO-JA-UnitTest-0245.3548.9659.4198.9042.4531.2818.6796.55
KYOTO-JA-UnitTest-0137.2738.5946.38100.0030.1928.1616.38100.00
JAIST-JA-UnitTest-0329.4630.7138.8386.8925.0020.1011.6772.41

Traditional Chinese (CT)

BC

TeamMacroF1Y-F1Y-Prec.Y-Rec.N-F1N-Prec.N-Rec.
IASL-CT-BC-0267.1471.6668.6474.9562.6366.4859.20
MIG-CT-BC-0267.0770.9969.0373.0763.1465.5160.95
MIG-CT-BC-0366.9971.2368.7473.9062.7665.8559.95
IMTKU-CT-BC-0165.9969.1668.8069.5262.8363.2262.44
WHUTE-CT-BC-0165.5570.2067.3773.2860.8964.4457.71
MIG-CT-BC-0165.4267.9468.8867.0162.9161.9363.93
IMTKU-CT-BC-0363.8267.7666.4769.1059.8761.3658.46
Yuntech-CT-BC-0362.3165.2665.8264.7259.3658.7859.95
Yuntech-CT-BC-0262.0266.4664.7568.2757.5859.5755.72
Yuntech-CT-BC-0161.6467.1364.0270.5656.1660.0652.74
KC99-CT-BC-0157.6766.4260.4573.7048.9357.5842.54
CYUT-CT-BC-0155.1655.7760.1451.9854.5550.7558.96
CYUT-CT-BC-0252.6458.4456.6760.3346.8348.7945.02
IASL-CT-BC-0151.7765.7956.8478.0837.7652.9129.35
CYUT-CT-BC-0351.5863.8956.3973.7039.2750.5932.09
JUNLP-CT-BC-0148.7250.8253.2048.6446.6344.4749.00
IMTKU-CT-BC-0248.6136.3663.5425.4760.8648.1982.59
NTOUA-CT-BC-0132.6325.0632.3420.4640.2034.0849.00
NTOUA-CT-BC-0331.7119.3928.8114.6144.0435.8956.97
NTOUA-CT-BC-0230.7015.2025.3710.8646.2036.8361.94

MC

TeamMacroF1B-F1B-Prec.B-Rec.F-F1F-Prec.F-Rec.C-F1C-Prec.C-Rec.I-F1I-Prec.I-Rec.
IASL-CT-MC-0246.3252.3553.0651.6664.6353.9980.4929.9036.2525.4438.4152.7330.21
WHUTE-CT-MC-0145.5058.8656.3661.5967.0654.3687.5012.0825.717.8943.9963.4033.68
MIG-CT-MC-0245.1557.6848.6470.8654.4958.6050.9114.1926.839.6554.2550.4558.68
NTOUA-CT-MC-0344.8061.1050.4377.4864.2155.0077.131.505.260.8852.4070.5941.67
NTOUA-CT-MC-0144.6362.0754.8271.5265.7954.0184.150.000.000.0050.6669.2839.93
MIG-CT-MC-0344.2151.0739.9370.8652.5261.1346.0419.1922.6216.6754.0454.6153.47
KC99-CT-MC-0143.7545.4842.9448.3463.6157.0071.9516.6715.8717.5449.2466.0839.24
MIG-CT-MC-0142.1653.4947.6760.9351.9157.1447.5610.0617.787.0253.1947.3060.76
Yuntech-CT-MC-0340.1452.7053.7951.6665.2751.0390.554.3813.042.6338.1961.0727.78
Yuntech-CT-MC-0139.7648.9253.5445.0365.8051.1992.075.6314.293.5138.6860.2928.47
Yuntech-CT-MC-0238.7746.8952.4642.3865.5950.7592.684.3212.002.6338.3060.0028.13
IMTKU-CT-MC-0135.7660.5652.1572.1963.4757.5770.7313.4122.009.6541.3854.5533.33
NTOUA-CT-MC-0233.491.2925.000.6662.5050.9680.7913.9818.0611.4056.2056.4955.90
IASL-CT-MC-0133.0432.2044.7125.1760.7147.5883.8421.7222.4321.0517.5431.5312.15
MCUIM-CT-MC-0132.5159.2158.8259.6070.0761.3381.7125.0020.6931.588.2920.275.21
IMTKU-CT-MC-0332.3652.8344.5564.9065.5859.0273.781.6514.290.8841.7552.3634.72
CYUT-CT-MC-0226.2621.3918.9724.5043.6838.4350.6117.0015.7918.4222.9838.8416.32
CYUT-CT-MC-0125.6022.5819.0027.8145.8538.6256.4012.4415.1910.5321.5441.1814.58
JUNLP-CT-MC-0124.2121.2217.7026.4932.2841.2326.5216.7212.6724.5626.6130.4923.61
CYUT-CT-MC-0323.5127.2219.7643.7145.6239.5153.960.000.000.0021.1941.4114.24
IMTKU-CT-MC-0219.370.000.000.0030.6364.0820.1215.9526.5311.4050.2635.7984.38

RITE4QA

Ranked by WorseRanking Top1 scores. For metrics related information, please see the task guidelines

BetterRanking scores show how good a system is in terms of the improvement on the answer ranking of a good-performing factoid QA system, while WorseRanking scores show how good a system when it is applied to the answer ranking of a bad-performing factoid QA system.

We ranked system performance based on WorseRanking Top1 accuracy because a high BetterRanking score may be the result of a system that outputs most pairs with “Y” label with the same confidence score, which results in a fallback into the original QA system ranking. A baseline RITE system that always outputs “Y” with the same confidence score was created to show the effect (see orgQAsys-* runs). The answer ranking of such a system is identical to the original QA system answer ranking. In addition, according to BetterRanking scores, none of the runs performs better than the original QA ranking.

WorseRanking BetterRanking
CT R R+U R R+U
Run Top1 MRR Top5 Top1 MRR Top5 Top1 MRR Top5 Top1 MRR Top5
WHUTE-CT-RITE4QA-01 27.33 34.57 46.67 30.67 38.76 52.67 26.67 34.29 46.67 30.00 38.48 52.67
IMTKU-CT-RITE4QA-03 16.67 25.70 40.67 19.33 29.39 46.00 17.33 26.03 40.67 20.00 29.72 46.00
IMTKU-CT-RITE4QA-01 14.67 22.69 37.33 22.00 31.84 49.33 14.67 22.58 37.33 22.00 31.73 49.33
CYUT-CT-RITE4QA-03 12.67 17.69 28.67 18.67 26.60 42.67 12.00 17.50 28.67 18.67 26.99 43.33
IASL-CT-RITE4QA-01 12.00 22.14 39.33 14.67 26.86 48.00 30.00 38.27 50.67 32.67 42.34 57.33
IMTKU-CT-RITE4QA-02 12.00 19.82 32.00 19.33 29.43 44.67 12.00 19.84 32.67 20.00 29.79 45.33
IASL-CT-RITE4QA-02 10.67 16.01 27.33 14.67 22.09 37.33 38.00 42.74 49.33 42.00 47.66 56.67
NTOUA-CT-RITE4QA-01 8.00 9.28 11.33 13.33 17.06 22.67 8.67 9.61 11.33 15.33 18.17 22.67
NTOUA-CT-RITE4QA-03 8.00 9.97 13.33 12.67 17.19 24.00 9.33 10.63 13.33 14.67 18.30 24.00
NTOUA-CT-RITE4QA-02 7.33 8.78 10.67 11.33 14.11 18.00 8.00 9.22 10.67 12.00 14.56 18.00
orgQAsys-CT-RITE4QA-01 7.33 11.54 22.67 10.67 16.99 31.33 40.67 47.60 57.33 44.67 52.32 64.00
CYUT-CT-RITE4QA-01 6.67 11.71 24.00 10.00 16.99 32.67 38.00 45.19 54.67 43.33 51.08 62.00
CYUT-CT-RITE4QA-02 6.67 11.71 24.00 10.00 16.99 32.67 38.00 45.19 54.67 43.33 51.08 62.00

Simplified Chinese (CS)

BC

TeamMacroF1AccuracyY-F1Y-Prec.Y-Rec.N-F1N-Prec.N-Rec.
bcNLP-CS-BC-0373.8474.6578.4372.5885.3169.2578.2562.12
MIG-CS-BC-0268.0968.5071.7269.6473.9364.4566.9762.12
CYUT-CS-BC-0367.8668.1270.7470.1671.3364.9865.6364.35
bcNLP-CS-BC-0167.0469.6576.3265.9890.5257.7580.2045.13
bcNLP-CS-BC-0266.8969.9176.8965.7192.6556.8883.3343.18
MIG-CS-BC-0165.7165.8167.5669.3365.8863.8762.1165.74
CYUT-CS-BC-0263.1163.1262.5069.3656.8763.7358.1670.47
WHUTE-CS-BC-0261.6566.5875.4062.6094.7947.9084.5133.43
CYUT-CS-BC-0161.1761.5957.1471.9447.3965.2055.8678.27
*IASL-CS-BC-0260.4563.2570.9861.9083.1849.9166.8239.83
WHUTE-CS-BC-0158.2064.7974.7960.9996.6841.6187.5027.30
MIG-CS-BC-0357.1963.6473.8060.4294.7940.5981.5127.02
IMTKU-CS-BC-0354.2862.7473.9559.4297.8734.6189.5321.45
Yuntech-CS-BC-0353.5259.5470.2458.2888.3936.8065.2525.63
Yuntech-CS-BC-0252.1059.0370.3257.7789.8133.8865.6022.84
Yuntech-CS-BC-0150.9158.6470.3957.4091.0031.4266.0720.61
IMTKU-CS-BC-0150.8260.3172.4257.9896.4529.2281.0117.83
*IASL-CS-BC-0150.6054.0363.6355.5874.4137.5750.0030.08
WUST-CS-BC-0250.1458.7770.8957.3192.8929.3969.0718.66
WUST-CS-BC-0150.1458.7770.8957.3192.8929.3969.0718.66
*WUST-CS-BC-0150.1458.7770.8957.3192.8929.3969.0718.66
WUST-CS-BC-0350.1458.7770.8957.3192.8929.3969.0718.66
IMTKU-CS-BC-0250.1260.3172.6657.8797.6327.5785.5116.43
JUNLP-CS-BC-0148.4948.6651.3952.6150.2445.5944.4446.80

MC

TeamMacroF1AccuracyB-F1B-Prec.B-Rec.F-F1F-Prec.F-Rec.C-F1C-Prec.C-Rec.I-F1I-Prec.I-Rec.
bcNLP-CS-MC-0356.8261.0866.6777.2758.6267.3053.9189.5338.4164.4427.3654.8969.2845.45
*IASL-CS-MC-0250.9453.9155.3061.3450.3464.4457.5173.2938.4240.2136.7945.5950.0041.90
WHUTE-CS-MC-0146.7954.8061.5462.4160.6964.3649.9090.6118.7139.3912.2642.5873.0830.04
WHUTE-CS-MC-0246.5356.5962.2559.8764.8365.0951.2489.178.2633.334.7250.5375.5937.94
bcNLP-CS-MC-0244.8857.6259.6871.8451.0367.8652.5895.670.000.000.0051.9963.7943.87
MIG-CS-MC-0244.7451.6058.5049.0772.4152.8457.6948.7411.3522.867.5556.2652.0161.26
CYUT-CS-MC-0242.5248.7853.6451.5955.8656.1348.8066.0612.4218.189.4347.8755.1542.29
MIG-CS-MC-0141.8249.1756.4450.8363.4550.7057.2745.495.4810.003.7754.6447.6564.03
Yuntech-CS-MC-0240.9151.2255.0251.8358.6264.8149.9092.4213.4332.148.4930.4065.7919.76
Yuntech-CS-MC-0340.8951.2253.9551.5756.5565.1550.2092.7813.4332.148.4931.0463.4120.55
WUST-CS-MC-0240.8752.3759.7456.4463.4562.7547.9990.613.5733.331.8937.4371.9125.30
WUST-CS-MC-0340.8752.3759.7456.4463.4562.7547.9990.613.5733.331.8937.4371.9125.30
*WUST-CS-MC-0140.8752.3759.7456.4463.4562.7547.9990.613.5733.331.8937.4371.9125.30
CYUT-CS-MC-0140.3747.6360.3459.3361.3856.7244.9476.9012.3133.337.5532.1246.6224.51
WUST-CS-MC-0140.3351.7359.3154.6564.8362.2047.8688.813.5733.331.8936.2669.6624.51
Yuntech-CS-MC-0140.3350.7053.4250.6256.5564.5649.6192.4213.6434.628.4929.7063.6419.37
CYUT-CS-MC-0340.1051.0948.7351.5446.2157.0744.3180.140.000.000.0054.5973.3343.48
bcNLP-CS-MC-0139.9553.9143.4381.1329.6664.7049.0794.950.000.000.0051.6959.9045.45
*IASL-CS-MC-0134.9541.7437.2948.3530.3459.3947.0580.5125.2426.0024.5317.8928.4513.04
MIG-CS-MC-0334.4243.1553.9039.8083.4558.5851.9667.1512.9413.6812.2612.2770.836.72
IMTKU-CS-MC-0327.2640.209.8110.838.9767.1052.6792.4232.1425.8642.450.000.000.00
JUNLP-CS-MC-0124.3824.7122.4219.5926.2127.0032.4923.1022.0216.2933.9626.0732.5421.74
IMTKU-CS-MC-0123.8937.645.8510.004.1463.2048.7389.8922.8217.7132.083.6927.781.98
IMTKU-CS-MC-0219.6736.117.7312.905.5257.7841.7393.868.7410.397.554.4131.582.37

RITE4QA

Ranked by WorseRanking Top1 scores. For metrics related information, please see the task guidelines

BetterRanking scores show how good a system is in terms of the improvement on the answer ranking of a good-performing factoid QA system, while WorseRanking scores show how good a system when it is applied to the answer ranking of a bad-performing factoid QA system.

We ranked system performance based on WorseRanking Top1 accuracy because a high BetterRanking score may be the result of a system that outputs most pairs with “Y” label with the same confidence score, which results in a fallback into the original QA system ranking. A baseline RITE system that always outputs “Y” with the same confidence score was created to show the effect (see orgQAsys-* runs). The answer ranking of such a system is identical to the original QA system answer ranking. In addition, according to BetterRanking scores, none of the runs performs better than the original QA ranking.

WorseRanking BetterRanking
CS R R+U R R+U
Run Top1 MRR Top5 Top1 MRR Top5 Top1 MRR Top5 Top1 MRR Top5
IMTKU-CS-RITE4QA-03 28.00 33.77 42.67 33.33 40.87 53.33 28.00 33.77 42.67 33.33 40.87 53.33
WHUTE-CS-RITE4QA-01 18.67 27.59 43.33 22.00 33.67 54.00 18.67 27.64 43.33 22.67 34.06 54.00
IMTKU-CS-RITE4QA-02 14.67 21.44 36.00 20.00 29.08 47.33 14.67 21.44 36.00 20.00 29.08 47.33
*IASL-CS-RITE4QA-02 12.67 18.80 32.00 15.33 23.13 39.33 38.00 42.71 50.00 42.00 47.27 56.00
*IASL-CS-RITE4QA-01 12.00 22.71 42.00 15.33 27.96 50.67 31.33 38.09 48.67 36.00 43.74 56.67
IMTKU-CS-RITE4QA-01 10.67 19.91 38.67 17.33 27.21 47.33 10.67 19.91 38.67 17.33 27.21 47.33
CYUT-CS-RITE4QA-02 7.33 11.82 23.33 10.67 17.32 32.67 39.33 46.13 55.33 44.00 51.52 62.67
orgQAsys-CS-RITE4QA-01 7.33 11.32 22.00 10.67 16.99 31.33 40.67 47.60 57.33 44.67 52.32 64.00
bcNLP-CS-RITE4QA-03 6.67 8.78 11.33 7.33 9.78 12.67 8.00 9.33 10.67 9.33 10.67 12.00
CYUT-CS-RITE4QA-01 6.67 11.49 23.33 10.00 16.99 32.67 38.00 45.19 54.67 43.33 51.08 62.00
bcNLP-CS-RITE4QA-01 3.33 3.67 4.00 4.67 6.17 8.00 2.67 3.22 4.00 6.00 6.78 8.00
bcNLP-CS-RITE4QA-02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
wiki/results.txt · Last modified: 2013/05/01 22:23 by rite2organizer