cocoxu / simplification Goto Github PK

I tried to use Joshua decoder with the following script:
-cand sari30it.test.output.1best -format plain -ref est.8turkers.tok.turk. -rps 8 -m SARI 4 test.8turkers.tok.norm -v 0
It seems only return SARI = 0.3125145864872819.
I wonder if I configure something wrong?

Would you be able to publish the sentence-level human judgement data for calculate correlation of Automatic Metrics.

Thank you.

Hello,

Could you provide details on the pre-processing applied to the dataset? For example, which tokenizer was used (with which options)? Thank you.

In SARI.py, line 107, I think it should be "if len(addgramcounterall) > 0" instead of "if len(addgramcounter) > 0"

Thank you for making this code available! I was trying to understand how the different components of the SARI score are computed, and I wonder if I've misunderstood something or if there's an inconsistency between the code and the paper. Consider the following example.

Input: "a b"
Output: "b"
Ref-1: "a b"
Ref-2: "a"

Now if I manually compute the recall of kept tokens using Eq. 5 from the paper, I get

    r_{keep}(1) = [min(0, 1) + min(1, 1/2)] / [1 + 1/2] = 1/3,

where the first terms of the numerator and denominator correspond to "a" and the second terms to "b". However, the GitHub implementation gives me

    r_{keep}(1) = 1/2.

The reason is that in the code the terms of the numerator are divided individually by the corresponding denominator terms on line 58, instead of dividing the sum of the numerator terms by the sum of the denominator terms as done in Eq. 5 in the paper.

Replacing line 58 by:

    keeptmpscore2 += keepgramcountergood_rep[keepgram]

and line 65 by:

    keepscore_recall = keeptmpscore2 / sum(keepgramcounterall_rep.values())

seems to fix this and yield p_{keep}(1) = 1/3 as I would expect it to yield.

Have I missed something? Thanks in advance!

Hello @cocoxu ,
Just out of curiosity, is there a reason that turkcorpus is split in 2000 (valid) / 359 (test) samples and not 50%/50% for example?
Thank you!

Hi,

Thanks for the great work!
I have a problem with SARI score. Looks like it does not support sequence with one word for example? See the example below:

test_src = "global"
test_pred = "scenarioPattern"
test_ref = "scenarioPattern"
print(SARIsent(test_src, test_pred, [test_ref]))

the score is 0.17 rather than 1.

Is a bug or as expected?

Thank you.

I would like to know whether your project is just comparing the paraphrases or is generating the paraphrases?
Kindly, guide me.

Hi, I have some questions about the STAR.java, I found this script use the f_score of the deletion rather than the precision, which is different from the SARI definition.
`

    double recall_del_n = 0.0 ;
    if (delRefTotalNgram > 0) {
            recall_del_n = delCandCorrectNgram / (double) delRefTotalNgram;
    }

    double f1_del_n = meanHarmonic(prec_del_n, recall_del_n);

    sc += weights[n] * f1_del_n;`

Could you tell me the reason?

Also, I tested STAR on the wiki-small corpus (1 reference ) and I found that using the random sentence as the simplified output can get very high STAR (about 33.7). Is this a normal phenomenon?

The link you offered for downloading is invalid, could you please upload it again, thanks a lot!

In the readme it is mentioned

Note that STAR is corpus-level version of SARI, SARI is sentence-level.

But, I couldn't find information on STAR anywhere. So, how is it calculated or is it simply the mean of all SARI sentence scores?

In Equation 7 of the paper, my understanding is that you need to compute the precision/recall of each ngram order, and then this is averaged over the maximum order of ngrams (which is 4). Only after that, you calculate the F1 score of each operation, and then compute SARI/STAR by averaging them:

add_precision = (add_precision_1 + add_precision_2 + add_precision_3 + add_precision_4) / 4
add_recall = (add_recall_1 + add_recall_2 + add_recall_3 + add_recall_4) / 4
add_f1 = 2 * add_precision * add_recall / (add_precision + add_recall)

keep_precision = (keep_precision_1 + keep_precision_2 + keep_precision_3 + keep_precision_4) / 4
keep_recall = (keep_recall_1 + keep_recall_2 + keep_recall_3 + keep_recall_4) / 4
keep_f1 = 2 * keep_precision * keep_recall / (keep_precision + keep_recall)

del_precision = (del_precision_1 + del_precision_2 + del_precision_3 + del_precision_4) / 4

sari = (add_f1 + keep_f1 + dep_precision) / 3

However, the code follows a different procedure. There, a F1 score (for each operation) is computed for each ngram order. These are accumulated (averaged by the maximum ngram order) and divided by 3 (the number of operations) in the end.

add_f1_1 = 2 * add_precision_1 * add_recall_1 / (add_precision_1 + add_recall_1)
add_f1_2 = 2 * add_precision_2 * add_recall_2 / (add_precision_2 + add_recall_2)
add_f1_3 = 2 * add_precision_3 * add_recall_3 / (add_precision_3 + add_recall_3)
add_f1_4 = 2 * add_precision_4 * add_recall_4 / (add_precision_4 + add_recall_4)

add_1 = (add_f1_1 + add_f1_2 + add_f1_3 + add_f1_4) / 4

keep_f1_1 = 2 * keep_precision_1 * keep_recall_1 / (keep_precision_1 + keep_recall_1)
keep_f1_2 = 2 * keep_precision_2 * keep_recall_2 / (keep_precision_2 + keep_recall_2)
keep_f1_3 = 2 * keep_precision_3 * keep_recall_3 / (keep_precision_3 + keep_recall_3)
keep_f1_4 = 2 * keep_precision_4 * keep_recall_4 / (keep_precision_4 + keep_recall_4)

keep_1 = (keep_f1_1 + keep_f1_2 + keep_f1_3 + keep_f1_4) / 4

del_precision = (del_precision_1 + del_precision_2 + del_precision_3 + del_precision_4) / 4

sari = (add_f1 + keep_f1 + dep_precision) / 3

These are not mathematically equivalent, so the scores produced by both ways of calculating the metric are different. Which is the correct process then? The one in the paper or the one in the code?

Thanks for your help and clarification.

Hello,

Would it be possible to have access to the data collected for the human evaluation? This means, the ratings submitted by the Turkers for Grammaticality, Meaning Preservation and Simplicity Gain for all the systems that were evaluated?

Thank you.