BLEU

  • Bilingual Evaluation Understudy
  • MT quality metric
  • Precision Based
  • Precisionn=∑p∈hyp∑n−gram∈pCountclip(n−gram)∑p∈hyp∑n−gram∈pCount(n−gram)Precision_n=\frac{\sum\limits_{p \in hyp}\sum\limits_{n-gram \in p}Count_{clip}(n-gram)}{\sum\limits_{p \in hyp} \sum\limits_{n-gram \in p}Count(n-gram)}
  • Countclip(n−gram)=min(matched n-gram count,maxr∈Ref(n-gram count in r))Count_{clip}(n-gram) = min(\text{matched n-gram count}, max_{r \in Ref} (\text{n-gram count in r}))
  • A weighted logarithmic average : Consider the exponential decay observed in the n-gram precision
∏n=1Npnwn∑n=1Nwn=1∑n=1Nwnexp⁡(∑n=1Nwn∗ln⁡pn)=exp⁡(1N∗∑n=1Nln⁡pn)\sqrt[\sum_{n=1}^N w_n]{\prod_{n=1}^N p_n^{w_n}}=\frac{1}{\sum_{n=1}^N w_n} \exp \left(\sum_{n=1}^N w_n * \ln p_n\right)=\exp \left(\frac{1}{N} * \sum_{n=1}^N \ln p_n\right)

Details

  • Brevity Penalty
  • BP={1if c>re(1−rc)if c≤rBP = \begin{cases} 1 & \text{if } c > r \\ e^{(1 - \frac{r}{c})} & \text{if } c \leq r \end{cases}
  • BLEU=BP⋅exp⁡(∑n=1Nwnlog⁡pn)BLEU = BP \cdot \exp \left(\sum\limits_{n=1}^N w_n \log p_n\right)
  • log⁡BLEU=min(1−rc,0)+∑n=1Nwnlog⁡pn\log BLEU = min(1-\frac{r}{c}, 0) + \sum\limits_{n=1}^N w_n \log p_n

ROUGE

Recall-Oriented Understudy for Gisting Evaluation

ROUGE-N

An n-gram recall between a candidate summary and a set of reference summaries

Recall=TPTP+FNRecall=\frac{TP}{TP+FN}

OverView