인간번역의 자동평가에서 번역 텍스트 길이가 평가 타당성에 미치는 영향

초록

This study explores the influence of the length of human-translated text on the validity of automatic evaluations. To this end, the Brevity Penalty (BP) was replaced with the Length Penalty (LP) to better align the evaluation methodology with the characteristics of human translation. The study examined if this modification improved evaluation validity. The impact of the BP on the total score was ascertained via the coefficient of determination (R²) of the regression analysis. Validity was further investigated by comparing the correlation coefficient of the similarity between BLEU and LP-applied BLEU (lBLEU) scores and human evaluation scores. The findings indicated an overall improvement in validity, and an increase in the coefficient of determination. The qualitative analysis revealed that students were more likely to lose points for longer translations when compared to the three reference translations, particularly in the case of extensive, informative original texts on unfamiliar topics. Therefore, the length of the human-translated text significantly influences the validity of the automatic evaluation, and substituting BP with LP positively improved this validity. The implications of this study can potentially extend to educational scenarios and the utilization of automated assessments.

키워드