Czech Grammar Agreement Dataset for Evaluation of Language Models

Investor logo

Warning

This publication doesn't include Faculty of Economics and Administration. It includes Faculty of Informatics. Official publication website can be found on muni.cz.
Authors

BAISA Vít

Year of publication 2016
Type Article in Proceedings
Conference RASLAN 2016 Recent Advances in Slavonic Natural Language Processing
MU Faculty or unit

Faculty of Informatics

Citation
Web https://nlp.fi.muni.cz/raslan/2016/paper02-Baisa.pdf
Field Informatics
Keywords language model; grammar agreement; verb suffix; Czech language; subject; predicate; dataset; evaluation; perplexity
Description AGREE is a dataset and task for evaluation of language models based on grammar agreement in Czech. The dataset consists of sentences with marked suffixes of past tense verbs. The task is to choose the right verb suffix which depends on gender, number and animacy of subject. It is challenging for language models because 1) Czech is morphologically rich, 2) it has relatively free word order, 3) high out-of-vocabulary (OOV) ratio, 4) predicate and subject can be far from each other, 5) subjects can be unexpressed and 6) various semantic rules may apply. The task provides a straightforward and easily reproducible way of evaluating language models on a morphologically rich language.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.