Towards Domain Robustness of Neural Language Models

Warning

This publication doesn't include Faculty of Economics and Administration. It includes Faculty of Informatics. Official publication website can be found on muni.cz.
Authors

ŠTEFÁNIK Michal SOJKA Petr

Year of publication 2021
Type Article in Proceedings
Conference Recent Advances in Slavonic Natural Language Processing (RASLAN 2021)
MU Faculty or unit

Faculty of Informatics

Citation
Web
Keywords Generalization; Debiasing; Domain extrapolation; Domain adaptation; Domain robustness; Neural language models
Description This work summarises recent progress in generalization evaluation and training of deep neural networks, categorized in data-centric and model-centric overviews. Grounded in the results of the referenced work, we propose three future directions towards reaching higher robustness of language models to an unknown domain or its adaptation to an existing domain of interest. In the example propositions that practically complement each of the directions, we introduce novel ideas of a) dynamic objective selection, b) language modeling respecting the token similarities to the ground truth and c) a framework of additive component of the loss utilizing the well-performing generalization measures.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.