1 |
Symbolic Semantic Memory in Transformer Language ModelsMorain, Robert Kenneth 16 March 2022 (has links)
This paper demonstrates how transformer language models can be improved by giving them access to relevant structured data extracted from a knowledge base. The knowledge base preparation process and modifications to transformer models are explained. We evaluate these methods on language modeling and question answering tasks. These results show that even simple additional knowledge augmentation leads to a reduction in validation loss by 73%. These methods also significantly outperform common ways of improving language models such as increasing the model size or adding more data.
|
Page generated in 0.1195 seconds