There is an increasing interest in using deep neural networks in various downstream natural language processing tasks. Such models are commonly used as black boxes, meaning that their decision-making is difficult to interpret. In order to build trust in models, it is crucial to analyse their inner workings which lead to predictions. The need to interpret natural language processing models has induced research on linguistically-informed interpretability. This field revolves around choosing specific linguistic phenomena and inspecting models' capability to capture them without being explicitly trained for it.  This thesis project contributes to the field by assessing the ability of BERT-style models to learn subject number in Lithuanian and English. The experiments revolve around designing diagnostic classifiers which are used to determine if the models are capable of learning this particular linguistic phenomenon. The results show that BERT-style models are capable of implicitly learning the number of a subject both in Lithuanian and English. However, this seems to be harder in Lithuanian, as diagnostic classifiers show a lower accuracy. The study observes that the accuracy of logistic regression diagnostic classifiers fluctuates to a large extent. Fully connected neural network classifiers outperform logistic regression classifiers.