Title: Converting Prose into Poetry with Neural Networks Author: Memduh Gokirmak Institute: Institute of Formal and Applied Linguistics Supervisor: Martin Popel, Institute of Formal and Applied Linguistics Abstract: We present here our attempts to create a system that generates poetry based on a sequence of text provided to it by a user. We explore the use of machine translation and language model technologies based on the neural network architecture. We use different types of data across three languages in our research, and employ and develop metrics to track the quality of the output of the systems we develop. We find that combining machine translation techniques to generate training data to this end with fine-tuning of pre-trained language models provides the most satisfactory generated poetry. Keywords: poetry machine translation language models iii

There exist many real-world applications such as recommendation systems, document retrieval, and computational biology where the correct ordering of instances is of equal or greater importance than predicting the exact value of some discrete or continuous outcome. Learning-to-Rank (LTR) refers to a group of algorithms that apply machine learning techniques to tackle these ranking problems. Despite their empirical success, most existing LTR models are not built to be robust to errors in labeling or annotation, distributional data shift, or adversarial data perturbations. To fill this gap, we develop four LTR frameworks that are robust to various types of perturbations. First, Pairwise Elastic Net Regression Ranking (PENRR) is an elastic-net-based regression method for drug sensitivity prediction. PENRR infers robust predictors of drug responses from patient genomic information. The special design of this model (comparing each drug with other drugs in the same cell line and comparing that drug with itself in other cell lines) significantly enhances the accuracy of the drug prediction model under limited data. This approach is also able to solve the problem of fitting on the insensitive drugs that is commonly encountered in regression-based models. Second, Regression-based Ranking by Pairwise Cluster Comparisons (RRPCC) is a ridge-regression-based method for ranking clusters of similar protein complex conformations generated by an underlying docking program (i.e., ClusPro). Rather than using regression to predict scores, which would equally penalize deviations for either low-quality and high-quality clusters, we seek to predict the difference of scores for any pair of clusters corresponding to the same complex. RRPCC combines these pairwise assessments to form a ranked list of clusters, from higher to lower quality. We apply RRPCC to clusters produced by the automated docking server ClusPro and, depending on the training/validation strategy, we show. improvement by 24%–100% in ranking acceptable or better quality clusters first, and by 15%–100% in ranking medium or better quality clusters first. Third, Distributionally Robust Multi-Output Regression Ranking (DRMRR) is a listwise LTR model that induces robustness into LTR problems using the Distributionally Robust Optimization framework. Contrasting to existing methods, the scoring function of DRMRR was designed as a multivariate mapping from a feature vector to a vector of deviation scores, which captures local context information and cross-document interactions. DRMRR employs ranking metrics (i.e., NDCG) in its output. Particularly, we used the notion of position deviation to define a vector of relevance score instead of a scalar one. We then adopted the DRO framework to minimize a worst-case expected multi-output loss function over a probabilistic ambiguity set that is defined by the Wasserstein metric. We also presented an equivalent convex reformulation of the DRO problem, which is shown to be tighter than the ones proposed by the previous studies. Fourth, Inversion Transformer-based Neural Ranking (ITNR) is a Transformer-based model to predict drug responses using RNAseq gene expression profiles, drug descriptors, and drug fingerprints. It utilizes a Context-Aware-Transformer architecture as its scoring function that ensures the modeling of inter-item dependencies. We also introduced a new loss function using the concept of Inversion and approximate permutation matrices. The accuracy and robustness of these LTR models are verified through three medical applications, namely cluster ranking in protein-protein docking, medical document retrieval, and drug response prediction.

Predicting wall heat flux accurately in wall-bounded turbulent flows is critical for a variety of engineering applications, including thermal management systems and energy-efficient designs. Traditional methods, which rely on expensive numerical simulations, are hampered by increasing complexity and extremly high computation cost. Recent advances in deep neural networks (DNNs), however, offer an effective solution by predicting wall heat flux using non-intrusive measurements derived from off-wall quantities. This study introduces a novel approach, the convolution-compacted vision transformer (ViT), which integrates convolutional neural networks (CNNs) and ViT to predict instantaneous fields of wall heat flux accurately based on off-wall quantities including velocity components at three directions and temperature. Our method is applied to an existing database of wall-bounded turbulent flows obtained from direct numerical simulations (DNS). We first conduct an ablation study to examine the effects of incorporating convolution-based modules into ViT architectures and report on the impact of different modules. Subsequently, we utilize fully-convolutional neural networks (FCNs) with various architectures to identify the distinctions between FCN models and the convolution-compacted ViT. Our optimized ViT model surpasses the FCN models in terms of instantaneous field predictions, learning turbulence statistics, and accurately capturing energy spectra. Finally, we undertake a sensitivity analysis using a gradient map to enhance the understanding of the nonlinear relationship established by DNN models, thus augmenting the interpretability of these models. / <p>Presentation online</p>

Predicting wall heat flux accurately in wall-bounded turbulent flows is critical fora variety of engineering applications, including thermal management systems andenergy-efficient designs. Traditional methods, which rely on expensive numericalsimulations, are hampered by increasing complexity and extremly high computationcost. Recent advances in deep neural networks (DNNs), however, offer an effectivesolution by predicting wall heat flux using non-intrusive measurements derivedfrom off-wall quantities. This study introduces a novel approach, the convolution-compacted vision transformer (ViT), which integrates convolutional neural networks(CNNs) and ViT to predict instantaneous fields of wall heat flux accurately based onoff-wall quantities including velocity components at three directions and temperature.Our method is applied to an existing database of wall-bounded turbulent flowsobtained from direct numerical simulations (DNS). We first conduct an ablationstudy to examine the effects of incorporating convolution-based modules into ViTarchitectures and report on the impact of different modules. Subsequently, we utilizefully-convolutional neural networks (FCNs) with various architectures to identify thedistinctions between FCN models and the convolution-compacted ViT. Our optimizedViT model surpasses the FCN models in terms of instantaneous field predictions,learning turbulence statistics, and accurately capturing energy spectra. Finally, weundertake a sensitivity analysis using a gradient map to enhance the understandingof the nonlinear relationship established by DNN models, thus augmenting theinterpretability of these models

Transformers are electrical devices used in practice to increase or decrease voltages. Transformers are of various sizes and used mainly in power distribution. To provide cooling and insulation, transformer oils are used together with cellulose that acts as a solid insulation. The most common type of transformer oil is mineral oil and is a product derived from the refining of crude oil. Its low cost and good compatibility with cellulose are two factors that have led to its predominant position as the common transformer oil. There are also synthetic ester based transformer oils, and following an increased interest in environmentally friendly products, transformer oils made from natural esters such as sunflower, soybean and rapeseed. Mineral oil is not biodegradable and is deemed as hazardous waste. The ester based oils are biodegradable and promoted as a more environmentally friendly alternative to mineral oil. In this thesis, the possibility of re-refining used mineral transformer oil is assessed from a financial perspective in the form of a business case and an LCA study has been done to compare the environmental impacts between ester based transformer oils and mineral based transformer oil. The results from the LCA study showed that from a cradle-to-gate perspective, mineral oil has a lower environmental impact than ester-based transformer oils. The re-refining of used mineral transformer oil further reduces the environmental impact. The results from the business case showed that a small scale re-refining facility is financially feasible but highly dependent on the supply and demand of used transformer oil. It is recommended to pursue further studies before making any decision. There is lack of data regarding the re-refining market in Eastern Europe and the accuracy of the LCA study can be further improved by having emissions data from re-refining used mineral transformer oil. / Transformatorer är elektriska komponenter som tillämpas vid spänningsregleringar. Dessa transformatorer har olika storlekar och används i eldistribution. Transformatorolja tillsammans med cellulosa används som elektrisk isolering och kylning av transformatorer. Den vanligaste typen av transformatorolja är mineralolja och är en produkt som erhålls vid raffinering av råolja. Dess låga kostnad och goda kompatibilitet med cellulosa är två faktorer som har lett till dess dominerande ställning. Det finns också syntetisk esterbaserad transformatorolja och efter ett ökat intresse för miljövänliga produkter så tillverkas även transformatoroljor av naturliga estrar så som solros, soja och raps. Mineralolja är inte nedbrytbar och anses vara farligt avfall. De esterbaserade oljorna är nedbrytbara och anses vara ett mer miljövänligt alternativ till mineralolja. I denna rapport utvärderades möjligheten till att återraffinera använd mineralolja ur ett ekonomiskt perspektiv i form av en affärsplan och en LCA-studie där esterbaserad olja och mineralolja har jämförts ur ett miljöperspektiv. Resultaten från LCA-studien visade att mineralolja från ett ”cradle-to-gate” perspektiv har en lägre miljöpåverkan än esterbaserade transformatoroljor. Återraffinering av använd mineralolja minskar dess miljöpåverkan ytterligare. Resultatet från affärsplanen visade att en småskalig återraffineringsanläggning är ekonomiskt hållbar men samtidigt väldigt beroende av utbud respektive efterfrågan på använd mineralolja. Det rekommenderas att göra en djupare analys innan man fattar ett beslut. Det finns brist på information med avseende på återraffineringsmarknaden i Östeuropa. Noggrannheten på LCA-studien kan förbättras ytterligare genom att emissionsdata från en återraffineringsanläggning är tillgänglig.

Readability assessment (RA) is to assign a score or a grade to a given document, which measures the degree of difficulty to read the document. RA originated in language education studies and was used to classify reading materials for language learners. Later, RA was applied to many other applications, such as aiding automatic text simplification.  This thesis is aimed at improving the way of using Transformer for RA. The motivation is the “pipeline” effect (Tenney et al., 2019) of pretrained Transformers: lexical, syntactic, and semantic features are best encoded with different layers of a Transformer model.  After a preliminary test of a basic RA model that resembles the previous works, we proposed several methods to enhance the performance: by using a Transformer layer that is not the last, by concatenating or mixing the outputs of all layers, and by using syntax-augmented Transformer layers. We examined these enhanced methods on three datasets: WeeBit, OneStopEnglish, and CommonLit.  We observed that the improvements showed a clear correlation with the dataset characteristics. On the OneStopEnglish and the CommonLit datasets, we achieved absolute improvements of 1.2% in F1 score and 0.6% in Pearson’s correlation coefficients, respectively. We also show that an 𝑛-gram frequency- based baseline, which is simple but was not reported in previous works, has superior performance on the classification datasets (WeeBit and OneStopEnglish), prompting further research on vocabulary-based lexical features for RA.

In recent years, deep learning has revolutionized computer vision and natural language processing tasks, but the black-box nature of these models poses significant challenges for their interpretability and reliability, especially in critical applications such as healthcare. To address this, attention-based methods have been proposed to enhance the focus and interpretability of deep learning models. In this dissertation, we investigate the effectiveness of attention mechanisms in improving prediction and modeling tasks across different domains. We propose three essays that utilize task-specific designed trainable attention modules in manufacturing, healthcare, and system identification applications. In essay 1, we introduce a novel computer vision tool that tracks the melt pool in X-ray images of laser powder bed fusion using attention modules. In essay 2, we present a mask-guided attention (MGA) classifier for COVID-19 classification on lung CT scan images. The MGA classifier incorporates lesion masks to improve both the accuracy and interpretability of the model, outperforming state-of-the-art models with limited training data. Finally, in essay 3, we propose a Transformer-based model, utilizing self-attention mechanisms, for parameter estimation in system dynamics models that outpaces the conventional system calibration methods. Overall, our results demonstrate the effectiveness of attention-based methods in improving deep learning model performance and reliability in diverse applications. / Doctor of Philosophy / Deep learning, a type of artificial intelligence, has brought significant advancements to tasks like recognizing images or understanding texts. However, the inner workings of these models are often not transparent, which can make it difficult to comprehend and have confidence in their decision-making processes. Transparency is particularly important in areas like healthcare, where understanding why a decision was made can be as crucial as the decision itself. To help with this, we've been exploring an interpretable tool that helps the computer focus on the most important parts of the data, which we call the ``attention module''. Inspired by the human perception system, these modules focus more on certain important details, similar to how our eyes might be drawn to a familiar face in a crowded room. We propose three essays that utilize task-specific attention modules in manufacturing, healthcare, and system identification applications. In essay one, we introduce a computer vision tool that tracks a moving object in a manufacturing X-ray image sequence using attention modules. In the second essay, we discuss a new deep learning model that uses focused attention on lung lesions for more accurate COVID-19 detection on CT scan images, outperforming other top models even with less training data. In essay three, we propose an attention-based deep learning model for faster parameter estimation in system dynamics models. Overall, our research shows that attention-based methods can enhance the performance, transparency, and usability of deep learning models across diverse applications.

The rapid growth of the world's urban population shows that people are increasingly moving to cities. In recent decades, the frequent occurrence of smog caused by increasing industrialization has brought environmental pollution to record highs. Therefore, the need to develop forecasting models about air quality occurs when the ambient air contains gasses, dust particles, smoke or odors in quantities large enough to be harmful to organic life. Accurate forecasts help people anticipate environmental conditions and act consequently to decrease dangerous pollution levels, reducing health impacts and associated costs. Rather than investigating deterministic models that attempt to simulate physical processes and develop complex mathematical simulations, this paper will focus on statistical methods, studying historical information and extracting information from data patterns. In looking for new reliable air quality forecasting methods, the goal was to develop and test an artifact based on the Transformer architecture, a novel technique initially developed for natural language processing tasks. Testing was performed against recurrent and convolutional, well-established deep-learning models successfully implemented in many applications, including time-series forecasting. Two different Transformer models were tested, one using time embeddings in the same manner as proposed in the original paper, while in the second model, the Time2Vec method has been adapted. The obtained results reveal that, even though not necessarily better than reference models, both Transformers could output accurate predictions and perform almost as well as recurrent and convolutional models.

In this project dynamic thermal rating for transformers and its effect on transformer reliability are investigated. A literature review is done on different thermal models used for this purpose and at the end differential equations model from IEC 60076-7 and bottom oil model from IEEE C57.91-1995 standard are selected to calculate hot spot temperature. A wind farm connected transformer is selected to implement the models. This transformer belongs to Ellevio AB and manufactured by ABB AB. Load data are gathered for every 5 minutes during 2016. Loss of life of transformer is calculated and it is concluded that by considering this factor, the transformer is overdesigned. The optimum size of transformer by utilizing dynamic rating is selected which results in a reduction in investment cost. This method can be used to select the appropriate size oft ransformer by taking advantages of ambient temperature variations and overload the transformer beyond nameplate ratings without exceeding transformer temperature limitations. Moreover, the risk of overloading the transformer at any time during 2016 is calculated. The risk of overloading is quantified as loss of life of transformer. It is shown that this risk is a function of ambient temperature and the duration of overloading. Finally, an economic analysis is done to demonstrate economic benefit of expanding wind farm by overloading the existing transformer by reducing the transformer life expectancy while keeping it in a safe limit. / I detta projekt undersoks dynamisk varmeklassicering for transformatorer ochdess eekt pa transformatorns tillforlitlighet. En litteraturoversikt gors pa olikatermiska modeller som anvands for detta andamal och i slutet av dierentialekvationsmodellenfran IEC 60076-7 och bottenoljemodellen fran IEEE C57.91-1995 standard valjes for att berakna varmpunktstemperatur. En transformatormed vindkraftpark valjs for att genomfora modellerna. Denna transformatortillhor Ellevio AB och tillverkas av ABB AB. Lastdata samlas in for var 5: eminut under 2016. Transformatorns livslangd beraknas och det slutsatsen atttransformatorn ar overdesignad med hansyn till denna faktor. Den optimalastorleken pa transformatorn genom att anvanda dynamisk rating valjs vilket resulterari en minskning av investeringskostnaden. Denna metod kan anvandasfor att valja lamplig storlek for transformatorn genom att dra fordel av omgivandetemperaturvariationer och overbelasta transformatorn bortom markskyltarutan att overskrida transformatortemperaturbegransningar. Dessutomberaknas risken for overbelastning av transformatorn nar som helst under 2016.Risken for overbelastning kvantieras som forlust av livslangd for transformatorn.Det visas att denna risk ar en funktion av omgivande temperatur ochvaraktigheten av overbelastning. Slutligen gors en ekonomisk analys for attvisa ekonomisk nytta av att expandera vindkraftparken genom att overbelastaden bentliga transformatorn genom att minska transformatorens livslangd samtidigtsom den halls i en saker grans.

京都大学 / 新制・課程博士 / 博士(人間健康科学) / 甲第23825号 / 人健博第96号 / 新制||人健||7(附属図書館) / 京都大学大学院医学研究科人間健康科学系専攻 / (主査)教授 椎名 毅, 教授 精山 明敏, 教授 中本 裕士 / 学位規則第4条第1項該当 / Doctor of Human Health Sciences / Kyoto University / DFAM