Advancements in high-throughput technologies have led to an exponential increase in the generation of multi-modal data in computational biology. These datasets, comprising diverse biological measurements such as genomics, transcriptomics, proteomics, metabolomics, and imaging data, offer a comprehensive view of biological systems at various levels of complexity. However, integrating and analyzing such heterogeneous data present significant challenges due to differences in data modalities, scales, and noise levels. Another challenge for multi-modal analysis is the complex interaction network that the modalities share. Understanding the intricate interplay between different biological modalities is essential for unraveling the underlying mechanisms of complex biological processes, including disease pathogenesis, drug response, and cellular function. Machine learning algorithms have emerged as indispensable tools for studying multi-modal data in computational biology, enabling researchers to extract meaningful insights, identify biomarkers, and predict biological outcomes. In this dissertation, we first propose a multi-modal integration framework that takes two interconnected data modalities and their interaction network to iteratively update the modalities into new representations with better disease outcome predictive abilities. The deep learning-based model underscores the importance and performance gains achieved through the incorporation of network information into integration process. Additionally, a multi-modal framework is developed to estimate protein expression from mRNA and microRNA (miRNA) expressions, along with the mRNA-miRNA interaction network. The proposed network propagation model simulates in-vivo miRNA regulation on mRNA translation, offering a cost-effective alternative to experimental protein quantification. Analysis reveals that predicted protein expression exhibits a stronger correlation with ground truth protein expression compared to mRNA expression. Moreover, the effectiveness of integrative models is contingent upon the quality of input data modalities and the completeness of interaction networks, with missing values and network noise adversely affecting downstream tasks. To address these challenges, two multi-modal imputation models are proposed, facilitating the imputation of missing values in time series data. The first model allows the imputation of missing values in time series gene expression utilizing single nucleotide polymorphism (SNP) data for children at high risk of type 1 diabetes. The imputed gene expression allows us to predict the progression towards type 1 diabetes at birth with six years prediction horizon. Subsequently, a follow-up study introduces a generalized multi-modal imputation framework capable of imputing missing values in time series data using either another time series or cross-sectional data collected from the same set of samples. These models excel at imputation tasks, whether values are missing randomly or an entire time step in the series is absent. Additionally, leveraging the additional modality, they are able to estimate a completely missing time series without prior values. Finally, to mitigate noise in the interaction network, a link prediction framework for drug-target interaction prediction is developed. This study demonstrates exceptional performance in cold start predictions and investigates the efficacy of large language models for such predictions. Through a comprehensive review and evaluation of state-of-the-art algorithms, this dissertation aims to provide researchers with valuable insights, methodologies, and tools for harnessing the rich information embedded within multi-modal biological datasets.

This dissertation aims to provide critical methodological advancements for sensor fusion and physics-informed machine learning in metal additive manufacturing (MAM) to assist practitioners in detecting quality control structural anomalies. In MAM, there is an urgent need to improve knowledge of the internal layer fusion process and geometric variation occurring during the directed energy deposition processes. A core challenge lies in the cyclic heating process, which results in various structural abnormalities and deficiencies, reducing the reproducibility of manufactured components. Structural abnormalities include microstructural heterogeneities, porosity, deformation and distortion, and residual stresses. Data-driven monitoring in MAM is needed to capture process variability, but challenges arise due to the inability to capture the thermal history distribution process and structural changes below the surface due to limitations in in-situ data collection capabilities, physical domain knowledge integration, and multi-data and multi-physical data fusion. The research gaps in developing system-based generalizable artificial intelligence (AI) and machine learning (ML) to detect abnormalities are threefold. (1) Limited fusion of various types of sensor data without handcrafted selection of features. (2) There is a lack of physical domain knowledge integration for various systems, geometries, and materials. (3) It is essential to develop sensor and system integration platforms to enable a holistic view to make quality control predictions in the additive manufacturing process. In this dissertation, three studies utilize various data types and ML methodologies for predicting in-process anomalies. First, a complementary sensor fusion methodology joins thermal and ultrasonic image data capturing layer fusion and structural knowledge for layer-wise porosity segmentation. Secondly, a physics-informed data-driven methodology for joining thermal infrared image data with Goldak heat flux improves thermal history simulation and deformation detection. Lastly, a physics-informed machine learning methodology constrained by thermal physical functions utilizes in-process multi-modal monitoring data from a digital twin environment to predict distortion in the weld bead. This dissertation provides current practitioners with data-driven and physics-based interpolation methods, multi-modal sensor fusion, and anomaly detection insights trained and validated with three case studies.

Master of Landscape Architecture / Department of Landscape Architecture/Regional and Community Planning / Blake M. Belanger / The main streets of the Rocky Mountain West are the social, economic, and cultural centers of their respective communities. Often, these main streets may deteriorate or become abandoned as a result of edge shopping malls and strip style economic development. Thus, a downtown or main street redevelopment effort by the community can help to ensure these economic centers remain. Yet, too often, the redevelopment efforts are oversimplified and fail to integrate the most current street development principles and design initiatives that can benefit not only the community but also the surrounding environment. I n the modern American city, almost half of all daily trips are less than three miles and a third are under one mile. (McCann 2010) “These are distances easily traversed by foot or bicycle, yet 65 percent of trips under one mile are made by automobile.” (McCann 2010) This mobility trend has led to the foundation of programs and organizations that try to promote non-motorized travel. Although these initiatives respond to the human/physical environment, they are far from comprehensive. Today, an integration of smart ecological ideals is essential. How can the revitalization efforts of Rocky Mountain communities be guided to ensure they consider not only the built environment; but also the natural environment? The face of the future main street will be multi-modal and ecologically responsible. Yet, there is presently no clear method of combining the two. A union of the multi-modal principles behind Complete Streets and the ecologically responsible ideals green infrastructure can provide a framework for a new and more inclusive redevelopment approach. The merging of modern ecological and street design principles can lead to a comprehensive Main Street redevelopment program and therefore successfully guide the revitalization efforts of small Rocky Mountain communities in a way that is responsive to future development needs as well as the cultural and ecological aspects of the region. Main Street Evolved will provide a set of tools to guide Colorado Rocky Mountain Main Street redevelopment efforts by providing strategies and implementation guidelines that focus on balancing multi-modal ideals and ecological stormwater management techniques within a small-town mountain context.

This thesis presents novel solutions for two fundamental problems associated with autonomous road driving. The first is accurate and persistent localisation and the second is automatic extrinsic sensor calibration. We start by describing a stereo Visual Odometry (VO) system, which forms the basis of later chapters. This sparse approach to ego-motion estimation leverages the efficacy and speed of the BRIEF descriptor to measure frame-to-frame correspondences and infer subsequent motion. The system is able to output locally metric trajectory estimates as demonstrated on many kilometres of data. We then present a robust vision only localisation system based on a two-stage approach. Firstly we gather a representative survey in ideal weather and lighting conditions. We then leverage locally accurate VO trajectories to synthesise a high resolution orthographic image strip of the road surface. This road image provides a highly descriptive and stable template against which to match subsequent traversals. During the second phase, localisation, we use the VO to provide high frequency pose updates, but correct for the drift inherent in all locally derived pose estimates with low frequency updates from a dense image matching technique. Here a live image stream is registered against synthesised views of the road image generated form the survey. We use an information theoretic measure, Mutual Information, to determine the alignment of live images and synthesised views. Using this measure we are able to successfully localise subsequent traversals of surveyed routes under even the most intense lighting changes expected in outdoor applications. We demonstrate our system localising in multiple environments with accuracy commensurate to that of an Inertial Navigation System. Finally we present a technique for automatically determining the extrinsic calibration between a camera and Light Detection And Ranging (LIDAR) sensor in natural scenes. Rather than requiring a stationary platform as with prior art, we actually exploit platform motion allowing us to aggregate data and adopt a retrospective approach to calibration. Coupled with accurate timing this retrospective approach allows for sensors with non-overlapping fields of view to be calibrated as long as at some point the observed workspaces overlap. We then show how we can improve the accuracy of our calibration estimates by treating each single shot estimate as a noisy measurement and fusing them together using a recursive Bayes filter. We evaluate the calibration algorithm in multiple environments and demonstrate millimetre precision in translation and deci-degrees in rotation.

<p>Advancements in retinal imaging technologies have drastically improved the quality of eye care in the past couple decades. Scanning laser ophthalmoscopy (SLO) and optical coherence tomography (OCT) are two examples of critical imaging modalities for the diagnosis of retinal pathologies. However current-generation SLO and OCT systems have limitations in diagnostic capability due to the following factors: the use of bulky tabletop systems, monochromatic imaging, and resolution degradation due to ocular aberrations and diffraction. </p><p>Bulky tabletop SLO and OCT systems are incapable of imaging patients that are supine, under anesthesia, or otherwise unable to maintain the required posture and fixation. Monochromatic SLO and OCT imaging prevents the identification of various color-specific diagnostic markers visible with color fundus photography like those of neovascular age-related macular degeneration. Resolution degradation due to ocular aberrations and diffraction has prevented the imaging of photoreceptors close to the fovea without the use of adaptive optics (AO), which require bulky and expensive components that limit the potential for widespread clinical use. </p><p>In this dissertation, techniques for extending the diagnostic capability of SLO and OCT systems are developed. These techniques include design strategies for miniaturizing and combining SLO and OCT to permit multi-modal, lightweight handheld probes to extend high quality retinal imaging to pediatric eye care. In addition, a method for extending true color retinal imaging to SLO to enable high-contrast, depth-resolved, high-fidelity color fundus imaging is demonstrated using a supercontinuum light source. Finally, the development and combination of SLO with a super-resolution confocal microscopy technique known as optical photon reassignment (OPRA) is demonstrated to enable high-resolution imaging of retinal photoreceptors without the use of adaptive optics.</p> / Dissertation

Le recalage des images du cerveau vise à réduire la variabilité anatomique entre les differentes sujets, et à créer un espace commun pour l'analyse de groupe. Les approches multi-modales essaient de minimiser les variations de forme du cortex et des structures internes telles que des faisceaux de fibres nerveuses. Ces approches nécessitent une identification préalable de ces structures, ce qui s'avère une tâche difficile en l'absence d'un atlas complet de référence. Nous proposons une extension de l'algorithme de recalage difféomorphe des Démons pour recaler conjointement des images et des faisceaux de fibres. Dans cette thèse, nous analysons différentes représentations des faisceaux de fibres comme une séquence de points, un nuage de points, les courants et les mesures. Différentes distances sont analysées et étudiées dans l'algorithme de recalage. Pour simplifier la représentation de la matière blanche nous utilisons et étendons les algorithmes de classification existants. En étendant le recalage d'images afin d'ajouter des descripteurs de la géométrie des fibres nerveuses, nous espérons améliorer les futures analyses concernant les matières grise et blanche. Nous avons démontré l'efficacité de notre algorithme en recalant conjointement des images anatomiques pondérées en T1 et des faisceaux de fibres. Nous avons comparé nos résultats à des approches concurrentes, l'une multimodale s'appuyant sur l'anisotropie fractionnaire et la pondération T1, l'autre sur les tenseurs de diffusion, et obtenu de meilleures performances à l'aide de notre algorithme. Enfin, nous mettons en évidence sur des études de groupe en IRMf que notre méthodologie et notre implémentation apportent un gain en sensibilité de détection des activations cérébrales. / Brain image registration aims at reducing anatomical variability across subjects to create a common space for group analysis. Multi-modal approaches intend to minimize cortex shape variations along with internal structures, such as fiber bundles. These approaches require prior identification of the structures, which remains a challenging task in the absence of a complete reference atlas. We propose an extension of the Diffeomorphic Demons image registration to jointly register images and fiber bundles. In this thesis we analyze differents representations of the fiber bundles such as ordered points, clouds of points, Currents and Measures. Different distances are analyzed and implemented into the registration algorithm. To simplify white matter representation we also analyze, use and extend existing clustering algorithms. By extending the image registration to include geometric fiber bundles descriptors we hope to improve future analyses regarding both, grey and white matter. We demonstrate the efficacy of our algorithm by registering simultaneously T1 images and fiber bundles and compare results with a multi-modal T1+Fractional Anisotropy (FA) and a tensor-based registration algorithms and obtain superior performance with our approach. We provide preliminary evidence that our implementation improves the sensitivity of activation detection in fMRI group studies.

This dissertation describes a new methodology for multi-modal (2-D + 3-D) face modeling and recognition. There are advantages in using each modality for face recognition. For example, the problems of pose variation and illumination condition, which cannot be resolved easily by using the 2-D data, can be handled by using the 3-D data. However, texture, which is provided by 2-D data, is an important cue that cannot be ignored. Therefore, we use both the 2-D and 3-D modalities for face recognition and fuse the results of face recognition by each modality to boost the overall performance of the system. In this dissertation, we consider two different cases for multi-modal face modeling and recognition. In the first case, the 2-D and 3-D data are registered. In this case we develop a unified graph model called Attributed Relational Graph (ARG) for face modeling and recognition. Based on the ARG model, the 2-D and 3-D data are included in a single model. The developed ARG model consists of nodes, edges, and mutual relations. The nodes of the graph correspond to the landmark points that are extracted by an improved Active Shape Model (ASM) technique. In order to extract the facial landmarks robustly, we improve the Active Shape Model technique by using the color information. Then, at each node of the graph, we calculate the response of a set of log-Gabor filters applied to the facial image texture and shape information (depth values); these features are used to model the local structure of the face at each node of the graph. The edges of the graph are defined based on Delaunay triangulation and a set of mutual relations between the sides of the triangles are defined. The mutual relations boost the final performance of the system. The results of face matching using the 2-D and 3-D attributes and the mutual relations are fused at the score level. In the second case, the 2-D and 3-D data are not registered. This lack of registration could be due to different reasons such as time lapse between the data acquisitions. Therefore, the 2-D and 3-D modalities are modeled independently. For the 3-D modality, we developed a fully automated system for 3-D face modeling and recognition based on ridge images. The problem with shape matching approaches such as Iterative Closest Points (ICP) or Hausdorff distance is the computational complexity. We model the face by 3-D binary ridge images and use them for matching. In order to match the ridge points (either using the ICP or the Hausdorff distance), we extract three facial landmark points: namely, the two inner corners of the eyes and the tip of the nose, on the face surface using the Gaussian curvature. These three points are used for initial alignment of the constructed ridge images. As a result of using ridge points, which are just a fraction of the total points on the surface of the face, the computational complexity of the matching is reduced by two orders of magnitude. For the 2-D modality, we model the face using an Attributed Relational Graph. The results of the 2-D and 3-D matching are fused at the score level. There are various techniques to fuse the 2-D and 3-D modalities. In this dissertation, we fuse the matching results at the score level to enhance the overall performance of our face recognition system. We compare the Dempster-Shafer theory of evidence and the weighted sum rule for fusion. We evaluate the performance of the above techniques for multi-modal face recognition on various databases such as Gavab range database, FRGC (Face Recognition Grand Challenge) V2.0, and the University of Miami face database.

Research in the areas of target detection, tracking, and geo-location is most important for enabling an unmanned aerial vehicle (UAV) platform to autonomously execute a mission or task without the need for a pilot or operator. Small-class UAVs and video camera sensors complemented with "soft sensors" realized only in software as a combination of a priori knowledge and sensor measurements are called upon to replace the cumbersome precision sensors on-board a large class UAV. The objective of this research is to develop a geo-location solution for use on-board multiple UAVs with mounted video camera sensors only to accurately geo-locate and track a target. This research introduces an estimation solution that combines the power of the particle filter with the utility of the video sensor as a general solution for passive target geo-location on-board multiple UAVs. The particle filter is taken advantage of, with its ability to use all of the available information about the system model, system uncertainty, and the sensor uncertainty to approximate the statistical likelihood of the target state. The geo-location particle filter is tested online and in real-time in a simulation environment involving multiple UAVs with video cameras and a maneuvering ground vehicle as a target. Simulation results show the geo-location particle filter estimates the target location with a high accuracy, the addition of UAVs or particles to the system improves the location estimation accuracy with minimal addition of processing time, and UAV control and trajectory generation algorithms restrict each UAV to a desired range to minimize error.

Wind waves are highly complex, random phenomena. One way to describe the irregular nature of the sea surface is the use of wave energy spectrum. Spectral information for wind waves in the Black Sea is extremely limited. Knowledge on spectral characteristics of wind waves would contribute to scientific, engineering and operational coastal and marine activities in the Black Sea. The aim of the present thesis is to investigate characteristics of wind wave spectra for the Eastern Black Sea. This would allow detailed understanding of the nature of the waves occurring in this enclosed basin. Long-term wave measurements obtained by directional buoys deployed offshore at Sinop, Hopa and Gelendzhik were utilized as the three sets of wave data. Records were analyzed to identify them as uni-modal or multi-modal spectra, and occurrences of spectral peaks were computed. Single peaked spectra were studied as belonging to fully arisen or developing sea states. Model parameters of JONSWAP and PM spectra were estimated for the observed spectra by using a least square error method. The records of developing seas were further analyzed to select the ones belonging to stable wind conditions. Fetch dependencies of non-dimensional spectral variables, mean parameters of JONSWAP model spectrum and the envelop of dimensionless spectra were investigated for this data sub-set.

The purpose of my project is to determine how a multimodal/intermedial text can be used when teaching in upper secondary school. Can using a text that to some point is current in the pupils’ lives lift the interest for the specific subject? I opted to pick up Gunther Kress’ research, Diana Laurillard’s book and Christina Olin-Scheller’s work on the subject. I have used several hours of lesson planning and the lesson itself with subsequent work consisting, among other things, of three qualitative interviews to achieve my purpose. Based on a qualitative approach, I discuss the results, which show that it is in fact very uplifting. It is possible to work with multimodal/intermedial/intertextual texts, in many ways. / Syftet med denna uppsats är att undersöka hur en multimodal/intermedial text kan användas i undervisningen på gymnasienivå. Detta skall ske genom att utgå från elevernas perspektiv och referensramar, söka rätt på en aktuell text som finns i elevernas vardag och se om det blir intressant för eleverna att arbeta med en sådan text. Jag har valt att arbeta med Gunther Kress forskning, Diana Laurillards bok om ämnet och även Christina Olin-Schellers avhandling om ämnet, samt därutöver relevant och nödvändigt material för att vetenskapligt förankra begrepp och synpunkter.                       Jag har använt mig av en lektionsplanering, en utförd lektion samt tre kvalitativa intervjuer i efterarbetet av lektionen för att uppnå mitt syfte. Lektionen utspelade sig våren 2012 på en svensk gymnasieskola. Utifrån ett kvalitativt förhållningssätt diskuterar jag resultatet, vilket visar en mycket positiv respons på att hämta upp en multimodal/intermedial aktuell text från elevernas vardagsmiljö, och arbeta med denna i undervisningen.