A common practice for solving machine learning problems is currently to consider each problem in isolation, starting from scratch every time a new learning problem is encountered or a new model is proposed. This is a perfectly feasible solution when the problems are sufficiently easy or, if the problem is hard when a large amount of resources, both in terms of the training data and computation, are available. Although this naive approach has been the main focus of research in machine learning for a few decades and had a lot of success, it becomes infeasible if the problem is too hard in proportion to the available resources. When using a complex model in this naive approach, it is necessary to collect large data sets (if possible at all) to avoid overfitting and hence it is also necessary to use large computational resources to handle the increased amount of data, first during training to process a large data set and then also at test time to execute a complex model. An alternative to this strategy of treating each learning problem independently is to leverage related data sets and computation encapsulated in previously trained models. By doing that we can decrease the amount of data necessary to reach a satisfactory level of performance and, consequently, improve the accuracy achievable and decrease training time. Our attack on this problem is to exploit diversity - in the structure of the data set, in the features learnt and in the inductive biases of different neural network architectures. In the setting of learning from multiple sources we introduce multiple-source cross-validation, which gives an unbiased estimator of the test error when the data set is composed of data coming from multiple sources and the data at test time are coming from a new unseen source. We also propose new estimators of variance of the standard k-fold cross-validation and multiple-source cross-validation, which have lower bias than previously known ones. To improve unsupervised learning we introduce scheduled denoising autoencoders, which learn a more diverse set of features than the standard denoising auto-encoder. This is thanks to their training procedure, which starts with a high level of noise, when the network is learning coarse features and then the noise is lowered gradually, which allows the network to learn some more local features. A connection between this training procedure and curriculum learning is also drawn. We develop further the idea of learning a diverse representation by explicitly incorporating the goal of obtaining a diverse representation into the training objective. The proposed model, the composite denoising autoencoder, learns multiple subsets of features focused on modelling variations in the data set at different levels of granularity. Finally, we introduce the idea of model blending, a variant of model compression, in which the two models, the teacher and the student, are both strong models, but different in their inductive biases. As an example, we train convolutional networks using the guidance of bidirectional long short-term memory (LSTM) networks. This allows to train the convolutional neural network to be more accurate than the LSTM network at no extra cost at test time.

Nowadays machine learning (especially deep learning) techniques are being incorporated to many intelligent systems affecting the quality of human life. The ultimate purpose of these systems is to perform automated decision making, and in order to achieve this, predictive systems need to return estimates of their confidence. Powered by the rules of probability, Bayesian inference is the gold standard method to perform coherent reasoning under uncertainty. It is generally believed that intelligent systems following the Bayesian approach can better incorporate uncertainty information for reliable decision making, and be less vulnerable to attacks such as data poisoning. Critically, the success of Bayesian methods in practice, including the recent resurgence of Bayesian deep learning, relies on fast and accurate approximate Bayesian inference applied to probabilistic models. These approximate inference methods perform (approximate) Bayesian reasoning at a relatively low cost in terms of time and memory, thus allowing the principles of Bayesian modelling to be applied to many practical settings. However, more work needs to be done to scale approximate Bayesian inference methods to big systems such as deep neural networks and large-scale dataset such as ImageNet. In this thesis we develop new algorithms towards addressing the open challenges in approximate inference. In the first part of the thesis we develop two new approximate inference algorithms, by drawing inspiration from the well known expectation propagation and message passing algorithms. Both approaches provide a unifying view of existing variational methods from different algorithmic perspectives. We also demonstrate that they lead to better calibrated inference results for complex models such as neural network classifiers and deep generative models, and scale to large datasets containing hundreds of thousands of data-points. In the second theme of the thesis we propose a new research direction for approximate inference: developing algorithms for fitting posterior approximations of arbitrary form, by rethinking the fundamental principles of Bayesian computation and the necessity of algorithmic constraints in traditional inference schemes. We specify four algorithmic options for the development of such new generation approximate inference methods, with one of them further investigated and applied to Bayesian deep learning tasks.

Identification of different cell types is an indispensable task of biomedical research and clinical application. During the last decades, much attention was given to molecular characterization, and many cell types can now be identified using established markers that bind to cell-specific antigens. The required staining process is a lengthy and costly treatment, which can cause alterations of cellular properties, contaminate the sample and therefore limit its subsequent use. For example, for photoreceptor transplantations, highly pure samples of photoreceptor cells are required, which can currently only be obtained using molecular labelling, rendering the resulting sample incompatible for clinical application. A promising alternative to molecular markers is the label-free identification of cells using mechanical or morphological features. Real-time deformability cytometry (RT DC) is a microfluidic technique, which allows capturing both types of information simultaneously for single cells at high-throughput. In this thesis, I present machine learning methods which allow identifying different cell types, based on bright-field images from RT DC. In particular, I introduce algorithms that are fast enough to be applied in real-time during the measurement (at >1000 cells/s), which can be used for image-based cell sorting. The performance of the algorithms is shown for the identification of rod precursor cells in retina-samples, indicating that image-based sorting based on those algorithms would allow enriching photoreceptors to a final concentration, applicable for transplantation purposes.:Contents Abstract iii Kurzfassung iv List of figures viii List of tables x 1. Introduction 1 1.1. Texture and mechanical properties: label-free markers 4 1.2. The retina, diseases and cure by photoreceptor transplantation 5 1.3. Technologies for label-free assessment of cells 8 2. Materials and Methods 10 2.1. Experimental setup 10 2.1.1. Chip design for RT DC and RT-FDC 10 2.1.2. Chip design for soRT-FDC 11 2.1.3. Chip fabrication 13 2.1.4. RT-DC, RT-FDC and soRT FDC Setup 14 2.1.5. Physics of surface acoustic wave mediated sorting 16 2.1.6. Measurement buffer (MB) for RT DC 17 2.2. Online parameters 18 2.3. Offline parameters 21 2.4. Linear mixed models (LMM) 28 2.5. Normality test using probability plots 31 2.6. Gaussian mixture model (GMM) and Bayesian information criterion (BIC) 32 2.7. Random forests 33 2.8. Confusion matrix 34 2.9. Deep learning 36 2.10. Preparation of retina samples 43 2.11. Preparation of blood samples 44 2.12. Staining of neutrophils and monocytes 45 3. Results 46 3.1. Meta-analysis of RT-DC data 46 3.1.1. Correlations of area and volume 47 3.1.2. Correlations of deformation and inertia ratio 48 3.1.3. Further screening of correlations 50 3.1.4. Shape of distributions 52 3.1.5. Discussion 54 3.2. Characterization of retina cells in RT-DC 57 3.2.1. Maturation of retina cells 57 3.2.2. Comparing retina cell types using statistical tests 61 3.2.3. Discussion 63 3.3. Classification of retina cells using supervised machine learning 66 3.3.1. The dataset 66 3.3.2. Cell classification using optimized area gating 72 3.3.3. Cell classification using random forests 74 3.3.4. Cell classification using deep neural nets 80 3.3.5. Improving DNN accuracy using image augmentation 85 3.3.6. Tuning of final models and classification performance 93 3.3.7. Visualization of model attention 98 3.3.8. Discussion 100 3.4. Software tools to train and apply deep neural nets for sorting 105 3.4.1. AIDeveloper 105 3.4.2. Sorting Software 113 3.4.3. Discussion 114 3.5. Sorting experiments 117 3.5.1. Sorting of rod precursor cells 117 3.5.2. Sorting of neutrophils 120 3.5.3. Discussion 125 4. Conclusion and outlook 128 A. Appendix 131 I. Comparison of dense and convolutional layer 131 Bibliography 133 Acronyms 148 Acknowledgements 150 Erklärung 152

Cell-to-cell variations and intracellular processes such as cytoskeletal organization and organelle dynamics exhibit massive heterogeneity. Advances in imaging and optics have enabled researchers to access spatiotemporal information in living cells efficiently. Even though current imaging technologies allow us to acquire an unprecedented amount of cell images, it is challenging to extract valuable information from the massive and complex dataset to interpret heterogeneous biological processes. Machine learning (ML), referring to a set of computational tools to acquire knowledge from data, provides promising solutions to meet this challenge. In this dissertation, we developed ML pipelines for deconvolution of subcellular protrusion heterogeneity from live cell imaging and molecular diagnostic from lens-free digital in-line holography (LDIH) imaging. Cell protrusion is driven by spatiotemporally fluctuating actin assembly processes and is morphodynamically heterogeneous at the subcellular level. Elucidating the underlying molecular dynamics associated with subcellular protrusion heterogeneity is crucial to understanding the biology of cellular movement. Traditional ensemble averaging methods without characterizing the heterogeneity could mask important activities. Therefore, we established an ACF (auto-correlation function) based time series clustering pipeline called HACKS (deconvolution of heterogeneous activities in coordination of cytoskeleton at the subcellular level) to identify distinct subcellular lamellipodial protrusion phenotypes with their underlying actin regulator dynamics from live cell imaging. Using our method, we discover “accelerating protrusion”, which is driven by the temporally ordered coordination of Arp2/3 and VASP activities. Furthermore, deriving the merits of ML, especially Deep Learning (DL) to learn features automatically, we advanced our pipeline to learn fine-grained temporal features by integrating the prior ML analysis results with bi-LSTM (bi-direction long-short term memory) autoencoders to dissect variable-length time series protrusion heterogeneity. By applying it to subcellular protrusion dynamics in pharmacologically and metabolically perturbed epithelial cells, we discovered fine differential response of protrusion dynamics specific to each perturbation. This provides an analytical framework for detailed and quantitative understanding of molecular mechanisms hidden in their heterogeneity. Lens-free digital in-line holography (LDIH) is a promising microscopic tool that overcomes several drawbacks (e.g., limited field of view) of traditional lens-based microscopy. Numerical reconstruction for hologram images from large-field-of-view LDIH is extremely time-consuming. Until now, there are no effective manual-design features to interpret the lateral and depth information from complex diffraction patterns in hologram images directly, which limits LDIH utility for point-of-care applications. Inherited from advantages of DL to learn generalized features automatically, we proposed a deep transfer learning (DTL)-based approach to process LDIH images without reconstruction in the context of cellular analysis. Specifically, using the raw holograms as input, the features extracted from a well-trained network were able to classify cell categories according to the number of cell-bounded microbeads, which performance was comparable with that of object images as input. Combined with the developed DTL approach, LDIH could be realized as a low-cost, portable tool for point-of-care diagnostics. In summary, this dissertation demonstrate that ML applied to cell imaging can successfully dissect subcellular heterogeneity and perform cell-based diagnosis. We expect that our study will be able to make significant contributions to data-driven cell biological research.

Cell-to-cell variations and intracellular processes such as cytoskeletal organization and organelle dynamics exhibit massive heterogeneity. Advances in imaging and optics have enabled researchers to access spatiotemporal information in living cells efficiently. Even though current imaging technologies allow us to acquire an unprecedented amount of cell images, it is challenging to extract valuable information from the massive and complex dataset to interpret heterogeneous biological processes. Machine learning (ML), referring to a set of computational tools to acquire knowledge from data, provides promising solutions to meet this challenge. In this dissertation, we developed ML pipelines for deconvolution of subcellular protrusion heterogeneity from live cell imaging and molecular diagnostic from lens-free digital in-line holography (LDIH) imaging. Cell protrusion is driven by spatiotemporally fluctuating actin assembly processes and is morphodynamically heterogeneous at the subcellular level. Elucidating the underlying molecular dynamics associated with subcellular protrusion heterogeneity is crucial to understanding the biology of cellular movement. Traditional ensemble averaging methods without characterizing the heterogeneity could mask important activities. Therefore, we established an ACF (auto-correlation function) based time series clustering pipeline called HACKS (deconvolution of heterogeneous activities in coordination of cytoskeleton at the subcellular level) to identify distinct subcellular lamellipodial protrusion phenotypes with their underlying actin regulator dynamics from live cell imaging. Using our method, we discover “accelerating protrusion”, which is driven by the temporally ordered coordination of Arp2/3 and VASP activities. Furthermore, deriving the merits of ML, especially Deep Learning (DL) to learn features automatically, we advanced our pipeline to learn fine-grained temporal features by integrating the prior ML analysis results with bi-LSTM (bi-direction long-short term memory) autoencoders to dissect variable-length time series protrusion heterogeneity. By applying it to subcellular protrusion dynamics in pharmacologically and metabolically perturbed epithelial cells, we discovered fine differential response of protrusion dynamics specific to each perturbation. This provides an analytical framework for detailed and quantitative understanding of molecular mechanisms hidden in their heterogeneity. Lens-free digital in-line holography (LDIH) is a promising microscopic tool that overcomes several drawbacks (e.g., limited field of view) of traditional lens-based microscopy. Numerical reconstruction for hologram images from large-field-of-view LDIH is extremely time-consuming. Until now, there are no effective manual-design features to interpret the lateral and depth information from complex diffraction patterns in hologram images directly, which limits LDIH utility for point-of-care applications. Inherited from advantages of DL to learn generalized features automatically, we proposed a deep transfer learning (DTL)-based approach to process LDIH images without reconstruction in the context of cellular analysis. Specifically, using the raw holograms as input, the features extracted from a well-trained network were able to classify cell categories according to the number of cell-bounded microbeads, which performance was comparable with that of object images as input. Combined with the developed DTL approach, LDIH could be realized as a low-cost, portable tool for point-of-care diagnostics. In summary, this dissertation demonstrate that ML applied to cell imaging can successfully dissect subcellular heterogeneity and perform cell-based diagnosis. We expect that our study will be able to make significant contributions to data-driven cell biological research.

The proliferation of multimedia data, including digital images and videos, has led to an increase in their misuse, such as the unauthorized sharing of sensitive content, the spread of fake news, and the dissemination of misleading propaganda. To address these issues, the research field of multimedia forensics has developed tools to distinguish genuine multimedia from fakes and identify the sources of those who share sensitive content. However, the accuracy and reliability of multimedia forensics tools are threatened by recent technological advancements in new multimedia processing software and camera devices. For example, source attribution involves attributing an image or video to a specific camera device, which is crucial for addressing privacy violations, cases of revenge porn, and instances of child pornography. These tools exploit forensic traces unique to each camera’s manufacturing process, such as Photo Response Non-Uniformity (PRNU). Nevertheless, image and video processing transformations can disrupt the consistency of PRNU, necessitating the development of new methods for its recovery. Conversely, to distinguish genuine multimedia from fakes, AI-based image and video forgery localization methods have also emerged. However, they constantly face challenges from new, more sophisticated AI-forgery techniques and are hindered by factors like AI-aided post-processing and, in the case of videos, lower resolutions, and stronger compression. This doctoral study investigates the relevance of exploiting temporal information during the parameters estimation used to reverse complex spatial transformations for source attribution, and video forgery localization in low-resolution H.264 post-processed inpainted videos. Two novel methods will be presented that model the set of parameters involved in reversing in-camera and out-camera complex spatial transformations applied to images and videos as time series, improving source attribution accuracy and computational efficiency. Regarding video inpainting localization, a novel dataset of videos inpainted and post-processed with Temporal Consistency Networks will be introduced, and we will present our solution to improve video inpainting localization by taking into account spatial and temporal inconsistencies at dense optical flow level. The research presented in this dissertation has resulted in several publications that contribute to the field of multimedia forensics, addressing challenges related to source attribution and video forgery localization.

This thesis addresses the problem of text spotting - being able to automatically detect and recognise text in natural images. Developing text spotting systems, systems capable of reading and therefore better interpreting the visual world, is a challenging but wildly useful task to solve. We approach this problem by drawing on the successful developments in machine learning, in particular deep learning and neural networks, to present advancements using these data-driven methods. Deep learning based models, consisting of millions of trainable parameters, require a lot of data to train effectively. To meet the requirements of these data hungry algorithms, we present two methods of automatically generating extra training data without any additional human interaction. The first crawls a photo sharing website and uses a weakly-supervised existing text spotting system to harvest new data. The second is a synthetic data generation engine, capable of generating unlimited amounts of realistic looking text images, that can be solely relied upon for training text recognition models. While we define these new datasets, all our methods are also evaluated on standard public benchmark datasets. We develop two approaches to text spotting: character-centric and word-centric. In the character-centric approach, multiple character classifier models are developed, reinforcing each other through a feature sharing framework. These character models are used to generate text saliency maps to drive detection, and convolved with detection regions to enable text recognition, producing an end-to-end system with state-of-the-art performance. For the second, higher-level, word-centric approach to text spotting, weak detection models are constructed to find potential instances of words in images, which are subsequently refined and adjusted with a classifier and deep coordinate regressor. A whole word image recognition model recognises words from a huge dictionary of 90k words using classification, resulting in previously unattainable levels of accuracy. The resulting end-to-end text spotting pipeline advances the state of the art significantly and is applied to large scale video search. While dictionary based text recognition is useful and powerful, the need for unconstrained text recognition still prevails. We develop a two-part model for text recognition, with the complementary parts combined in a graphical model and trained using a structured output learning framework adapted to deep learning. The trained recognition model is capable of accurately recognising unseen and completely random text. Finally, we make a general contribution to improve the efficiency of convolutional neural networks. Our low-rank approximation schemes can be utilised to greatly reduce the number of computations required for inference. These are applied to various existing models, resulting in real-world speedups with negligible loss in predictive power.

No description available.

Levees provide protection for vast amounts of commercial and residential properties. However, these structures degrade over time, due to the impact of severe weather, sand boils, subsidence of land, seepage, etc. In this research, we focus on detecting sand boils. Sand boils occur when water under pressure wells up to the surface through a bed of sand. These make levees especially vulnerable. Object detection is a good approach to confirm the presence of sand boils from satellite or drone imagery, which can be utilized to assist in the automated levee monitoring methodology. Since sand boils have distinct features, applying object detection algorithms to it can result in accurate detection. To the best of our knowledge, this research work is the first approach to detect sand boils from images. In this research, we compare some of the latest deep learning methods, Viola Jones algorithm, and other non-deep learning methods to determine the best performing one. We also train a Stacking-based machine learning method for the accurate prediction of sand boils. The accuracy of our robust model is 95.4%.

Sequence labeling is a type of machine learning problem that involves as- signing a label to each sequence member. Deep learning has shown good per- formance for this problem. However, one disadvantage of this approach is its requirement of having a large amount of labeled data. Semi-supervised learning mitigates this problem by using cheaper unlabeled data together with labeled data. Currently, usage of semi-supervised deep learning for sequence labeling is limited. Therefore, the focus of this thesis is on the application of semi-super- vised deep learning in sequence labeling. Existing semi-supervised deep learning approaches are examined, and approaches for sequence labeling are proposed. The proposed approaches were implemented and experimentally evaluated on named-entity recognition and part-of-speech tagging tasks.