Video super-resolution (VSR) methods align and fuse consecutive low-resolution frames to generate high-resolution frames. One of the main difficulties for the VSR process is that video contains various motions, and the accuracy of motion estimation dramatically affects the quality of video restoration. However, standard CNNs share the same receptive field in each layer, and it is challenging to estimate diverse motions effectively. Neuroscience research has shown that the receptive fields of biological visual areas will be adjusted according to the input information. Diverse receptive fields in temporal and spatial dimensions have the potential to adapt to various motions, which is rarely paid attention in most known VSR methods. In this thesis, we propose to provide adaptive receptive fields for the VSR model. Firstly, we design a multi-kernel 3D convolution network and integrate it with a multi-kernel deformable convolution network for motion estimation and multiple frames alignment. Secondly, we propose a 2D multi-kernel convolution framework to improve texture restoration quality. Our experimental results show that the proposed framework outperforms the state-of-the-art VSR methods.

This work deals with the task of image scaling using GPU paralelization. Portion of text is devoted to signal processing and his affection of whole result including measuring it's quality. Describtion of the most important methods including super-resolution is given further in the text. An important part of this thesis is library implementing choosen methods with usage of paralelization on graphic chip. Achieved results of paralelization are demonstrated on set of speed tests.

La reconstruction 3D multi-vue atteint désormais un niveau de maturité industrielle : des utilisateurs non-experts peuvent produire des modèles 3D large-échelle de qualité à l'aide de logiciels commerciaux. Ces reconstructions utilisent des capteurs haut de gamme comme des LIDAR ou des appareils photos de type DSLR, montés sur un trépied et déplacés autour de la scène. Ces protocoles d'acquisition sont mal adaptés à l’inspection d’infrastructures de grande taille, à géométrie complexe. Avec l'évolution rapide des capacités des micro-drones, il devient envisageable de leur confier ce type de tâche. Un tel choix modifie les données d’acquisition : on passe d’un ensemble restreint de photos de qualité, soigneusement acquises par l’opérateur, à une séquence d'images à cadence vidéo, sujette à des variations de qualité image dues, par exemple, au bougé et au défocus.Les données vidéo posent problème aux logiciels de photogrammétrie du fait de la combinatoire élevée engendrée par le grand nombre d’images. Nous proposons d’exploiter l’intégralité des images en deux étapes. Au cours de la première, la reconstruction 3D est obtenue en sous-échantillonnant temporellement la séquence, lors de la seconde, la restitution haute résolution de texture est obtenue en exploitant l'ensemble des images. L'intérêt de la texture est de permettre de visualiser des détails fins du modèle numérisé qui ont été perdus dans le bruit géométrique de la reconstruction. Cette augmentation de qualité se fait via des techniques de Super Résolution (SR).Pour atteindre cet objectif nous avons conçu et réalisé une chaîne algorithmique prenant, en entrée, la séquence vidéo acquise et fournissant, en sortie, un modèle 3D de la scène avec une texture sur-résolue. Cette chaîne est construite autour d’un algorithme de reconstruction 3D multi-vues de l’état de l’art pour la partie géométrique.Une contribution centrale de notre chaîne est la méthode de recalage employée afin d’atteindre la précision sub-pixellique requise pour la SR. Contrairement aux données classiquement utilisées en SR, nos prises de vues sont affectées par un mouvement 3D, face à une scène à géométrie 3D, ce qui entraîne des mouvements image complexes. La précision intrinsèque des méthodes de reconstruction 3D est insuffisante pour effectuer un recalage purement géométrique, ainsi nous appliquons un raffinement supplémentaire par flot optique. Le résultat de cette méthode de restitution de texture SR est d'abord comparée qualitativement à une approche concurrente de l’état de l’art.Ces appréciations qualitatives sont renforcées par une évaluation quantitative de qualité image. Nous avons à cet effet élaboré un protocole d’évaluation quantitatif de techniques de SR appliquées sur des surfaces 3D. Il est fondé sur l'utilisation de mires fractales binaires, initialement proposées par S. Landeau. Nous avons étendu ces idées au contexte de SR sur des surfaces courbes. Cette méthode est employée ici pour valider les choix de notre méthode de SR, mais elle s'applique à l'évaluation de toute texturation de modèle 3D.Enfin, les surfaces spéculaires présentes dans les scènes induisent des artefacts au niveau des résultats de SR en raison de la perte de photoconsistence des pixels au travers des images à fusionner. Pour traiter ce problème nous avons proposé deux méthodes correctives permettant de recaler photométriquement nos images et restaurer la photoconsistence. La première méthode est basée sur une modélisation des phénomènes d’illumination dans un cas d'usage particulier, la seconde repose sur une égalisation photométrique locale. Les deux méthodes testées sur des données polluées par une illumination variable s'avèrent effectivement capables d'éliminer les artefacts. / Multi-view 3D reconstruction techniques have reached industrial level maturity : non-expert users are now able to use commercial software to produce quality, large scale, 3D models. These reconstructions use top of the line sensors such as LIDAR or DSLR cameras, mounted on tripods and moved around the scene. Such protocols are not designed to efficiently inspect large infrastructures with complex geometry. As the capabilities of micro-drones progress at a fast rate, it is becoming possible to delegate such tasks to them. This choice induces changes in the acquired data : rather than a set of carefully acquired images, micro-drones will produce a video sequence with varying image quality, due to such flaws as motion blur and defocus. Processing video data is challenging for photogrammetry software, due to the high combinatorial cost induced by the large number of images. We use the full image sequence in two steps. Firstly, a 3D reconstruction is obtained using a temporal sub-sampling of the data, then a high resolution texture is built from the full sequence. Texture allows the inspector to visualize small details that may be lost in the noise of the geometric reconstruction. We apply Super Resolution techniques to achieve texture quality augmentation. To reach this goal we developed an algorithmic pipeline that processes the video input and outputs a 3D model of the scene with super resolved texture. This pipeline uses a state of the art 3D reconstruction software for the geometric reconstruction step. The main contribution of this pipeline is the image registration method used to achieve the sub-pixel accuracy required for Super Resolution. Unlike the data on which Super Resolution is generally applied, our viewpoints are subject to relative 3D motion and are facing a scene with 3D geometry, which makes the motion field all the more complex. The intrinsic precision of current 3D reconstruction algorithms is insufficient to perform a purely geometric registration. Instead we refine the geometric registration with an optical flow algorithm. This approach is qualitatively to a competing state of the art method. qualitative comparisons are reinforced by a quantitative evaluation of the resulting image quality. For this we developed a quantitative evaluation protocol of Super Resolution techniques applied to 3D surfaces. This method is based on the Binary Fractal Targets proposed by S. Landeau. We extended these ideas to the context of curved surfaces. This method has been used to validate our choice of Super Resolution algorithm. Finally, specularities present on the scene surfaces induce artefacts in our Super Resolution results, due to the loss of photoconsistency among the set of images to be fused. To address this problem we propose two corrective methods designed to achieve photometric registration of our images and restore photoconsistency. The first method is based on a model of the illumination phenomena, valid in a specific setting, the second relies on local photometric equalization among the images. When tested on data polluted by varying illumination, both methods were able to eliminate these artefacts.

Pour analyser la structure et la dynamique des échantillons, la biologie cellulaire repose sur l'utilisation d'outils d'imagerie. En particulier, la microscopie de fluorescence offre une grande spécificité et une toxicité réduite. L'émergence récente des méthodes de super-résolution a permis d'outrepasser la limite de diffraction et ouvert de nouvelles perspectives d'études. Les stratégies de molécule unique sont particulièrement adaptées à l'imagerie nanométrique tridimensionnelle, et permettent de nombreux couplages avec des modalités complémentaires ; toutefois, leur manque de reproductibilité entrave leur généralisation.Nous proposons ici de nouvelles méthodes dans le but de remédier à ces problèmes en facilitant leur application en biologie cellulaire, en chimie et en science des matériaux. Tout d'abord, nous présentons des protocoles et échantillons dédiés aux acquisitions de calibration et de mesure de performances. Nous décrivons également plusieurs exemples d'utilisation de super-localisation tridimensionnelle dans le cadre d'études d'adhésion cellulaire et de résistance bactérienne.Ensuite, nous nous concentrons au développement d'une nouvelle méthode de microscopie de localisation de molécules uniques tri-dimensionnelle permettant l'élimination de biais de détection. Ceci est permis par le couplage entre deux stratégies complémentaires: la mise en forme de fonction d'étalement de point, et la détection de la fluorescence d'angle super-critique. L'intercorrélation et la recombinaison des informations latérales et axiales permet l'obtention d'une résolution quasi-isotrope, avec des précisions jusqu'à 15 nanomètres sur une plage de capture d'un micron. Nous mettons en évidence l'insensibilité de la méthode aux biais d'imagerie comme la dérive axiale, l'aberration chromatique et l'inclinaison de l'échantillon, et nous l'illustrons à travers des applications à la neurobiologie et au marquage de bactéries.Pour finir, nous présentons deux nouvelles approches pour le découplage d'acquisitions multi-espèces simultanées. Toutes deux basées entièrement sur le post-traitement des données acquises, elles exploitent respectivement la mesure des tailles des taches et le comportement dynamique du clignotement. Après une preuve de principe, nous évaluons l'impact des différents paramètres susceptibles d'influencer les résultats. Nous concluons en proposant des pistes d'amélioration des performances de découplage, et en suggérant de possibles couplages avec des méthodes complémentaires en imagerie de molécules uniques. / Cell biology relies on imaging tools to provide structural and dynamic information about samples. Among them, fluorescence microscopy offers a compromise between high specificity and low toxicity. Recently, super-resolution methods overcame the diffraction barrier to unlock new fields of investigation. Single molecule approaches prove especially useful for three-dimensional nanoscale imaging, and allow couplings between different detection modalities. Still, their use is hindered by the complexity of the methods as well as the lack of reproducibility between experiments.We propose new methods to render super-localisation microscopy more easily applicable to relevant studies in cell biology, chemistry and material science. First, we introduce dedicated protocols and samples to eliminate sources of error in calibration and performance measurement acquisitions. We also provide examples of uses of three-dimensional super-localisation for state-of-the-art studies in the frameworks of cell adhesion and bacterial resistance to drugs.Then, we focus on the development of a novel optical method that provides unbiased results in three-dimensional single molecule localisation microscopy. This is achieved through the combination of two complementary axial detection strategies: point spread function shaping on the one hand, and supercritical angle fluorescence detection on the other hand. By cross-correlating and merging the lateral and axial positions provided by the different sources, we achieve quasi-isotropic localisation precisions down to 15 nanometres over a 1-micrometre capture range. We demonstrate the insensibility of the method to imaging non-idealities such as axial drift, chromatic aberration and sample tilt, and we propose applications in neurobiology and bacteria labelling.Finally, we introduce two new post-processing approaches for the demixing of simultaneous multi-species acquisitions. They are based respectively on the measurement of the spot sizes, and on the assessment of the dynamic blinking behaviour of molecules. After demonstrating a proof of principle, we assess the impact of the different parameters likely to influence the results. Eventually, we discuss leads to improve the demixing performances, and we discuss the coupling possibilities with complementary single molecule localisation techniques.

No description available.

Cells require adenosine triphosphate (ATP) to drive the myriad processes associated with growth, replication, and homeostasis. Eukaryotic cells rely on mitochondria to produce the vast majority of their ATP. Mitochondria consist of a relatively smooth outer mitochondrial membrane (OMM) and a highly complex inner mitochondrial membrane (IMM), containing numerous invaginations, called cristae, which house the molecular machinery of oxidative phosphorylation (OXPHOS). Although mitochondrial form and function are intimately connected, limitations in the resolution of live-cell imaging have hindered the ability to directly visualize the relationship between the architecture of the IMM and its associated bioenergetic properties. Using advanced imaging technologies, including Airyscan, stimulated emission depletion (STED), and structured illumination microscopy (SIM), we developed an approach to image the IMM in living cells. Staining mitochondria with various ΔΨm-dependent dyes, we found that the fluorescence pattern along the IMM was heterogeneous, with cristae possessing a significantly greater fluorescence intensity than the contiguous inner boundary membrane (IBM). Applying the Nernst equation, we determined that the ΔΨm of cristae is approximately 12 mV stronger than that of IBM, indicating that the electrochemical gradient that drives ATP synthesis is compartmentalized in cristae membranes. Notably, deletion of key components of the mitochondrial contact site and cristae organizing system (MICOS), as well as OPA1, which regulate crista junctions (CJs), decreased ΔΨm heterogeneity. Complementing our super-resolution imaging of cristae in living cells, we also developed a machine-learning protocol to quantify IMM architecture. Tracking real-time changes in cristae density, size, and shape, we determined that cristae dynamically remodel on a scale of seconds. Furthermore, we found that cristae move away from sites of mitochondrial fission, and, prior to mitochondrial fusion, the IMM forms finger-like protrusions bridging the membranes of the fusing organelles. Lastly, we investigated the role of the motor adaptor protein, Milton1/TRAK1, in mitochondrial dynamics. Patient-derived Milton1-null fibroblasts not only had impaired mitochondrial motility but exhibited fragmentation corresponding to a roughly 40% decrease in mitochondrial aspect ratio and a 17% increase in circularity, associated with increased DRP1 activity. Conversely, we found that overexpression of Milton1 led to mitochondrial hyperfusion, decreased DRP1 activity, and aberrant clustering of mtDNA. Overall, our studies directly demonstrate that maintaining mitochondrial architecture is essential for preserving the functionality of mitochondria, the hubs of eukaryotic metabolism.

<div>Light in heavily scattering media such as tissue can be modeled with a diffusion equation. A diffusion equation forward model in a computational imaging framework can be used to form images of deep tissue, an approach called diffuse optical tomography, which is important for biomedical studies. However, severe attenuation of high-spatial-frequency information occurs as light propagates through scattering media, and this limits image resolution. Here, we introduce a super-resolution approach based on a point emitter localization method that enables an improvement in spatial resolution of over two orders of magnitude. We demonstrate this experimentally by localizing a small fluorescent inhomogeneity in a highly scattering slab and characterize the localization uncertainty. The approach allows imaging in deep tissue with a spatial resolution of tens of microns, enabling cells to be resolved.</div><div><br></div><div>We also propose a localization-based method that relies on separation in time of the temporal responses of fluorescent signals, as would occur with biological reporters. By localizing each emitter individually, a high-resolution spatial image can be achieved. We develop a statistical detection method for localization based on temporal switching and characterization of multiple fluorescent emitters in a tissue-like domain. By scaling the spatial dimensions of the problem, the scope of applications is widened beyond tissue imaging to other scattering domains. </div><div><br></div><div>Finally, we demonstrate that motion of an object in structured illumination and intensity-based measurements provide sensitivity to material and subwavelength-scale-dimension information. The approach is illustrated as retrieving unknown parameters of interest, such as the refractive index and thickness of a film on a substrate, by utilizing measured power data as a function of object position. </div>

In digital image correlation, an optical full-field analysis method that can determine displacements of an object under load, high-resolution images are preferable. One way to improve the resolution is to improve the camera hardware. This can be expensive, hence another way to enhance the image is by various image processing techniques increase the resolution of the image. There are several ways of doing this and these techniques are called super-resolution. In this thesisthe theory behind several different approaches to super-resolution is presented and discussed. The goal of this Thesis has been to investigate if super-resolutionis possible in a scene with moving objects as well as movement of the camera. It became clear early on that image registration, a step in many super-resolution methods that will be explained in this thesis, was of utmost importance, and a major part of the work became comparing image registration methods. Data has been recorded and then two different super-resolution algorithms have been evaluated on a data set showing that super-resolution is possible.

<p>In this dissertation, we document a few of our recent attempts in bridging the gap between the fast evolving deep learning research and the vast industry needs for dealing with computer vision challenges. More specifically, we developed novel deep-learning-based techniques for the following application-driven computer vision challenges: image super-resolution with quality restoration, motion estimation by optical flow, object detection for shape reconstruction, and object segmentation for motion tracking. Those four topics cover the computer vision hierarchy from the low level where digital images are processed to restore missing information for better human perception, to middle level where certain objects of interest are recognized and their motions are analyzed, finally to high level where the scene captured in the video footage will be interpreted for further analysis. In the process of building the whole-package of  ready-to-deploy solutions, we center our efforts on designing and training the most suitable convolutional neural networks for the particular computer vision problem at hand. Complementary procedures for data collection, data annotation,  post-processing of network outputs tailored for specific application needs, and deployment details will also be discussed where necessary. We hope our work demonstrates the applicability and versatility of convolutional neural networks for real-world computer vision tasks on a broad spectrum, from seeing better to understanding better.</p>

While digital displays have continuously increased in resolution, video content produced before these improvements is however stuck at its original resolution, and the use of some form of scaling is needed for a satisfactory viewing experience on high-resolution displays. In recent years, the field of video scaling has taken a leap forward in output quality, due to the adoption of deep learning methods in research. In this paper, we describe a study wherein we train a convolutional neural network for super-resolution, and conduct a large-scale A/B video quality test in order to investigate if SVT video-ondemand viewers prefer video upscaled using a convolutional neural network to video upscaled using the standard bicubic method. Our results show that viewers generally prefer CNNscaled video, but not necessarily for the types of content this technology would primarily be used to scale. We conclude that the technology of deep upscaling shows promise, but also believe that more optimization and flexibility is need for deep scaling to be viable for mainstream use. / Allteftersom bildskärmstekniken förbättras så får mediekonsumenter tillgång till skärmar med allt högre upplösningar; dock är videomaterial som producerats för en viss bildupplösning, fast på denna nivå, och någon form av skalning måste användas för en bra tittarupplevelse på högupplösta skärmar. På senare tid så har videoskalning förändrats, tack vare användandet av djupinlärningsmetoder inom forskningen. I den här rapporten beskriver vi en studie där vi tränade en djup modell för videouppskalning, och sedan utförde ett storskaligt A/B-test, med syftet att undersöka huruvida SVTs onlinetittare föredrar video skalad med djupinlärning över video skalad med konventionella metoder. Våra resultat visar att tittarna föredrog video skalad med djupinlärning, dock inte nödvändigtvis för det material tekniken främst skulle användas med. Vi drar slutsatsen att videoskalning med hjälp av djupinlärning är lovande, men anser också att mer optimering och flexibilitet behövs innan tekniken kan anses mogen för bred adoption.