Recent years have witnessed tremendous success in using deep learning approaches to handle low-level vision problems. Most of the deep learning based methods address the low-level vision problem by training a neural network to approximate the mapping from the inputs to the desired ground truths. However, directly learning this mapping is usually difficult and cannot achieve ideal performance. Besides, under the setting of unsupervised learning, the general deep learning approach cannot be used. In this thesis, we investigate and address several problems in low-level vision using the proposed approaches. To learn a better mapping using the existing data, an indirect domain shift mechanism is proposed to add explicit constraints inside the neural network for single image dehazing. This allows the neural network to be optimized across several identified neighbours, resulting in a better performance. Despite the success of the proposed approaches in learning an improved mapping from the inputs to the targets, three problems of unsupervised learning is also investigated. For unsupervised monocular depth estimation, a teacher-student network is introduced to strategically integrate both supervised and unsupervised learning benefits. The teacher network is formed by learning under the binocular depth estimation setting, and the student network is constructed as the primary network for monocular depth estimation. In observing that the performance of the teacher network is far better than that of the student network, a knowledge distillation approach is proposed to help improve the mapping learned by the student. For single image dehazing, the current network cannot handle different types of haze patterns as it is trained on a particular dataset. The problem is formulated as a multi-domain dehazing problem. To address this issue, a test-time training approach is proposed to leverage a helper network in assisting the dehazing network adapting to a particular domain using self-supervision. In lossy compression system, the target distribution can be different from that of the source and ground truths are not available for reference. Thus, the objective is to transform the source to target under a rate constraint, which generalizes the optimal transport. To address this problem, theoretical analyses on the trade-off between compression rate and minimal achievable distortion are studied under the cases with and without common randomness. A deep learning approach is also developed using our theoretical results for addressing super-resolution and denoising tasks. Extensive experiments and analyses have been conducted to prove the effectiveness of the proposed deep learning based methods in handling the problems in low-level vision. / Thesis / Doctor of Philosophy (PhD)

No description available.

No description available.

No description available.

<p dir="ltr">Single-particle tracking (SPT) provides high-resolution spatial-temporal information on biomolecule dynamics. However, localization inaccuracies, limited track lengths, heterogeneous fluorescence backgrounds, and potential molecular motion blur pose significant challenges that hinder the accurate extraction of movement trajectories and their underlying motion behavior. The conventional SPT pipeline struggles to comprehensively address detection, localization, linkage, and parameter inference simultaneously, resulting in information loss during sequential processing. To overcome these challenges, we propose SPTnet, an end-to-end deep learning framework that leverages a transformer-based architecture to optimize trajectory and motion parameter estimations in parallel through a global loss. SPTnet bypasses traditional SPT processes, directly inferring molecular trajectories and motion parameters from fluorescence microscopy video frames with precision approaching the statistical information limit. Our results demonstrate that SPTnet outperforms conventional methods under commonly encountered but challenging conditions such as short trajectories, low signal-to-noise ratio (SNR), heterogeneous backgrounds, motion blur, and especially when molecules exhibit non-Brownian behaviors.</p><p dir="ltr">Besides SPT, we used single-molecule localization microscopy (SMLM) to study cytokinetic protein in fission yeast. During cytokinesis, myosin-II constricts the contractile ring that separates one cell into two daughter cells. The fission yeast cytokinetic contractile ring contains two types of myosin Ⅱ, Myo2 and Myp2. However, the precise ultrastructural arrangement of the two type Ⅱ myosins remains in question. We investigated the relative spatial arrangement of Myo2p and Myp2p within contractile ring using two-color super-resolution microscopy based on salvaged fluorescence imaging. Quantitative analysis of the nanoscale images should provide useful information for modeling contractile ring assembly and constriction.</p><p><br></p>

Epithelial-mesenchymal transition (EMT) is a trans-differentiation program essential for development and wound healing that is pathologically activated during cancer progression. During this process, cells undergo complex changes at the transcriptional and translational levels leading to dissolution of cell-cell junctions, loss of apical-basal polarity, and cytoskeleton reorganization. Transforming Growth Factor-β (TGF-β) is well-established in driving cancer progression through EMT induction. Remodeling of cellular junctions, including gap junctions, is critical to acquiring migratory and invasive characteristics during EMT. The gene GJA1 encodes for Connexin43 (Cx43), the most ubiquitously expressed gap junction protein where altered regulation of Cx43 is associated with cancer progression. Intriguingly, Cx43 mRNA undergoes alternative ‘internal’ translation initiation, generating N-terminally truncated isoforms, including GJA1-20k, which regulates Cx43 gap junction formation. We have previously demonstrated GJA1-20k expression is inhibited during TGF-β-induced EMT, limiting gap junction formation; however, the relationship between GJA1-20k modulation of gap junction localization and cellular invasion and migration remains unknown. Given the role GJA1-20k has in regulating gap junctions, we hypothesize that suppression of GJA1-20k expression promotes metastatic trait acquisition through limiting gap junction formation. Utilizing lentivirally transduced stable mouse mammary gland epithelial (NMuMG) and triple-negative human breast epithelial (MDA-MB-231) cells expressing GJA1-20k, or Lac Z as control, we tested effects on TGF-β-induced EMT induction and metastatic trait induction. Boyden chambers, would/scratch assays were employed to analyze cell invasion and migration respectively. We found GJA1-20k overexpression during EMT results in decreased cell invasion and migration to LacZ controls. Future directions include evaluation of GJA1-20k restoration in a metastatic breast cancer model in vivo. Investigating the underlying role of GJA1-20k in EMT-induced cell junction remodeling could be promising as a potential pharmacological target process independent of transcriptional or post-translational pathways. Ultimately, by adding novel information in the expanding and compelling field of translational control, this work could aid in developing the future of precision medicine as new therapeutic solutions to treat cancer will require limiting cancer cell’s ability to metastasize. / R01 HL132236 JWS R41 CA250874 SL R01HL132236 Diversity Supplement KLY II 23PRE1025483 AHA Predoctoral Fellowship KLY II / Doctor of Philosophy / Every organ system relies upon cell-to-cell communication to properly function and is the basis of multi-cellular life. Gap junctions are nanoscale conduits allowing the passage of small signaling molecules and ions between adjacent cells, similar to telephone tubes. Gap junctions are formed from proteins called connexins. Interestingly, it is well known that shorter pieces of connexin proteins can regulate the formation of gap junctions and are uniquely created by a process called alternative ‘internal’ translation. Changes in the amounts of short-length and full-length connexin proteins are often found in cancer cells. Cancer is the uncontrolled growth of abnormal cells. Commonly, the morphology of cancer cells, and the way they communicate with neighboring cells, is altered. Cancer progression is aided by changes in cell signaling molecules, including TGF-β which can drive cancer cells to leave primary tumor sites and grow elsewhere in the body. This is important for the cancer cells to continue dividing and eventually metastasizing (invading other organ systems). Treating cancer once it has spread to other regions of the body is difficult and is the main cause of cancer deaths worldwide. Using TGF-β to model metastatic changes in mouse and human cell lines, we studied how short-length connexin protein affects metastatic cancer cell behavior. With this information we will be able to guide the development of druggable alternative ‘internal’ translation targets, by restoring the proper communication between neighboring cells and therefore preventing spread of cancer cells.

Vision has always been one of the most important cognitive tools of human beings. In this regard, the development of image sensors opens up the potential to view objects that our eyes cannot see. One of the most promising capability in some image sensors is their single-photon sensitivity that provides information at the ultimate fundamental limit of light. Time-resolved single-photon avalanche diode (SPAD) image sensors bring a new dimension as they measure the arrival time of incident photons with a precision in the order of hundred picoseconds. In addition to this characteristic, they can be fabricated in complementary metal-oxide-semiconductor (CMOS) technology enabling the integration of complex signal processing blocks at the pixel level. These unique features made CMOS SPAD sensors a prime candidate for a broad spectrum of applications. This thesis is dedicated to the optimization and characterization of quantum imagers based on the SPADs as part of the E.U. funded SUPERTWIN project to surpass the fundamental diffraction limit known as the Rayleigh limit by exploiting the spatio-temporal correlation of entangled photons. The first characterized sensor is a 32×32-pixel SPAD array, named “SuperEllen”, with in-pixel time-to-digital converters (TDC) that measure the spatial cross-correlation functions of a flux of entangled photons. Each pixel features 19.48% fill-factor (FF) in 44.64-μm pitch fabricated in a 150-nm CMOS standard technology. The sensor is fully characterized in several electro-optical experiments, in order to be used in quantum imaging measurements. Moreover, the chip is calibrated in terms of coincidence detection achieving the minimal coincidence window determined by the SPAD jitter. The second developed sensor in the context of SUPERTWIN project is a 224×272-pixel SPAD-based array called “SuperAlice”, a multi-functional image sensor fabricated in a 110-nm CMOS image sensor technology. SuperAlice can operate in multiple modes (time-resolving or photon counting or binary imaging mode). Thanks to the digital intrinsic nature of SPAD imagers, they have an inherent capability to achieve a high frame rate. However, running at high frame rate means high I/O power consumption and thus inefficient handling of the generated data, as SPAD arrays are employed for low light applications in which data are very sparse over time and space. Here, we present three zero-suppression mechanisms to increase the frame rate without adversely affecting power consumption. A row-skipping mechanism that is implemented in both SuperEllen and SuperAlice detects the absence of SPAD activity in a row to increase the duty cycle. A current-based mechanism implemented in SuperEllen ignores reading out a full frame when the number of triggered pixels is less than a user-defined value. A different zero-suppression technique is developed in the SuperAlice chip that is based on jumping through the non-zero pixels within one row. The acquisition of TDC-based SPAD imagers can be speeded up further by storing and processing events inside the chip without the need to read out all data. An on-chip histogramming architecture based on analog counters is developed in a 150-nm CMOS standard technology. The test structure is a 16-bin histogram with 9 bit depth for each bin. SPAD technology demonstrates its capability in other applications such as automotive that demands high dynamic range (HDR) imaging. We proposed two methods based on processing photon arrival times to create HDR images. The proposed methods are validated experimentally with SuperEllen obtaining &gt;130 dB dynamic range within 30 ms of integration time and can be further extended by using a timestamping mechanism with a higher resolution.

Image Super-Resolution (ISR) is a technology that aims to increase image resolution while preserving as much content and detail as possible. In this study, we evaluate four different Deep Learning models (EDSR, LapSRN, ESPCN, and FSRCNN) to determine their effectiveness in increasing the resolution of lowresolution images. The study builds on previous research in the field as well as the results of the comparison between the different deep learning models. The problem statement for this study is: “Which of the four Deep Learning-based models, EDSR, LapSRN, ESPCN, and FSRCNN, generates an upscaled image with the best quality from a low-resolution image on a dataset of Abyssinian cats, with a factor of four, based on quantitative results?” The study utilizes a dataset consisting of pictures of Abyssinian cats to evaluate the performance and results of these different models. Based on the quantitative results obtained from RMSE, PSNR, and Structural Similarity (SSIM) measurements, our study concludes that EDSR is the most effective Deep Learning-based model. / Bildsuperupplösning (ISR) är en teknik som syftar till att öka bildupplösningen samtidigt som så mycket innehåll och detaljer som möjligt bevaras. I denna studie utvärderar vi fyra olika Deep Learning modeller (EDSR, LapSRN, ESPCN och FSRCNN) för att bestämma deras effektivitet när det gäller att öka upplösningen på lågupplösta bilder. Studien bygger på tidigare forskning inom området samt resultatjämförelser mellan olika djupinlärningsmodeller. Problemet som studien tar upp är: “Vilken av de fyra Deep Learning-baserade modellerna, EDSR, LapSRN, ESPCN och FSRCNN generarar en uppskalad bild med bäst kvalité, från en lågupplöst bild på ett dataset med abessinierkatter, med skalningsfaktor fyra, baserat på kvantitativa resultat?” Studien använder en dataset av bilder på abyssinierkatter för att utvärdera prestandan och resultaten för dessa olika modeller. Baserat på de kvantitativa resultaten som erhölls från RMSE, PSNR och Structural Similarity (SSIM) mätningar, drar vår studie slutsatsen att EDSR är den mest effektiva djupinlärningsmodellen.

A near-field technique known as evanescent interferometric lithography allows for high resolution imaging. However its primary limitation is that the image exponentially decays within the photoresist due to physical limits. This thesis aims to overcome this limitation and presents a method to considerably enhance the depth of focus of images created using evanescent interferometric lithography by using a material underlay beneath the photoresist. A key enabler of this is the understanding that evanescent fields couple to surface states and operating within proximity of a resonance, the strength of the coupling allows for considerable energy extraction from the incident beam and redistribution of this energy in a photoresist cavity. This led to the analysis of the Fresnel equations, which suggested that such coupling was in fact the result of an enhanced reflectance that takes place at boundaries of carefully chosen materials. While it is known that metals and lossy dielectrics result in surface plasmon polaritons (SPP) and surface exciton polaritons (SEP) as conventional solutions to the Fresnel reflection equations for the TM polarization of light, there is no such naturally occurring surface state that allows evanescent wave enhancement with the TE polarization of light. Further investigation of the Fresnel reflection equations revealed both for TM and TE that in fact another solution exists that is but unconventional to enhance the reflectivity. This solution requires that one of the media have a negative loss. This is a new type of surface resonance that requires that one of the media be a gain medium; not one in the optical pumped sense but one that would naturally supply energy to a wave to make it grow. This new surface resonance is also a key result of this thesis. Clearly, however this is only a hypothetical solution as a real gain medium would violate the conservation of energy. However, as it is only the reflectance of this gain medium that is useful for evanescent wave enhancement, in fact a multilayered stack consisting of naturally occurring materials is one way to achieve the desired reflectivity. This would of course be only an emulation of the reflectivity aspect of the gain medium. This multilayered stack is then an effective gain medium for the reflectivity purposes when imaging is carried out at a particular NA at a particular wavelength. This proposal is also a key idea of this thesis. At λ = 193 nm, this method was used to propose a feasible design to image high resolution structures, NA = 1.85 at an aspect ratio of ~3.2. To experimentally demonstrate the enhancements, a new type of solid immersion test bed, the solid immersion Lloyd's mirror interference lithography test-bed was constructed. High quality line and space patterns with a half-pitch of 55.5 nm were created using λ = 405 nm, corresponding to a NA of 1.824, that is well in the evanescent regime of light. Image depths of 33-40 nm were seen. Next, the evanescent image was coupled to an effective gain medium made up of a thin layer of hafnium oxide (HfO) upon silicon dioxide (SiO2). This resulted in a considerable depth enhancement, and 105 nm tall structures were imaged. The work in this thesis details the construction of the solid immersion lithography test-bed, describes the implementation of the modeling tools, details the theory and analysis required to achieve the relevant solutions and understanding of the physical mechanism and finally experimentally demonstrates an enhancement that allows evanescent interferometric lithography beyond conventional limits.

Le domaine de la microscopie de fluorescence a connu une réelle révolution ces dernières années, permettant d'atteindre des résolutions nanométriques, bien en dessous de la limite de diffraction prédite par Abbe il y a plus d’un siècle. Les techniques basées sur la localisation de molécules individuelles telles que le PALM (Photo-Activation Light Microscopy) ou le (d)STORM (direct Stochastic Optical Reconstruction Microscopy) permettent la reconstruction d’images d’échantillons biologiques en 2 et 3 dimensions, avec des résolutions quasi-moléculaires. Néanmoins, même si ces techniques nécessitent une instrumentation relativement simple, elles requièrent des traitements informatiques conséquents, limitant leur utilisation en routine. En effet, plusieurs dizaines de milliers d’images brutes contenant plus d’un million de molécules doivent être acquises et analysées pour reconstruire une seule image. La plupart des outils disponibles nécessitent une analyse post-acquisition, alourdissant considérablement le processus d’acquisition. Par ailleurs la quantification de l’organisation, de la dynamique mais aussi de la stœchiométrie des complexes moléculaires à des échelles nanométriques peut constituer une clé déterminante pour élucider l’origine de certaines maladies. Ces nouvelles techniques offrent de telles capacités, mais les méthodes d’analyse pour y parvenir restent à développer. Afin d’accompagner cette nouvelle vague de microscopie de localisation et de la rendre utilisable en routine par des expérimentateurs non experts, il est primordial de développer des méthodes de localisation et d’analyse efficaces, simples d’emploi et quantitatives. Dans le cadre de ce travail de thèse, nous avons développé dans un premier temps une nouvelle technique de localisation et reconstruction en temps réel basée sur la décomposition en ondelettes et l‘utilisation des cartes GPU pour la microscopie de super-résolution en 2 et 3 dimensions. Dans un second temps, nous avons mis au point une méthode quantitative basée sur la visualisation et la photophysique des fluorophores organiques pour la mesure de la stœchiométrie des récepteurs AMPA dans les synapses à l’échelle nanométrique. / The field of fluorescence microscopy has witnessed a real revolution these last few years, allowing nanometric spatial resolutions, well below the diffraction limit predicted by Abe more than a century ago. Single molecule-based super-resolution techniques such as PALM (Photo-Activation Light Microscopy) or (d)STORM (direct Stochastic Optical Reconstruction Microscopy) allow the image reconstruction of biological samples in 2 and 3 dimensions, with close to molecular resolution. However, while they require a quite straightforward instrumentation, they need heavy computation, limiting their use in routine. In practice, few tens of thousands of raw images with more than one million molecules must be acquired and analyzed to reconstruct a single super-resolution image. Most of the available tools require post-acquisition processing, making the acquisition protocol much heavier. In addition, the quantification of the organization, dynamics but also the stoichiometry of biomolecular complexes at nanometer scales can be a key determinant to elucidate the origin of certain diseases. Novel localization microscopy techniques offer such capabilities, but dedicated analysis methods still have to be developed. In order to democratize this new generation of localization microscopy techniques and make them usable in routine by non-experts, it is essential to develop simple and easy to use localization and quantitative analysis methods. During this PhD thesis, we first developed a new technique for real-time localization and reconstruction based on wavelet decomposition and the use of GPU cards for super-resolution microscopy in 2 and 3 dimensions. Second, we have proposed a quantitative method based on the visualization and the photophysics of organic fluorophores for measuring the stoichiometry of AMPA receptors in synapses at the molecular scale.