University of North Texas
Sparse Autoencoder Enhanced Universal Image Restoration with DACLIP and Diffusion Models
Abstract
dc:descriptionImage restoration recovers high-quality images from degraded inputs affected by environmental conditions, sensor limitations, or content removal. While unified restoration frameworks like DA-CLIP achieve state-of-the-art performance through vision-language guidance and diffusion-based reconstruction, two limitations persist: cross-attended features may contain residual artifacts, and diffusion outputs often lack fine-grained spatial refinement under severe degradations. This thesis presents SAE-Restore, a dual-stage sparse autoencoder framework addressing these limitations through explicit feature denoising and image refinement. SAE-Restore integrates two sparse autoencoders at strategic positions. SAE1 performs feature-space denoising after cross-attention with 8-times expansion and 80 to 90 percent activation sparsity to remove artifacts before diffusion guidance. SAE2 applies image-space refinement after diffusion through residual learning to recover fine details. A learnable fusion module enables end-to-end training of all components while preserving frozen DA-CLIP parameters, reducing trainable parameters by 82 percent. Comprehensive evaluation on six degradation types reveals task-dependent effectiveness. SAE1 improves artifact removal tasks with strongest impact on snowy images at plus 2.29 decibels and low-light at plus 0.99 decibels, but degrades inpainting by 0.80 decibels, revealing that sparsity bottlenecks hinder content synthesis. SAE2 provides minimal benefit at negative 0.06 decibels average. Systematic ablation demonstrates that SAE1 only achieves best performance at 19.81 decibels, outperforming the full pipeline at 19.74 decibels. This research contributes the first integration of sparse autoencoders with vision-language guided diffusion, a multi-stage evaluation framework isolating component contributions, and empirical evidence that artifact removal and content synthesis require different architectural approaches. The findings validate task-dependent benefits and provide guidance for adaptive component routing in unified restoration frameworks.
Degree
thesis:*- Grantor dc:publisher
- University of North Texas
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Dodia, Panam Pareshkumar
- Contributors dc:contributor
-
- Cui, Xinrui
- Shi, Weishi
- Liu, Zhuqing
Subjects
dc:subject × 3Rights
dc:rights- Statement dc:rights
-
- Public
- Dodia, Panam Pareshkumar
- Copyright
- Copyright is held by the author, unless otherwise noted. All rights Reserved.
- Language dc:language
- English
Identifiers
dc:identifier.*- Identifier
-
local-cont-no: submission_4620
ark: ark:/67531/metadc2566489