<p>Advances of mobile and wearable technologies have enabled a wide range of new methods for dietary assessment and monitoring, such as active or passive capturing of food images of an eating scene. Compared to traditional methods, these approaches are less burdensome and can reduce biased measurements. Food image analysis, consists of food region detection, food category classification, and food segmentation, can benefit subsequent nutrient analysis. However, due to the different complexity levels of food images and inter-class similarity of food categories, it is challenging for an image-based food analysis system to achieve high performance outside of a lab setting.</p> <p>In this thesis, we investigate four research topics related to image-based dietary assess- ment: (1) construction of the VIPER-FoodNet dataset, (2) food recognition, (3) nutrient integrated hierarchy food classification, and (4) weakly supervised segmentation. For topic (1), we developed a learning-based method to automatically remove non-food images for dataset construction. For topic (2), we proposed a novel two-step food recognition system that consists of food localization and hierarchical food classification. For topic (3), we de- veloped a cross-domain food classification framework that integrates nutrition information to help the classification system make better mistakes. Finally, for topic (4), a weakly- supervised segmentation system is developed which only requires image-level supervision during training.</p> <p>In addition, we developed a high-quality Photoplethysmography (PPG) signal selection method for a wearable device when subjects are undergoing daily life activities, which could be used to inform the health status of the individual.</p>