Georgia Institute of Technology
Analysis of mixture of experts on individual, sparsely-annotated and multi-dataset object detection
Abstract
dc:description.abstractLearning with multiple datasets is a solution to develop a universal vision system. By using multiple datasets, each with different properties and label sets, we can build a robust system. However, datasets from different sources may focus on different object classes due to incompatible label schema. We imitate this mixing problem by creating synthetic sparsely-annotated splits of COCO and VOC datasets which have annotations of a set of classes removed. These splits also evaluate the missing annotation noise that is often found in large-scale annotated datasets. Ensemble learning can be an optimal solution to this problem but it requires a lot of compute for individual models. In this work, we build upon an architectural solution of ensemble learning, Mixture-of-Experts (MoE). MoE is a combination of a router and multiple feed-forward experts which takes in incoming tokens, routes them to appropriate experts, process them and consolidates them for next pass. Traditional MoE use load balancing loss to ensure expert utilization, we build over it by using cross-entropy objective conditioned over dataset source. We call this approach MoE-Xent. We evaluate MoE at backbone or decoder position and study its effect w.r.t. dataset complexity. With MoE as backbone, we achieve a mAP50 score of 63.6(+0.7) on VOC and 66.7(+0.3) on KITTI dataset over Swin-v2 baseline. On sparsely-annotated VOC, we achieve an average improvement of 2.5 in AP score with 25% sparsity-level. Further experiments reveal that MoE as backbone performs poorly on COCO due to shallow input features to MoE on complex datasets. Hence, we move MoE to decoder in end-to-end DINO object- detection pipeline and achieved a 49.7(+0.8) mAP score on COCO and 39.7(+2.3) mAP on COCO-minitrain datasets. Finally, we discuss the sensitivity of MoE training and suggest hyperparameter changes that can avoid their representation collapse.
Degree
thesis:*- Level thesis:degree_level
- Masters
- Department dc:contributor.department
- Computer Science
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Jain, Yash
- Advisor dc:contributor.advisor
-
- Kira, Zsolt
- Committee members dc:contributor.committeemember
-
- Hays, James
- AlRegib, Ghassan
Subjects
dc:subject × 6Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/72019