Back to search

Georgia Institute of Technology

From 3D Mapping to Scene Representations for Embodied AI

Abstract

dc:description.abstract

In the past few years, a burgeoning field of research has emerged within the broader AI community known as "Embodied AI." This field encompasses various challenges, including the development of scene datasets and simulators used to train AI agents in diverse tasks, necessitating a comprehensive set of skills. Generally speaking, Embodied AI research projects assume a similar setup where an agent equipped with a set of sensors (usually an RGB and Depth camera) is trained to accomplish certain tasks such as navigation, pick-and-place, question answering etc... While there are many approaches and designs of such Embodied AI agents, this thesis focuses on methods involving intermediate scene representations. In contrast to end-to-end approaches, AI systems with intermediate explicit scene representations comprise two distinct modules: one for raw input sensor processing and a second one for planning and acting. Agents utilizing scene representations offer several advantages, including reduced susceptibility to the forgetting effect observed in their RNN counterparts, easier incorporation of inductive bias directly into the representations (e.g., geometrical constraints), and improved interpretability. Towards this end, this thesis serves as an investigation towards the design of such scene representations. We specifically research how to leverage 3D mapping techniques in order to build rich, dense and useful representations for Embodied AI applications. We start by studying representations in the form of 2D topdown metric maps. These 2D maps can store features, labels or geometrical information to form a useful training signal for the tasks of semantic mapping, navigation or question answering. We then study 3D representations in the forms of implicit maps applied for SLAM and 3D object-based maps for multi-object re-identification. Next we extend our research to dynamic scenes and explore 4D representations in the form of localized keyframes. Finally, the thesis also explores connections between these different representations while highlighting their strengths and limitations.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Electrical and Computer Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Cartillier, Vincent Daniel
Advisors dc:contributor.advisor
  • Essa, Irfan
  • Romberg, Justin
Committee members dc:contributor.committeemember
  • AlRegib, Ghassan
  • Lee, Stefan
  • Hoffman, Judy
  • Schindler, Grant

Subjects

dc:subject × 8

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/73112

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Cartillier, Vincent Daniel. From 3D Mapping to Scene Representations for Embodied AI. Doctoral thesis, Georgia Institute of Technology, 2023. https://hdl.handle.net/1853/73112