Back to search

Boston University

Efficient vision and language models for autonomous systems

Abstract

dc:description.abstract

The transition of vision-based systems, including autonomous vehicles, from controlled lab environments to real-world deployment poses significant challenges due to constraints such as limited data availability and computational resources. Current approaches often rely on sending data to remote cloud servers for processing, leading to energy inefficiencies. To address these challenges, this thesis proposes novel methodologies for achieving reliable and efficient inference in dynamic scenarios. Firstly, large-scale vision-based models are evaluated within the CARLA autonomous driving simulator. We introduce a 'switch' policy to offload inference between local and cloud models, trained using reinforcement learning. We evaluate the effectiveness of this policy using a newly introduced evaluation metric: Ecological Navigation Score (ENS). This metric considers route deviations, collisions, and energy consumption, critical factors for assessing driving agent effectiveness. Secondly, we present a novel dataset to assess the performance of language models (LLMs) and vision language models (VLMs) in autonomous driving tasks. The dataset comprises questions from learner's license examinations, covering various question formats and including both text-based and visual-text pairs. We conduct evaluations on small-scale LLMs using this dataset to analyze their performance, aiming to achieve a passing score of 80% or higher on the driving test. Through this comprehensive approach, the thesis aims to address the pressing need for robust and efficient vision-based systems in dynamic real-world environments, contributing to advancements in autonomous driving research and technology.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Bharadwaj, Sandesh
Advisor dc:contributor.advisor
  • Ohn-Bar, Eshed
ORCID dc:identifier.orcid

Subjects

dc:subject × 7

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/49892

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Bharadwaj, Sandesh. Efficient vision and language models for autonomous systems. masters thesis, Boston University, 2024. https://hdl.handle.net/2144/49892