{"id":{"repo_id":"bu","oai_identifier":"oai:open.bu.edu:2144/49892"},"canonical_url":"https://search.ndltd.org/etd/bu/oai:open.bu.edu:2144/49892","repository":{"repo_id":"bu","name":"Boston University","base_url":"https://open.bu.edu/oai/request"},"display":{"title":"Efficient vision and language models for autonomous systems","abstract":"The transition of vision-based systems, including autonomous vehicles, from controlled lab environments to real-world deployment poses significant challenges due to constraints such as limited data availability and computational resources. Current approaches often rely on sending data to remote cloud servers for processing, leading to energy inefficiencies. To address these challenges, this thesis proposes novel methodologies for achieving reliable and efficient inference in dynamic scenarios. Firstly, large-scale vision-based models are evaluated within the CARLA autonomous driving simulator. We introduce a 'switch' policy to offload inference between local and cloud models, trained using reinforcement learning. We evaluate the effectiveness of this policy using a newly introduced evaluation metric: Ecological Navigation Score (ENS). This metric considers route deviations, collisions, and energy consumption, critical factors for assessing driving agent effectiveness. Secondly, we present a novel dataset to assess the performance of language models (LLMs) and vision language models (VLMs) in autonomous driving tasks. The dataset comprises questions from learner's license examinations, covering various question formats and including both text-based and visual-text pairs. We conduct evaluations on small-scale LLMs using this dataset to analyze their performance, aiming to achieve a passing score of 80% or higher on the driving test. Through this comprehensive approach, the thesis aims to address the pressing need for robust and efficient vision-based systems in dynamic real-world environments, contributing to advancements in autonomous driving research and technology.","abstract_html":"The transition of vision-based systems, including autonomous vehicles, from controlled lab environments to real-world deployment poses significant challenges due to constraints such as limited data availability and computational resources. Current approaches often rely on sending data to remote cloud servers for processing, leading to energy inefficiencies. To address these challenges, this thesis proposes novel methodologies for achieving reliable and efficient inference in dynamic scenarios. Firstly, large-scale vision-based models are evaluated within the CARLA autonomous driving simulator. We introduce a &#x27;switch&#x27; policy to offload inference between local and cloud models, trained using reinforcement learning. We evaluate the effectiveness of this policy using a newly introduced evaluation metric: Ecological Navigation Score (ENS). This metric considers route deviations, collisions, and energy consumption, critical factors for assessing driving agent effectiveness. Secondly, we present a novel dataset to assess the performance of language models (LLMs) and vision language models (VLMs) in autonomous driving tasks. The dataset comprises questions from learner&#x27;s license examinations, covering various question formats and including both text-based and visual-text pairs. We conduct evaluations on small-scale LLMs using this dataset to analyze their performance, aiming to achieve a passing score of 80% or higher on the driving test. Through this comprehensive approach, the thesis aims to address the pressing need for robust and efficient vision-based systems in dynamic real-world environments, contributing to advancements in autonomous driving research and technology.","abstract_has_math":false,"creators":["Bharadwaj, Sandesh"],"institution":"Boston University","degree_name":"Master of Science","degree_level":"masters","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":[],"advisors":["Ohn-Bar, Eshed"],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024","date_published":"2024","updated_at":"2026-11-18T20:31:29Z","subjects":["Computer science","Artificial intelligence","Autonomous systems","Computer vision","Large language models","Machine learning","Reinforcement learning"],"languages":["en_US"],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[{"key":"dc:identifier.orcid","label":"ORCID","values":["0000-0002-1491-4140"],"render_values":[{"text":"0000-0002-1491-4140","href":"https://orcid.org/0000-0002-1491-4140","code":true}]}]},"links":{"outbound_url":"https://hdl.handle.net/2144/49892","outbound_label":"Handle","outbound_source":"dc:identifier.uri","record_url":"https://hdl.handle.net/2144/49892","record_label":"Repository record","record_kind":"handle","record_source":"dc:identifier.uri","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://open.bu.edu/oai/request?verb=GetRecord&metadataPrefix=dim&identifier=oai%3Aopen.bu.edu%3A2144%2F49892","prefix":"dim"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Ohn-Bar, Eshed"]},{"key":"dc:creator","label":"Author","values":["Bharadwaj, Sandesh"]},{"key":"dc:identifier.orcid","label":"ORCID","values":["0000-0002-1491-4140"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2025-03-06T15:14:15Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2025-03-06T15:14:15Z"]},{"key":"dc:date.issued","label":"Date","values":["2024"]},{"key":"dc:date.updated","label":"Dc Date Updated","values":["2025-03-05T02:01:20Z"]},{"key":"dc:type","label":"Dc Type","values":["Thesis/Dissertation"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["masters"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master of Science"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["Boston University"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer science","Artificial intelligence","Autonomous systems","Computer vision","Large language models","Machine learning","Reinforcement learning"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en_US"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/2144/49892"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["2024"]},{"key":"dc:description.abstract","label":"Abstract","values":["The transition of vision-based systems, including autonomous vehicles, from controlled lab environments to real-world deployment poses significant challenges due to constraints such as limited data availability and computational resources. Current approaches often rely on sending data to remote cloud servers for processing, leading to energy inefficiencies. To address these challenges, this thesis proposes novel methodologies for achieving reliable and efficient inference in dynamic scenarios. Firstly, large-scale vision-based models are evaluated within the CARLA autonomous driving simulator. We introduce a 'switch' policy to offload inference between local and cloud models, trained using reinforcement learning. We evaluate the effectiveness of this policy using a newly introduced evaluation metric: Ecological Navigation Score (ENS). This metric considers route deviations, collisions, and energy consumption, critical factors for assessing driving agent effectiveness. Secondly, we present a novel dataset to assess the performance of language models (LLMs) and vision language models (VLMs) in autonomous driving tasks. The dataset comprises questions from learner's license examinations, covering various question formats and including both text-based and visual-text pairs. We conduct evaluations on small-scale LLMs using this dataset to analyze their performance, aiming to achieve a passing score of 80% or higher on the driving test. Through this comprehensive approach, the thesis aims to address the pressing need for robust and efficient vision-based systems in dynamic real-world environments, contributing to advancements in autonomous driving research and technology."]},{"key":"dc:title","label":"Title","values":["Efficient vision and language models for autonomous systems"]}]}],"canonical_facts":{"dc:contributor.advisor":["Ohn-Bar, Eshed"],"dc:creator":["Bharadwaj, Sandesh"],"dc:date.accessioned":["2025-03-06T15:14:15Z"],"dc:date.available":["2025-03-06T15:14:15Z"],"dc:date.issued":["2024"],"dc:date.updated":["2025-03-05T02:01:20Z"],"dc:description":["2024"],"dc:description.abstract":["The transition of vision-based systems, including autonomous vehicles, from controlled lab environments to real-world deployment poses significant challenges due to constraints such as limited data availability and computational resources. Current approaches often rely on sending data to remote cloud servers for processing, leading to energy inefficiencies. To address these challenges, this thesis proposes novel methodologies for achieving reliable and efficient inference in dynamic scenarios. Firstly, large-scale vision-based models are evaluated within the CARLA autonomous driving simulator. We introduce a 'switch' policy to offload inference between local and cloud models, trained using reinforcement learning. We evaluate the effectiveness of this policy using a newly introduced evaluation metric: Ecological Navigation Score (ENS). This metric considers route deviations, collisions, and energy consumption, critical factors for assessing driving agent effectiveness. Secondly, we present a novel dataset to assess the performance of language models (LLMs) and vision language models (VLMs) in autonomous driving tasks. The dataset comprises questions from learner's license examinations, covering various question formats and including both text-based and visual-text pairs. We conduct evaluations on small-scale LLMs using this dataset to analyze their performance, aiming to achieve a passing score of 80% or higher on the driving test. Through this comprehensive approach, the thesis aims to address the pressing need for robust and efficient vision-based systems in dynamic real-world environments, contributing to advancements in autonomous driving research and technology."],"dc:identifier.orcid":["0000-0002-1491-4140"],"dc:identifier.uri":["https://hdl.handle.net/2144/49892"],"dc:language.iso":["en_US"],"dc:subject":["Computer science","Artificial intelligence","Autonomous systems","Computer vision","Large language models","Machine learning","Reinforcement learning"],"dc:title":["Efficient vision and language models for autonomous systems"],"dc:type":["Thesis/Dissertation"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["masters"],"thesis:degree_name":["Master of Science"],"thesis:institution_name":["Boston University"]},"updated_at":"2026-11-18T20:31:29Z"}