Back to search

Virginia Tech

Evaluating Function Calling in Language Models through a Conversational Agent

Abstract

dc:description.abstract

This study investigates the effectiveness of function calling in Large Language Models (LLMs) for data-driven conversational systems. While LLMs excel in natural language understanding, reliably mapping user queries to structured computational functions remains a key challenge. To address this, the study develops the India Policy Insights (IPI) Chatbot, a GenAI-powered system that enables natural language interaction with complex, spatiotemporal public policy datasets. The chatbot integrates LLMs with backend functions to translate user queries into structured parameters, execute database operations, and generate multi-modal outputs, including text, charts, and maps. Proposed workflow demonstrates how natural language queries are converted into actionable analytical tasks. A systematic evaluation of GPT-4o and GPT-4o-mini is conducted across prompt specificity, query difficulty, and query type. Results show that function-calling accuracy improves significantly with more explicit and structured prompts, with GPT-4o achieving the highest performance. Spatial and constraint-based queries yield consistently high accuracy, while complex multi-indicator tasks remain challenging, particularly for smaller models. Overall, this study highlights the importance of prompt design, parameter clarity, and model selection in optimizing function-calling performance. It also demonstrates the potential of conversational AI to improve accessibility to policy-relevant data, contributing to more inclusive and data-informed decision-making.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Department dc:contributor.department
Not found
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sharma Dulal, Sandesh
Chair dc:contributor.committeechair
  • Kim, Junghwan
Committee members dc:contributor.committeemember
  • Rijal, Santosh
  • Crawford, Thomas Wall

Subjects

dc:subject × 1

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:46319

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Sharma Dulal, Sandesh. Evaluating Function Calling in Language Models through a Conversational Agent. masters thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143048