Multimodal NLP is an approach to language understanding that incorporates data from nontextual in order to enhance our linguistic understanding through additional contextual information. In particular, incorporating visual data has allowed for great strides in our ability to model language related to physical phenomena. The performance of these models has so far been contingent upon access to large datasets, focusing on classification problems without relative information, and constraining the problem space to literal descriptions and interpretations. In this thesis, we examine these limitations by investigating how types of data previously unused in these models can be reconfigured and worked with intelligently and on a small scale to enhance our understanding of the pragmatics of language. We contribute to comparative language grounding, emotional interpretation, and metaphoric understanding by releasing multiple annotated datasets, developing a new paradigm for modeling relative data, creating a new task in examining the generation of emotional descriptions for image information, and demonstrating a novel approach to working with figurative text for image generation. We start by examining how traditional grounding models could be adapted to incorporate relative information. As no previous work has ever utilized relative textual description for image understanding, we first constrain the problem by focusing on the language of color. We create a new dataset of comparative color terms with associated RGB datapoints, and use this data to develop a novel paradigm of grounding comparative color terms in RGB space, providing the first avenue towards utilizing relative information in a multimodal setting. Continuing our study of color, we then turn to examining the relationship between color and emotion. In order to further our understanding of this relationship, we define a new task, called Justified Affect Transformation, in which an image is recolored specifically to alter its emotional evocation and text is generated to explain the recoloring from an emotional perspective. We create a dataset of abstract art with contiguous emotion labels and textual rationales for the emotional evocation of multiple images, and using our new dataset for training, introduce a new unified model that recolors an image and provides a textual rationale explaining the recoloring with respect to the specified emotion. We use this model to examine the relationship between color and emotion devoid of confounding factors. Finally, we turn to figurative language as a resource, examining the pragmatics of visualizing metaphoric phrases. We demonstrate a novel approach to generating visual metaphor through the collaboration of large language models and diffusion-based text-to-image models, and in doing so create a novel dataset of visual metaphor with both literal and figurative captions. We then develop an evaluation framework using human-AI collaboration to examine the efficacy of the model collaboration, and choose a downstream task of visual entailment to evaluate the human-AI collaboration.

No description available.

Our experience of the world is subjective–we are constantly interpreting the world around us according to what we have already perceived, experienced, and learned. How we interpret the world–and how we draw on prior experience to do so–is studied in psychology and cognitive neuroscience, theorized about in philosophy, and explored in the arts. To study subjective interpretation, we combine multiple disciplines – using behavioral paradigms from cognitive neuroscience and psychology in order to test an overarching framework of subjective interpretation that arose in art history–the Beholder’s Share. In this dissertation, I present three studies that investigate the behavioral and neural phenomena of the Beholder’s Share. I begin, in Chapter 1, by giving an overview of the Beholder’s Share and its intersections with theories of the mind and brain. I then discuss our approach to studying the Beholder's Share; namely, by measuring cognitive and neural responses to abstract and representational art by the same artist, as a key prediction following from the Beholder’s Share is that it will be different for abstract and representational art. Following this, I then present a review of the literature that has begun to characterize the cognitive and neural responses to abstract and representational art, and the open questions we address in our studies. Chapter 2 presents a behavioral study that leverages the well-established theory of mental representations–Construal Level Theory (CLT). Drawing from CLT, we develop a behavioral paradigm that reliably characterizes differences in mental representations between abstract art and representational art, showing that abstract art evokes more abstract, context-independent representations than representational art. This study serves to establish reliable and measurable differences in the subjective experience of abstract and representational art, and yields a task that can be used to elicit these differences. Chapter 3 describes a study that combines behavior and fMRI, and takes advantage of advancements in multivariate analysis methods of brain activity and models of natural language processing to capture the Beholder’s Share in neural activity and written descriptions. This study demonstrates that both neural and semantic representations evoked by abstract paintings are more subject-unique than those evoked by representational paintings. Moreover, subject-unique patterns of brain activity are present in the Default Mode Network, a set of brain regions thought to be involved in internally oriented cognition. This study demonstrates that participants contribute personal associations to abstract paintings more than to representational paintings, and links this process to brain regions involved in higher-level cognitive processes. Chapter 4 examines the role of prior experience in subjective interpretation. I present a study in which we induced different prior experiences with an emotional autobiographical memory induction and measured the effects of that manipulation in written descriptions of abstract paintings. This study shows that abstract paintings are more vulnerable to manipulations in prior experiences, as well as individual differences in naturally occurring experiences, measured by self-report. Together, these results suggest that abstract paintings are interpreted more subjectively than representational paintings. This process of subjective interpretation recruits regions of the brain involved in internally oriented cognition (the DMN) and involves drawing on prior experiences. These results, and the methods we used to obtain them, have implications for understanding subjective experience and cognition more fully. Chapter 5 situates these results in the broader discussion of how we study subjectivity, and carves out a role for the Beholder’s Share in future research characterizing individual differences.

Indiana University-Purdue University Indianapolis (IUPUI) / Over the course of five weeks, three children were engaged in various exercises involving the observance and creation of metaphors. Before and after the exercises, the children were asked a series of questions designed to determine their understanding of and ability to express their understanding of five abstract concepts. It was found that working with metaphor enhanced their abilities to grasp the concepts, in various and surprising ways. It was also found that their abilities to express their understanding was enhanced subtly.

No description available.

Synchronous Dataflow (SDF) graphs are a convenient way to represent many signal processing and dataflow operations. Nodes within SDF graphs represent computation while arcs represent dependencies between nodes. Using a graph representation, SDF graphs formally specify a dataflow algorithm without any assumptions on the final implementation. This allows an SDF model to be synthesized into a variety of implementation techniques including both software and hardware. This thesis presents a technique for generating an abstract hardware representation from SDF models. The techniques presented here operate on SDF models defined structurally within the Ptolemy modeling environment. The behavior of the nodes within Ptolemy SDF models is specified in software and can be simple, such as a single arithmetic operation, or arbitrarily complex. This thesis presents a technique for extracting the behavior of a limited class of SDF nodes defined in software and generating a structural description of the SDF model based on primitive arithmetic and logical operations. This synthesized graph can be used for subsequent hardware synthesis transformations.

The United States of America has dropped behind many countries in terms of the Science and Engineering university degrees awarded since the beginning of the nineties. Multiple studies have been conducted to determine the cause of this decline in degrees awarded, and try to reverse the trend in US education. The goal of these studies was to determine the proper instructional methods that facilitate the knowledge acquisition process for the student. It has been determined that not one method works for all types of curriculum, for example methods that have been found to work effectively in curriculum that teaches procedures and physical systems often fail in curriculum that teaches abstract and conceptual content. The purpose of this study is to design an instructional method that facilitates teaching of abstract knowledge, and to demonstrate its effectiveness through empirical research. An experiment including 72 undergraduate students was conducted to determine the best method of acquiring abstract knowledge. All students were presented with the same abstract knowledge but presented in different types of organization. These organization types consisted of hierarchy referred as Bottom Up, Top Down, and Unorganized. Another factor that was also introduced is Graphing, which is a method that is believe to improve the learning process. The experiment was completed in 8 weeks and data was gathered and analyzed. The results strongly suggest that abstract knowledge acquisition is greatly improved when the knowledge is presented in a Bottom Up hierarchical fashion. On the other hand, neither Graphing nor the Top Down or Unorganized conditions affect learning in these novice students. iv This dissertation is dedicated to my parents who established my education career since the beginning of my first day of school. Their prayers and encouragements have an impact toward my life and education.

Developing interactive web application systems requires a large amount of effort on designing database, system logic and user interface. These tasks are expensive and error-prone. Web application systems are accessed and used by many different sets of people with different backgrounds and numerous demands. Meeting these demands requires frequent updating for Web application systems which results in a very high cost process. Thus, many attempts have been made to automate, to some degree, the construction of Web user interfaces. Three main directions have been cited for this purpose. The first direction suggested of generating user interfaces from the application¿s data model. This path was able to generate the static layout of user interfaces with dynamic behaviour specified programmatically. The second tendency suggested deployment of the domain model to generate both, the layout of a user interface and its dynamic behaviour. Web applications built based on this approach are most useful for domain-specific interfaces with a relatively fixed user dialogue. The last direction adopted the notion of deploying database metadata to developing dynamic user interfaces. Although the notion was quite valuable, its deployment did not present a generic solution for generating a variety of types of dynamic Web user interface targeting several platforms and electronic devices. This thesis has inherited the latter direction and presented significant improvements on the current deployment of this tendency. This thesis aims to contribute towards the development of an abstract framework to generate abstract and dynamic Web user interfaces not targeted to any particular domain or platform. To achieve this target, the thesis proposed and evaluates a general notion for implementing a prototype system that uses an internal model (i.e. database metadata) in conjunction with XML technology. Database metadata is richer than any external model and provides the information needed to build dynamic user interfaces. In addition, XML technology became the mainstream of presenting and storing data in an abstract structure. It is widely adopted in Web development society because of its ability to be transformed into many different formats with a little bit of effort. This thesis finds that only Java can provide us with a generalised database metadata based framework. Other programming languages apply some restrictions on accessing and extracting database metadata from numerous database management systems. Consequently, JavaServlets and relational database were used to implement the proposed framework. In addition, Java Data Base Connectivity was used to bridge the two mentioned technologies. The implementation of our proposed approach shows that it is possible and very straightforward to produce different automatic and dynamic Web entry forms that not targeted at any platform. In addition, this approach can be applied to a particular domain without affecting the main notion or framework architecture. The implemented approach demonstrates a number of advantages over the other approaches based on external or internal models.

While new approaches to displaying art free both the art and the viewer from overly didactic forms of curation, there have been very few attempts to examine how viewers negotiate meaning from art when no goal or directive is provided. While some see difference as the critical factor, others use similarity as a way to introduce new narratives. This dissertation research takes a close look at the kinds of things people observe in visual works of art to expose the specific ways that the offerings in the work are made knowable by its viewer and how different modes of presentation might affect the process. A paired design was developed to find out how juxtaposing works on dimensions of similarity and difference might affect what people see in individual paintings and whether the presence or absence of depictive content would be a factor. In three online experiments, participants were tasked with generating as many single words or short phrase responses as they could over a two-minute time period from a selection of modern and contemporary paintings – 32 abstract and 32 representational. In the first study, paintings were presented sequentially. In the next study, the same pictures were purposefully matched for color, composition, style, and thematic content. In the third study, the same pictures were re-paired to maximize difference. Pairing effected an overall decline in number of total comments for representational paintings compared to isolated single-view sequences. In contrast, significant increases were found for abstract art when the adjacent painting was also abstract. Significant consistency in response patterns for both art types across all three studies provide quantitative and content-based evidence for a normative level of engagement, with specific processing effects relative to art type.

The exponential increase in online text has created a pressing need for automatic summarization systems that can distill key information from lengthy documents. While neural abstractive summarizers have achieved gains in fluency and coherence, a critical challenge that has emerged is ensuring faithfulness, i.e., accurately preserving the meaning from the original text. Modern neural abstractive summarizers can distort or fabricate facts, undermining their reliability in real-world applications. Thus, this thesis tackles the critical issue of improving faithfulness in abstractive summarization. This thesis is comprised of four parts. The first part examines challenges in evaluating summarization faithfulness, including issues with reference-free metrics and human evaluation. We propose a novel approach for building automated evaluation metrics that are less reliant on spurious correlations and demonstrate significantly improved performance over existing faithfulness evaluation metrics. We further introduce a novel evaluation framework that enables a more holistic assessment of faithfulness by accounting for the abstractiveness of summarization systems. This framework enables more rigorous faithfulness evaluation, differentiating between gains from increased extraction versus improved abstraction. The second part focuses on explaining the root causes of faithfulness issues in modern summarization systems. We introduce a novel contrastive approach for attributing errors that vastlyoutperforms prior work at tracing hallucinations in generated summaries back to training data deficiencies. Moreover, incorporating our method’s ideas into an existing technique substantially boosts its performance. Through a case study, we also analyze pre-training biases and demonstrate their propagation to summarization models, yielding biased hallucinations. We show that while mitigation strategies during finetuning can reduce overall hallucination rates, the remaining hallucinations still closely reflect intrinsic pre-training biases. The third part applies insights from previous sections to develop impactful techniques for improving faithfulness in practice. We propose a novel approach for adaptively determining the appropriate level of abstractiveness for a given input to improve overall faithfulness. Our method yields systems that are both more faithful and more abstractive compared to baseline systems. We further leverage our error attribution approach to clean noisy training data, significantly reducing faithfulness errors in generated outputs. Models trained on datasets cleaned with our approach generate markedly fewer hallucinations than both baseline systems and models trained using other data cleaning techniques. Finally, the fourth part examines the summarization capabilities of LLMs and assesses their faithfulness. We demonstrate that instruction-tuning and RLHF are key for enabling LLMs to achieve high-quality zero-shot summarization in the news domain, with state-of-the-art LLMs generating summaries comparable to human-written ones. However, this ability does not extend to narrative summarization, where even advanced LLMs struggle to produce consistently faithful summaries. Finally, we highlight the difficulty in evaluating high-performing LLMs, showing that crowdsourcing evaluations of LLM outputs may no longer be reliable as fluency and coherence improve. We observe a substantial gap between crowd workers and experts in identifying deficiencies in LLM-generated narrative summaries.