Identifying individuals in historical photographs is important for preserving material culture, correcting historical records, and adding economic value. Historians, antiques dealers, and collectors often rely on manual, time-consuming approaches. While Artificial Intelligence (AI) offers potential solutions, it's not widely adopted due to a lack of specialized tools and inherent inaccuracies and biases. In my dissertation, I address this gap by combining the complementary strengths of human intelligence and AI.
I introduce Photo Sleuth, a novel person identification pipeline that combines crowdsourced expertise with facial recognition, supporting users in identifying unknown portraits from the American Civil War era (1861--65). Despite successfully identifying numerous unknown photos, users often face the `last-mile problem' --- selecting the correct match(es) from a shortlist of high-confidence facial recognition candidates while avoiding false positives. To assist experts, I developed Second Opinion, an online tool that employs a novel crowdsourcing workflow, inspired by cognitive psychology, effectively filtering out up to 75% of facial recognition's false positives.
Yet, as AI models continually evolve, changes in the underlying model can potentially impact user experience in such crowd--expert--AI workflows. I conducted an online study to understand user perceptions of changes in facial recognition models, especially in the context of historical person identification. Our findings showed that while human-AI collaborations were effective in identifying photos, they also introduced false positives.
To reduce these misidentifications, I built Photo Steward, an information stewardship architecture that employs a deliberative workflow for validating historical photo identifications. Building on this foundation, I introduced DoubleCheck, a quality assessment framework that combines community stewardship and comprehensive provenance information, for helping users accurately assess photo identification quality. Through my dissertation, I explore the design and deployment of human-AI collaborative tools, emphasizing the creation of sustainable online communities and workflows that foster accurate decision-making in the context of historical photo identification. / Doctor of Philosophy / Identifying historical photos offers significant cultural and economic value; however, the identification process can be complex and challenging due to factors like poor source material and limited research resources. In my dissertation, I address this problem by leveraging the complementary strengths of human intelligence and Artificial Intelligence (AI). I built Photo Sleuth, an online platform, that helps users in identifying unknown portraits from the American Civil War era. This platform employs a novel person identification workflow that combines crowdsourced human expertise and facial recognition. While AI-based facial recognition is effective at quickly scanning thousands of photos, it can sometimes present challenges. Specifically, it provides the human expert with a shortlist of highly similar-looking candidates from which the expert must discern the correct matches; I call this as the `last-mile problem' of person identification. To assist experts in navigating this challenge, I developed Second Opinion, a tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, named seed-gather-analyze. Further, I conducted an online study to understand the influence of changes in the underlying facial recognition models on the downstream person identification tasks. While these tools enabled numerous successful identifications, they also occasionally led to misidentifications. To address this issue, I introduced Photo Steward, an information stewardship architecture that encourages deliberative decision-making while identifying photos. Building upon the principles of information stewardship and provenance, I then developed DoubleCheck, a quality assessment framework that presents pertinent information, aiding users in accurately evaluating the quality of historical photo IDs. Through my dissertation, I explore the design and deployment of human-AI collaborative tools, emphasizing the creation of sustainable online communities and workflows that encourage accurate decision-making in the context of historical photo identification.
| Identifer | oai:union.ndltd.org:VTETD/oai:vtechworks.lib.vt.edu:10919/116169 |
| Date | 30 August 2023 |
| Creators | Mohanty, Vikram |
| Contributors | Computer Science and Applications, Luther, Kurt, Johnson, Sylvester A., Lee, Sang Won, Liao, Qingzi, North, Christopher L. |
| Publisher | Virginia Tech |
| Source Sets | Virginia Tech Theses and Dissertation |
| Language | English |
| Detected Language | English |
| Type | Dissertation |
| Format | ETD, application/pdf |
| Rights | Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International, http://creativecommons.org/licenses/by-nc-nd/4.0/ |
Page generated in 0.0021 seconds