Return to search

An evaluation paradigm for spoken dialog systems based on crowdsourcing and collaborative filtering.

Yang, Zhaojun. / Thesis (M.Phil.)--Chinese University of Hong Kong, 2011. / Includes bibliographical references (p. 92-99). / Abstracts in English and Chinese. / Chapter 1 --- Introduction --- p.1 / Chapter 1.1 --- SDS Architecture --- p.1 / Chapter 1.2 --- Dialog Model --- p.3 / Chapter 1.3 --- SDS Evaluation --- p.4 / Chapter 1.4 --- Thesis Outline --- p.7 / Chapter 2 --- Previous Work --- p.9 / Chapter 2.1 --- Approaches to Dialog Modeling --- p.9 / Chapter 2.1.1 --- Handcrafted Dialog Modeling --- p.9 / Chapter 2.1.2 --- Statistical Dialog Modeling --- p.12 / Chapter 2.2 --- Evaluation Metrics --- p.16 / Chapter 2.2.1 --- Subjective User Judgments --- p.17 / Chapter 2.2.2 --- Interaction Metrics --- p.18 / Chapter 2.3 --- The PARADISE Framework --- p.19 / Chapter 2.4 --- Chapter Summary --- p.22 / Chapter 3 --- Implementation of a Dialog System based on POMDP --- p.23 / Chapter 3.1 --- Partially Observable Markov Decision Processes (POMDPs) --- p.24 / Chapter 3.1.1 --- Formal Definition --- p.24 / Chapter 3.1.2 --- Value Iteration --- p.26 / Chapter 3.1.3 --- Point-based Value Iteration --- p.27 / Chapter 3.1.4 --- A Toy Example of POMDP: The NaiveBusInfo System --- p.27 / Chapter 3.2 --- The SDS-POMDP Model --- p.31 / Chapter 3.3 --- Composite Summary Point-based Value Iteration (CSPBVI) --- p.33 / Chapter 3.4 --- Application of SDS-POMDP Model: The Buslnfo System --- p.35 / Chapter 3.4.1 --- System Description --- p.35 / Chapter 3.4.2 --- Demonstration Description --- p.39 / Chapter 3.5 --- Chapter Summary --- p.42 / Chapter 4 --- Collecting User Judgments on Spoken Dialogs with Crowdsourcing --- p.46 / Chapter 4.1 --- Dialog Corpus and Automatic Dialog Classification --- p.47 / Chapter 4.2 --- User Judgments Collection with Crowdsourcing --- p.50 / Chapter 4.2.1 --- HITs on Dialog Evaluation --- p.51 / Chapter 4.2.2 --- HITs on Inter-rater Agreement --- p.53 / Chapter 4.2.3 --- Approval of Ratings --- p.54 / Chapter 4.3 --- Collected Results and Analysis --- p.55 / Chapter 4.3.1 --- Approval Rates and Comments from Mturk Workers --- p.55 / Chapter 4.3.2 --- Consistency between Automatic Dialog Classification and Manual Ratings --- p.57 / Chapter 4.3.3 --- Inter-rater Agreement Among Workers --- p.60 / Chapter 4.4 --- Comparing Experts to Non-experts --- p.64 / Chapter 4.4.1 --- Inter-rater Agreement on the Let's Go! System --- p.65 / Chapter 4.4.2 --- Consistency Between Expert and Non-expert Annotations on SDC Systems --- p.66 / Chapter 4.5 --- Chapter Summary --- p.68 / Chapter 5 --- Collaborative Filtering for Performance Prediction --- p.70 / Chapter 5.1 --- Item-Based Collaborative Filtering --- p.71 / Chapter 5.2 --- CF Model for User Satisfaction Prediction --- p.72 / Chapter 5.2.1 --- ICFM for User Satisfaction Prediction --- p.72 / Chapter 5.2.2 --- Extended ICFM for User Satisfaction Prediction --- p.73 / Chapter 5.3 --- Extraction of Interaction Features --- p.74 / Chapter 5.4 --- Experimental Results and Analysis --- p.76 / Chapter 5.4.1 --- Prediction of User Satisfaction --- p.76 / Chapter 5.4.2 --- Analysis of Prediction Results --- p.79 / Chapter 5.5 --- Verifying the Generalibility of CF Model --- p.81 / Chapter 5.6 --- Evaluation of The Buslnfo System --- p.86 / Chapter 5.7 --- Chapter Summary --- p.87 / Chapter 6 --- Conclusions and Future Work --- p.89 / Chapter 6.1 --- Thesis Summary --- p.89 / Chapter 6.2 --- Future Work --- p.90 / Bibliography --- p.92

Identiferoai:union.ndltd.org:cuhk.edu.hk/oai:cuhk-dr:cuhk_327561
Date January 2011
ContributorsYang, Zhaojun., Chinese University of Hong Kong Graduate School. Division of Systems Engineering and Engineering Management.
Source SetsThe Chinese University of Hong Kong
LanguageEnglish, Chinese
Detected LanguageEnglish
TypeText, bibliography
Formatprint, xi, 99 p. : ill. ; 30 cm.
RightsUse of this resource is governed by the terms and conditions of the Creative Commons “Attribution-NonCommercial-NoDerivatives 4.0 International” License (http://creativecommons.org/licenses/by-nc-nd/4.0/)

Page generated in 0.0023 seconds