Structured document retrieval (SDR) systems minimize the effort users spend to locate relevant information by retrieving sub-documents (i.e., parts of, as opposed to entire, documents) to focus the user's attention on the relevant parts of a retrieved document. SDR search tasks are differentiated by the multiplicity of ways that users prefer to spend effort and gain relevant information in SDR. The sub-document retrieval paradigm has required researchers to undertake costly user studies to validate whether new IR measures, based on gain and effort, accurately capture IR performance. We propose the Extended Structural Relevance (ESR) framework as a way, akin to classical set-based measures, to formulate SDR measures that share the common basis of our proposed pillars of SDR evaluation: relevance, navigation and redundancy. Our experimental results show how ESR provides a flexible way to formulate measures, and addresses the challenge of testing measures across related search tasks by replacing costly user studies with low-cost simulation.

Structured document retrieval (SDR) systems minimize the effort users spend to locate relevant information by retrieving sub-documents (i.e., parts of, as opposed to entire, documents) to focus the user's attention on the relevant parts of a retrieved document. SDR search tasks are differentiated by the multiplicity of ways that users prefer to spend effort and gain relevant information in SDR. The sub-document retrieval paradigm has required researchers to undertake costly user studies to validate whether new IR measures, based on gain and effort, accurately capture IR performance. We propose the Extended Structural Relevance (ESR) framework as a way, akin to classical set-based measures, to formulate SDR measures that share the common basis of our proposed pillars of SDR evaluation: relevance, navigation and redundancy. Our experimental results show how ESR provides a flexible way to formulate measures, and addresses the challenge of testing measures across related search tasks by replacing costly user studies with low-cost simulation.