<p> One pathway for a new device to gain access to the marketplace requires demonstration that it is equivalent to, or substantially better than, a legally marketed device. To evaluate the equivalence of a medical imaging device, we propose measuring the intra- or inter-reader agreement in a reader study, where the clinicians (readers) make diagnoses on the medical images (cases) using both the new and old imaging devices. Such an endpoint, as well as its variance estimate, enable us to make a statistical inference on the equivalence of two devices. A method for multiple-reader multiple-case agreement analysis was presented in Gallas et al. (2016) for fully-crossed study designs, where every reader reads every case. In practice, having every reader read every case may be impossible when readers have a limited amount of time to participate in the study. One alternative study design is the split-plot study design, where both the readers and the cases are partitioned into a fixed number of groups, and each group of readers reads its own group of cases. In this thesis, we adapt the multiple-reader multiple-case agreement analysis method in Gallas et al. (2016) to analyze split-plot study designs, and propose a new variance estimator based on splitting the analysis across the groups. In each split sub-study, we compute an estimate, and then combine these estimates to obtain the final estimate for the full study. Our numerical studies show that the "split-analysis" variance estimator provides more accurate estimation of the variance of concordance measurements than the full-study-based method for unbalanced split-plot study designs.</p><p>