As software engineering is a socio-technical research field, there is a myriad of research strategies and data sources that researchers need to consider when designing their studies. These choices determine different tradeoffs in terms of generalizability, realism, and control, among other aspects of research quality. It is not possible to create a perfect study, so these strengths and weaknesses are acceptable at the study level; however, when a research community's collective body of work suffers from an imbalance in these tradeoffs it can negatively impact overall research quality.
Through this thesis, I investigate the research strategies and data sources that are used by the software engineering research community, and reflect on how this may affect aspects of research quality in our collective body of work. I apply Runkel and McGrath's models of research strategies and data sources to the software engineering domain through a systematic mapping study of three years of International Conference on Software Engineering (ICSE) proceedings and a mixed-methods survey of the authors of these papers.
I found that a majority of papers report computational studies relying on trace measures rather than active human participation, showing an imbalance where generalizability and realism are prioritized over control. Through my survey, I confirmed that researcher participants explicitly prioritized realism and generalizability over control, impacting their research design choices. This imbalance in prioritization has the potential to lead to a collective failure to control for extraneous factors in the measurement of human behavior in software development, and without understanding what causes the behaviors we measure, we cannot fully understand why certain approaches and techniques work better than others, thus slowing our ability to advance as a research domain. Therefore, I present a call to action for the community to critically examine and discuss the issues raised by this research, and implement changes to increase the quality and diversity of our future work as a community. / Graduate / 2019-08-08
| Identifer | oai:union.ndltd.org:uvic.ca/oai:dspace.library.uvic.ca:1828/9997 |
| Date | 30 August 2018 |
| Creators | Bornholdt, Courtney |
| Contributors | Storey, Margaret-Anne |
| Source Sets | University of Victoria |
| Language | English, English |
| Detected Language | English |
| Type | Thesis |
| Format | application/pdf |
| Rights | Available to the World Wide Web |
Page generated in 0.0024 seconds