We present a controlled experiment for the empirical evaluation of Example-Driven Modeling (EDM), an approach that systematically uses examples for model comprehension and domain knowledge transfer. We conducted the experiment with 26 graduate and undergraduate students from electrical and computer engineering (ECE), computer science (CS), and software engineering (SE) programs at the University of Waterloo. The experiment involves a domain model, with a UML class diagram representing the domain abstractions and UML object diagrams representing examples of using these abstractions. The goal is to provide empirical evidence of the effects of suitable examples on model comprehension, compared to having model abstractions only, by having the participants perform model comprehension tasks. Our results show that EDM is superior to having model abstractions only, with an improvement of (+39%) for diagram completeness, (+30%) for study questions completeness, (+71%) for efficiency, and a reduction of (-80%) for the number of mistakes. We provide qualitative results showing that participants receiving model abstractions augmented with examples experienced lower perceived difficulty in performing the comprehension tasks, higher perceived confidence in their tasks' solutions, and asked fewer clarifying domain questions (a reduction of 90%). We also present participants' feedback regarding the usefulness of the provided examples, the number of examples, the types of examples, and the use of partial examples.

The goal of this thesis is to demonstrate how user testing can be used to identify and remediate learnability issues of a web application. Experimentation revolved around ASSISTments (www.assistments.org), an intelligent tutoring web application in which teachers create virtual classrooms where they can assign problem sets to their students and gain valuable data which can be used to make informed decisions. Recent log analysis uncovered very low task completion rates for new users on tasks that were intended to be trivial. Suspecting that this could be due to poor user interface design, user tests were conducted to help identify usability problems. Sessions were analyzed, and changes were made between each user test to address issues found. Feedback from user testing led to the implementation of an embedded support system. This support system consisted of a splash page which gave an overview of how the system should be used and a collection of context-sensitive tooltips which tried to give the user instructions on what to do as well as explain various parts of the interface. A randomized control trial was performed to measure the effectiveness of the embedded support. 69 participants were shown one of two interfaces: one with embedded support and one without. Task completion rates were analyzed for each of the groups. We found that the support system was able to influence which links a user clicked. However, although the support system was intended to address poor task completion rates, users in the conditions had similar task completion rates regardless of whether the support system was enabled.

An A/B test is implemented on a SaaS rm's product page to examine the dierence in conversion rates from website visitors who are randomly assigned to two dierent product-landing pages that show dierent prices. To count as a successful conversion the visitors that view a product-landing page have to click on a \Free Trial" button. Half of the group will be assigned the treatment page, which will state higher prices and the other half will be assigned the controlled page, which will state today's current price. The only variant that will dier from the two pages will be the stated price of the product and all other factors will be kept constant. The controlled experiment is executed to get a sense of customers' price sensitivity, hence this thesis contributes to microeconomic research of the private sector, more specically to the ICT industry by using a novel approach with the help of A/B testing on prices. The results showed no statistical signicance difference between the two variations, which can be translated to accepting the null hypothesis; the demand for a particular Software-As-A-Service product will hold unchanged after the proposed price increase. At first, this could be a surprising result but when looking into the industry, which the rm participates in and their early mover advantages this result could have been assumed.

<p>This thesis presents empirical research in the field of software development with a focus on handling semantic aspects. There is a general lack of empirical data in the field of software development. This makes it difficult for industry to choose an appropriate method for their particular needs. The lack of empirical data also makes it difficult to convey academic results to the industrial world.</p><p>This thesis tries to remedy this problem by presenting a number of empirical evaluations that have been conducted to evaluate some common approaches in the field of semantics handling. The evaluations have produced some interesting results, but their main contribution is the addition to the body of knowledge on how to perform empirical evaluations in software development. The evaluations presented in this thesis include a between-groups controlled experiment, industrial case studies and a full factorial design controlled experiment. The factorial design seems like the most promising approach to use when the number of factors that need to be controlled is high and the number of available test subjects is low. A factorial design has the power to evaluate more than one factor at a time and hence to gauge the effects from different factors on the output.</p><p>Another contribution of the thesis is the development of a method for handling semantic aspects in an industrial setting. A background investigation performed concludes that there seems to be a gap between what academia proposes and how industry handles semantics in the development process. The proposed method aims at bridging this gap. It is based on academic results but has reduced formalism to better suit industrial needs. The method is applicable in an industrial setting without interfering too much with the normal way of working, yet providing important benefits. This method is evaluated in the empirical studies along with other methods for handling semantics. In the area of semantic handling, further contributions of the thesis include a taxonomy for semantic handling methods as well as an improved understanding of the relation between semantic errors and the concept of contracts as a means of avoiding and handling these errors.</p>

This thesis presents empirical research in the field of software development with a focus on handling semantic aspects. There is a general lack of empirical data in the field of software development. This makes it difficult for industry to choose an appropriate method for their particular needs. The lack of empirical data also makes it difficult to convey academic results to the industrial world. This thesis tries to remedy this problem by presenting a number of empirical evaluations that have been conducted to evaluate some common approaches in the field of semantics handling. The evaluations have produced some interesting results, but their main contribution is the addition to the body of knowledge on how to perform empirical evaluations in software development. The evaluations presented in this thesis include a between-groups controlled experiment, industrial case studies and a full factorial design controlled experiment. The factorial design seems like the most promising approach to use when the number of factors that need to be controlled is high and the number of available test subjects is low. A factorial design has the power to evaluate more than one factor at a time and hence to gauge the effects from different factors on the output. Another contribution of the thesis is the development of a method for handling semantic aspects in an industrial setting. A background investigation performed concludes that there seems to be a gap between what academia proposes and how industry handles semantics in the development process. The proposed method aims at bridging this gap. It is based on academic results but has reduced formalism to better suit industrial needs. The method is applicable in an industrial setting without interfering too much with the normal way of working, yet providing important benefits. This method is evaluated in the empirical studies along with other methods for handling semantics. In the area of semantic handling, further contributions of the thesis include a taxonomy for semantic handling methods as well as an improved understanding of the relation between semantic errors and the concept of contracts as a means of avoiding and handling these errors.

Context: Defects occurred in the SRS may cause problems in project due to implementation of poor requirements which require extra time, effort, resources and budget to complete it. Reading techniques i.e., checklist based reading (CBR) helps to guide reviewers in identifying defects in software requirement specification (SRS) during individual requirement inspections. Checklists contain potential defects/problems to look for, but often lack clear definitions with examples of the problem, and also their abstractions are different. Therefore, there is a need for identifying existing defects and classifiers and to create a consolidated version of taxonomy. Objectives: We developed taxonomy for requirement defects that are in requirement specifications and compared it with the checklist based approach. The main objective was to investigate and compare the effectiveness and efficiency of inspection techniques (checklist and taxonomy) with M.Sc. software engineering students and industry practitioners by performing a both controlled student and industry experiment. Methods: Literature review, controlled student experiment and controlled industry experiment were the research methods utilized to fulfill the objectives of this study. INSPEC and Google scholar database was used to find the articles from the literature. Controlled student experiment was conducted with the M.Sc. software engineering students and controlled industry experiment was performed with the industry practitioners to evaluate the effectiveness and efficiency of the two treatments that are checklist and taxonomy. Results: An extensive literature review helped us to identify several types of defects with their definitions and examples. In this study, we studied various defect classifiers, checklists, requirement defects and inspection techniques and then built taxonomy for requirement defects. We evaluated whether the taxonomy performed better with respect to checklist using controlled experiments with students and practitioners. Moreover, the results of student experiment (p= 0.90 for effectiveness and p=0.10 for efficiency) and practitioner experiment (p=1.0 for effectiveness and p=0.70 for efficiency) did not show significant values with respect to effectiveness and efficiency. But because of less number of practitioners it is not possible to apply a statistical test since we also have used standard formulas to calculate effectiveness and efficiency. 2 out of the 3 reviewers using taxonomy found more defect types compared to 3 reviewers using checklist. 10-15% more defects have been found by reviewers using taxonomy. 2 out of the 3 reviewers using taxonomy are more productive (measuring in hours) compared to reviewers of checklist. Although the results are quite better than the student experiment but it is hard to claim that reviewers using taxonomy are more effective and efficient than the reviewers using checklist because of less subjects in number. The results of the post experiment questionnaire revealed that the taxonomy is easy to use and easy to understand but hard to remember while inspecting SRS than the checklist technique. Conclusions: Previously researchers created taxonomies for their own purpose or on industry demand. These taxonomies lack clear and understandable definitions. To overcome this problem, we built taxonomy with requirement defects which consists of definitions and examples. No claims are made based on student experiment because of insignificant values with respect to effectiveness and efficiency. Although the controlled industry experiment results showed that taxonomy performed slightly better than the checklist in efficiency i.e., in defect detection rate and effectiveness i.e., number of defect found. From this we can conclude that taxonomy helps guiding the reviewers to indentify defects from SRS but not quite much so it is recommended to perform a further study with practitioners in a large scale for effective results. / skarimuddin@yahoo.com, hassanshah357@gmail.com

The emerging industrial trend towards agile software development processes brings forth new concerns, challenges as well as opportunities. One of the main concerns is with respect to the achievable quality levels of the final product, for which testing is the well-known assurance mechanism. However, it is not well defined for the community on how to perform testing using existing expertise in an agile environment. This uncertainty may create confusion and contra productivity that can lead to testing teams and their practices considered as an obstacle for full implementation of agile processes within an organisation.   This thesis outlines our current research activities towards identifying and addressing important organisational and technical challenges in the agile environment.  In this context, we propose a new role for traditional testers which will enable them to integrate into the agile team as well as to fully exploit their knowledge in the new context. We have conducted an elaborate industrial survey on the preferences and practices with respect to the contemporary aspects of software testing and identified test-driven development as an important technical area for improvement.  A systematic review on empirical evidences related to test-driven development was performed subsequently, which revealed a list of factors limiting its widespread industrial acceptance. Knowledge of testing was identified as one of those factors and was further investigated in a controlled experiment performed with undergraduate students.   Our future works aim to confirm these research findings in wider as well as industrial settings and investigate other limiting factors in detail, with the aim of providing guidelines for achieving better utilisation of testers and testing practices.

An A/B test is implemented on a SaaS rm's product page to examine the di erence in conversion rates from website visitors who are randomly assigned to two di erent product-landing pages that show di erent prices. To count as a successful conversion the visitors that view a product-landing page have to click on a \Free Trial" button. Half of the group will be assigned the treatment page, which will state higher prices and the other half will be assigned the controlled page, which will state today's current price. The only variant that will di er from the two pages will be the stated price of the product and all other factors will be kept constant. The controlled experiment is executed to get a sense of customers' price sensitivity, hence this thesis contributes to microeconomic research of the private sector, more specically to the ICT industry by using a novel approach with the help of A/B testing on prices. The results showed no statistical signicance difference between the two variations, which can be translated to accepting the null hypothesis; the demand for a particular Software-As-A-Service product will hold unchanged after the proposed price increase. At rst, this could be a surprising result but when looking into the industry, which the rm participates in and their early mover advantages this result could have been assumed.

More and more companies use a REST architecture to implement applications for an easy to use API. One important quality attribute of an application is the performance. To be able to understand how the application will perform it is important to know how the selected framework perform. By testing the performance of different frameworks it will become easier for software developers to choose the right framework to achieve their requirements and goals. At the time when this paper was written the research in this area was limited. This paper answered the question of which framework between Express, .NET Core, Spring and Flask that had the best performance. To be able to know how frameworks performed the author needed to measure them. One way of measuring performance is with response time from the server. The author used a controlled experiment to collect raw data from which the results was drawn. The author found out that Spring had the best overall performance between the different categories. By analysing the results the author also found out that performance differed a lot between the frameworks in some categories.

It is only a matter of time before the Swedish Companies Registration Office makes digital registration of annual reports/auditors reports mandatory. At the time of writing, there are only ten public integrators currently able to handle this requirement. In collaboration with one of the big four accounting firms, this project aims to evaluate performance of the response time while using GraphQL versus REST in Web APIs. The application under test is a .NET application integrating to the Swedish Companies Registration Office API. Through a controlled experiment using two different GraphQL frameworks, HotChocolate and GraphQL for .NET, this thesis provides a knowledge base for the partnered accounting firm, developers and other stakeholders that are evaluating the use of GraphQL in their future applications. Results  from the experiment indicate that HotChocolate in its current version is not only faster than its competitor GraphQL for .NET, but also faster than REST. This is surprising, given that other related work seems to suggest that this is not always the case. Testing of GraphQL for .NET gives a more traditional result when compared to other related work. Given the results, a senior developer of HotChocolate was contacted to gain insight to why the framework outperforms not only GraphQL for .NET but also REST. The senior developer states that a large amount of effort has been put in to make the GraphQL execution engine more optimized, something that is corroborated by this thesis experiment. HotChocolate is also periodically measuring and comparing performance benchmarks to other libraries to conclude on its performance in different scenarios. The analysis of the experiment concluded that there exists another important variable previously not identified in other research, more precisely the chosen framework, that has a large impact on performance and can impact both memory allocation and response time.