The goal of this thesis is to demonstrate how user testing can be used to identify and remediate learnability issues of a web application. Experimentation revolved around ASSISTments (www.assistments.org), an intelligent tutoring web application in which teachers create virtual classrooms where they can assign problem sets to their students and gain valuable data which can be used to make informed decisions. Recent log analysis uncovered very low task completion rates for new users on tasks that were intended to be trivial. Suspecting that this could be due to poor user interface design, user tests were conducted to help identify usability problems. Sessions were analyzed, and changes were made between each user test to address issues found. Feedback from user testing led to the implementation of an embedded support system. This support system consisted of a splash page which gave an overview of how the system should be used and a collection of context-sensitive tooltips which tried to give the user instructions on what to do as well as explain various parts of the interface. A randomized control trial was performed to measure the effectiveness of the embedded support. 69 participants were shown one of two interfaces: one with embedded support and one without. Task completion rates were analyzed for each of the groups. We found that the support system was able to influence which links a user clicked. However, although the support system was intended to address poor task completion rates, users in the conditions had similar task completion rates regardless of whether the support system was enabled.

The thesis analyses the customer behavior of Kiwi.com, a global online retail company for booking flights and proposes marketing interventions to increase the conversion rates in various customer segments. The effectiveness of new behavioral interventions is tested against current marketing efforts using experimental A/B methods. Additionally, areas for further improvements are explored and a design of future product features and marketing behavioral interventions is proposed.

An A/B test is implemented on a SaaS rm's product page to examine the dierence in conversion rates from website visitors who are randomly assigned to two dierent product-landing pages that show dierent prices. To count as a successful conversion the visitors that view a product-landing page have to click on a \Free Trial" button. Half of the group will be assigned the treatment page, which will state higher prices and the other half will be assigned the controlled page, which will state today's current price. The only variant that will dier from the two pages will be the stated price of the product and all other factors will be kept constant. The controlled experiment is executed to get a sense of customers' price sensitivity, hence this thesis contributes to microeconomic research of the private sector, more specically to the ICT industry by using a novel approach with the help of A/B testing on prices. The results showed no statistical signicance difference between the two variations, which can be translated to accepting the null hypothesis; the demand for a particular Software-As-A-Service product will hold unchanged after the proposed price increase. At first, this could be a surprising result but when looking into the industry, which the rm participates in and their early mover advantages this result could have been assumed.

A VR car-driving simulator for evaluating the user experience of new drivers by helping them to learn driving rules and regulations. The Classic VR Driver helps new drivers to learn driving rules and regulations using various audio and visual feedback. The simulator helps them to get acquainted with the risks and mistakes associated with real life driving. In addition, the users have to play the game in an immersive environment using a Virtual Reality system. This project attempts to fulfill two important goals. The major goal is to evaluate whether the user can learn driving rules and regulations of the road. The game allows the users to take a road test. The road test determines the type of mistakes the user makes and it also determines if they passed or failed in it. I have conducted A/B testing and let the testers participate in user-interviews and user-survey. The testing procedure allowed me to analyze the effectiveness of learning driving rules from the simulator as compared to learning rules from the RMV (Registry of Motor Vehicles) manual. Secondly, the user experience was evaluated by allowing users to participate in user-interviews and user-surveys. It helped me to understand the positives and drawbacks of the game. These feedback are taken into consideration for future improvement. All these factors were considered to make the game as enjoyable and useful in terms of skill training.

Web development is a challenging task and it’s easy to neglect feedback from users in development stages. That’s why the aim is to create a tool which would help the communication between developers and users by using A/B-testing. The idea is to let developers release two choices containing images, which would be the intended design changes. By letting the users vote for the preferred option, they will be able to provide some feedback for the developers. Response times becomes a critical factor for the tool’s overall success. Therefore, this study compares MySQL and PostgreSQL through a technical experiment to see which database would be the better option regarding the image processing. The experiment indicated that PostgreSQL would the better alternative regarding the subject, as it had the most responsive processing of images. This prototype provides a good foundation for a potentially useful system that could be implemented in future work.

A/B testing combined with funnel analysis is a highly interesting technique to support data driven decision making. This thesis outlines a scalable architecture that gathers custom defined events and applies funnel analysis to gain valuable insights about user behaviour. The insights of the users are discussed from an A/B testing point of view, however, these insights are just as valuable for scenarios outside A/B testing as well. Custom defined events together with A/B testing is an interesting combination, since it provides opportunities to test different versions of an application against each other, based on relevant metrics. Where the vision is to determine which of the tested application versions that is best. The power to make smart data driven decisions lies in the hand of good data analysis of the end-users. Having pre-defined metrics such as counts, of some sort, is one way to do it. However, it reduces the flexibility to let e.g. application managers to "dig deeper" into what is actually happening. Funnel analysis is a technique to analyze sequences, and can be used to analyze user behaviour in a sequential matter. There are different techniques to provide such tools, e.g. with Google Analytics, users can pre-define funnel steps that they want Google Analytics to register when events are logged. This thesis will instead strive to not require anything being pre-defined and also make it possible, at high scale, provide dynamic low latency queries. A proof of concept architecture has been presented in this thesis, to support two problematic ends of a spectrum, that is; at scale, both to store custom events, and at the same time be able to interactively run dynamic low latency funnel analysis on the events.

This report describes the work process behind two A/B tests that was developed, launched and analyzed. The tests were designed for the e-merchants Wegot and Cirkusfabriken. The work was carried out at the conversion optimization company Internetexperten. The report reviews the tools used for development, as well as the actual work that the development of the two tests entailed. After that the results of the tests will be described based on how the new design performed on the sections of the sites in the original version that they where compared against. / Den här rapporten redogör för arbetsprocessen kring två stycken A/B test som utvecklas, lanseras samt analyseras. Testerna är framtagna till e-handelsföretagen Wegot och Cirkusfabriken. Arbetet är utfört internt på konverteringsoptimeringsföretaget Internetexperten. Rapporten går igenom vilka hjälpmedel som använts för framtagandet, samt det faktiska arbetet som utvecklingen av de två testerna medförde. Efter detta beskrivs resultatet av hur testerna presterade mot de sektioner av webbplatserna i originalutförande som ställdes mot den nya designen.

Split testing is a popular practise to compare and evaluate design elements of web-sites. In this report, it is explored how split testing can be employed to test features of motion-based online games. This is done by applying theory taken from split testing of common web-pages and translating it to work with games. A pilot test is conducted on the newly developed game Phaseformer to evaluate two interaction techniques. With only 105 test participants, the results from the pilot test show with high statistical power that one variant provides the player with a higher level of control over the game than the other.

Mikrotransaktioner är en stor del av dagens internethandel. Det används flitigt på både websidor och i datorspel. Denna rapport tar upp hur ett företag eller en privatperson kan, genom A/B-testning, testa vilken transaktionstyp som är bäst för just deras applikation. Problemet är att det finns många olika transaktionstyper att välja mellan och att det inte finns något bra sätt att få fram vilket som är det bästa för just din applikation. För att försöka hitta den optimala transaktionstypen utnyttjades en testapplikation med olika transaktionstyper. Genom att mäta en transaktiontyps effektivitet kan vi få fram en mätpunkt med vilken man kan jämföra med andra transaktionstyper. Resultaten visar att A/B-testning av mikrotransaktioner går att genomföra genom att följa de arkitekturerna som tillhandahålls av rapporten. / Micro transactions are a big part of today's internet commerce. They are used diligently in both web applications and in computer games. This report will explore how a company or private developer, through A/B-testing, can test which transaction type is the best for their specific application. The problem is that there's a large amount of transaction types to choose from and there is not any good way to find out which one is the best for your application. To do this there needs to be some way to test the different types to receive the optimal one. By measuring a transaction type's efficiency, we can get a point of measure which can be compared to other transaction types. The results show that A/B-testing is applicable on micro transactions by following the architectures provided by this report.

This text concerns methods of parallel quantitative testing of web pages in order to increase their profitability. These methods bring new possibilities and overcome drawbacks of their ancestors. This work deals in detail with preparation, execution and result evaluations of quantitative tests. In chapters concerning test preparation this text describes the right technique of hypothesis formulation and selection of method for their verification. Text continues with description of tools available for conducting quantitative testing. Following chapters contain set of best practices and suggestions for successful testing. In its practical part, the text proves benefits of these methods by results interpretation of two successful tests conducted on a real e-commerce project and corporate website. Practical part also describes technical aspects of conducted tests in detail.