Background: Web application security (WAS) testing is crucial for protecting web applications from cyber threats. However, organizations often struggle to select effective WAS testing tools due to the lack of a well-defined set of evaluation criteria. This research aims to address this need by identifying the key metrics for evaluating and comparing WAS testing tools.  Objectives: The primary objectives of this research are to identify the key metrics for comparing WAS testing tools, validate the significance of these metrics through semi-structured interviews, and perform a comparison between WAS testing tools using the validated metrics. This research aims to find a set of validated metrics for evaluating and comparing WAS testing tools.  Methods: The research methodology consisted of three main phases: a literature review to compile a comprehensive set of technical and non-technical metrics commonly used for assessing and comparing WAS testing tools, semi-structured interviews with security experts to validate the significance of the identified metrics, and an experiment to compare three WAS testing tools - ZAP, Burp Suite, and Acunetix - using the OWASP Benchmark project. These three tools were selected based on the author’s recommendations in the literature.  Results: The initial literature review found 37 evaluation metrics for WAS testing tools. Through interviews, experts confirmed some of these were important, but also said some were not very useful. The experts additionally suggested some new metrics that were not in the literature. Incorporating this feedback, the final list was refined down to 35 metrics for evaluating WAS testing tools. An experiment was then conducted to compare three WAS testing tools - ZAP, Burp Suite, and Acunetix with the test subject as the OWASP Benchmark Project and by using the validated set of metrics. The results of this experiment revealed differences in the performance of the tools, with Burp Suite emerging as the best performer.  Conclusions: This research has provided a valid set of metrics for comparing and evaluating WAS testing tools, empowering organizations to make more informed decisions. Security professionals can optimise their WAS testing tool selection by understanding the key metrics and their relative significance, as established through the literature and interviews. Based on the experimental analysis, Burp Suite performed better than other tools. Therefore, for organizations initiating the selection process of the WAS testing tool, Burp Suite stands out as a good choice.

A risk is something that can happen and a problem is something that we know will happen or that has already happened. Security testing is used to evaluate a programs security using various methods and risk-based security testing is used to analyze, calculate and correct potential defects or problems in a system.Testing can be very costly and it is the most primary way of removing software defects. Many people focus their testing looking for correct behavior and not deviant behaviors in software, therefore security testing has not been as relevant in traditional testing. It is usually not possible to perform exhaustive testing on a system, instead you must selectively choose tests to conduct. How should the selection of tests be conducted? The study therefore intends to investigate how one can start working with risk-based security testing in development projects in order to prioritize and choose test cases and test methods. The study also aims to answer whether you can get any financial or practical benefits from working with risk-based security testing. To conduct the study a case study was used and to collect data, a document study was used to provide the opportunity to answer the questions. In order to analyze the collected data, a qualitative data analysis method has been used to explain and describe the content with a descriptive research approach. The results of the study provided an example of risk management with different steps one can take to start working on risk-based security testing in existing or new development projects. The study’s conclusion also shows that if you work with risk-based security testing there are practical benefits. For instance, higher quality of the system and economic benefits by finding defects or implementing countermeasures for possible risks at an early stage during the systems development. / En risk är någonting som kan hända och ett problem är någonting som vi vet kommer hända eller som redan har hänt. Säkerhetstest används för att med olika metoder värdera ett programs säkerhet och riskbaserad säkerhetstestning används för att analysera, kalkylera och åtgärda potentiella defekter eller problem i ett system. Testning kan vara väldigt kostnadskrävande och det är det mest primära sättet som används för att ta bort defekter i mjukvaror. Många fokuserar sin testning på att leta efter ett korrekt beteende och inte avvikande beteenden i programvara, därför har säkerhetstestning inte varit så aktuellt i traditionell testning. Det är oftast inte möjligt att utföra uttömmande testning på ett system utan man måste selektivt välja tester, men hur ska selektionen gå till?Studien ämnar därför att undersöka hur man kan börja arbeta med riskbaserad säkerhetstestning i utvecklingsprojekt för att kunna prioritera och välja testfall och testmetoder. Studien ämnar också svara på om man kan få några ekonomiska eller praktiska fördelar av att arbeta med riskbaserad säkerhetstestning.För att genomföra studien gjordes en fallstudie och för att samla in data utfördes en dokumentstudie för att ge möjlighet att besvara frågeställningarna. För att analysera den insamlade data har en kvalitativ dataanalysmetod använts för att med en deskriptiv undersökningsansats kunna djupare förklara och beskriva innehållet. Resultaten från studien har medfört ett riskhanterings exempel med olika steg man kan ta för att börja arbeta med riskbaserad säkerhetstestning i befintliga eller nya utvecklingsprojekt. Studiens slutsats visar också att om man arbetar med riskbaserad säkerhetstestning så finns det praktiska fördelar. Exempelvis högre kvalitet på systemet och ekonomiska fördelar genom att man i ett tidigt skede under systemets utveckling hittar defekter eller implementerar motåtgärder för eventuella risker.

When making new software systems, security testing should always be included in the process. In this thesis, attacks were identified and performed against a system consisting of two servers and an Android application. A penetration test was also performed against parts of the system. If an attack was successful, this was considered a vulnerability. The attacks that were identified and performed were a NoSQL injection attack a man-in-the-middle attack and reverse engineering. Through the man-in-the-middle attack and reverse engineering, breaching security properties such as confidentiality and integrity was possible. The NoSQL injection attack was not successful in breaching neither. No results from these could be used to exploit the system further. Countermeasures were taken to secure against the discovered vulnerabilities, and new instances of the attacks were performed after this as well. The overall conclusion is that the system is now secure against our implementations of the attacks performed in this thesis.

En ce qui concerne le développement des technologies informatique, les systèmes et les réseaux informatiques sont intensément utilisés dans la vie quotidienne. Ces systèmes sont responsables de nombreuses tâches essentielles pour notre communauté sociale (par exemple, système de traitement médical, E-Commerce, Système d'avion, système de vaisseau spatial, etc.). Quand ces systèmes cessent de fonctionner ou sont corrompus, les pertes économiques peuvent atteindre des sommes inacceptables. Pour éviter ces situations, les systèmes doivent être sécurisés avant leur installation. Alors que la plupart de ces systèmes sont mis en œuvre à partir de spécifications des protocoles, les problèmes de vérification de la sécurité de systèmes concrets renvient à vérifier la sécurité de l'implémentation de ces protocoles. Dans cette thèse, nous nous concentrons sur les méthodes de vérification de la sécurité des implémentations des protocoles et nous sommes intéressés à deux principaux types d'attaques sur les réseaux : Déni de service (DoS) et attaque de Protocol d’authentification. Nous étudions les caractéristiques de ces attaques et les méthodes de vérification formelles. Puis nous proposons modèle étendu de IOLTS et les algorithmes correspondants à la génération de les cas de test pour la vérification de sécurité automatique. Afin d'éviter les explosions d'état possibles, nous formalisons également les expériences de sécurité du testeur comme le « Objectif de Sécurité » pour contrôler la génération de test sur la volée. Parallèlement, une méthode d'analyse basée sur le modèle pour la Systèmes de Détection d'intrusion Anomalie (Anomaly IDS) est également proposée dans cette thèse, ce qui peut améliorer les capacités de détecter des anomalies de l'IDS. Ces méthodes de vérification proposées sont mises en évidence par l'étude de RADIUS protocole et un outil intégré de graphique est également proposé pour facilement les opérations de la génération de test. / Regarding the development of computer technologies, computer systems have been deeply used in our daily life. Those systems have become the foundation of our modern information society. Some of them even take responsibilities for many essential and sensitive tasks (e.g., Medical Treatment System, E-Commerce, Airplane System, Spaceship System, etc.). Once those systems are executed with problems, the loss on the economy may reach an unacceptable number. In order to avoid these disappointing situations, the security of the current systems needs to be verified before their installations. While, most of the systems are implemented from protocol specifications, the problems of verifying the security of concrete system can be transformed to verify the security of protocol implementation. In this thesis, we focus on the security verification methods of protocol implementations and we are interested with two main types of network attacks: Denis-of-Services (DoS) attacks and Protocol Authentication attacks. We investigate the features of these attacks and the existed formal verification methods and propose two extended models of IOLTS and the corresponding algorithms to generate the security verification test cases automatically. In order to avoid the possible state explosions, we also formalize the security experiences of the tester as Security Objective to control the test generation on-the-fly. Meanwhile, a modeled based Anomaly Intrusion Detection Systems (IDS) analysis method is also proposed in this thesis, which can enhance the detect abilities of Anomaly IDS. These proposed verification methods are demonstrated with the case study of RADIUS protocol and an integrated GUI tool is also proposed to simply the operations of test generation.

Agile Softwareentwicklung steht mit seinen Prinzipien für offene Kollaboration, leichtgewichtige Rahmenwerke und schnelle Anpassung an Änderungen. Mit diesen Charakteristika konnte sich Problemen und Unzufriedenheit in der traditionellen Software-Entwicklung gewidmet werden. Auf der Seite der IT-Sicherheit haben sich allerdings vielfältige Herausforderungen offenbart. Mit Static Application Security Testing (SAST) und Dynamic Application Security Testing (DAST) wurden erste Lösungsansätze dafür geliefert. Eine zufriedenstellende Möglichkeit zur Integration von Security-Testing in agile Softwareentwicklung, insbesondere im Cloud-Kontext, stellen diese allerdings nicht dar. Die vorliegende Arbeit soll unter folgender Fragestellung bearbeitet werden: Wie kann ein praktisches Konzept zur Betrachtung der Sicherheit von Anwendungs-Code, Container und Cluster innerhalb von agilen Entwicklungszyklen realisiert werden, wenn ein metadatenbasiertes Testverfahren verwendet werden soll? Das Ziel teilt sich damit in die Konzeption und Realisierung von zwei Aspekten: das metadatenbasierte Security-Testing von Code/Container/Cluster und den Entwicklungsablauf zur Anwendung des Testing-Verfahrens. Ein Fallbeispiel der Webentwicklung wurde zur qualitativen Evaluation eines Prototypen herangezogen, welcher mittels Python und GitLab umgesetzt wurde. Nach Erläuterung der Rahmenbedingungen, konnten konkrete Szenarien eines Entwicklungsprozesses durchlaufen werden. Die qualitative Untersuchung zeigte eine erfolgreiche Erkennung von Schwachstellen unterschiedlicher Kategorien (z.B. Broken Access Control). Insgesamt konnte eine gute Einbettung in den beispielhaften Entwicklungsablauf beobachtet werden. Der Aufwand für die Pflege der Metadaten ist nicht zu vernachlässigen, jedoch sollte dieser aufgrund der Orientierung am etablierten OpenAPI Schema nicht zu stark gewichtet werden. Dies gilt insbesondere dann, wenn durch den Einfluss von Metadaten Mehrwerte (Durchführbarkeit, Schnelligkeit, Komfortabilität) generiert werden können.:1 Einleitung 1.1 Problembeschreibung 1.2 Zielstellung 1.3 Stand der Technik und Entwicklungsmethoden 1.4 Methodik 2 Theoretische und Technische Grundlagen 2.1 Grundlagen der agilen Software-Entwicklung 2.2 GitLab 2.3 Grundlagen zum metadatengestützten Security-Testing 3 Konzeption 3.1 Low-Level Modell (Testablauf) 3.2 Synthese der beispielhaften Testfälle 3.3 Beschreibungsdatei 3.4 High-Level Modell (Entwicklungsablauf) 4 Implementation 4.1 Testablauf 4.2 CI/CD Pipeline 4.3 Fallbeispiel der agilen Softwareentwicklung 5 Auswertung und Ausblick

Garantir la sécurité d’une application tout au long de son cycle de vie est une tâche fastidieuse. Le choix, l’implémentation et l’évaluation des solutions de sécurité est difficile et sujette a des erreurs. Les compétences en sécurité ne sont pas répondues dans toutes les équipes de développement. Afin de réduire ce manque de compétences en sécurité, les développeurs ont a leurs disposition une multitude de documents décrivant des problèmes de sécurité et des solutions requises (i.e., vulnérabilités, attaques, principes de sécurité, patrons sécurité, etc.). Abstraites et informelles, ces documents sont fournis par des sources différentes et leur nombre est en constante croissance. Les développeurs sont noyés dans une multitude de documents ce qui fait obstruction à leur capacité à choisir, implémenter et évaluer la sécurité d’une application. Cette thèse aborde ces questions et propose un ensemble de méthodes pour aider les développeurs à choisir, implémenter et évaluer les solutions de sécurité face aux problèmes de sécurité. Ces problèmes sont matérialisés par les failles, les vulnérabilités, les attaques, etc. et les solutions fournies par des patrons de sécurité. Cette thèse introduit en premier une méthode pour aider les développeurs dans l’implémentation de patrons de sécurité et l’estimation de leur efficacité face aux vulnérabilités. Puis elle présente trois méthodes associant les patrons de sécurité, les vulnérabilités, les attaques, etc. au sein d’une base de connaissance. Cette dernière permet une extraction automatique de classifications de patrons et améliore la rapidité et la précision des développeurs dans le choix des patrons de sécurité face à une vulnérabilité ou une attaque. En utilisant la base de connaissance, nous présentons une méthode pour aider les développeurs dans la modélisation des menaces ainsi que la générations et l’exécution des cas de test de sécurité. La méthode est évaluée et les résultats montrent que la méthode améliore l’efficacité, la compréhensibilité et la précision des développeurs dans le choix des patrons de sécurité et d’écriture des cas de test de sécurité. / Ensuring the security of an application through its life cycle is a tedious task. The choice, the implementation and the evaluation of security solutions is difficult and error prone. Security skills are not common in development teams. To overcome the lack of security skills, developers and designers are provided with a plethora of documents about security problems and solutions (i.e, vulnerabilities, attacks, security principles, security patterns, etc.). Abstract and informal, these documents are provided by different sources, and their number is constantly growing. Developers are drown in a sea of documentation, which inhibits their capacity to design, implement, and the evaluate the overall application security. This thesis tackles these issues and presents a set of approaches to help designers in the choice, the implementation and the evaluation of security solutions required to overcome security problems. The problems are materialized by weaknesses, vulnerabilities, attacks, etc. and security solutions are given by security patterns.This thesis first introduces a method to guide designers implement security patterns and assess their effectiveness against vulnerabilities. Then, we present three methods associating security patterns, attacks, weaknesses, etc. in a knowledge base. This allows automated extraction of classifications and help designers quickly and accurately select security patterns required to cure a weakness or to overcome an attack. Based on this nowledge base, we detaila method to help designers in threat modeling and security test generation and execution. The method is evaluated and results show that the method enhances the comprehensibility and the accuracy of developers in the security solutions choice, threat modeling and in the writing of security test cases.

abstract: Access control is necessary for information assurance in many of today's applications such as banking and electronic health record. Access control breaches are critical security problems that can result from unintended and improper implementation of security policies. Security testing can help identify security vulnerabilities early and avoid unexpected expensive cost in handling breaches for security architects and security engineers. The process of security testing which involves creating tests that effectively examine vulnerabilities is a challenging task. Role-Based Access Control (RBAC) has been widely adopted to support fine-grained access control. However, in practice, due to its complexity including role management, role hierarchy with hundreds of roles, and their associated privileges and users, systematically testing RBAC systems is crucial to ensure the security in various domains ranging from cyber-infrastructure to mission-critical applications. In this thesis, we introduce i) a security testing technique for RBAC systems considering the principle of maximum privileges, the structure of the role hierarchy, and a new security test coverage criterion; ii) a MTBDD (Multi-Terminal Binary Decision Diagram) based representation of RBAC security policy including RHMTBDD (Role Hierarchy MTBDD) to efficiently generate effective positive and negative security test cases; and iii) a security testing framework which takes an XACML-based RBAC security policy as an input, parses it into a RHMTBDD representation and then generates positive and negative test cases. We also demonstrate the efficacy of our approach through case studies. / Dissertation/Thesis / M.S. Computer Science 2014

How secure is your application? How can you evaluate if it is secure? The threats are many and may be hard to find. In a world where things are more and more automated; how does manual labour contribute to security auditing applications? This study aims to assess two proof of concept Android client-server applications, developed by students to suit the needs of a fictitious Police Department and Fire Department, respectively. The approach is unconventional yet supported by well-established theory. The gist of a vulnerability assessment methodology initially developed to assess the security of middleware is followed and applied to the entire architecture of these client-server applications. How the manual labour contributed to the end results, in comparison to the use of automated tools and a list of known threats, is then evaluated.   It is concluded that the applications encompass multiple of the Open Web Application Security Project (OWASP) Top 10 Mobile Risks and that automated tools find most of those vulnerabilities. However, relying on automation may lead to a false sense of security, which in effect may cause developers to lose understanding of why vulnerabilities occur and how they should be mitigated. Understanding how the design and architecture of the application influence its security is key.   As of Android 9.0+, default is that applications use SSL encrypted communication. Only 40% of Android users are in 2020 affected by this change according to Android studio developer information, leaving a majority of users unaware of if or how their data is being protected, also observed in analysis results from this thesis work. One should consider if or how to inform users of how their data is being handled, not only in newer Android versions or regarding SSL communication.    This work also shows that developers' decisions may be greatly affected by time pressed situations, which is reflected upon in the last chapter. Another important finding was that the third-party software Sinch, which enabled the use of voice and video communication in one of the applications, sent IP addresses and usernames of the users in clear text during the binding request, when the Session Traversal Utilities for NAT (STUN) protocol was used.

In recent years, companies and organizations have increasingly integrated software security testing into the software development life cycle using DevOps practices. The current integration approach introduces multiple challenges in an information technology environment that consists of a large number of software development projects and multiple software security testing tools. This thesis aims to address these challenges by proposing a microservice-based framework to unify application security testing. The thesis first identifies the challenges, then proposes a design for a framework based on relevant literature and common characteristics of application security testing tools. The main components of the proposed framework are implemented and evaluated. The evaluation result shows that the framework offers many benefits: more secure credential management process, reduced execution time for Continuous Integration (CI) pipelines, and more efficient project onboarding and management. Furthermore, the integration of the proposed framework does not introduce major security threats to the current environment. / Under de senaste åren har företag och organisationer i allt högre grad integrerat testning av programvarusäkerhet i livscykeln för programvaruutveckling med hjälp av DevOps-metoder. Den nuvarande integrationsmetoden medför flera utmaningar i en informationsteknisk miljö som består av ett stort antal programvaruutvecklingsprojekt och flera verktyg för testning av programvarusäkerhet. Detta examensarbete syftar till att ta itu med dessa utmaningar genom att föreslå en mikrotjänstbaserat ramverk för enhetlig testning av programsäkerhet. I arbetet identifieras först utmaningarna och därefter föreslås en konstruktion baserad på relevant litteratur och gemensamma egenskaper hos verktyg för testning av applikationssäkerhet. De viktigaste komponenterna i det föreslagna ramverket implementeras och utvärderas. Utvärderingsresultatet visar att ramverket erbjuder många fördelar: säkrare process för hantering av autentiseringsuppgifter, kortare genomförandetid för Continuous Integration (CI)-pipelines och effektivare projektstart och -hantering. Dessutom medför integrationen av det föreslagna ramverket inga större säkerhetshot i den nuvarande miljön.

It is well-established that machine learning techniques have been used with great success in other domains and has been leveraged to deal with sources of evolving abuse, such as spam. This study aims to determine whether machine learning techniques can be used to create a model that detects vulnerability scanning attacks using proprietary real-world data collected from tCell, a web application firewall. In this context, a vulnerability scanning attack is defined as an automated process that detects and classifies security weaknesses and flaws in the web application. To test the hypothesis that machine learning techniques can be used to create a detection model, twenty four models were trained. The models showed a high level of precision and recall, ranging from 91% to 0.96% and 85% to 0.93%, respectively. Although the classification performance was strong, the models were not calibrated sufficiently which resulted in an underconfidence in the predictions. The results can therefore been viewed as a performance baseline. Nevertheless, the results demonstrate an advancement over the simplistic threshold-based techniques developed in the early days of the internet, but require further research and development to tune and calibrate the models. / Det är väletablerat att tekniker för maskininlärning har använts med stor framgång inom andra domäner och har utnyttjats för att hantera källor till växande missbruk, såsom spam. Denna studie syftar till att avgöra om maskininlärningstekniker kan tillämpas för att skapa en modell som upptäcker sårbarhets-skanningsattacker med hjälp av proprietär data som samlats in från tCell, en webbapplikationsbrandvägg. I detta sammanhang definieras en sårbarhetsskanningsattack som en automatiserad process som upptäcker och klassificerar säkerhetsbrister och brister i webb-applikationen. För att testa hypotesen att maskininlärningstekniker kan användas för att skapa en detektionsmodell, tränades tjugofyra modeller. Modellerna visade en hög nivå av precision och sensitivitet, från 91% till 0,96% och 85% till 0,93%, respektive. Även om klassificeringsprestandan var god, var modellerna inte tillräckligt kalibrerade, vilket resulterade i ett svagt förtoende för förutsägelserna. De presenterade resultaten kan därför ses som en prestationsbaslinje. Resultaten visar ett framsteg över de förenklade tröskelbaserade teknikerna som utvecklades i begynnelsen av internet, men kräver ytterligare forskning och utveckling för att kalibrera modellerna.