The Signal protocol provides end-to-end encryption for billions of users in popular instant messaging applications like WhatsApp, Facebook Messenger, and Google Allo. The protocol relies on an app-specific central server to distribute public keys and relay encrypted messages between the users. Signal prevents passive attacks. However, it is vulnerable to some active attacks due to its reliance on a trusted key server. A malicious key server can distribute fake keys to users to perform man-in-the-middle or impersonation attacks. Signal applications support an authentication ceremony to detect these active attacks. However, this places an undue burden on the users to manually verify each other's public key. Recent studies reveal that the authentication ceremony is time-consuming and confusing, and almost nobody adopts it. Our goal is to explore various approaches for automatically detecting or preventing fake key attacks. We modified a local copy of the Signal server to demonstrate that active attacks are feasible. We then designed three defenses that automatically detect or prevent the attacks. We completed a threat analysis of the defenses and implemented some proof-of-concept prototypes for two of them. We analyze their strengths and weaknesses and outline avenues for future work.

End-to-end encryption is becoming a standard feature in popular mobile chat appli-cations (apps) with millions of users. In the two years a number of leading chat apps have added end-end encryption features including LINE, KakaoTalk, Viber, Facebook Messenger, and WhatsApp.However, most of these apps are closed-source and there is little to no independent ver-ification of their end-to-end encryption system design. These implementations may be a major concern as proprietary chat apps may make use of non-standard cryptographic algorithms that may not follow cryptography and security best practices. In addition, governments authorities may force chat app providers to add easily decryptable export-grade cryptography to their products. Further, mainstream apps have a large attack surface as they offer a variety of features. As a result, there may be software vulnera-bilities that could be exploited by an attacker in order to compromise user’s end-to-end privacy. Another problem is that, despite being closed-source software, providers often market their apps as being so secure that even the provider is not able to decrypt messages. These marketing claims may be potentially misleading as most users do not have the technical knowledge to verify them.In this Master’s thesis we use KakaoTalk – the most popular chat app in South Korea – as a case study to perform a security and privacy assessment and audit of its “Secure Chat” opt-in end-to-end encryption feature. Also, we examine KakaoTalk’s Terms of Service policies to verify claims such as “[. . . ] Kakao’s server is unable to decrypt the encryption [. . . ]” from a technical perspective.The main goal of this work is to show how various issues in a product can add up to the potential for serious attack vectors against end-to-end privacy despite there being multiple layers of security. In particular, we show how a central public-key directory server makes the end-to-end encryption system vulnerable to well-known operator-site man-in-the-middle attacks. While this naive attack may seem obvious, we argue that (KakaoTalk) users should know about the strength and weaknesses of a particular design in order to make an informed decision whether to trust the security of a chat app or not. / End-to-end kryptering är en allt mer vanligt förekommande funktionalitet bland populära mobila chatttjänster (händanefter appar) med miljontals användare. Under de två senaste åren har många ledande chattappar, bland annat LINE, KakaoTalk, Viber, Facebook Messenger, och WhatsApp, börjat använda end-to-end kryptering. Dock så är de flesta av dessa appar closed-source och det finns begränsad, eller ingen, fristående granskning av systemdesignen för deras end-to-end kryptering. Dessa implementationer kan innebära en stor risk då proprietära chattappar kan använda sig av kryptografiska algoritmer som inte följer best practice för säkerhet eller kryptografi. Vidare så kan statliga myndigheter tvinga de som tillhandahåller chattappar att använda lättdekrypterad export-grade kryptografi för sina produkter. Lägg till det att de flesta vanliga appar har många ytor som kan attackeras, till följd av all funktionalitet de erbjuder. Som ett resultat av detta finns en risk för mjukvarubrister som kan utnyttjas av en hackare för att inkräkta på en användares end-to-end integritet. Ytterligare ett problem är att trots att det är closed-source mjukvara så marknadsför ofta appleverantörerna sina appar som att vara är så säkra att inte ens leverantörerna själva kan dekryptera användarnas meddelanden. Det som hävdas i marknadsföringen riskerar vara missledande eftersom de flesta användarna inte har den tekniska kunskap som krävs för att kunna verifiera att det som hävdas är sant. I den här Master-uppsatsen använder vi KakaoTalk – den mest populära chattappen i Sydkorea – som en fallstudie för att granska och bedömma säkerhetens- och integritets-aspekterna hos deras valbara “Secure Chat” med end-to-end krypteringsfunktionalitet. Vi granskar även KakaoTalk’s användarvillkor för att kunna verifiera påståenden som att “[. . . ] Kakao’s server is unable to decrypt the encryption [. . . ]” från ett tekniskt perspektiv. Det huvudsakliga syftet med denna studien är att belysa hur olika brister i en produkt sammantagna kan skapa en risk för allvarliga vektorattacker mot end-to-end integriteten även fast det finns flera skyddslager. Mer specifikt visar vi hur en central katalogserver för public-keys gör end-to-end krypteringssystemet sårbart mot välkända operator-site man-in-the-middle-attacker. Trots att denna naiva typ av attack kan verka uppenbar, argumenterar vi för att (KakaoTalk) användare borde veta om styrkorna och svagheterna med en särskild systemdesign för att kunna göra ett informerat val för om de ska lita på säkerheten hos en chattapplikation eller inte.

In the smartphone era, instant messaging is fully embedded in our daily life. Messaging protocols must preserve the confidentiality and authenticity of sent messages both in two-party conversations and in group chats, in which the list of group members may suffer modifications over time. Hence, a precise characterization of their security is required. In this thesis, we analyze the cryptographic properties that are desirable in secure messaging protocols, particularly in asynchronous group key agreement protocols. Our main contribution is a study of the administration of a messaging group, which is a common scenario in which a subset of the group members (the administrators) are the only users allowed to modify the group structure by adding and removing group members. As we discuss, enabling secure group administration mechanisms can enhance the security of messaging protocols. For this purpose, we introduce a new primitive which extends the continuous group key agreement (CGKA) primitive to capture secure administration, which we denote by administrated CGKA (A-CGKA). The definition is followed by a correctness notion and an informal security description. We present two constructions of our A-CGKA that can be built on top of any CGKA: individual admin signatures (IAS), and dynamic group signature (DGS), both constructed using signature schemes. Furthermore, we provide a detailed overview of secure group messaging in which we discuss group evolution, efficiency, concurrency, and different adversarial models. We introduce a novel CGKA correctness definition (in the so-called propose-and-commit paradigm), followed by a security game that incorporates the correctness properties. We also survey some variants of the TreeKEM protocol and compare their security. / I de smarta telefonernas tid är direktmeddelanden en självklar del av vår vardag. Meddelandeprotokoll måste upprätthålla konfidentialitet och autenticitet för skickade meddelanden både i tvåpartskonversationer samt i gruppchatter vars medlemslistor kan förändras över tid. Därför krävs en precis karaktärisering av deras säkerhet. I detta arbete analyserar vi de kryptografiska egenskaper som är önskvärda i meddelandeprotokoll med fokus på asynkrona gruppnyckelavtalsprotokoll (group key agreement protocols). Arbetets huvudsakliga bidrag till området är en studie av administrationen av en meddelandegrupp. Detta är ett vanligt förekommande scenario där endast en delmängd av gruppmedlemmarna (administratörerna) tillåts modifiera gruppens struktur genom att lägga till och ta bort medlemmar. Som diskuteras i arbetet kan användandet av säkra gruppadministrationsmekanismer (group administration mechanisms) förbättra säkerheten för meddelandeprotokoll. I detta syfte introducerar vi en ny kryptografisk primitiv vilken uttökar den s.k. “continuous group key agreement”-primitiven (CGKA) till att även innefatta säker administration. Denna primitiv kallar vi administrated CGKA (A-CGKA), vars definition följs av en korrekthetsdefinition och en informell säkerhetsbeskrivning. Vi presenterar två konstruktioner av A-CGKA som kan byggas ovanpå vilken CGKA som helst: individual admin signatures (IAS) och dynamic group signature (DGS), som båda konstrueras via signaturscheman. Utöver detta ger vi även en detaljerad överblick över säkra gruppmeddelanden i vilken vi diskuterar gruppevolution, effektivitet, samtidighet och olika fientliga modeller. Vi introducerar en ny definition av korrekthet för CGKA (vilket följer paradigmen propose-and-commit) följt av ett s.k. “security game” som inkorporerar korrekthetsegenskaperna. Vi undersöker även varianter av TreeKEM-protokollet och jämför deras säkerhet.

The subject of secure messaging has gained notable attention lately in the cryptographic community. For communications between two parties, paradigms such as the double ratchet, used in the Signal protocol, provide provably strong security guarantees such as forward secrecy and post-compromise security. Variations of the Signal protocol have enjoyed widespread adoption and are embedded in several well known messaging services, including Signal, WhatsApp and Facebook Secret Conversations. However, providing equally strong guarantees that scale well in group settings remains somewhat less well studied and is often neglected in practice. This motivated the need for the IETF Messaging Layer Security (MLS) working group. The first continuous group key agreement (CGKA) protocol to be proposed was Asynchronous Ratcheting Trees (ART) [Cohn-Gordon et al., 2018] and formed the basis of TreeKEM [Barnes et al., 2019], the CGKA protocol currently suggested for MLS. In this thesis we propose a new asynchronous group key agreement protocol based on a one-round Tripartite Diffie-Hellman [Joux, 2000]. Furthermore, we show that our protocol can be generalised for an n-ary asynchronous ratchet tree, assuming the existence of a one-round (n + 1)-way Diffie-Hellman key exchange, based on a n-multilinear map [Boneh and Silverberg, 2003]. We analyse ART, TreeKEM, and our proposals from a complexity theoretic perspective and show that our proposals improve the cost of update operations. Finally we present some discussion and improvements to the IETF MLS standard. / Ämnet om säkra meddelanden har på senare tid skapat uppmärksamhet inom kryptografiska samfundet. För kommunikationer mellan två parter ger paradigmer såsom Double Ratchet, som används i Signal-protokollet, starka bevisbara säkerhetsgarantier som forward secrecy och post-compromise security. Variationer av Signal-protokollet används mycket i praktiken och är inbäddade i flera välkända meddelandetjänster såsom Signal, WhatsApp och Facebook Secret Conversations. Däremot är protokoll som erbjuder lika starka garantier och som skalar väl i gruppsituationer något mindre studerade och ofta eftersatta i praktiken. Detta motiverade behovet av arbetsgruppen IETF Messaging Layer Security (MLS). Det första kontinuerliga gruppnyckelprotokollet (CGKA) som föreslogs var Asynchronous Ratcheting Trees (ART) [Cohn-Gordon et al., 2018] och lade grunden för TreeKEM [Barnes et al., 2019], det CGKA-protokoll som för närvarande föreslagits för MLS. I detta examensarbete föreslår vi ett nytt asynkront gruppnyckelprotokoll baserat på en en-rundad Tripartite Diffie{Hellman [Joux, 2000]. Vidare visar vi att vårt protokoll kan generaliseras för n-ary träd med hjälp av ett en-rundat (n + 1)-väg Diffie-Hellman nyckelutbyte, baserat på en multilinjär mappning [Boneh and Silverberg, 2003]. Vi analyserar ART, TreeKEM och våra förslag ur ett teoretiskt perspektiv samt visar att våra förslag förbättrar kostnaden för uppdateringsoperationer. Slutligen presenterar vi några diskussioner och förbättringar av IETF MLS-standarden.

The threat of government and corporate surveillance around the world, as well as the publicity surrounding major cybersecurity attacks, have increased interest in secure and private end-to-end communications. In response to this demand, numerous secure messaging applications have been developed in recent years. These applications have been welcomed and publically used not just by political activists and journalists but by everyday users as well. Most of these popular secure messaging applications are usable because they hide many of the details of how encryption is provided. The strength of the security properties of these applications relies on the authentication ceremony, wherein users validate the keys being used for encryption that is exchanged through the service providers. The validation process typically involves verifying the fingerprints of encryption keys to protect the communication from being intercepted.In this dissertation, we explore how to help users enhance the privacy of their communica- tions, with a particular focus on secure messaging applications. First, we explore whether secure messaging applications are meeting the security and privacy needs of their users, especially in countries that practice censorship and restrict civil liberties, including blocking access to social media and communication applications. Second, we studied existing popular secure messaging applications to explore how users interact with these applications and how well they are using the authentication ceremony during lab studies. Third, we applied design principles to improve the interfaces for the authentication ceremony, and also to help users find and perform the authentication ceremony faster. Forth, we applied the lessons from our interviews with participants in our user studies to help users comprehend the importance of authentication. As part of the effort, we developed an authentication ceremony using social media accounts to map key fingerprints to social features, pushing the ceremony to a more natural domain for users. We modified the Signal secure messaging application to include this social authentication ceremony and used a user study to compare this method to other common methods. We found that social authentication has some promising features, but that social media companies are too distrusted by users. Based on our results, we make several recommendations to improve the use of security and privacy features in secure messaging applications and outline areas for future work.

In the contemporary digital landscape, ensuring secure communication amid widespread data exchange is imperative. This dissertation focuses on enhancing the security and privacy of end-to-end encryption (E2EE) applications while maintaining or improving usability. The dissertation first investigates and proposes improvements in two areas of existing E2EE applications: countering man-in-the-middle and impersonation attacks through automated key verification and studying user perceptions of cryptographic deniability. Insights from privacy-conscious users reveal concerns about the lack of E2EE support, app siloing, and data accessibility by client apps. To address these issues, we propose an innovative user-controlled encryption system, enabling encryption before data reaches the client app. Finally, the dissertation evaluates local threats in the FIDO2 protocol and devises defenses against these risks. Additionally, it explores streamlining FIDO2 authentication management across multiple websites for user convenience and security.