As a system grows in size, it is common that it is transformed into a microservice architecture. In order to be able monitor this new architecture there is a need to collect information from the microservices. The software company IDA Infront is transitioning their product iipax to a microservice architecture and is faced with this problem. In order to solve this, they propose the use of a Message-oriented Middleware (MOM). There exists many different MOMs that are suitable to execute this task. The aim of this thesis is to determine, in terms of latency, throughput and scalability, which MOM is best suitable for this. Out of four suitable MOMs Apache Kafka and RabbitMQ are chosen for further testing and benchmarking. The tests display that RabbitMQ is able to send single infrequent messages (latency) faster than Kafka. But it is also shown that Kafka is faster at sending a lot of messages rapidly and with an increased number of producers sending messages (throughput and scalability). However, the scalability test suggests that RabbitMQ possibly scales better with a larger amount of microservices, thus more testing is needed to get a definite conclusion.

In this thesis, the trade-offs between latency and transmitted data volume in vehicle-to-cloud communication for different choices of application layer messaging protocols and binary serialization formats are studied. This is done with the purpose of getting enough performance improvement to enable delay-sensitive Intelligent Transport System (ITS) features, and to reduce data usage in mobile networks. The studied protocols are Constrained Application Protocol (CoAP), Advanced Message Queuing Protocol (AMQP) and Message Queuing Telemetry Transport (MQTT), and the serialization formats studied are Protobuf and Flatbuffers.  The results show that CoAP — the only User Datagram Protocol (UDP) based protocol — has the lowest latency and overhead while not being able to guarantee reliable transfer. The best performer that can guarantee reliable transfer is MQTT. For the serialization formats, Protobuf is shown to have three times smaller serialized message size than Flatbuffers and also faster serialization speed. Flatbuffers is the winner in the case of memory use and deserialization time, which could make up for the poor performance in other aspects of data processing in the cloud. Further, the implications of these results in ITS communication are discussed suggestions made into future research topics.

Att flera datorer används för att gemensamt lösa problem är inte någonting nytt. Det finns många distribuerade system i bruk och många olika lösningar för hur dessa ska kommunicera med varandra. Vissa använder sig av meddelande-orienterade lager för kommunikation vilket det finns väldigt många implementationer av. RabbitMQ är ett exempel där att kommunikation går genom en (eller ett kluster av) central nod och kommunicerar med hjälp av protokollet Advanced Message Queue Protocol, AMQP. I en helt annan kategori finns ZeroMQ som inte definierar någon central nod för all kommunikation att passera utan peer to peer är istället möjlig vilket innebär snabbare responstider men försvårar hur olika klienter hittar till varandra. Det bloggindexerande företaget Twingly kör idag med ett distribuerat system som använder flera olika kösystem för att koordinera ut arbete till de olika datorerna. De vill kolla närmare på hur de kan bygga sitt system med hjälp av meddelande-orienterade lager. Resultatet av arbetet är att RabbitMQ innebär mindre komponenter att hålla reda på vilket innebär att koden blir mindre komplex. Det som kommer gratis med att använda RabbitMQ är just att klienterna inte behöver känna till varandra utan endast behöver känna till RabbitMQ-servern. Nackdelen är att RabbitMQ-servern kommer bli en flaskhals för systemet. ZeroMQ är däremot friare att implementera den funktionalitet man själv behöver vilket är till fördel i de fall tid och pengar finns för att skapa ett eget system byggt ovanpå ZeroMQ. För Twingly som vill ha ett system inom en snar framtid är RabbitMQ ett bättre val av dessa två alternativ.

Today’s applications are required to continuously adapt and adjust, to be able to meet a constant change in demand. As result of an increasing amount of data, choosing the right communication method becomes a vital step. A solution that have been functional for a long time, may at any point in time be unable to reach the level it requires and instead turns into bottlenecks and inefficient solutions.Using a database as a communication method between system entities, does not have to be a bad solution. A database has it perks with being a simple solution and efficient query operations. However, using it as a queue management system, requires entities to continuously poll new table entries. This solution may not be the most suitable nor best available option. There exists communication system developed for the specific purpose of efficiently distributing messages to available parties.Implementing a message-oriented middleware enables for asynchronous communication which promotes applications to be more loosely coupled. As a result, available resources could be better utilised and improve the system performance. This degree project investigates the development and integration of two message-oriented middlewares, RabbitMQ and AcviteMQ, within an e-commerce system. The purpose is to explore the potentials of changing queue management system from a database to a message broker. The expected outcome is a more flexible job handling and, perhaps, an improvement of job processing by using a more efficient distribution.The results show that changing queue management system from the database to a message-oriented middleware could improve the performance of handling of invoice jobs. Testing the application servers of the Proceedo system, with a batch of invoice jobs, showed a potential of up to 17 percent faster process time using a message broker. This corresponds to a reduced process time of around 11 minutes for one application server and 6 minutes using two. Additionally, both brokers provide flexible message handling through functionality to priorities messages. / Dagens applikationer måste kontinuerligt anpassa och justera sig för att kunna möta en ständig förändring i efterfrågan. Resultat som blir av den ökande mängd data som behöver kunna hanteras, är kravet på att välja rätt kommunikationsmetod. En lösning som varit funktionell under lång tid, kan när som helst bli oförmögen att nå den nivå som krävs. Istället förvandlas den till en flaskhals och på så sätt bli en ineffektiv lösning.Att använda en databas som en kommunikationsmetod mellan systemenheter behöver inte vara en dålig lösning. En databas har förmåner som att att vara en enkel lösning och effektivt kunna hantera förfrågningar. När det appliceras som ett köhanteringssystem, krävs det att alla enheter kontinuerligt skickar nya förfrågningar för att hämta nya tabelluppdateringar. Denna lösning kanske inte är det mest lämpliga eller bästa tillgängliga alternativet. Det finns kommunikationssystem utvecklade för det här specifika syftet, att effektivt distribuera meddelanden till tillgängliga parter.Införandet av ett meddelandeorienterad middlewares gör det möjligt för asynkron kommunikation som främjar applikationer till att kunna vara mer löst kopplade. Som ett resultat kan tillgängliga resurser utnyttjas bättre och förbättra systemets prestanda. Detta examensprojekt undersöker utvecklingen och integrationen av två meddelandeorienterade middlewares, RabbitMQ och AcviteMQ, inom ett e-handelssystem. Syftet är att undersöka de positiva möjligheterna som finns av att by-ta köhanteringssystem från en databas till en meddelandeorienterad middleware. Det förväntade resultatet är en mer flexibel jobbhantering och kanske en förbättring av jobbearbetningen, genom att använda en effektivare meddelande distribution.Resultaten visar att bytet av köhanteringssystem, från databasen till en meddelandeorienterad middleware, kan förbättra hanteringen av fakturahandlingar. Testningen av Proceedo-systemets applikationsservrar visade potential på upp till 17 procent snabbare processtid med hjälp av en meddelande broker. Det motsvarar en hanteringstid på 11 minuter snabbare vid användande av en applikationserver och 6 minuter vid använding av två. Dessutom ger båda middlewares en mer flexibel meddelandehantering, i form av, funktionalitet att kunna prioritera meddelanden

L émergence des environnements omniprésents pose un nouveau défi aux systèmes informatiques. Les intergiciels asynchrones (Message-Oriented Middleware, MOM) sont reconnus comme étant la solution la plus apte à répondre aux besoins de passage à grande échelle, de flexibilité, et d' hétérogénéité des nouvelles applications distribuées. Mais l'implémentation des intergiciels asynchrones actuels reste souvent figée quels que soient les sites d'exécution et l'application (les applications) l'utilisant et sont peu voire pas configurables. Cette thèse s'intéresse à la configuration et au déploiement des intergiciels asynchrones sur système hétérogène à grande échelle. Elle vise la définition d'un modèle d'intergiciel asynchrone configurable permettant une configuration statique et à l'exécution. Elle a pour objectif d'associer les nombreux travaux dans le domaine de l'asynchrone qui ont mené à la définition des modèles de communication asynchrones et les réflexions menées dans les intergiciels synchrones autour des nouveaux besoins de configuration et d'adaptabilité. La synthèse de tous nos travaux nous a mené à la création de DREAM (Dynamic REflective Asynchronous Middleware), un intergiciel asynchrone adaptable. Les mécanismes de contrôles fournis par Dream ainsi que son architecture permettent de réaliser une configuration en se basant sur les besoins applicatifs et les contraintes imposées par le système. Nous validons notre prototype par l'implémentation d'un service à évènement à base d'agents dans lequel nous utilisons les capacités d'adaptation de DREAM pour ajouter de nouvelles fonctionnalités.

O aumento crescente no número de processadores das arquiteturas paralelas que estão no topo dos rankings de desempenho, apesar de permitir uma maior capacidade de processamento, também traz consigo um aumento na taxa de falhas diretamente proporcional ao número de processadores. Atualmente, as técnicas de tolerância a falhas com recuperação retroativa são as mais empregadas em aplicações MPI, principalmente a técnica de checkpoint coordenado. No entanto, previsões afirmam que essa última técnica será inadequada para as arquiteturas emergentes. Em contrapartida, as técnicas de log de mensagens possuem características que as tornam mais apropriadas no novo cenário que se estabelece. O presente trabalho consiste em uma proposta de log de mensagens pessimista remoto com checkpoint não-coordenado e a avaliação de desempenho da comunicação MPI sobre Publish/Subscriber no qual se baseia o log de mensagens. O trabalho compreende: um estudo das técnicas de tolerância a falhas mais empregadas em ambientes de alto desempenho e a motivação para a escolha dessa variante de log de mensagens; a proposta de log de mensagens; uma implementação de comunicação Open MPI sobre OpenAMQ e sua respectiva avaliação de desempenho com comunicação tradicional TCP/IP e com o log de mensagens pessimista local da distribuição do Open MPI. Os benchmarks utilizados foram o NetPIPE, o NAS Parallel Benchmarks e a aplicação Virginia Hydrodynamics (VH-1). / The growing number of processors in parallel architectures at the top of performance rankings allows a higher processing capacity. However, it also brings an increase in the fault rate which is directly proportional to the number of processors. Nowadays, coordinated checkpoint is the most widely used rollback technique for system recovery in the occurrence of faults in MPI applications. Nevertheless, projections point that this technique will be inappropriate for the emerging architectures. On the other hand, message logging seems to be more appropriate to this new scenario. This work consists in a proposal of pessimistic message logging (remote based) with non-coordinated checkpoint and the performance evaluation of an MPI communication mechanism that works over Publish/Subscriber channels in which the proposed message logging is based. The work is organized as following: an study of fault tolerant techniques used in HPC and the motivation for choosing this variant of message logging; a message logging proposal; an implementation of Open MPI communication over OpenAMQ; performance evaluation and comparision with the tradicional TCP/IP communication and a pessimistic message logging (sender based) from Open MPI distribution. The benchmark set is composed of NetPIPE, NAS Parallel Benchmarks and Virginia Hydrodynamics (VH-1).

O aumento crescente no número de processadores das arquiteturas paralelas que estão no topo dos rankings de desempenho, apesar de permitir uma maior capacidade de processamento, também traz consigo um aumento na taxa de falhas diretamente proporcional ao número de processadores. Atualmente, as técnicas de tolerância a falhas com recuperação retroativa são as mais empregadas em aplicações MPI, principalmente a técnica de checkpoint coordenado. No entanto, previsões afirmam que essa última técnica será inadequada para as arquiteturas emergentes. Em contrapartida, as técnicas de log de mensagens possuem características que as tornam mais apropriadas no novo cenário que se estabelece. O presente trabalho consiste em uma proposta de log de mensagens pessimista remoto com checkpoint não-coordenado e a avaliação de desempenho da comunicação MPI sobre Publish/Subscriber no qual se baseia o log de mensagens. O trabalho compreende: um estudo das técnicas de tolerância a falhas mais empregadas em ambientes de alto desempenho e a motivação para a escolha dessa variante de log de mensagens; a proposta de log de mensagens; uma implementação de comunicação Open MPI sobre OpenAMQ e sua respectiva avaliação de desempenho com comunicação tradicional TCP/IP e com o log de mensagens pessimista local da distribuição do Open MPI. Os benchmarks utilizados foram o NetPIPE, o NAS Parallel Benchmarks e a aplicação Virginia Hydrodynamics (VH-1). / The growing number of processors in parallel architectures at the top of performance rankings allows a higher processing capacity. However, it also brings an increase in the fault rate which is directly proportional to the number of processors. Nowadays, coordinated checkpoint is the most widely used rollback technique for system recovery in the occurrence of faults in MPI applications. Nevertheless, projections point that this technique will be inappropriate for the emerging architectures. On the other hand, message logging seems to be more appropriate to this new scenario. This work consists in a proposal of pessimistic message logging (remote based) with non-coordinated checkpoint and the performance evaluation of an MPI communication mechanism that works over Publish/Subscriber channels in which the proposed message logging is based. The work is organized as following: an study of fault tolerant techniques used in HPC and the motivation for choosing this variant of message logging; a message logging proposal; an implementation of Open MPI communication over OpenAMQ; performance evaluation and comparision with the tradicional TCP/IP communication and a pessimistic message logging (sender based) from Open MPI distribution. The benchmark set is composed of NetPIPE, NAS Parallel Benchmarks and Virginia Hydrodynamics (VH-1).

Made available in DSpace on 2015-05-14T12:36:56Z (GMT). No. of bitstreams: 1 arquivototal.pdf: 2305795 bytes, checksum: 0e634264f2c488a816479aec2da0cc18 (MD5) Previous issue date: 2010-08-16 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / Advances in wireless communication technologies and miniatuziation of mobile devices are leading to great opportunities in the development of applications to explore this new computing frontier. However, the development of applications for such scenarios introduces new challenges, as mobile applications run on resouce-scarce devices and communicate with each other by means of wireless networks, which are characterized by intermittent connections. In order to avoid having to deal with such issues in an ad hoc fashion for every application, middleware platforms are adopted, concealing difficulties raised by mobility from application engineers as much as possible. Due to the asynchronous and loosely coupled communication style, Message-oriented Middleware (MOM) platforms have been commonly adopted for supporting the development of networked mobile applications. However, one of the most significant limitations of current MOM for mobile platforms is that they typically support a single, predefined communication paradigm (e.g., publish/subscribe). Such a restriction limits the scope of applications supported by the middleware. In order to mitigate such a limitation, this paper presents a middleware for mobile devices capable of supporting an extensible set of message-oriented communication paradigms (e.g. tuple spaces, message queue, publish/subscribe). Supported by an integrated architecture, which has been conceived based on a Software Product Line (SPL) approach, the middleware encapsulates common features that deal with mobility issues and provides them as shared, reusable components. Evaluation results show that the overhead introduced by such a multi-paradigm approach is minimal, both in memory footprint and runtime performance. Additionally, an application scenario illustrates how mobile applications may benefit from such an approach. / Os avanços nas tecnologias de comunicação sem fio e a miniaturização dos dispositivos móveis de alta capacidade estão trazendo grandes oportunidades para o desenvolvimento de aplicações que explorem essas novas fronteiras da computação. Entretanto, o desenvolvimento de aplicações nestes cenários traz novos desafios, pois estas operam em dispositivos de recursos limitados e comunicam-se através de redes sem fio, caracterizadas por conexões intermitentes. Para evitar que esses desafios tenham que ser resolvidos de forma improvisada para cada aplicação, plataformas de middleware são utilizadas, liberando os desenvolvedores de aplicações das dificuldades trazidas pela mobilidade. Devido ao seu estilo de comunicação assíncrono e fracamente acoplado, plataformas de Middleware orientado a Mensagens (MOM) têm sido comumente adotadas nestes casos. Entretanto, uma das limitações mais significantes das atuais plataformas de MOM é que elas geralmente dão suporte a um paradigma de comunicação único e predefinido (ex.: publish/subscribe). Essa restrição limita o escopo de aplicações que o middleware pode dar suporte. Para lidar com essa limitação, esta dissertação propõe um middleware para dispositivos móveis capaz de dar suporte a um conjunto extensível de paradigmas de comunicação baseados em mensagens (ex.: espaço de tuplas, filas de mensagens, publish/subscribe). Através de uma arquitetura integrada, a qual foi concebida baseada numa abordagem de Linha de Produto de Software (LPS), o middleware encapsula funcionalidades comuns para lidar com os desafios trazidos pela mobilidade, provendo componentes reusáveis e compartilhados entre os diversos paradigmas. Os resultados da avaliação mostram que o overhead introduzido pela abordagem multi-paradigma é mínimo, tanto em termos de espaço ocupado em memória, quanto em desempenho em tempo de execução. Por fim, com o intuito de ilustrar como aplicações móveis podem se beneficiar dessa abordagem, um cenário de aplicação é descrito.

O aumento crescente no número de processadores das arquiteturas paralelas que estão no topo dos rankings de desempenho, apesar de permitir uma maior capacidade de processamento, também traz consigo um aumento na taxa de falhas diretamente proporcional ao número de processadores. Atualmente, as técnicas de tolerância a falhas com recuperação retroativa são as mais empregadas em aplicações MPI, principalmente a técnica de checkpoint coordenado. No entanto, previsões afirmam que essa última técnica será inadequada para as arquiteturas emergentes. Em contrapartida, as técnicas de log de mensagens possuem características que as tornam mais apropriadas no novo cenário que se estabelece. O presente trabalho consiste em uma proposta de log de mensagens pessimista remoto com checkpoint não-coordenado e a avaliação de desempenho da comunicação MPI sobre Publish/Subscriber no qual se baseia o log de mensagens. O trabalho compreende: um estudo das técnicas de tolerância a falhas mais empregadas em ambientes de alto desempenho e a motivação para a escolha dessa variante de log de mensagens; a proposta de log de mensagens; uma implementação de comunicação Open MPI sobre OpenAMQ e sua respectiva avaliação de desempenho com comunicação tradicional TCP/IP e com o log de mensagens pessimista local da distribuição do Open MPI. Os benchmarks utilizados foram o NetPIPE, o NAS Parallel Benchmarks e a aplicação Virginia Hydrodynamics (VH-1). / The growing number of processors in parallel architectures at the top of performance rankings allows a higher processing capacity. However, it also brings an increase in the fault rate which is directly proportional to the number of processors. Nowadays, coordinated checkpoint is the most widely used rollback technique for system recovery in the occurrence of faults in MPI applications. Nevertheless, projections point that this technique will be inappropriate for the emerging architectures. On the other hand, message logging seems to be more appropriate to this new scenario. This work consists in a proposal of pessimistic message logging (remote based) with non-coordinated checkpoint and the performance evaluation of an MPI communication mechanism that works over Publish/Subscriber channels in which the proposed message logging is based. The work is organized as following: an study of fault tolerant techniques used in HPC and the motivation for choosing this variant of message logging; a message logging proposal; an implementation of Open MPI communication over OpenAMQ; performance evaluation and comparision with the tradicional TCP/IP communication and a pessimistic message logging (sender based) from Open MPI distribution. The benchmark set is composed of NetPIPE, NAS Parallel Benchmarks and Virginia Hydrodynamics (VH-1).

Many different message-oriented middlewares (MOM) exist today on the market. With different underlying designs, purposes and features, it is not easy to make the right choice of MOM for your system. The IT company Nordicstation is in such a situation where they need to make a choice of MOM. They currently own a system called Sharelock which generates reports about violations of financial assets trad- ing. They wish to make the process of generating a report more asynchronous by using a MOM and they have a couple of requirements. This thesis was carried out with the purpose of finding the most suitable products for the system in mind, compare their features, performance, licenses and ease-of- use. This was accomplished by analysing their installation process, monitoring in- terfaces, documentation on their websites and performance in a simple throughput test. The results showed that RabbitMQ was the strongest candidate. It had good per- formance, an attractive web interface for monitoring, an easy installation and it also offered commercial support. Apache Artemis was also found to be an attractive choice but it did not have a web interface which made it hard to manage the system. / Idag finns det många meddelandebaserade mellanprogramvaror (MOM) på marknaden. Dessa har olika grundläggande designer, syften och kännetecken vilket kan göra det svårt att göra ett bra val av MOM för ett visst system. IT-företaget Nordicstation befinner sig i en sådan situation där de behöver göra ett val av MOM. De har ett system som kallas Sharelock som genererar rapporter om regelbrott in- om värdepappershandeln. De vill göra processen mer asynkron genom att använda en MOM och de har ett antal krav på produkten. Det här examensarbetet utfördes med syftet att hitta de mest passande produkter- na för det tänka systemet, jämföra deras särdrag, prestanda, licenser och anvä- ndarvänlighet. Detta gjordes genom att analysera deras installationsprocess, övervakningsgränssnitt, dokumentation på deras hemsida och prestanda i ett en- kelt prestandatest. Resultaten visade att RabbitMQ var den starkaste kandidaten. Den hade bra prestanda, ett attraktivt webbgränssnitt, en enkel installation och den erbjöd också kommersiell support åt kunder. Apache Artemis var också ett attraktivt val men den hade inget webbgränssnitt vilket gjorde det svårt att övervaka och hantera sys- temet.