The aim of this degree project is to investigate alternatives to the relational database paradigm when storing hierarchical geospatial data. The document paradigm is found suitable and is therefore further examined. A benchmark suite is developed in order to test the relative performance of the paradigms for the relevant type of data. MongoDB and Microsoft SQL Server are chosen to represent the two paradigms in the benchmark. The results indicate that the document paradigm has potential when working with hierarchical structures. When adding geospatial elements to the data, the results are inconclusive. / Det här examensarbetet ämnar undersöka alternativ till den relationella databasparadigmen för lagring av hierarkisk geospatial data. Dokumentparadigmen identiferas som särskilt lämplig och undersöks därför vidare. En benchmark-svit utvecklas för att undersöka de två paradigmens relativa prestanda vid lagring av den undersökta typen av data. MongoDB och Microsoft SQL Server väljs som representanter för de två paradigmen i benchmark-sviten. Resultaten indikerar att dokumentparadigmen har god potential för hierarkisk data. Inga tydliga slutsatser kan dock dras gällande den geospatiala aspekten.

No description available.

Förmågan att lagra data är en stor bidragande faktor till att vetenskapen ständigt rört sig framåt. Under några tusen år har människan utvecklats från att lagra data på grottväggar till hårddiskar och kraven på prestanda, tillgång och felsäkerhet ökar i rasande takt. För att hantera data i det moderna samhället utvecklas ständigt nya metoder och en av dessa metoder är replikation. Den här undersökningen testar hur replikation påverkar prestandan i en distribuerad MongoDB-lösning. Testerna i undersökningen är automatiserade och körs mot databasen i olika konfigurationer för att se hur prestandan förändras. / The ability to store data is a contributing factor in making science constantly move forward. In a few thousand years man has evolved from storing information on cave walls to hard drives and requirements in performance, availability and fault tolerance are rapidly increasing. To manage information in modern society new methods are constantly evolving and one of these methods is replication. This study tests how replication affects the performance in a distributed MongoDB solution. The tests in this survey are automated and run against the database in different configurations to see how performance changes.

Com o crescimento do volume de dados manipulado por aplicações de data warehousing, soluções centralizadas tornam-se muito custosas e enfrentam dificuldades para tratar a escalabilidade do volume de dados. Nesse sentido, existe a necessidade tanto de se armazenar grandes volumes de dados quanto de se realizar consultas analíticas (ou seja, consultas OLAP) sobre esses dados volumosos de forma eficiente. Isso pode ser facilitado por cenários caracterizados pelo uso de bancos de dados NoSQL gerenciados em ambientes paralelos e distribuídos. Dentre os desafios relacionados a esses cenários, destaca-se a necessidade de se promover uma análise de desempenho de aplicações de data warehousing que armazenam os dados do data warehouse (DW) em bancos de dados NoSQL orientados a colunas. A análise experimental e padronizada de diferentes sistemas é realizada por meio de ferramentas denominadas benchmarks. Entretanto, benchmarks para DW foram desenvolvidos majoritariamente para bancos de dados relacionais e ambientes centralizados. Nesta pesquisa de mestrado são investigadas formas de se estender o Star Schema Benchmark (SSB), um benchmark de DW centralizado, para o banco de dados NoSQL distribuído e orientado a colunas HBase. São realizadas propostas e análises principalmente baseadas em testes de desempenho experimentais considerando cada uma das quatro etapas de um benchmark, ou seja, esquema e carga de trabalho, geração de dados, parâmetros e métricas, e validação. Os principais resultados obtidos pelo desenvolvimento do trabalho são: (i) proposta do esquema FactDate, o qual otimiza consultas que acessam poucas dimensões do DW; (ii) investigação da aplicabilidade de diferentes esquemas a cenários empresariais distintos; (iii) proposta de duas consultas adicionais à carga de trabalho do SSB; (iv) análise da distribuição dos dados gerados pelo SSB, verificando se os dados agregados pelas consultas OLAP estão balanceados entre os nós de um cluster; (v) investigação da influência de três importantes parâmetros do framework Hadoop MapReduce no processamento de consultas OLAP; (vi) avaliação da relação entre o desempenho de consultas OLAP e a quantidade de nós que compõem um cluster; e (vii) proposta do uso de visões materializadas hierárquicas, por meio do framework Spark, para otimizar o desempenho no processamento de consultas OLAP consecutivas que requerem a análise de dados em níveis progressivamente mais ou menos detalhados. Os resultados obtidos representam descobertas importantes que visam possibilitar a proposta futura de um benchmark para DWs armazenados em bancos de dados NoSQL dentro de ambientes paralelos e distribuídos. / Due to the explosive increase in data volume, centralized data warehousing applications become very costly and are facing several problems to deal with data scalability. This is related to the fact that these applications need to store huge volumes of data and to perform analytical queries (i.e., OLAP queries) against these voluminous data efficiently. One solution is to employ scenarios characterized by the use of NoSQL databases managed in parallel and distributed environments. Among the challenges related to these scenarios, there is a need to investigate the performance of data warehousing applications that store the data warehouse (DW) in column-oriented NoSQL databases. In this context, benchmarks are widely used to perform standard and experimental analysis of distinct systems. However, most of the benchmarks for DW focus on relational database systems and centralized environments. In this masters research, we investigate how to extend the Star Schema Benchmark (SSB), which was proposed for centralized DWs, to the distributed and column-oriented NoSQL database HBase. We introduce proposals and analysis mainly based on experimental performance tests considering each one of the four steps of a benchmark, i.e. schema and workload, data generation, parameters and metrics, and validation. The main results described in this masters research are described as follows: (i) proposal of the FactDate schema, which optimizes queries that access few dimensions of the DW; (ii) investigation of the applicability of different schemas for different business scenarios; (iii) proposal of two additional queries to the SSB workload; (iv) analysis of the data distribution generated by the SSB, verifying if the data aggregated by OLAP queries are balanced between the nodes of a cluster; (v) investigation of the influence caused by three important parameters of the Hadoop MapReduce framework in the OLAP query processing; (vi) evaluation of the relationship between the OLAP query performance and the number of nodes of a cluster; and (vii) employment of hierarchical materialized views using the Spark framework to optimize the processing performance of consecutive OLAP queries that require progressively more or less aggregated data. These results represent important findings that enable the future proposal of a benchmark for DWs stored in NoSQL databases and managed in parallel and distributed environments.

Sistemas de Gerenciamento de Banco de Dados Não Relacionais (SGBDs NoSQL) são pacotes de software para gerenciamento de dados utilizando um modelo não relacional. Dado o atual contexto de crescimento na geração de dados e a necessidade que as organizações possuem em coletar grande quantidade de informações de clientes, pesquisas científicas, vendas e outras informações para análises futuras, é importante repensar a forma de se definir um SGBD adequado levando em consideração fatores econômicos, técnicos e estratégicos da organização. Esta é uma pesquisa relacionada com o estudo do novo modelo de gerenciamento de banco de dados, conhecido como NoSQL e traz como contribuição apresentar critérios de seleção para auxiliar consumidores de serviços de banco de dados, em organizações privadas, a selecionar um SGBD NoSQL. Para atender a este objetivo foi realizada revisão da literatura com levantamento bibliográfico sobre processo de seleção de software e de SGBDs, levantando critérios utilizados para este fim. Feito o levantamento bibliográfico, definiu-se o método de pesquisa como sendo a aplicação de um Painel Delphi, na modalidade ranking form. Por meio do painel foi possível determinar, após a realização de duas rodadas e participando um grupo de especialistas misto formado por gerentes, fornecedores de SGBD, acadêmicos, desenvolvedores e DBAs e DAs, os critérios mais relevantes para a escolha de um SGBD NoSQL, ordenados conforme pontuação obtida para cada critério. Os dados foram coletados por meio de questionário. A partir dos critérios identificados, foram feitas análises sobre os principais critérios de seleção de SGBDs NoSQL. Posteriormente, as conclusões e considerações finais contemplaram a análise dos resultados obtidos com o Painel Delphi. Como principal resultado alcançado, este estudo oferece uma visão realística acerca do modelo não relacional para gerenciamento de dados e apresenta os critérios mais importantes que indicam plausível a adoção de SGBDs NoSQL. / Database Management Systems Not Relational (NoSQL DBMSs) are software packages for data management using a non-relational model. Given the current context of growth in data generation and the need that organizations have to collect vast amount of customer information, scientific research, sales and other information for further analysis, it is important to rethink how to define a suitable DBMS considering economic, technical and strategic organization. This research is concerned with the study of the new management model database, known as NoSQL, and brings the present contribution selection criteria to assist service consumers Database, private organizations, to select a NoSQL DBMS. To satisfy this objective was reviewed the literature with bibliographic on software selection process and DBMSs, identifying criteria used for this purpose. After completion of the literature, was defined the search method with application of a Delphi panel, by the ranking form mode. Through the panel could be determined, after the completion of two rounds and attending a mixed group of experts formed by managers, DBMS vendors, academics, developers, DBAs and DAs, the most relevant criteria for choosing a NoSQL DBMS, ordered according score for each criteria. Data were collected through a survey. From the identified criteria, analyzes were made on the main selection criteria of NoSQL DBMSs. Subsequently, the conclusions and final considerations were made with analysis of the results obtained with the Delphi panel. The main result achieved, this study offers a realistic view about the non-relational model for managing data and presents the most important criteria that indicate plausible the adoption of NoSQL DBMSs.

When storing and handling Big Data sets a database management system (DBMS) can be implemented to administrate and query databases. The Swedish Internet is a big unstructured data set that contains all published Swedish Websites since the late 1990’s. NoSQL DBMSs such as MongoDB and CouchDB are particularly suited to store the Swedish Internet. Comparing the search performance of MongoDB and CouchDB in this scenario required the insertion of a subset of the Swedish Internet, querying of the data and measuring the search performance. The results show that CouchDB has in general a superior performance but comes with a drawback which is its indexation time. If a query will only be executed a few amount of times MongoDB is generally the better choice. Further studies needs to be conducted in order to assess the performance of NoSQL DBMSs over the whole dataset.

Submitted by Marlene Santos (marlene.bc.ufg@gmail.com) on 2016-08-05T19:33:36Z No. of bitstreams: 2 Dissertação - Flávio de Assis Vilela - 2015.pdf: 4909033 bytes, checksum: 3266fed0915712ec88adad7eec5bfc55 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Approved for entry into archive by Luciana Ferreira (lucgeral@gmail.com) on 2016-08-08T14:30:29Z (GMT) No. of bitstreams: 2 Dissertação - Flávio de Assis Vilela - 2015.pdf: 4909033 bytes, checksum: 3266fed0915712ec88adad7eec5bfc55 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Made available in DSpace on 2016-08-08T14:30:29Z (GMT). No. of bitstreams: 2 Dissertação - Flávio de Assis Vilela - 2015.pdf: 4909033 bytes, checksum: 3266fed0915712ec88adad7eec5bfc55 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) Previous issue date: 2015-10-08 / The increase in quantity and variety of data available on the Web contributed to the emergence of NOSQL approach, aiming at new demands, such as availability, schema flexibility and scalability. At the same time, relational databases are widely used for storing and manipulating structured data, providing stability and integrity of data, which is accessed through a standard language such as SQL. This work presents a method for integrating data stored in heterogeneous sources, in which an input query in standard SQL produces a unified answer, based in the partial answers of relational and NOSQL databases. / O aumento da quantidade e variedade de dados disponíveis na Web contribuiu com o surgimento da abordagem NOSQL, visando atender novas demandas, como disponibilidade, flexibilidade de esquema e escalabilidade. Paralelamente, bancos de dados relacionais são largamente utilizados para armazenamento e manipulação de dados estruturados, oferecendo estabilidade e integridade de dados, que são acessados através de uma linguagem padrão, como SQL. Este trabalho apresenta um método de integração de dados armazenados em fontes heterogêneas, no qual uma consulta de entrada em SQL produz uma resposta unificada, baseada nas respostas parciais de bancos de dados relacionais e NOSQL. Palavras–chave

Sistemas de Gerenciamento de Banco de Dados Não Relacionais (SGBDs NoSQL) são pacotes de software para gerenciamento de dados utilizando um modelo não relacional. Dado o atual contexto de crescimento na geração de dados e a necessidade que as organizações possuem em coletar grande quantidade de informações de clientes, pesquisas científicas, vendas e outras informações para análises futuras, é importante repensar a forma de se definir um SGBD adequado levando em consideração fatores econômicos, técnicos e estratégicos da organização. Esta é uma pesquisa relacionada com o estudo do novo modelo de gerenciamento de banco de dados, conhecido como NoSQL e traz como contribuição apresentar critérios de seleção para auxiliar consumidores de serviços de banco de dados, em organizações privadas, a selecionar um SGBD NoSQL. Para atender a este objetivo foi realizada revisão da literatura com levantamento bibliográfico sobre processo de seleção de software e de SGBDs, levantando critérios utilizados para este fim. Feito o levantamento bibliográfico, definiu-se o método de pesquisa como sendo a aplicação de um Painel Delphi, na modalidade ranking form. Por meio do painel foi possível determinar, após a realização de duas rodadas e participando um grupo de especialistas misto formado por gerentes, fornecedores de SGBD, acadêmicos, desenvolvedores e DBAs e DAs, os critérios mais relevantes para a escolha de um SGBD NoSQL, ordenados conforme pontuação obtida para cada critério. Os dados foram coletados por meio de questionário. A partir dos critérios identificados, foram feitas análises sobre os principais critérios de seleção de SGBDs NoSQL. Posteriormente, as conclusões e considerações finais contemplaram a análise dos resultados obtidos com o Painel Delphi. Como principal resultado alcançado, este estudo oferece uma visão realística acerca do modelo não relacional para gerenciamento de dados e apresenta os critérios mais importantes que indicam plausível a adoção de SGBDs NoSQL. / Database Management Systems Not Relational (NoSQL DBMSs) are software packages for data management using a non-relational model. Given the current context of growth in data generation and the need that organizations have to collect vast amount of customer information, scientific research, sales and other information for further analysis, it is important to rethink how to define a suitable DBMS considering economic, technical and strategic organization. This research is concerned with the study of the new management model database, known as NoSQL, and brings the present contribution selection criteria to assist service consumers Database, private organizations, to select a NoSQL DBMS. To satisfy this objective was reviewed the literature with bibliographic on software selection process and DBMSs, identifying criteria used for this purpose. After completion of the literature, was defined the search method with application of a Delphi panel, by the ranking form mode. Through the panel could be determined, after the completion of two rounds and attending a mixed group of experts formed by managers, DBMS vendors, academics, developers, DBAs and DAs, the most relevant criteria for choosing a NoSQL DBMS, ordered according score for each criteria. Data were collected through a survey. From the identified criteria, analyzes were made on the main selection criteria of NoSQL DBMSs. Subsequently, the conclusions and final considerations were made with analysis of the results obtained with the Delphi panel. The main result achieved, this study offers a realistic view about the non-relational model for managing data and presents the most important criteria that indicate plausible the adoption of NoSQL DBMSs.

Le travail présenté dans cette thèse vise à proposer des approches pour construire et développer des entrepôts de données selon le modèle NoSQL orienté colonnes. L'intérêt porté aux modèles NoSQL est motivé d'une part, par l'avènement des données massives et d'autre part, par l'incapacité du modèle relationnel, habituellement utilisés pour implémenter les entrepôts de données, à permettre le passage à très grande échelle. En effet, les différentes modèles NoSQL sont devenus des standards dans le stockage et la gestion des données massives. Ils ont été conçus à l'origine pour construire des bases de données dont le modèle de stockage est le modèle « clé/valeur ». D'autres modèles sont alors apparus pour tenir compte de la variabilité des données : modèles orienté colonne, orienté document et orienté graphe. Pour développer des entrepôts de données massives, notre choix s'est porté sur le modèle NoSQL orienté colonnes car il apparaît comme étant le plus approprié aux traitements des requêtes décisionnelles qui sont définies en fonction d'un ensemble de colonnes (mesures et dimensions) issues de l'entrepôt. Cependant, le modèle NoSQL en colonnes ne propose pas d'opérateurs de type analyse en ligne (OLAP) afin d'exploiter les entrepôts de données.Nous présentons dans cette thèse des solutions innovantes sur la modélisation logique et physique des entrepôts de données NoSQL en colonnes. Nous avons proposé une approche de construction des cubes de données qui prend compte des spécificités de l'environnement du stockage orienté colonnes. Par ailleurs, afin d'exploiter les entrepôts de données en colonnes, nous avons défini des opérateurs d'agrégation permettant de créer des cubes OLAP. Nous avons proposé l'opérateur C-CUBE (Columnar-Cube) permettant de construire des cubes OLAP stockés en colonnes dans un environnement relationnel en utilisant la jointure invisible. MC-CUBE (MapReduce Columnar-Cube) pour construire des cubes OLAP stockés en colonnes dans un environnement distribué exploitant la jointure invisible et le paradigme MapReduce pour paralléliser les traitements. Et enfin, nous avons développé l'opérateur CN-CUBE (Columnar-NoSQL Cube) qui tient compte des faits et des dimensions qui sont groupés dans une même table lors de la génération de cubes à partir d'un entrepôt dénormalisé selon un certain modèle logique. Nous avons réalisé une étude de performance des modèles de données dimensionnels NoSQL et de nos opérateurs OLAP. Nous avons donc proposé un index de jointure en étoile adapté aux entrepôts de données NoSQL orientés colonnes, baptisé C-SJI (Columnar-Star Join Index). Pour évaluer nos propositions, nous avons défini un modèle de coût pour mesurer l'impact de l'apport de cet index. D'autre part, nous avons proposé un modèle logique baptisé FLM (Flat Logical Model) pour implémenter des entrepôts de données NoSQL orientés colonnes et de permettre une meilleure prise en charge par les SGBD NoSQL de cette famille.Pour valider nos différentes contributions, nous avons développé une plate-forme logicielle CG-CDW (Cube Generation for Columnar Data Warehouses) qui permet de générer des cubes OLAP à partir d'entrepôts de données en colonnes. Pour terminer et afin d'évaluer nos contributions, nous avons tout d'abord développé un banc d'essai décisionnel NoSQL en colonnes (CNSSB : Columnar NoSQL Star Schema Benchmark) basé sur le banc d'essai SSB (Star Schema Benchmark), puis, nous avons procédé à plusieurs tests qui ont permis de montrer l'efficacité des différents opérateurs d'agrégation que nous avons proposé. / The work presented in this thesis aims at proposing approaches to build data warehouses by using the columnar NoSQL model. The use of NoSQL models is motivated by the advent of big data and the inability of the relational model, usually used to implement data warehousing, to allow data scalability. Indeed, the NoSQL models are suitable for storing and managing massive data. They are designed to build databases whose storage model is the "key/value". Other models, then, appeared to account for the variability of the data: column oriented, document oriented and graph oriented. We have used the column NoSQL oriented model for building massive data warehouses because it is more suitable for decisional queries that are defined by a set of columns (measures and dimensions) from warehouse. However, the NoSQL model columns do not offer online analysis operators (OLAP) for exploiting the data warehouse.We present in this thesis new solutions for logical and physical modeling of columnar NoSQL data warehouses. We have proposed a new approach that allows building data cubes by taking the characteristics of the columnar environment into account. Thus, we have defined new cube operators which allow building columnar cubes. C-CUBE (Columnar-CUBE) for columnar relational data warehouses. MC-CUBE (MapReduce Columnar-CUBE) for columnar NoSQL data warehouses when measures and dimensions are stored in different tables. Finally, CN-CUBE (Columnar NoSQL-CUBE) when measures and dimensions are gathered in the same table according a new logical model that we proposed. We have studied the NoSQL dimensional data model performance and our OLAP operators, and we have proposed a new star join index C-SJI (Columnar-Star join index) suitable for columnar NoSQL data warehouses which store measures and dimensions separately. To evaluate our contribution, we have defined a cost model to measure the impact of the use of this index. Furthermore, we have proposed a logic model called FLM (Flat Logical Model) to represent a data cube NoSQL oriented columns and enable a better management by columnar NoSQL DBMS.To validate our contributions, we have developed a software framework CG-CDW (Cube Generation for Data Warehouses Columnar) to generate OLAP cubes from columnar data warehouses. Also, we have developed a columnar NoSQL decisional benchmark CNSSB (Columnar NoSQL Star Schema Benchmark) based on the SSB and finally, we conducted several tests that have shown the effectiveness of different aggregation operators that we proposed.

L’Ingénierie Dirigée par les Modèles (IDM) est une méthode de développement logicielle ayant pour but d’améliorer la productivité et la qualité logicielle en utilisant les modèles comme artefacts de premiers plans durant le processus développement. Dans cette approche, les modèles sont typiquement utilisés pour représenter des vues abstraites d’un système, manipuler des données, valider des propriétés, et sont finalement transformés en ressources applicatives (code, documentation, tests, etc). Bien que les techniques d’IDM aient montré des résultats positifs lors de leurs intégrations dans des processus industriels, les études montrent que la mise à l’échelle des solutions existantes est un des freins majeurs à l’adoption de l’IDM dans l’industrie. Ces problématiques sont particulièrement importantes dans le cadre d’approches génératives, qui nécessitent des techniques efficaces de stockage, requêtage, et transformation de grands modèles typiquement construits dans un contexte mono-utilisateur. Plusieurs solutions de persistance, requêtage, et transformations basées sur des bases de données relationnelles ou NoSQL ont été proposées pour améliorer le passage à l’échelle, mais ces dernières sont souvent basées sur une seule sérialisation model/base de données, adaptée à une activité de modélisation particulière, mais peu efficace pour d’autres cas d’utilisation. Par exemple, une sérialisation en graphe est optimisée pour calculer des chemins de navigations complexes,mais n’est pas adaptée pour accéder à des valeurs atomiques de manière répétée. De plus, les frameworks de modélisations existants ont été initialement développés pour gérer des activités simples, et leurs APIs n’ont pas évolué pour gérer les modèles de grande taille, limitant les performances des outils actuels. Dans cette thèse nous présentons une nouvelle infrastructure de modélisation ayant pour but de résoudre les problèmes de passage à l’échelle en proposant (i) un framework de persistance permettant de choisir la représentation bas niveau la plus adaptée à un cas d’utilisation, (ii) une solution de requêtage efficace qui délègue les navigations complexes à la base de données stockant le modèle,bénéficiant de ses optimisations bas niveau et améliorant significativement les performances en terme de temps d’exécution et consommation mémoire, et (iii) une approche de transformation de modèles qui calcule directement les transformations au niveau de la base de données. Nos solutions sont construites en utilisant des standards OMG tels que UML et OCL, et sont intégrées dans les solutions de modélisations majeures telles que ATL ou EMF. / The Model Driven Engineering (MDE) paradigm is a softwaredevelopment method that aims to improve productivity and software quality by using models as primary artifacts in all the aspects of software engineering processes. In this approach, models are typically used to represent abstract views of a system, manipulate data, validate properties, and are finally transformed to application artifacts (code, documentation, tests, etc). Among other MDE-based approaches, automatic model generation processes such as Model Driven Reverse Engineering are a family of approaches that rely on existing modeling techniques and languages to automatically create and validate models representing existing artifact. Model extraction tasks are typically performed by a modeler, and produce a set of views that ease the understanding of the system under study. While MDE techniques have shown positive results when integrated in industrial processes, the existing studies also report that scalability of current solutions is one of the key issues that prevent a wider adoption of MDE techniques in the industry. This isparticularly true in the context of generative approaches, that require efficient techniques to store, query, and transform very large models typically built in a single-user context. Several persistence, query, and transformation solutions based on relational and NoSQL databases have been proposed to achieve scalability, but they often rely on a single model-to-database mapping, which suits a specific modeling activity, but may not be optimized for other use cases. For example a graph-based representation is optimized to compute complex navigation paths, but may not be the best solution for repeated atomic accesses. In addition, low-level modeling framework were originally developed to handle simple modeling activities (such as manual model edition), and their APIs have not evolved to handle large models, limiting the benefits of advance storage mechanisms. In this thesis we present a novel modeling infrastructure that aims to tackle scalability issues by providing (i) a new persistence framework that allows to choose the appropriate model-to-database mapping according to a given modeling scenario, (ii) an efficient query approach that delegates complex computation to the underlying database, benefiting of its native optimization and reducing drastically memory consumption and execution time, and (iii) a model transformation solution that directly computes transformations in the database. Our solutions are built on top of OMG standards such as UML and OCL, and are integrated with the de-facto standard modeling solutions such as EMF and ATL.