Back to search

National Technical University of Athens (NTUA)

Διαχείριση δεδομένων σε κατανεμημένα συστήματα μεγάλης κλίμακας για χρήση σε εφαρμογές αναλυτικής επεξεργασίας

Abstract

dc:description

The increasing size of the data collected and generated by industrial and academic information systems has created new sets of demands from data management platforms. Besides the well-documented need for offline analytics, the requirement to immediately detect interesting trends is ever-growing, rendering real-time analytics a necessity. In such applications, data are usually determined by a temporal aspect and presented at different levels of granularity. Thousands or millions of such records are produced per second and modern systems are expected to be able to both incorporate and process them. This thesis deals with the storage, indexing and querying of multidimensional data used for analytical processing in large scale distributed systems and aims to create an always-on, real-time data access and support system. To that end, the basic requirements of such a system are studied and identified: Powerful data processing and high-rate updates. Existing methodologies inadvertently fail to simultaneously meet both these requirements. To alleviate the problem, techniques from the field of distributed data management and data warehousing are applied in order to disseminate, query and update high volumes of multidimensional data characterized by hierarchies. The goal is to maintain the best of both worlds: Powerful indexing/analytics engine for immense volumes of data both over historical and real-time incoming updates and a shared-nothing architecture that ensures scalability and availability at low cost. Geographically spanned users, without the use of any proprietary tool, can share information that arrives from distributed locations at a high rate and query it in different levels of granularity. The research process towards this goal starts with HiPPIS, a system that supports concept hierarchies in DHTs, in order to store historical data in various levels of granularity. The resulting system, HiPPIS, greatly simplifies the insertion and update operations due to the lack of data pre-processing. Moreover, it employs an adaptive scheme that automatically adjusts the level of indexing according to the granularity of the incoming queries, without assuming any prior knowledge of the workload. In an attempt to include an a-priori consideration for group-by queries, as well as to explicitly deal with the query performance versus variable data availability or load skew, a well known, highly effective centralized structure is distributed over an unstructured network of interconnected commodity nodes on-the-fly, reducing cube creation and query times by enforcing parallelization. Brown Dwarf performs online querying and updating and employs an adaptive replication scheme that adjusts to sudden shifts in workload skew as well as network churn by expanding or shrinking the units of the distributed data structure. Thus, the system ensures elasticity of resources and content availability. Finally, to improve the handling of time series data, namely data determined by a temporal aspect, HORAE is proposed, a system that employs a hybrid solution for data storage and processing: High-rate updates and queries targeting the most recent items are handled by a DHT-based system that enables fast insertion times and multidimensional indexing. The large bulk of the data is handled through a distributed data cube structure that adaptively materializes and replicates according to demand. The two components seamlessly integrate to offer the advantages of powerful aggregate data processing along with scalability and elasticity of commodity resources.

Degree

thesis:*
Grantor dc:publisher
National Technical University of Athens (NTUA)
Year dc:date
2011

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Doka, Katerina
  • Δόκα, Αικατερίνη

Subjects

dc:subject × 14

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/25551

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Doka, Katerina; Δόκα, Αικατερίνη. Διαχείριση δεδομένων σε κατανεμημένα συστήματα μεγάλης κλίμακας για χρήση σε εφαρμογές αναλυτικής επεξεργασίας. National Technical University of Athens (NTUA), 2011. http://hdl.handle.net/10442/hedi/25551