National Technical University of Athens (NTUA)
Αποδοτικές τεχνικές μεταφοράς και διαχείρισης δεδομένων σε κατανεμημένα συστήματα μεγάλης κλίμακας
Abstract
dc:descriptionIn recent years, Grid systems have gained popularity and have been widely utilized withinboth the research and the business domains. The Grid is a wide-area, large-scale distributed computingsystem, in which a vast number of remotely located, disjoint and diverse processing and datastorage facilities are integrated under a common service-oriented software architecture. Given thegeographic dispersion, Grid Services must be scalable enough to cope with extreme load conditions.One of the most critical components in Grid systems is the data management layer. Facedwith the problem of managing extremely large scale datasets, the Grid community has proposedthe Data Grid architecture, defining a set of basic services. The most fundamental of them arethe Data Transfer service, responsible for moving files among grid nodes, the Replica Locationservice, which keeps track of the physical locations of files and the Optimization service, whichselects the best data source for each transfer in terms of completion time and manages the dynamicreplica creation/deletion according to file usage statistics. However, all of the aforementioned servicesheavily rely on centralized mechanisms, which constitute performance bottlenecks and singlepoints of failure. The so far centralized services can neither scale to large numbers of concurrentrequests nor keep pace with frequent updates performed in highly dynamic environments. In ourwork, we introduce a novel data management architecture which integrates the location servicewith data transfer under a fully distributed and adaptive philosophy. Our scheme comprises of twoparts that cooperate to efficiently handle multiple concurrent requests and data transfer: The DistributedReplica Location Service (DRLS) that handles the locating of files and GridTorrent thatmanages the file transfer and related optimizations. DRLS utilizes a set of nodes that, organizedin a DHT, equally share the replica location information. The unique characteristic of the DRLSis that, besides the decentralization and scalability that it offers, it fully supports updates on the multiple sites of a file that exist in the system. Since in many dynamic applications data locationschange rapidly with time, our Byzantine-tolerant protocol guarantees consistency and efficientlyhandles updates on the various data locations stored, unlike conventional DHT implementations.GridTorrent is a protocol that, inspired by BitTorrent, focuses on real-time optimization of datatransfers on the Grid, fully supporting the induced security mechanisms. Based on collaborativesharing, GridTorrent allows for low latency and maximum bandwidth utilization, even under extremeload and flash crowd conditions. It allows transfers from multiple sites to multiple clientsand maximizes performance by piece exchange among the participants. A very important characteristicof the proposed architecture is that it is designed to interface and exploit well-definedand deployed Data Grid components and protocols, thus being completely backwards compatibleand readily deployable. This work includes an extensive experimental section that contains areal implementation of the system and results over both LAN and WAN environments with highlydynamic and adverse workloads.
Degree
thesis:*- Grantor dc:publisher
- National Technical University of Athens (NTUA)
- Year dc:date
- 2011
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Zissimos, Antonios
- Ζήσιμος, Αντώνιος
Subjects
dc:subject × 18- Πλέγμα
- Κατανεμημένος πίνακας κατακερματισμού
- Υπηρεσία διαχείρισης αντιγράφων
- Δίκτυα ομότιμων κόμβων
- Κατανεμημένο σύστημα μεγάλης κλίμακας
- Distributed hash table
- Peer-to-peer
- Grid
- Gridtorrent
- Replica location service
- Φυσικές Επιστήμες
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Επιστήμες Μηχανικού και Τεχνολογία
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Natural Sciences
- Computer and Information Sciences
- Engineering and Technology
- Electrical Engineering, Electronic Engineering, Information Engineering
Rights
- Language dc:language
- gre
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/42831