Many of the grid compute resources (CPU cycles, network bandwidths, computing power, processor times, and software) exist as non-storable commodities, which we call grid compute commodities (gcc) and are distributed geographically across organizations. These organizations have dissimilar resource compositions and usage policies, which makes pricing grid resources and guaranteeing their availability a challenge. Several initiatives (Globus, Legion, Nimrod/G) have developed various frameworks for grid resource management. However, there has been a very little effort in pricing the resources. In this thesis, we propose financial option based model for pricing grid resources by devising three research threads: pricing the gcc as a problem of real option, modeling gcc spot price using a discrete time approach, and addressing uncertainty constraints in the provision of Quality of Service (QoS) using fuzzy logic. We used GridSim, a simulation tool for resource usage in a Grid to experiment and test our model. To further consolidate our model and validate our results, we analyzed usage traces from six real grids from across the world for which we priced a set of resources. We designed a Price Variant Function (PVF) in our model, which is a fuzzy value and its application attracts more patronage to a grid that has more resources to offer and also redirect patronage from a grid that is very busy to another grid. Our experimental results show that the application of the PVF has helped achieve equilibrium between users satisfaction measured as QoS and recovery of the infrastructure investment made by the providers. In the absence of pricing benchmarks, we setup Commodity Base Prices (CBP) and then integrated our PVF and CBP with GridSim to price grid compute resources. In summary, this thesis provides the design of a model to price grid compute resources using financial options theory. The model achieves mutual benefit for users and providers in the grid environment. The mutual benefit is expressed in terms of QoS to the users and recovery of investments on the grid infrastructure for the providers. This thesis has opened up many different opportunities for further research especially in the era of enterprise computing with clouds.

Many of the grid compute resources (CPU cycles, network bandwidths, computing power, processor times, and software) exist as non-storable commodities, which we call grid compute commodities (gcc) and are distributed geographically across organizations. These organizations have dissimilar resource compositions and usage policies, which makes pricing grid resources and guaranteeing their availability a challenge. Several initiatives (Globus, Legion, Nimrod/G) have developed various frameworks for grid resource management. However, there has been a very little effort in pricing the resources. In this thesis, we propose financial option based model for pricing grid resources by devising three research threads: pricing the gcc as a problem of real option, modeling gcc spot price using a discrete time approach, and addressing uncertainty constraints in the provision of Quality of Service (QoS) using fuzzy logic. We used GridSim, a simulation tool for resource usage in a Grid to experiment and test our model. To further consolidate our model and validate our results, we analyzed usage traces from six real grids from across the world for which we priced a set of resources. We designed a Price Variant Function (PVF) in our model, which is a fuzzy value and its application attracts more patronage to a grid that has more resources to offer and also redirect patronage from a grid that is very busy to another grid. Our experimental results show that the application of the PVF has helped achieve equilibrium between users satisfaction measured as QoS and recovery of the infrastructure investment made by the providers. In the absence of pricing benchmarks, we setup Commodity Base Prices (CBP) and then integrated our PVF and CBP with GridSim to price grid compute resources. In summary, this thesis provides the design of a model to price grid compute resources using financial options theory. The model achieves mutual benefit for users and providers in the grid environment. The mutual benefit is expressed in terms of QoS to the users and recovery of investments on the grid infrastructure for the providers. This thesis has opened up many different opportunities for further research especially in the era of enterprise computing with clouds.

Advances in technologies have made an unprecedented range and variety of computing resources available. A number of fields have sought to take maximum advantage of these resources, with grid computing being one of the more successful. However, the increasing complexity of these heterogeneous, distributed systems has compromised users’ ability to manage them effectively. Autonomic computing, which seeks to hide the complexity of systems by making them self-managing, offers a potential solution. In order to produce autonomic managers for grid systems, realistic input is required for development and testing. This thesis proposes a model that can be used to provide simulated input, utilizing existing system logs. The simulator adheres to the standards and specifications recognized in both autonomic and grid services, and provides the detailed, accurate information that is required by developers.

Current grid computing architectures have been based on cluster management and batch queuing systems, extended to a distributed, federated domain. These have shown shortcomings in terms of scalability, stability, and modularity. To address these problems, this dissertation applies architectural styles from the Internet and Web to the domain of generic computational grids. Using the REST style, a flexible model for grid resource interaction is developed which removes the need for any centralised services or specific protocols, thereby allowing a range of implementations and layering of further functionality. The context for resource interaction is a generalisation and formalisation of the Condor ClassAd match-making mechanism. This set theoretic model is described in depth, including the advantages and features which it realises. This RESTful style is also motivated by operational experience with existing grid infrastructures, and the design, operation, and performance of a proto-RESTful grid middleware package named DIRAC. This package was designed to provide for the LHCb particle physics experiment’s “off-line” computational infrastructure, and was first exercised during a 6 month data challenge which utilised over 670 years of CPU time and produced 98 TB of data through 300,000 tasks executed at computing centres around the world. The design of DIRAC and performance measures from the data challenge are reported. The main contribution of this work is the development of a REST model for grid resource interaction. In particular, it allows resource templating for scheduling queues which provide a novel distributed and scalable approach to resource scheduling on the grid.

Zugl.: Erlangen, Nürnberg, Univ., Diss., 2008

Zugl.: Aachen, Techn. Hochsch., Diss., 2009

Στη σύγχρονη εποχή, η ανάπτυξη των ετερογενών και κατανεμημένων περιβαλλόντων, όπως τα περιβάλλοντα πλέγματος, καθιστά εφικτή την επίλυση υπολογιστικά εντατικών προβλημάτων με αξιόπιστο και οικονομικό τρόπο. Το Πλέγμα Υπολογιστών είναι μια αναπτυσσόμενη υποδομή που παρέχει πρόσβαση σε υπολογιστική ισχύ και αποθηκευτικό χώρο κατανεμημένα σε όλο τον κόσμο. Εισήχθη για να ικανοποιήσει την ανάγκη για εφαρμογές που απαιτούν μεγάλο αριθμό υπολογισμών καθώς και την επικοινωνία των ατόμων που τις εκτελούν. Ένα πρόβλημα που μπορεί να εκμεταλλευτεί τα πλεονεκτήματα του Πλέγματος είναι το πρόβλημα χρονοπρογραμματισμού πληρωμάτων. Το συγκεκριμένο πρόβλημα είναι πολύπλοκο και χρονοβόρο εξαιτίας των πολλών περιορισμών που συνδέονται με αυτό. Στην παρούσα διπλωματική εργασία παρουσιάζεται με λεπτομέρεια η δομή και ο τρόπος λειτουργίας και εξυπηρέτησης χρηστών του Πλέγματος. Επίσης, καταγράφουμε τη μεθοδολογία και τον τρόπο υποβολής εργασιών στο Πλέγμα από τη σκοπιά του χρήστη. Επικεντρώνουμε το ενδιαφέρον μας στην αποδοτική επίλυση του προβλήματος χρονοπρογραμματισμού ανθρωπίνων πόρων, συγκεκριμένα του νοσηλευτικού προσωπικού ενός νοσοκομείου, με χρήση παράλληλης επεξεργασίας σε περιβάλλον δικτύου υπολογιστών. / -

A grande demanda por computação de alto desempenho culminou na construção de ambientes de execução de larga escala como as Grades Computacionais. Não diferente de outras plataformas de execução, seus usuários precisam obter os dados de entrada para suas aplicações e muitas vezes precisam armazenar os resultados por elas gerados. Apesar de o termo Grade ter surgido de uma metáfora onde os recursos computacionais estão tão facilmente acessíveis como os da rede elétrica, as ferramentas para gerenciamento de dados e de recursos de armazenamento disponíveis estão muito aquém do necessário para concretizar essa idéia. A imaturidade desses serviços se torna crítica para aplicações científicas que necessitam processar grandes volumes de dados. Nesses casos, utiliza-se apenas os recursos de alto desempenho e assegura-se confiabilidade, disponibilidade e segurança para os dados através de presença humana. Este trabalho apresenta o JavaRMS, um sistema de gerência de dados para Grades. Ao empregar um modelo par-a-par, consegue-se agregar os recursos menos capacitados disponíveis no ambiente de Grade, diminuindo-se assim o custo da solução. O sistema utiliza a técnica de nodos virtuais para lidar com a grande heterogeneidade de recursos, distribuindo os dados de acordo com o espaço de armazenamento fornecido. Empregase fragmentação para viabilizar o uso dos recursos menos capacitados e para melhorar o desempenho das operações que envolvem a transferência de arquivos. Utiliza-se replicação para prover persistência aos dados e para melhorar sua disponibilidade. JavaRMS lida ainda com a dinamicidade e a instabilidade dos recursos através de um modelo de estados, de forma a diminuir o impacto das operações de manutenção. A arquitetura contempla também serviços para gerenciamento de usuários e protege os recursos contra fraudes através de um sistema de cotas. Todas as operações foram projetadas para serem seguras. Por fim, disponibiliza-se toda a infra-estrutura necessária para que serviços de busca e ferramentas de interação com o usuário sejam futuramente fornecidos. Os experimentos realizados com o protótipo do JavaRMS comprovam que usar um modelo par-a-par para organizar os recursos e localizar os dados resulta em boa escalabilidade. Já a técnica de nodos virtuais se mostrou eficiente para distribuir de forma balanceada os dados entre as máquinas, de acordo com a capacidade de armazenamento oferecida. Através de testes com a principal operação que envolve a transferência de arquivos, comprovou-se que o modelo é capaz de melhorar significativamente o desempenho de aplicações que necessitam processar grandes volumes de dados. / Large scale execution environments such as Grids emerged to meet high-performance computing demands. Like in other execution platforms, its users need to get input data to their applications and to store their results. Although the Grid term is a metaphor where computing resources are so easily accessible as those from the eletric grid, its data and resource management tools are not sufficiently mature to make this idea a reality. They usually target high-performance resources, where data reliability, availability and security is assured through human presence. It turns to be critical when scientific applications need to process huge amounts of data. This work presents JavaRMS, a Grid data management system. By using a peer-topeer model, it aggregates low capacity resources to reduce storage costs. Resource heterogeneity is dealt with the virtual node technique, where peers receive data proportionally to their provided storage space. It applies fragmentation to make feasible the usage of low capacity resources and to improve file transfer operations performance. Also, the system achieves data persistence and availability through replication. In order to decrease the impact of maintenance operations, JavaRMS deals with resource dinamicity and instability with a state model. The architecture also contains user management services and protects resources through a quota system. All operations are designed to be secure. Finally, it provides the necessary infrastructure for further deployment of search services and user interactive tools. Experiments with the JavaRMS prototype showed that using a peer-to-peer model for resource organization and data location results in good scalability. Also, the virtual node technique showed to be efficient to provide heterogeneity-aware data distribution. Tests with the main file transfer operation proved the model can significantly improve data-intensive applications performance.

As grades computacionais e as redes peer-to-peer (P2P) surgiram como áreas distintas, com diferentes propósitos, modelos e ferramentas. No decorrer dos últimos anos, estas áreas foram convergindo, uma vez que a infraestrutura e o modelo de execução descentralizada das redes P2P provaram ser uma alternativa adequada para o tratamento de questões relacionadas à manutenção de grades de larga escala, tais como escalabilidade, descoberta, alocação e monitoramento de recursos. O modelo MultiCluster trata a convergência entre grades computacionais e redes peer-to-peer de uma forma mais restrita: os problemas de escalabilidade, de descoberta e alocação de recursos são minimizados considerando-se apenas recursos localmente disponíveis para a construção de uma grade, a qual pode ser usada para a execução de aplicações com diferentes características de acoplamento e comunicação. Esse trabalho apresenta a arquitetura do modelo e seus aspectos funcionais, bem como um primeira implementação do modelo, realizada através da adaptação da biblioteca de programação DECK sobre os protocolos do projeto JXTA. A avaliação do funcionamento dessa implementação é apresentada e discutida, com base em algumas aplicações com diferentes características. / Grid computing and peer-to-peer computing emerged as distinct areas with different purposes, models and tools. Over the last years, these areas has been converging since the infrastructure and the execution model used in peer-to-peer networks have proven to be a suitable way to treat some problems related to the maintenance of large scale grids, such as scalability, monitoring, and resource discovery and allocation. The MultiCluster model addresses the convergence of grids and peer-to-peer networks in a more restricted way: the problems related to scalability, resource allocation and discovery are minimized by considering only local resources for the conception of a small scale grid, which can be used to run applications with different characteristics of granularity and communication. This work presents the MultiCluster architecture and its functional aspects, as well as a first implementation carried out by adapting the DECK programming library to use JXTA protocols and its consequent evaluation, based on applications with different characteristics.

A grande demanda por computação de alto desempenho culminou na construção de ambientes de execução de larga escala como as Grades Computacionais. Não diferente de outras plataformas de execução, seus usuários precisam obter os dados de entrada para suas aplicações e muitas vezes precisam armazenar os resultados por elas gerados. Apesar de o termo Grade ter surgido de uma metáfora onde os recursos computacionais estão tão facilmente acessíveis como os da rede elétrica, as ferramentas para gerenciamento de dados e de recursos de armazenamento disponíveis estão muito aquém do necessário para concretizar essa idéia. A imaturidade desses serviços se torna crítica para aplicações científicas que necessitam processar grandes volumes de dados. Nesses casos, utiliza-se apenas os recursos de alto desempenho e assegura-se confiabilidade, disponibilidade e segurança para os dados através de presença humana. Este trabalho apresenta o JavaRMS, um sistema de gerência de dados para Grades. Ao empregar um modelo par-a-par, consegue-se agregar os recursos menos capacitados disponíveis no ambiente de Grade, diminuindo-se assim o custo da solução. O sistema utiliza a técnica de nodos virtuais para lidar com a grande heterogeneidade de recursos, distribuindo os dados de acordo com o espaço de armazenamento fornecido. Empregase fragmentação para viabilizar o uso dos recursos menos capacitados e para melhorar o desempenho das operações que envolvem a transferência de arquivos. Utiliza-se replicação para prover persistência aos dados e para melhorar sua disponibilidade. JavaRMS lida ainda com a dinamicidade e a instabilidade dos recursos através de um modelo de estados, de forma a diminuir o impacto das operações de manutenção. A arquitetura contempla também serviços para gerenciamento de usuários e protege os recursos contra fraudes através de um sistema de cotas. Todas as operações foram projetadas para serem seguras. Por fim, disponibiliza-se toda a infra-estrutura necessária para que serviços de busca e ferramentas de interação com o usuário sejam futuramente fornecidos. Os experimentos realizados com o protótipo do JavaRMS comprovam que usar um modelo par-a-par para organizar os recursos e localizar os dados resulta em boa escalabilidade. Já a técnica de nodos virtuais se mostrou eficiente para distribuir de forma balanceada os dados entre as máquinas, de acordo com a capacidade de armazenamento oferecida. Através de testes com a principal operação que envolve a transferência de arquivos, comprovou-se que o modelo é capaz de melhorar significativamente o desempenho de aplicações que necessitam processar grandes volumes de dados. / Large scale execution environments such as Grids emerged to meet high-performance computing demands. Like in other execution platforms, its users need to get input data to their applications and to store their results. Although the Grid term is a metaphor where computing resources are so easily accessible as those from the eletric grid, its data and resource management tools are not sufficiently mature to make this idea a reality. They usually target high-performance resources, where data reliability, availability and security is assured through human presence. It turns to be critical when scientific applications need to process huge amounts of data. This work presents JavaRMS, a Grid data management system. By using a peer-topeer model, it aggregates low capacity resources to reduce storage costs. Resource heterogeneity is dealt with the virtual node technique, where peers receive data proportionally to their provided storage space. It applies fragmentation to make feasible the usage of low capacity resources and to improve file transfer operations performance. Also, the system achieves data persistence and availability through replication. In order to decrease the impact of maintenance operations, JavaRMS deals with resource dinamicity and instability with a state model. The architecture also contains user management services and protects resources through a quota system. All operations are designed to be secure. Finally, it provides the necessary infrastructure for further deployment of search services and user interactive tools. Experiments with the JavaRMS prototype showed that using a peer-to-peer model for resource organization and data location results in good scalability. Also, the virtual node technique showed to be efficient to provide heterogeneity-aware data distribution. Tests with the main file transfer operation proved the model can significantly improve data-intensive applications performance.