In grid architecture, a grid workflow system is a type of high-level grid middleware which aims to support large-scale sophisticated scientific or business processes in a variety of complex e-science or e-business applications such as climate modelling, disaster recovery, medical surgery, high energy physics, international stock market modelling and so on. Such sophisticated processes often contain hundreds of thousands of computation or data intensive activities and take a long time to complete. In reality, they are normally time constrained. Correspondingly, temporal constraints are enforced when they are modelled or redesigned as grid workflow specifications at build-time. The main types of temporal constraints include upper bound, lower bound and fixed-time. Then, temporal verification would be conducted so that we can identify any temporal violations and handle them in time. Conventional temporal verification research and practice have presented some basic concepts and approaches. However, they have not paid sufficient attention to overall temporal verification effectiveness and efficiency. In the context of grid economy, any resources for executing grid workflows must be paid. Therefore, more resources should be mainly used for execution of grid workflow itself rather than for temporal verification. Poor temporal verification effectiveness or efficiency would cause more resources diverted to temporal verification. Hence, temporal verification effectiveness and efficiency become a prominent issue and deserve an in-depth investigation. This thesis systematically investigates the limitations of conventional temporal verification in terms of temporal verification effectiveness and efficiency. The detailed analysis of temporal verification effectiveness and efficiency is conducted for each step of a temporal verification cycle. There are four steps in total: Step 1 - defining temporal consistency; Step 2 - assigning temporal constraints; Step 3 - selecting appropriate checkpoints; and Step 4 - verifying temporal constraints. Based on the investigation and analysis, we propose some new concepts and develop a set of innovative methods and algorithms towards more effective and efficient temporal verification. Comparisons, quantitative evaluations and/or mathematical proofs are also presented at each step of the temporal verification cycle. These demonstrate that our new concepts, innovative methods and algorithms can significantly improve overall temporal verification effectiveness and efficiency. Specifically, in Step 1, we analyse the limitations of two temporal consistency states which are defined by conventional verification work. After, we propose four new states towards better temporal verification effectiveness. In Step 2, we analyse the necessity of a number of temporal constraints in terms of temporal verification effectiveness. Then we design a novel algorithm for assigning a series of finegrained temporal constraints within a few user-set coarse-grained ones. In Step 3, we discuss the problem of existing representative checkpoint selection strategies in terms of temporal verification effectiveness and efficiency. The problem is that they often ignore some necessary checkpoints and/or select some unnecessary ones. To solve this problem, we develop an innovative strategy and corresponding algorithms which only select sufficient and necessary checkpoints. In Step 4, we investigate a phenomenon which is ignored by existing temporal verification work, i.e. temporal dependency. Temporal dependency means temporal constraints are often dependent on each other in terms of their verification. We analyse its impact on overall temporal verification effectiveness and efficiency. Based on this, we develop some novel temporal verification algorithms which can significantly improve overall temporal verification effectiveness and efficiency. Finally, we present an extension to our research about handling temporal verification results since these verification results are based on our four new temporal consistency states. The major contributions of this research are that we have provided a set of new concepts, innovative methods and algorithms for temporal verification in grid workflow systems. With these, we can significantly improve overall temporal verification effectiveness and efficiency. This would eventually improve the overall performance and usability of grid workflow systems because temporal verification can be viewed as a service or function of grid workflow systems. Consequently, by deploying the new concepts, innovative methods and algorithms, grid workflow systems would be able to better support large-scale sophisticated scientific and business processes in complex e-science and e-business applications in the context of grid economy.

Developments in electronics, computing and communication technologies have transformed IT systems from desktop and tightly coupled mainframe computers of the past to modern day highly complex distributed systems. These ICT systems interact with humans at a much advanced level than what was envisaged during the early years of computer development. The ICT systems of today have gone through various phases of developments by absorbing intermediate and modern day concepts such as networked computing, utility, on demand and autonomic computing, virtualisation and so on. We now live in a ubiquitous computing and digital economy era where computing systems have penetrated into the human lives to a degree where these systems are becoming invisible. The price of these developments is in the increased costs, higher risks and higher complexity. There is a compelling need to study these emerging systems, their applications, and the emerging market sectors that they are penetrating into. Motivated by the challenges and opportunities offered by the modern day ICT technologies, we aim in this thesis to explore the major technological developments that have happened in the ICT systems during this century with a focus on developing techniques to manage applied ICT systems in digital economy. In the process, we wish to also touch on the evolution of ICT systems and discuss these in context of the state of the art technologies and applications. We have identified the two most transformative technologies of this century, grid computing and cloud computing, and two application areas, intelligent healthcare and transportation systems. The contribution of this thesis is multidisciplinary in four broad areas. Firstly, a workload model of a grid-based ICT system in the healthcare sector is proposed and analysed using multiple healthcare organisations and applications. Secondly, an innovative intelligent system for the management of disasters in urban environments using cloud computing is proposed and analysed. Thirdly, cloud computing market sectors, applications, and workload are analysed using over 200 real life case studies. Fourthly, a detailed background and literature review is provided on grid computing and cloud computing. Finally, directions for future work are given. The work contributes in multidisciplinary fields involving healthcare, transportation, mobile computing, vehicular networking, grid, cloud, and distributed computing. The discussions presented in this thesis on the historical developments, technology and architectural details of grid computing have served to understand as to how and why grid computing was seen in the past as the global infrastructure of the future. These discussions on grid computing also provided the basis that we subsequently used to explain the background, motivations, technological details, and ongoing developments in cloud computing. The introductory chapters on grid and cloud computing, collectively, have provided an insight into the evolution of ICT systems over the last 50+ years - from mainframes to microcomputers, internet, distributed computing, cluster computing, and computing as a utility and service. The existing and proposed applications of grid and cloud computing in healthcare and transport were used to further elaborate the two technologies and the ongoing ICT developments in the digital economy. The workload models and analyses of grid and cloud computing systems can be used by the practitioners for the design and resource management of ICT systems.

Orientador: Edmundo Roberto Mauro Madeira / Tese (doutorado) - Universidade Estadual de Campinas, Instituto de Computação / Made available in DSpace on 2018-08-16T05:33:50Z (GMT). No. of bitstreams: 1 Bittencourt_LuizFernando_D.pdf: 2691554 bytes, checksum: b936bb837e62d8c4b7bacaeaae71e167 (MD5) Previous issue date: 2010 / Resumo: Grades computacionais são sistemas distribuídos compartilhados potencialmente grandes compostos por recursos heterogêneos que são ligados através de uma rede com enlaces heterogêneos. Esses sistemas tornaram-se ambientes largamente difundidos para execução de tarefas que demandam grande capacidade de processamento. Por serem sistemas compartilhados, a submissão de tarefas nas grades é oriunda de diversos usuários independentemente, o que gera uma demanda concorrente pelos recursos computacionais que deve ser gerenciada pelo middleware da grade. O escalonador é o componente responsável por decidir de que forma a distribuição dessas tarefas será realizada, devendo tratar das peculiaridades desse ambiente, tais como a heterogeneidade e o comportamento dinâmico dos recursos que o compõem, com variações tanto em quantidade quanto em qualidade. A função objetivo mais comum encontrada no escalonamento de tarefas é a minimização do makespan, ou seja, o tempo de término das tarefas que estão sendo escalonadas. Dentre os possíveis tipos de tarefas executadas em grades podemos destacar as tarefas independentes, que executam sem comunicação entre si, e as tarefas dependentes, que possuem dependências de dados que geram precedências de execução e são frequentemente modeladas como grafos acíclicos direcionados (DAGs - do inglês directed acyclic graphs). Dentre as aplicações compostas por tarefas dependentes, os DAGs de e-Ciência se sobressaem pela complexidade e necessidade crescente de recursos computacionais. Adicionalmente, o problema de escalonamento de tarefas, em sua forma geral, é NP-Completo. Dessa forma, o estudo do escalonamento de DAGs em grades computacionais é importante para o aprimoramento da execução de aplicações científicas utilizadas em diversas áreas do conhecimento. Nesta tese apresentamos algoritmos para quatro tipos de problema relacionados ao escalonamento de DAGs em grades: escalonamento estático de DAGs, escalonamento dinâmico de DAGs, escalonamento bi-critério e escalonamento de múltiplos DAGs. Apresentamos avaliações do makespan gerado pelos algoritmos após o escalonamento inicial e após a execução das tarefas com carga externa simulada nos recursos / Abstract: Computational grids are potentially large distributed systems composed of heterogeneous resources connected by a network with heterogeneous links. These systems became largely used in the execution of tasks which require large processing capacities. Because they are shared systems, task submission in grids independently originate from a number of users, leading to a concurrent demand over the computational resources, which must be managed by the grid middleware. The scheduler is the component responsible for deciding how the distribution of such tasks will occur, and it must deal with peculiarities of this environment, such as the heterogeneity and dynamic behavior of the resources, with variations in both quality and quantity. The objective function usually adopted in task scheduling is makespan minimization, which means that the scheduler tries to minimize the finish time of the tasks being scheduled. Among the tasks executed in grids we can find independent tasks, which execute without communication among them, and dependent tasks, which have data dependencies that yield in precedence constraints and are frequently modeled as directed acyclic graphs (DAGs). Among the applications composed of dependent tasks, e-Science DAGs are distinguished because of their complexity and increasing demand for computational resources. Additionally, the task scheduling problem, in its general form, is NP-Complete. Therefore, the study of scheduling of dependent tasks represented by directed acyclic graphs in computational grids is important to improve the execution of scientific applications in many areas of knowledge. In this thesis we present algorithms for four types of problems related to the DAG scheduling in grids: static scheduling of DAGs, dynamic scheduling of DAGs, bi-criteria scheduling, and scheduling of multiple DAGs. We present evaluations of the makespan generated by the algorithms after the initial scheduling and after the execution of the tasks with simulated external load in the resources / Doutorado / Sistemas de Computação / Doutor em Ciência da Computação

Orientador: Marco Aurelio Amaral Henriques / Dissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Eletrica e de Computação / Made available in DSpace on 2018-08-05T12:56:47Z (GMT). No. of bitstreams: 1 Enomoto_Cristina_M.pdf: 856279 bytes, checksum: ce524a49db0f67734e28d8458d5deb0b (MD5) Previous issue date: 2005 / Resumo: Vários sistemas de grid e computação distribuída existentes só permitem a execução de aplicações com um fluxo de execução de tarefas básico, no qual é feita a distribuição das tarefas executadas em paralelo e depois a coleta de seus resultados. Outros sistemas permitem definir uma relação de dependências entre as tarefas, formando um grafo direcionado acíclico. Porém, mesmo com este modelo de fluxo de execução não é possível executar vários tipos de aplicações que poderiam ser paralelizadas, como, por exemplo, algoritmos genéticos e de cálculo numérico que utilizam algum tipo de processamento iterativo. Nesta dissertação é proposta uma linguagem de especificação para fluxo de execução de aplicações paralelas que permite um controle de fluxo de tarefas mais flexível, viabilizando desvios condicionais e laços com iterações controladas. A linguagem é baseada na notação XML (eXtensible Markup Language), o que lhe confere características importantes tais como flexibilidade e simplicidade. Para avaliar estas e outras características da linguagem proposta, foi feita uma implementação sobre o sistema de processamento paralelo JoiN. Além de viabilizar a criação e execução de novas aplicações paralelas cujos fluxos de tarefas contêm laços e/ou desvios condicionais, a linguagem se mostrou simples de usar e não causou sobrecarga perceptível ao sistema paralelo / Abstract: Many distributed and parallel systems allow only a basic task flow, in which the parallel tasks are distributed and their results collected. In some systems the application execution flow gives support to a dependence relationship among tasks, represented by a directed acyclic graph. Even with this model it is not possible to execute in parallel some important applications as, for example, genetic algorithms. Therefore, there is a need for a new specification model with more sophisticated flow controls that allow some kind of iterative processing at the level of task management. The purpose of this work is to present a proposal for a specification language for parallel application execution workflow, which provides new types of control structures and allows the implementation of a broader range of applications. This language is based on XML (eXtensible Markup Language) notation, which provides characteristics like simplicity and flexibility to the proposed language. To evaluate these and other characteristics of the language, it was implemented on the JoiN parallel processing system. Besides allowing the creation and execution of new parallel applications containing task flows with loops and conditional branches, the proposedlanguage was easy to use and did not cause any significant overhead to the parallel system / Mestrado / Engenharia de Computação / Mestre em Engenharia Elétrica

Orientadores: Nelson Luis Saldanha da Fonseca, Fabrizio Granelli / Dissertação (mestrado ) - Universidade Estadual de Campinas, Instituto de Computação / Made available in DSpace on 2018-08-08T18:06:54Z (GMT). No. of bitstreams: 1 Batista_DanielMacedo_M.pdf: 1723518 bytes, checksum: b35136d03e434003ef1a7d13da25994f (MD5) Previous issue date: 2006 / Resumo: Grades são ambientes computacionais caracterizados pela heterogeneidade de recursos e dinamismo. Por serem ambientes dinâmicos, as grades precisam de processos que otimizem a execução das aplicações de forma também dinâmica. Tais processos devem detectar mudanças no estado da grade e tomar medidas para manter o tempo de execução das aplicações o menor possível. Existem diversas propostas de otimização dinâmica de aplicações em grades que visam atender essa necessidade através da migração de tarefas. Esta dissertação propõe uma metodologia que considera variações na disponibilidade dos hosts bem como no estado da rede. A metodologia proposta é baseada nos princípios gerais da engenharia de tráfego e atua em várias camadas da arquitetura Internet. Ela tem como objetivo minimizar o tempo de execução das aplicações e visa ser simples e independente, tanto da aplicação, quanto da grade. Os ganhos obtidos na execução de aplicações em grades com a utilização da proposta, versus a execução sem a mesma, são avaliados através de simulação com exemplos implementados usando o simulador de redes NS-2. Esta dissertação propõe também uma família de escalonadores baseados em programação inteira e em programação mista para o escalonamento de tarefas em grades que modelam o estado dos hosts bem como o da rede, sendo este o diferencial em relação às demais propostas na literatura / Abstract: Grids are dynamic and heterogeneous computing environments which require systematic methods for minimizing the execution time of applications. Such methods needs to detect changes on resource availability so that the execution time of applications can be kept low. The method introduced in this dissertation considers changes on the availability of hosts as well as on the availability of network resources. This method ressembles the Traffic Engineering for the Internet. It was validated via simulation using the NS-2 simulator. This dissertation also introduces a set of schedulers based on integer and mix programming which considers both host availability as well as network resources availability, differing from other proposals in the literature / Mestrado / Mestre em Ciência da Computação

Orientador: Nelson Luis Saldanha da Fonseca / Tese (doutorado ) - Universidade Estadual de Campinas, Instituto de Computação / Made available in DSpace on 2018-08-15T11:13:39Z (GMT). No. of bitstreams: 1 Batista_DanielMacedo_D.pdf: 4822882 bytes, checksum: 0875aace17a80193a116db65097ea804 (MD5) Previous issue date: 2010 / Resumo: Para que escalonadores em grades derivem escalonamentos, é necessário que se forneçam as demandas das aplicações e as disponibilidades dos recursos das grades. No entanto, a falta de controle centralizado, o desconhecimento dos usuários e a imprecisão das ferramentas de medição fazem com que as informações fornecidas aos escalonadores difiram dos valores reais que deveriam ser considerados para se obter escalonamentos quase-ótimos. A presente Tese introduz dois escalonadores de tarefas robustos às incertezas das informações providas como entrada ao escalonador. Um dos escalonadores lida com informações imprecisas sobre as demandas das aplicações, enquanto que o outro considera tanto imprecisões das demandas quanto da disponibilidade de recursos. A eficácia e a eficiência dos escalonadores robustos às incertezas são avaliadas através de simulação.Comparam-se os escalonamentos gerados pelos escalonadores robustos com os produzidos por escalonadores sensíveis às informações incertas. A eficácia de estimadores de largura de banda disponível são, também, avaliadas, através de medição, a luz da adoção destes em sistemas de grades, a fim de que se possa utilizar suas estimativas como informação de entrada a escalonadores robustos / Abstract: Schedulers need information on the application demands and on the grid resource availability as input to derive efficient schedules for the tasks of a grid application. However, information provided to schedulers differ from the true values due to the lack of central control in a grid and the lack of ownership of resources as well as the precision of estimations provided by measurement tools. This thesis introduces two robust schedulers based on fuzzy optimization. The first scheduler deals with uncertainties on the application demands while the other with uncertainties of both application demands and resource availability. The effectiveness of these schedulers are evaluated via simulation and the schedules produced by them are compared to those of their non-fuzzy counterpart. Moreover, the efficacy of available bandwidth estimators is assessed in order to evaluate their use in grid systems for providing schedulers with useful input information / Doutorado / Sistemas de Computação, Redes Multimidia / Doutor em Ciência da Computação

Orientador: Eduardo Candido Xavier / Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Computação / Made available in DSpace on 2018-08-27T04:49:46Z (GMT). No. of bitstreams: 1 Povoa_MarceloGalvao_M.pdf: 1965830 bytes, checksum: 7509ae1701df384bfdc3d415ecd4eda8 (MD5) Previous issue date: 2015 / Resumo: Sistemas computacionais conhecidos como Data Grids fornecem uma infraestrutura computacional distribuída para processamento e armazenamento de dados, com várias aplicações envolvendo computação em larga escala. Devido ao uso de um grande volume de dados, é necessário não apenas um escalonamento eficiente de tarefas, mas também uma distribuição inteligente de réplicas dos dados para se atingir o melhor desempenho. Esses dois problemas já foram extensivamente estudados de forma independente na literatura, mas estamos concentrados em um formulação integrada em um problema estático, de forma a otimizar uma única função objetivo. Primeiramente, mostramos que este problema não pode admitir um algoritmo aproximado. Porém, considerando uma versão restrita do problema, apresentamos um algoritmo aproximado original com fator de aproximação constante. Também fazemos um estudo de algoritmos aproximados para problemas relacionados disponíveis na literatura. Sob um aspecto mais prático, introduzimos duas heurísticas originais para o problema. A primeira é baseada no agrupamento de máquinas próximas em clusters, enquanto a segunda procura identificar grupos de dados frequentemente acessados em conjunto. Comparamos esses algoritmos com duas abordagens adaptadas da literatura, através de simulações computacionais em um grande conjunto de instâncias baseadas em grids reais. Mostramos que nossa primeira heurística costuma obter melhores soluções que as outras com boa eficiência de tempo, enquanto a segunda heurística é ainda mais rápida e ainda obtém soluções competitivas / Abstract: Computational systems known as Data Grids provide a flexible, distributed computing infrastructure for processing and storage and has many applications in large-scale computing. Due to the use of great amounts of data, not only efficient task scheduling but also thorough file replication are crucial for achieving the best performance. Both these problems have already been studied independently in the literature, but we are interested in a combined formulation as a static problem, in order to minimize a single objective function. First, we show that this problem does not admit an approximation algorithm. However, considering a restricted version of the problem, we provide a constant ratio approximation algorithm. We also conduct a study of approximation algorithms for related problems avaliable in the literature. On a more practical side, we introduce two novel heuristics for the problem. The first is based on grouping neighbor nodes into clusters, while the second tries to identify groups of files frequently accessed together. We compare these algorithms with two adapted approaches from other works in the literature by doing computational simulations using an extensive set of instances based on real grids. We show that our first heuristic often obtains the best solutions with good time efficiency, while the second is even faster and still provides competitive solutions / Mestrado / Ciência da Computação / Mestre em Ciência da Computação

This thesis details the approach taken in developing PPerfGrid. Section 2 discusses other research related to this project. Section 3 provides general background on the technologies utilized in PPerfGrid, focusing on the components that make up the Grid services architecture. Section 4 provides a description of the architecture of PPerfGrid. Section 5 details the implementation of PPerfGrid. Section 6 presents tests designed to measure the overhead and scalability of the PPerfGrid application. Section 7 suggests future work, and Section 8 concludes the thesis.

Scientists' ability to generate and store simulation results is outpacing their ability to analyze them via ad hoc programs. We observe that these programs exhibit an algebraic structure that can be used to facilitate reasoning and improve performance. In this dissertation, we present a formal data model that exposes this algebraic structure, then implement the model, evaluate it, and use it to express, optimize, and reason about data transformations in a variety of scientific domains. Simulation results are defined over a logical grid structure that allows a continuous domain to be represented discretely in the computer. Existing approaches for manipulating these gridded datasets are incomplete. The performance of SQL queries that manipulate large numeric datasets is not competitive with that of specialized tools, and the up-front effort required to deploy a relational database makes them unpopular for dynamic scientific applications. Tools for processing multidimensional arrays can only capture regular, rectilinear grids. Visualization libraries accommodate arbitrary grids, but no algebra has been developed to simplify their use and afford optimization. Further, these libraries are data dependent—physical changes to data characteristics break user programs. We adopt the grid as a first-class citizen, separating topology from geometry and separating structure from data. Our model is agnostic with respect to dimension, uniformly capturing, for example, particle trajectories (1-D), sea-surface temperatures (2-D), and blood flow in the heart (3-D). Equipped with data, a grid becomes a gridfield. We provide operators for constructing, transforming, and aggregating gridfields that admit algebraic laws useful for optimization. We implement the model by analyzing several candidate data structures and incorporating their best features. We then show how to deploy gridfields in practice by injecting the model as middleware between heterogeneous, ad hoc file formats and a popular visualization library. In this dissertation, we define, develop, implement, evaluate and deploy a model of gridded datasets that accommodates a variety of complex grid structures and a variety of complex data products. We evaluate the applicability and performance of the model using datasets from oceanography, seismology, and medicine and conclude that our model-driven approach offers significant advantages over the status quo.

Economic scheduling needs to be considered for Grid computing environment, because it gives an incentive for resource providers to supply their resources. Moreover, it enforces efficient use of resources, because the users have to pay for their use. Tendering is a suitable model for Grid scheduling because users start the negotiations for finding suitable resources for executing their jobs. Furthermore, the users specify their job requirements with their requests and therefore the resources reply with bids that are based on the cost of taking on the job and the availability of their processors. In this thesis, a framework for economic Grid scheduling using tendering is proposed. The framework entities such as users, brokers and resources employ tender/contract-net model to negotiate the prices and deadlines. The brokers' role is acting on behalf of users. During the negotiations, the entities aim to maximise their performance which is measured by a number of metrics. In order to evaluate the entities' performance under different scenarios, a Java- based simulator, called MICOSim, supporting event-driven simulation of economic Grid scheduling is presented. MICOSim can perform a simulation of more than one hundred entities faster than real time. It is concluded from the evaluation that users who are interested in increasing the job success rate and paying less for executing their jobs have to consider received prices to select the most appropriate bids, while users who are interested in improving the job average satisfaction rate have to consider either received completion time or both price and completion time to select the most suitable bids when the submission of jobs is static. The best broker strategy is the one that doesn't take into account meeting the job deadlines in the bids it sends to job owners. Finally, the resource strategy that considers the price to determine if to reply to a request or not is superior to other resource strategies. The only exception is employing this strategy with price that is too low. However, there is a tiny difference between the performances of different user strategies in dynamic submission. It is also concluded from the evaluation that broker strategies have the best performance when the revenue they target from the users is reasonable. Thus, the broker's aim has to be receiving reasonable revenue (neither too low nor too high) from acting on behalf of users. It is observed from the results that the strategy performance is influenced by the behaviour of other entities such as the submission time of user jobs. Finally, it is observed that the characteristics of entities have an effect on the performance of strategies. For example, the two user strategies that consider the received completion time and both price and completion time to determine if to accept a broker bid have similar performance, because of the existence of resources with various prices from cheap to expensive and existence of resources which don't care about the price paid for the execution. So, the price threshold doesn't have a large effect on the performance.