Time-sensitive applications that are also CPU intensive like video games, video playback, eye-candy desktops etc. are increasingly common. These applications run on commodity operating systems that are targeted at diverse hardware, and hence they cannot assume that sufficient CPU is always available. Increasingly, these applications are designed to be adaptive. When executing multiple such applications, the operating system must not only provide good timeliness but also (optionally) allow co-ordinating their adaptations so that applications can deliver uniform fidelity. In this work, we present a starvation-free, fair, process scheduling algorithm that provides predictable and low latency execution without the use of reservations and assists adaptive time sensitive tasks with achieving consistent quality through cooperation. We combine an event-driven application model called cooperative polling with a fair-share scheduler. Cooperative polling allows sharing of timing or priority information across applications via the kernel thus providing good timeliness, and the fair-share scheduler provides fairness and full utilization. Our experiments show that cooperative polling leverages the inherent efficiency advantages of voluntary context switching versus involuntary pre-emption. In CPU saturated conditions, we show that the scheduling responsiveness of cooperative polling is five times better than a well-tuned fair-share scheduler, and orders of magnitude better than the best-effort scheduler used in the mainstream Linux kernel.

Time-sensitive applications that are also CPU intensive like video games, video playback, eye-candy desktops etc. are increasingly common. These applications run on commodity operating systems that are targeted at diverse hardware, and hence they cannot assume that sufficient CPU is always available. Increasingly, these applications are designed to be adaptive. When executing multiple such applications, the operating system must not only provide good timeliness but also (optionally) allow co-ordinating their adaptations so that applications can deliver uniform fidelity. In this work, we present a starvation-free, fair, process scheduling algorithm that provides predictable and low latency execution without the use of reservations and assists adaptive time sensitive tasks with achieving consistent quality through cooperation. We combine an event-driven application model called cooperative polling with a fair-share scheduler. Cooperative polling allows sharing of timing or priority information across applications via the kernel thus providing good timeliness, and the fair-share scheduler provides fairness and full utilization. Our experiments show that cooperative polling leverages the inherent efficiency advantages of voluntary context switching versus involuntary pre-emption. In CPU saturated conditions, we show that the scheduling responsiveness of cooperative polling is five times better than a well-tuned fair-share scheduler, and orders of magnitude better than the best-effort scheduler used in the mainstream Linux kernel.

Time-sensitive applications that are also CPU intensive like video games, video playback, eye-candy desktops etc. are increasingly common. These applications run on commodity operating systems that are targeted at diverse hardware, and hence they cannot assume that sufficient CPU is always available. Increasingly, these applications are designed to be adaptive. When executing multiple such applications, the operating system must not only provide good timeliness but also (optionally) allow co-ordinating their adaptations so that applications can deliver uniform fidelity. In this work, we present a starvation-free, fair, process scheduling algorithm that provides predictable and low latency execution without the use of reservations and assists adaptive time sensitive tasks with achieving consistent quality through cooperation. We combine an event-driven application model called cooperative polling with a fair-share scheduler. Cooperative polling allows sharing of timing or priority information across applications via the kernel thus providing good timeliness, and the fair-share scheduler provides fairness and full utilization. Our experiments show that cooperative polling leverages the inherent efficiency advantages of voluntary context switching versus involuntary pre-emption. In CPU saturated conditions, we show that the scheduling responsiveness of cooperative polling is five times better than a well-tuned fair-share scheduler, and orders of magnitude better than the best-effort scheduler used in the mainstream Linux kernel. / Science, Faculty of / Computer Science, Department of / Graduate

This thesis study focuses on the development of model-based optimization strategies for the integration of process scheduling and dynamic optimization, and applications of the integrated approaches to industrial polymerization processes. The integrated decision making approaches seek to explore the synergy between production schedule design and process unit control to improve process performance. The integration problem has received much attention from both the academia and industry since the past decade. For scheduling, we adopt two formulation approaches based on the state equipment network and resource task network, respectively. For dynamic optimization, we rely on the simultaneous collocation strategy to discretize the differential-algebraic equations. Two integrated formulations are proposed that result in mixed discrete/dynamic models, and solution methods based on decomposition approaches are addressed. A class of ring-opening polymerization processes are used for our industrial case studies. We develop rigorous dynamic reactor models for both semi-batch homopolymerization and copolymerization operations. The reactor models are based on first-principles such as mass and heat balances, reaction kinetics and vapor-liquid equilibria. We derive reactor models with both the population balance method and method of moments. The obtained reactor models are validated using historical plant data. Polymerization recipes are optimized with dynamic optimization algorithms to reduce polymerization times by modifying operating conditions such as the reactor temperature and monomer feed rates over time. Next, we study scheduling methods that involve multiple process units and products. The resource task network scheduling model is reformulated to the state space form that offers a good platform for incorporating dynamic models. Lastly for the integration study, we investigate a process with two parallel polymerization reactors and downstream storage and purification units. The dynamic behaviors of the two reactors are coupled through shared cooling resources. We formulate the integration problem by combining the state space resource task network model with the moment reactor model. The case study results indicate promising improvements of process performances by applying dynamic optimization and scheduling optimization separately, and more importantly, the integration of the two.

É um resultado conhecido que em Data Grids, onde o processamento envolve grandes quantidades de dados, pode ser mais eficaz escalonar os processos para execução nos sites que já dispõem dos dados do que transferir os dados para um site onde o processo que irá necessitar deles foi escalonado. Os estudos existentes se baseiam em pequenas quantidades de sites, com conhecimento centralizado sobre o estado dos diversos sites. Essa opção não é escalável para Grids com grande número de participantes. Este trabalho analisa versões distribuídas com informação local para os algoritmos de escalonamento de processo e replicação de dados, mostrando o efeito das topologias de interconexão de sites no desempenho desses. É observado que, considerando a existência apenas de informações locais devido às restrições topologicas, resultados diferentes quanto aos melhores algoritmos de escalonamento de processos e replicação de dados são encontrados. / It is a known result that in Data Grids, where the processing involves large amounts of data, can be more effective schedule processes to run on sites that already have the data than transfering data to a site where the process that will require them was installed. The existing studies are based on small numbers of sites, with centralized knowledge about the state of the various sites. This option is not scalable for grids with large numbers of participants. This paper will propose distributed versions with local information for process scheduling algorithms and data replication, showing the effect of interconnect topologies on the performance of these sites. It is observed that, considering the existence of only local information due to topological constraints, different results related to the best scheduling algorithms and data replication processes are found.

Pesquisas em sistemas paralelos e distribuídos de alto desempenho apresentam limitações no que se refere a análise, projeto, implementação e execução automática e transparente de aplicações. Essas limitações motivaram o projeto do MidHPC (do inglês Middleware for High Performance Computing, ou seja, Middleware para Computação de Alto Desempenho), que balanceia transparente e automaticamente cargas de trabalho considerando a capacidade dos recursos computacionais e o comportamento das aplicações envolvendo: processamento, acesso a disco, memória e rede. Para utilizar todo o potencial do MidHPC, aplicações devem ser escritas utilizando o modelo de programação concorrente, tal como o padrão POSIX de threads (pthreads). Aplicações desenvolvidas seguindo esse modelo de programação podem ser executadas em ambientes de Grid sem alteração de código fonte ou recompilação. Durante a execução, tarefas de uma mesma aplicação paralela comunicam-se, transparentemente, por meio de um sistema de memória compartilhada distribuída. O objetivo deste trabalho foi desenvolver alguns dos módulos do projeto MidHPC e integrar demais ferramentas que haviam sido previamente desenvolvidas pelo grupo. Este trabalho permite aplicar, em ambientes reais, todos os conceitos de escalonamento de processos estudados e desenvolvidos durante o projeto MidHPC / Current researches on parallel and distributed systems present limitations regarding the analysis, design, implementation and automatic execution of high performance applications. Those limitations motivated the design of MidHPC (Middleware for High Performance Computing), which transparently and automatically balances workloads considering computing resources capacities and application behavior such as: processing, network, memory and disc accesses. In order to use all of the MidHPC potential, applications must be developed following the concurrent programming model, using the POSIX thread standard (pthreads). Applications developed according to this programming model can be executed in the Grid environment with no source code modification nor recompilation. During the execution, tasks of the same application communicate, transparently, using a distributed shared memory system. The objective of this master thesis was to develop modules of the MidHPC project and integrate tools that were previously developed by the research group. This work allows applying, in real-world environments, all process scheduling concepts studied and developed during the MidHPC project

Este trabalho apresenta uma análise de comportamento de índices de carga relacionados ao uso e à atividade de memória. São descritos 11 índices que refletem direta ou indiretamente a carga de memória. Esses índices podem ser obtidos facilmente no sistema peracional GNU/Linux através do sistema de arquivos /proc. Uma ferramenta de monitoramento foi criada para facilitar a análise de comportamento, podendo também servir para fornecer informações de carga para políticas de escalonamento. Os valores de cada índice foram monitorados durante a execução de uma carga de trabalho composta por aplicações reais que utilizam altas quantidades de memória. A partir dos resultados é possível descobrir a utilidade de cada índice, indicando como eles podem ser usados para auxiliar políticas de escalonamento a avaliar a carga de memória de uma máquina. Métricas de avaliação de carga originárias da combinação de mais de um índice são descritas. Essas métricas foram criadas em casos em que a combinação de mais de um índice permitia representar a carga de memória com mais precisão do que índices usados isoladamente. As métricas e índices pesquisados proporcionam formas precisas de representar a carga de memória em vários níveis, desde níveis de baixa utilização até níveis de saturação da memória principal com sintomas de ocorrência de thrashing. / This work presents an analysis of the behavior of load indices related to memory usage and activity. It describes 11 indices which reflect memory load, directly or indirectly. These indices may be easily obtained on the GNU/Linux operating system using the /proc filesystem. A monitoring tool has been developed to make the behavior analysis easier, but it can also be employed to provide load information to scheduling policies. The values of each index have been monitored during the execution of a workload composed by real memory-intensive applications. From the monitoring results, it is possible to find out the usefulness of each index, pointing out how it may be used to help scheduling policies evaluate the memory load of a certain computer. Load evaluation metrics have been created combining more than one index, with the objective of characterizing memory load more precisely than just one isolated index. The explored metrics and indices have shown to provide precise forms of representing memory load at several levels, from low utilization levels up to excessive main memory usage with symptoms of thrashing.

Este trabalho analisa o impacto causado no desempenho de uma simulação distribuída quando técnicas de particionamento convencionais são empregadas. Essas técnicas não levam em conta informações inerentes ao estado da simulação. Pelo fato da execução de uma simulação também estar sujeita a sofrer interferências da plataforma, informações sobre a potência computacional de cada recurso utilizado e sobre o tipo de simulação, podem ser aplicadas em seu particionamento. Foram utilizadas informações estáticas, geradas através da avaliação da plataforma com benchmarks, e dinâmicas, obtidas através de índices de carga. Os resultados obtidos da utilização destas técnicas se mostram atrativos, principalmente quando o objetivo é a execução das simulações em ambientes que não disponibilizam políticas de escalonamento específicas e sim políticas convencionais. Nos estudos de casos avaliados, observaram-se ganhos satisfatórios, como a redução de até 24% do tempo de execução, um aumento de até 22% de eficiência e 79% menos rollbacks causados. Percebe-se que dependendo do tempo que se dispõe e dos objetivos pretendidos, as técnicas convencionais podem ser empregadas em simulações distribuídas. Este trabalho também contribui com o aperfeiçoamento das duas ferramentas utilizadas: WARPED e AMIGO. Uma interface de comunicação entre as duas ferramentas foi desenvolvida, ampliando assim seus campos de utilização. / This work analyses the impact caused on distributed simulation performance when conventional partitioning techniques are employed. These techniques do not take into account inherent information about the state of the simulation. Since a simulation execution is subject to platform interference, information about the type of simulations and about the computational power of resources may be applied to the partitioning process. Static performance information, generated from evaluating the platform with benchmarks has been employed, as well as dynamic load information provided by load indices. The results obtained with this approach are attractive, mainly when the objective is to execute simulations on environments which make conventional scheduling policies available, instead of specific policies. The evaluated case studies show satisfactory performance gains of up to 24% of reduction in execution time, 22% of improvement in efficiency and reduction of up to 79% in rollbacks. So, depending on the available time and the aimed objectives, it is worth using conventional techniques to assist distributed simulation partitioning. This work also contributes to the improvement of both tools used in it: Warped and AMIGO. A communication interface has been developed to integrate the tools, extending their capabilities.

É um resultado conhecido que em Data Grids, onde o processamento envolve grandes quantidades de dados, pode ser mais eficaz escalonar os processos para execução nos sites que já dispõem dos dados do que transferir os dados para um site onde o processo que irá necessitar deles foi escalonado. Os estudos existentes se baseiam em pequenas quantidades de sites, com conhecimento centralizado sobre o estado dos diversos sites. Essa opção não é escalável para Grids com grande número de participantes. Este trabalho analisa versões distribuídas com informação local para os algoritmos de escalonamento de processo e replicação de dados, mostrando o efeito das topologias de interconexão de sites no desempenho desses. É observado que, considerando a existência apenas de informações locais devido às restrições topologicas, resultados diferentes quanto aos melhores algoritmos de escalonamento de processos e replicação de dados são encontrados. / It is a known result that in Data Grids, where the processing involves large amounts of data, can be more effective schedule processes to run on sites that already have the data than transfering data to a site where the process that will require them was installed. The existing studies are based on small numbers of sites, with centralized knowledge about the state of the various sites. This option is not scalable for grids with large numbers of participants. This paper will propose distributed versions with local information for process scheduling algorithms and data replication, showing the effect of interconnect topologies on the performance of these sites. It is observed that, considering the existence of only local information due to topological constraints, different results related to the best scheduling algorithms and data replication processes are found.

Este trabalho apresenta o projeto e a implementação da política de escalonamento com suporte à migração de processos JUMP. A migração de processos é uma ferramenta importante que complementa a alocação inicial realizada pela política de escalonamento em um ambiente paralelo distribuído, permitindo um balanceamento de carga dinâmico e mais refinado, resultando em um melhor desempenho do ambiente e menor tempo de resposta das aplicações paralelas distribuídas. A nova política unifica a alocação inicial e migração de processos em um único algoritmo, de forma a compartilhar decisões para o objetivo comum de prover um melhor desempenho para aplicações de uso intensivo de processamento em clusters heterogêneos. A política é implementada sobre o ambiente de escalonamento flexível e dinâmico AMIGO, adaptado para o suporte à migração de processos. A avaliação de desempenho mostrou que a nova política oferece ganhos expressivos nos tempos de resposta quando comparada às outras duas políticas de escalonamento implementadas no AMIGO, em quase todos os cenários, para diversas aplicações e diversas situações de carga do ambiente / This work presents the project and implementation of the scheduling policy with process migration support JUMP. Process migration is an important tool that complements the initial placement performed by the scheduling policy in a distributed parallel environment, allowing for dynamic and more refined load balancing, resulting in better performance of the environment and shorter response time for distributed parallel applications. The new policy unifies initial placement and process migration in a single algorithm, enabling the sharing of decisions for the common goal of providing a better performance for CPU-bound applications in heterogeneous clusters. The policy is implemented over the dynamical and flexible environment AMIGO, adapted in order to support process migration. Performance evaluation showed that the new policy offers expressive gains in response times when compared to other two scheduling policies implemented in AMIGO in almost all scenarios, for different applications and different environment load situations