In this thesis, the focus is on the development of scheduling algorithms for Sim-PETEK which is a framework for parallel and distributed execution of simulations. Since it is especially designed for running parameter sweep applications in a heterogeneous distributed computational environment, multi-round and adaptive scheduling approaches are followed. Five different scheduling algorithms are designed and evaluated for scheduling purposes of Sim-PETEK. Development of these algorithms are arranged in a way that a newly developed algorithm provides extensions over the previously developed and evaluated ones. Evaluation of the scheduling algorithms is handled by running a Wireless Sensor Network (WSN) simulation over Sim-PETEK in a heterogeneous distributed computational system formed in TUBITAK UEKAE ILTAREN. This evaluation not only makes comparisons among the scheduling algorithms but it also and rates them in terms of the optimality principle of divisible load theory which mentions that in order to obtain optimal processing time all the processors used in the computation must stop at the same time. Furthermore, this study adapts a scheduling approach, which uses statistical calibration, from literature to Sim-PETEK and makes an assessment between this approach and the most optimal scheduling approach among the five algorithms that have been previously evaluated. The approach which is found to be the most efficient is utilized as the Sim-PETEK scheduler.

High performance parallel and distributed computing systems are used to solve large, complex, and data parallel scientific applications that require enormous computational power. Data parallel workloads which require performing similar operations on different data objects, are present in a large number of scientific applications, such as N-body simulations and Monte Carlo simulations, and are expressed in the form of loops. Data parallel workloads that lack precedence constraints are called arbitrarily divisible workloads, and are amenable to easy parallelization. Load imbalance that arise from various sources such as application, algorithmic, and systemic characteristics during the execution of scientific applications degrades performance. Scheduling of arbitrarily divisible workloads to address load imbalance in order to obtain better utilization of computing resources is a major area of research. Divisible load theory (DLT) and dynamic loop scheduling (DLS) algorithms are two algorithmic approaches employed in the scheduling of arbitrarily divisible workloads. Despite sharing the same goal of achieving load balancing, the two approaches are fundamentally different. Divisible load theory algorithms are linear, deterministic and platform dependent, whereas dynamic loop scheduling algorithms are probabilistic and platform agnostic. Divisible load theory algorithms have been traditionally used for performance prediction in environments characterized by known or expected variation in the system characteristics at runtime. Dynamic loop scheduling algorithms are designed to simultaneously address all the sources of load imbalance that stochastically arise at runtime from application, algorithmic, and systemic characteristics. In this dissertation, an analysis and performance evaluation of DLT and DLS algorithms are presented in the form of a scalability study and a robustness investigation. The effect of network topology on their performance is studied. A hybrid scheduling approach is also proposed that integrates DLT and DLS algorithms. The hybrid approach combines the strength of DLT and DLS algorithms and improves the performance of the scientific applications running in large scale parallel and distributed computing environments, and delivers performance superior to that which can be obtained by applying DLT algorithms in isolation. The range of conditions for which the hybrid approach is useful is also identified and discussed.