High performance grid computing is a key enabler of large scale collaborative computational science. With the promise of exascale computing, high performance grid systems are expected to incur electricity bills that grow super-linearly over time. In order to achieve cost effectiveness in these systems, it is essential for the scheduling algorithms to exploit electricity price variations, both in space and time, that are prevalent in the dynamic electricity price markets. Typically, a job submission in the batch queues used in these systems incurs a variable queue waiting time before the resources necessary for its execution become available. In variably-priced electricity markets, the electricity prices fluctuate over discrete intervals of time. Hence, the electricity prices incurred during a job execution will depend on the start and end time of the job. Our thesis consists of two parts. In the first part, we develop a method to predict the start and end time of a job at each system in the grid. In batch queue systems, similar jobs which arrive during similar system queue and processor states, experience similar queue waiting times. We have developed an adaptive algorithm for the prediction of queue waiting times on a parallel system based on spatial clustering of the history of job submissions at the system. We represent each job as a point in a feature space using the job characteristics, queue state and the state of the compute nodes at the time of job submission. For each incoming job, we use an adaptive distance function, which assigns a real valued distance to each history job submission based on its similarity to the incoming job. Using a spatial clustering algorithm and a simple empirical characterization of the system states, we identify an appropriate prediction model for the job from among standard deviation minimization method, ridge regression and k-weighted average. We have evaluated our adaptive prediction framework using historical production workload traces of many supercomputer systems with varying system and job characteristics, including two Top500 systems. Across workloads, our predictions result in up to 22% reduction in the average absolute error and up to 56% reduction in the percentage prediction errors over existing techniques. To predict the execution time of a job, we use a simple model based on the estimate of job runtime provided by the user at the time of job submission. In the second part of the thesis, we have developed a metascheduling algorithm that schedules jobs to the individual batch systems of a grid, to reduce both the electricity prices for the systems and response times for the users. We formulate the metascheduling problem as a Minimum Cost Maximum Flow problem and leverage execution period and electricity price predictions to accurately estimate the cost of job execution at a system. The network simplex algorithm is used to minimize the response time and electricity cost of job execution using an appropriate flow network. Using trace based simulation with real and synthetic workload traces, and real electricity price data sets, we demonstrate our approach on two currently operational grids, XSEDE and NorduGrid. Our experimental setup collectively constitute more than 433K processors spread across 58 compute systems in 17 geographically distributed locations. Experiments show that our approach simultaneously optimizes the total electricity cost and the average response time of the grid, without being unfair to users of the local batch systems. Considering that currently operational HPC systems budget millions of dollars for annual operational costs, our approach which can save $167K in annual electricity bills, compared to a baseline strategy, for one of the grids in our test suite with over 76000 cores, is very relevant for reducing grid operational costs in the coming years.

The future electricity grid is likely to be increasingly complex and uncertain due to the introduction of new technologies in the grid, the increased use of control and communication infrastructure, and the uncertain political climate. In recent years, the transactive energy market framework has emerged as the key framework for future electricity market design in the electricity grid. However, most of the work done in this area has focused on developing retail level transactive energy markets. There seems to be an underlying assumption that wholesale electricity markets are ready to support any retail market design. In this dissertation, we focus on designing wholesale electricity markets that can better support transactive retail market. On the highest level, this dissertation contributes towards developing tools and models for future electricity market designs. A particular focus is placed on the relationship between wholesale markets and investment planning. Part I of this dissertation uses relatively simple models and case studies to evaluate key impediments to flexible transmission operation. In doing so, we identify several potential areas of concern in wholesale market designs: 1. There is a lack of consideration of demand flexibility both in the long-run and in the short-run 2. There is a disconnect between operational practices and investment planning 3. There is a need to rethink forward markets to better manage resource adequacy under long-term uncertainties 4. There is a need for more robust modeling tools for wholesale market design In Part II and Part III of this dissertation, we make use of mathematical decomposition and agent-based simulations to tackle these concerns. Part II of this dissertation uses Benders Decomposition and Lagrangian Decomposition to spatially and temporally decompose a power system and operation problem with active participation of flexible loads. In doing so, we are able to not only improve the computational efficiency of the problem, but also gain various insights on market structure and pricing. In particular, the decomposition suggests the need for a coordinated investment market and forward energy market to bridge the disconnect between operational practices and investment planning. Part III of this dissertation combines agent-based modeling with state-machine based modeling to test various spot, forward, and investment market designs, including the coordinated investment market and forward energy market proposed in Part II of this dissertation. In addition, we test a forward energy market design where 75% of load is required to be purchased in a 2-year-ahead forward market and various transmission cost recovery strategies. We demonstrate how the different market designs result in different investment decisions, winners, and losers. The market insights lead to further policy recommendations and open questions. Overall, this dissertation takes initial steps towards demonstrating how mathematical decomposition and agent-based simulations can be used as part of a larger market design toolbox to gain insights into different market designs and rules for the future electricity grid. In addition, this dissertation identifies market design ideas for further studies, particularly in the design of forward markets and investment cost recovery mechanisms.

This project aims to make a comparison between the Swedish and Indianelectricity market, the design of new improvements will achieve a betteroperation for both markets as well as the price forecasting for markets. Thisresults will give a clear idea about the electricity prices, different energy uses andpeak hours and also the carbon dioxide emissions.Also the main organizations of the market and their roles has been characterized,discussing about the functions of the Market Operator and the System Operator.And also the different markets, the trading products and the price formation havebeen explained and giving an idea about the market structure with enough details.Moreover, Time Series Analysis explained in a detail manner and some of themost used methods in Time Series Analysis are also explained in a very goodmanner. Mainly the results section includes the description of the market situationin Swedish and Indian electricity markets comparison, which includes Powerinstalled capacity, electricity generation, main renewable technologies andpolicies to increase the renewable energy share in total electricity generated.After this analysis, the strengths and weakness of both markets are presented andthe main problems of Swedish electricity system like dependency for nuclearpower, uncertainty for solar electricity generation and the Indian electricitysystem problems like high losses in power system, power quality issues, and veryless focus on energy mix with renewable systems.Finally, due to the quick development of the energy sector in the last few yearsto reach a new design for the electricity market, different kinds ofrecommendations for the future have been considered.

This dissertation is about how Monte Carlo simulation can be used to analyse electricity markets. There are a wide range of applications for simulation; for example, players in the electricity market can use simulation to decide whether or not an investment can be expected to be profitable, and authorities can by means of simulation find out which consequences a certain market design can be expected to have on electricity prices, environmental impact, etc. In the first part of the dissertation, the focus is which electricity market models are suitable for Monte Carlo simulation. The starting point is a definition of an ideal electricity market. Such an electricity market is partly practical from a mathematical point of view (it is simple to formulate and does not require too complex calculations) and partly it is a representation of the best possible resource utilisation. The definition of the ideal electricity market is followed by analysis how the reality differs from the ideal model, what consequences the differences have on the rules of the electricity market and the strategies of the players, as well as how non-ideal properties can be included in a mathematical model. Particularly, questions about environmental impact, forecast uncertainty and grid costs are studied. The second part of the dissertation treats the Monte Carlo technique itself. To reduce the number of samples necessary to obtain accurate results, variance reduction techniques can be used. Here, six different variance reduction techniques are studied and possible applications are pointed out. The conclusions of these studies are turned into a method for efficient simulation of basic electricity markets. The method is applied to some test systems and the results show that the chosen variance reduction techniques can produce equal or better results using 99% fewer samples compared to when the same system is simulated without any variance reduction technique. More complex electricity market models cannot directly be simulated using the same method. However, in the dissertation it is shown that there are parallels and that the results from simulation of basic electricity markets can form a foundation for future simulation methods. Keywords: Electricity market, Monte Carlo simulation, variance reduction techniques, operation cost, reliability. / QC 20100608

In the future, renewable energy share, especially wind power share, in electricity generation is expected to increase. Due to nature of the wind, wind power generation pattern includes uncertainties which affects the energy prices in the electricity markets. New simulations are needed for efficient planning process for the resources in the power systems to address the uncertainties in demand, generation, legal, economical and technical limitations. In this study, the aspects of planning process for wind power generation is described and some example scenarios are implemented with the help of MATLAB software.

This thesis addresses two main issues: first, forecasting short-term electricity market prices; and second, the application of short-term electricity market price forecasts to operation planning of demand-side Bulk Electricity Market Customers (BEMCs). The Ontario electricity market is selected as the primary case market and its structure is studied in detail. A set of explanatory variable candidates is then selected accordingly, which may explain price behavior in this market. In the process of selecting the explanatory variable candidates, some important issues, such as direct or indirect effects of the variables on price behavior, availability of the variables before real-time, choice of appropriate forecasting horizon and market time-line, are taken into account. Price and demand in three neighboring electricity markets, namely, the New York, New England, and PJM electricity markets, are also considered among the explanatory variable candidates. Electricity market clearing prices in Ontario are calculated every five minutes. However, the hourly average of these 5-minute prices, referred to as the Hourly Ontario Energy Price (HOEP), applies to most Ontario market participants for financial settlements. Therefore, this thesis concentrates on forecasting the HOEP by employing various linear and non-linear modeling approaches. The multivariate Transfer Function (TF), the multivariate Dynamic Regression (DR), and the univariate Auto Regressive Integrated Moving Average (ARIMA) are the linear time series models examined. The non-linear approaches comprise the Multivariate Adaptive Regression Splines (MARS), and the Multi-Layer Perceptron (MLP) neural networks. Multivariate HOEP models are developed considering two forecasting horizons, i.e. 3 hours and 24 hours, taking into account the case market time-line and the ability of market participants to react to the generated forecasts. Univariate ARIMA models are also developed for day-ahead market prices in the three neighboring electricity markets. The developed models are used to generate price forecasts for low-demand, summer peak-demand, and winter peak-demand periods. The HOEP forecasts generated in this work are significantly more accurate than any other available forecast. However, the accuracy of the generated HOEP forecasts is relatively lower than those of the price forecasts for Ontario's neighboring electricity markets. The low accuracy of the HOEP forecasts is explained by conducting a price volatility analysis across the studied electricity markets. This volatility analysis reveals that the Ontario electricity market has the most volatile prices compared to the neighboring electricity markets. The high price volatility of the Ontario electricity market is argued to be the direct result of the real-time nature of this market. It is further observed that the inclusion of the just-in-time publicly available data in multivariate HOEP models does not improve the HOEP forecast accuracy significantly. This lack of significant improvement is attributed to the information content of the market data which are available just-in-time. The generated HOEP forecasts are used to plan the short-term operation of two typical demand-side case-study BEMCs. The first case-study BEMC is a process industry load with access to on-site generation facilities, and the second one is a municipal water plant with controllable electric demand. Optimization models are developed for the next-day operation of these BEMCs in order to minimize their total energy costs. The optimization problems are solved when considering market price forecasts as the expected future prices for electricity. The economic impact of price forecast inaccuracy on both the case study is analyzed by introducing the novel Forecast Inaccuracy Economic Impact (FIEI) index. The findings of this analysis show that electricity market price forecasts can effectively be used for short-term scheduling of demand-side BEMCs. However, sensitivity to price forecast inaccuracy significantly varies across market participants. In other words, a set of price forecasts may be considered ``accurate enough'' for a customer, while leading to significant economic losses for another.

In the current era of deregulated electricity markets, the power distribution systems have attained a very important and crucial role in the industry. A distribution company (referred to as a disco) plays an active and effective role in electricity markets, and can positively impact the market efficiency and make it more reliable, secure and beneficial to customers. Therefore, operation and planning issues of discos in such electricity market environment requires extensive analysis and research in order to improve their operational strategies both in the short-term and long-term. A generic operations framework for a disco operating in a competitive electricity market environment is presented in the thesis. The operations framework is a two-stage hierarchical model in which the first stage deals with disco’s activities in the day-ahead stage, the Day Ahead Operations Model (DAOM). The second stage deals with disco’s activities in real-time and is termed Real-Time Operations Model (RTOM). The DAOM determines the disco’s operational decisions on grid purchase, scheduling of distributed generation (DG) units owned by it, and contracting for interruptible load. These decisions are imposed as boundary constraints in the RTOM and the disco seeks to minimize its short-term costs keeping in mind its day-ahead decisions. A case-study is presented considering the well-known 33-bus distribution system and three different scenarios are constructed to analyze the disco’s actions and decision-making in this context. The thesis presents a new paradigm for distribution system operation taking into account the presence of DG sources and their goodness factors. The proposed concept of goodness factor of DG units is based on the computation of the incremental contribution of a DG unit to distribution system losses. The incremental contributions of a DG unit to active and reactive power losses in the distribution system are termed as the active / reactive Incremental Loss Indices (ILI). The goodness factors are integrated directly into the distribution system operations model. This model seeks to minimize the disco’s energy costs in the short-term taking into account the contribution (goodness factor) of each DG unit. The analysis was carried out considering an 18-bus distribution network, considering two different ownership structures of DG units, and a 69-bus distribution system considering specific characteristics of wind-DG units. The concept of goodness factors is further extended to determine a new set of goodness factors pertaining to a DG’s impact on feeder unloading by virtue of its power injection. A novel long-term planning model has been developed for the disco that considers investments in DG capacity, distribution system feeder addition / expansion and substation transformers capacity addition. The model includes the new set of goodness factors pertaining to both loss reduction and feeder unloading and arrives at an optimal set of new expansion plan, with specified locations, and year of commissioning. The work clearly demonstrates the effectiveness and contribution of DG units in distribution systems both in the short-term and long-term framework.

This thesis presents a multi-stage linear stochastic mixed integer programming (SMIP) model for planning power generation in a pool-type day-ahead electricity market. The model integrates a reserve demand curve and shares most of the features of a stochastic unit commitment (UC) problem, which is known to be NP-hard. We capture the stochastic nature of the problem through scenarios, resulting in a large-scale mixed integer programming (MIP) problem that is computationally challenging to solve. Given that an independent system operator (ISO) has to solve such a problem within a time requirement of an hour or so, in order to release operating schedules for the next day real-time market, the problem has to be solved efficiently. For that purpose, we use some approximations to maintain the linearity of the model, parsimoniously select a subset of scenarios, and invoke realistic assumptions to keep the size of the problem reasonable. Even with these measures, realistic-size SMIP models with binary variables in each stage are still hard to solve with exact methods. We, therefore, propose a scenario-rolling heuristic to solve the SMIP problem. In each iteration, the heuristic solves a subset of the scenarios, and uses part of the obtained solution to solve another group in the subsequent iterations until all scenarios are solved. Two numerical examples are provided to test the performance of the scenario-rolling heuristic, and to highlight the difference between the operative schedules of a deterministic model and the SMIP model. Motivated by previous studies on pricing MIP problems and their applications to pricing electric power, we investigate pricing issues and compensation schemes using MIP formulations in the second part of the thesis. We show that some ideas from the literature can be applied to pricing energy/reserves for a relatively realistic model with binary variables, but some are found to be impractical in the real world. We propose two compensation schemes based on the SMIP that can be easily implemented in practice. We show that the compensation schemes with make-whole payments ensure that generators can have non-negative profits. We also prove that under some assumptions, one of the compensation schemes has the interesting theoretical property of minimizing the variance of the profit of generators to zero. Theoretical and numerical results of these compensation schemes are presented and discussed.

In deregulated electricity markets, generating companies submit energy bids which are derived from a self-schedule. In this thesis, we propose an improved semidefinite programming-based model for the self-scheduling problem. The model provides the profit-maximizing generation quantities of a single generator over a multi-period horizon and accounts for uncertainty in prices using robust optimization. Within this robust framework, the price information is represented analytically as an ellipsoid. The risk-adversity of the decision maker is taken into account by a scaling parameter. Hence, the focus of the model is directed toward the trade-off between profit and risk. The bounds obtained by the proposed approach are shown to be significantly better than those obtained by a mean-variance approach from the literature. We then apply the proposed model within a branch-and-bound algorithm to improve the quality of the solutions. The resulting solutions are also compared with the mean-variance approach, which is formulated as a mixed-integer quadratic programming problem. The results indicate that the proposed approach produces solutions which are closer to integrality and have lower relative error than the mean-variance approach.

This thesis addresses two main issues: first, forecasting short-term electricity market prices; and second, the application of short-term electricity market price forecasts to operation planning of demand-side Bulk Electricity Market Customers (BEMCs). The Ontario electricity market is selected as the primary case market and its structure is studied in detail. A set of explanatory variable candidates is then selected accordingly, which may explain price behavior in this market. In the process of selecting the explanatory variable candidates, some important issues, such as direct or indirect effects of the variables on price behavior, availability of the variables before real-time, choice of appropriate forecasting horizon and market time-line, are taken into account. Price and demand in three neighboring electricity markets, namely, the New York, New England, and PJM electricity markets, are also considered among the explanatory variable candidates. Electricity market clearing prices in Ontario are calculated every five minutes. However, the hourly average of these 5-minute prices, referred to as the Hourly Ontario Energy Price (HOEP), applies to most Ontario market participants for financial settlements. Therefore, this thesis concentrates on forecasting the HOEP by employing various linear and non-linear modeling approaches. The multivariate Transfer Function (TF), the multivariate Dynamic Regression (DR), and the univariate Auto Regressive Integrated Moving Average (ARIMA) are the linear time series models examined. The non-linear approaches comprise the Multivariate Adaptive Regression Splines (MARS), and the Multi-Layer Perceptron (MLP) neural networks. Multivariate HOEP models are developed considering two forecasting horizons, i.e. 3 hours and 24 hours, taking into account the case market time-line and the ability of market participants to react to the generated forecasts. Univariate ARIMA models are also developed for day-ahead market prices in the three neighboring electricity markets. The developed models are used to generate price forecasts for low-demand, summer peak-demand, and winter peak-demand periods. The HOEP forecasts generated in this work are significantly more accurate than any other available forecast. However, the accuracy of the generated HOEP forecasts is relatively lower than those of the price forecasts for Ontario's neighboring electricity markets. The low accuracy of the HOEP forecasts is explained by conducting a price volatility analysis across the studied electricity markets. This volatility analysis reveals that the Ontario electricity market has the most volatile prices compared to the neighboring electricity markets. The high price volatility of the Ontario electricity market is argued to be the direct result of the real-time nature of this market. It is further observed that the inclusion of the just-in-time publicly available data in multivariate HOEP models does not improve the HOEP forecast accuracy significantly. This lack of significant improvement is attributed to the information content of the market data which are available just-in-time. The generated HOEP forecasts are used to plan the short-term operation of two typical demand-side case-study BEMCs. The first case-study BEMC is a process industry load with access to on-site generation facilities, and the second one is a municipal water plant with controllable electric demand. Optimization models are developed for the next-day operation of these BEMCs in order to minimize their total energy costs. The optimization problems are solved when considering market price forecasts as the expected future prices for electricity. The economic impact of price forecast inaccuracy on both the case study is analyzed by introducing the novel Forecast Inaccuracy Economic Impact (FIEI) index. The findings of this analysis show that electricity market price forecasts can effectively be used for short-term scheduling of demand-side BEMCs. However, sensitivity to price forecast inaccuracy significantly varies across market participants. In other words, a set of price forecasts may be considered ``accurate enough'' for a customer, while leading to significant economic losses for another.