Nowadays, electricity markets are becoming more deregulated, especially development of smart grid and introduction of renewable energy promote regulations of energy markets. On the other hand, the uncertainties of new energy sources and market participants’ bidding bring more challenges to power system operation and transmission system planning. These problems motivate us to study spot price (also called locational marginal pricing) of electricity markets, the strategic bidding of wind power producer as an independent power producer into power market, transmission expansion planning considering wind power investment, and analysis of the maximum loadability of a power grid. The work on probabilistic spot pricing for a utility grid includes renewable wind power generation in a deregulated environment, taking into account both the uncertainty of load forecasting and the randomness of wind speed. Based on the forecasted normal-distributed load and Weibull-distributed wind speed, probabilistic optimal power flow is formulated by including spinning reserve cost associated with wind power plants and emission cost in addition to conventional thermal power plant cost model. Simulations show that the integration of wind power can effectively decrease spot price, also increase the risk of overvoltage. Based on the concept of loacational marginal pricing which is determined by a marketclearing algorithm, further research is conducted on optimal offering strategies for wind power producers participating in a day-ahead market employing a stochastic market-clearing algoivrithm. The proposed procedure to drive strategic offers relies on a stochastic bilevel model: the upper level problem represents the profit maximization of the strategic wind power producer, while the lower level one represents the marketing clearing and the corresponding price formulation aiming to co-optimize both energy and reserve. Thirdly, to improve wind power integration, we propose a bilevel problem incorporating two-stage stochastic programming for transmission expansion planning to accommodate large-scale wind power investments in electricity markets. The model integrates cooptimizations of energy and reserve to deal with uncertainties of wind power production. In the upper level problem, the objective of independent system operator (ISO) modelling transmission investments under uncertain environments is to minimize the transmission and wind power investment cost, and the expected load shedding cost. The lower level problem is composed of a two stage stochastic programming problem for energy schedule and reserve dispatch simultaneously. Case studies are carried out for illustrating the effectiveness of the proposed model. The above market-clearing or power system operation is based on direct current optimal power flow (DC-OPF) model which is a linear problem without reactive power constraints. Power system maximum loadability is a crucial index to determine voltage stability. The fourth work in this thesis proposes a Lagrange semi-definite programming (SDP) method to solve the non-linear and non-convex optimization alternating current (AC) problem of the maximum loadability of security constrained power system. Simulation results from the IEEE three-bus system and IEEE 24-bus Reliability Test System (RTS) show that the proposed method is able to obtain the global optimal solution for the maximum loadability problem. Lastly, we summarize the conclusions from studies on the above mentioned optimization problems of electric power market under modern grid, as well as the influence of wind power integration on power system reliability, and transmission expansion planning, as well as the operations of electricity markets. Meanwhile, we also present some open questions on the related research, such as non-convex constraints in the lower-level problem of a bilevel problem, and integrating N-1 security criterion of transmission planning. / Graduate / lei.ming296@gmail.com