In simulation of closely- or separately-joined rock masses, stability of rock blocks is of primary concern. However, there seems to be no approach that can handle general modes of simultaneous sliding and truly large rotation under general forces, including non-conservative forces such as water forces. General causes of failure for rock blocks, such as limit points, bifurcation points, and dynamic instability (divergence and flutter), have never been addressed. This research implements a formulation, called BS3D(an incremental-iterative algorithm introduced by Tonon), for analyzing general failure modes of rock blocks under conservative and non-conservative forces. Among the constitutive models for rock fractures developed over the years, Barton's empirical model has been widely used because it is easy to apply and includes several important factors associated with fracture characteristics. Although Barton's failure criterion predicts peak shear strength of rock fractures with acceptable precision, it has some weaknesses in estimating the peak shear displacement, post-peak shear strength, dilation, and surface degradation in unloading and reloading. In this dissertation, modifications are made to Barton's original model in order to address these weaknesses. The modified Barton’s model is validated by a series of direct shear tests on rock fractures and implemented in BS3D to consider the dilatant behavior of fractures. The mechanical behavior of a rock block formed in the roof of a tunnel is governed by its geometry, the mechanical characteristics and the deformability of the fractures forming the block, the deformability of the block and that of the surrounding rock mass, and the stresses within the rock. BS3D, after verification and validation, is used to investigate the effect of dilatancy on stability of rock blocks formed in the roof of a circular tunnel. High-velocity plunging jets, issuing from hydraulic artificial or natural structures, can result in scouring of the rock riverbed or the dam toe foundation. Assessment of the extent of scour is necessary to ensure the safety of the dam and to guarantee the stability of its abutments. BS3D is used to investigate effect of high-velocity jet impact on stability of rock blocks in plunge pools. / text