<p> Effects of electron correlations on a two dimensional quantum spin Hall system are studied. We examine possible phases of a generalized Hubbard model on a bilayer honeycomb lattice with a spin-orbit coupling and short range electron-electron repulsions at half filling, based on the slave rotor mean-field theory. The phase diagram of the model is found for a special case where the interlayer Coulomb repulsion is comparable to the intralayer Coulomb repulsion.</p> <p> Besides the conventional quantum spin Hall phase and a broken-symmetry insulating phase, we find a new phase, a fractionalized quantum spin Hall phase, where the quantum spin Hall effect arises for fractionalized spinons which carry only spin but not charge. Experimental manifestations of the exotic phase and effects of fluctuations beyond the saddle point approximation are also discussed.</p> <p> We finally study a toy Bose-Hubbard model for the charge sector of the theory to gain some insight into the phase diagram away from the special Coulomb repulsion values.</p> / Thesis / Master of Science (MSc)