Heavy oil derived from oil sands is becoming an important resource of energy and transportation fuels due to the depletion of conventional oil resources. However, bitumen and heavy oils have a low hydrogen/carbon ratio and contain a large percentage of sulfur and nitrogen heterocyclic compounds. At the level of deep desulfurization, aromatic poly-nuclear molecules, especially nitrogen-containing heterocyclic compounds, exhibit strong inhibitive effect on hydrodesulfurization (HDS) due to competitive adsorption on catalytically active sites with sulfur-containing molecules. Therefore, it is necessary to study the HDS of refractory sulfur-containing compounds and also the effect of nitrogen-containing species on the deep HDS for achieving the ultra low sulfur specifications for transportation fuels. Additionally, the cost of H2 increased in recent years and a bitumen emulsion upgrading technique using an alternative in-situ H2 generated via the water gas shift (WGS) reaction during the hydro-treating was developed in our group. In the present study, a kind of nano-dispersed unsupported MoSx based catalyst was developed and used for hydrodesulfurization, hydrodenitrogenation (HDN) and upgrading bitumen emulsions. Objectives of this thesis were to (1) improve the catalytic activity of the nano-dispersed Mo based catalysts towards the HDS and HDN reactions of refractory sulfur-/nitrogen-containing compounds; and (2) compare the reactivity of in-situ hydrogen generated via the WGS reaction versus externally provided molecular hydrogen in HDS and HDN reactions to improve the efficiency of the bitumen emulsion upgrading technology developed by our group. In the present study, to stimulate the reaction system of bitumen emulsion, water was added into the organic reaction system, so there are different phases in this reaction system. To investigate the activity of the catalyst, the catalyst particles dispersed in different phases were characterized separatedly via HRTEM-EDX. After HRTEM-EDX study, all phases were mixed up and dried for further characterizations, BET, SEM, and XRD. The catalyst prepared in in-situ hydrogen was found to have higher surface area and smaller particle size than the one made in molecular hydrogen. The presence of sulfur-/nitrogen-containing compounds in the preparation system caused significant changes in the morphology of dispersed Mo sulfide catalyst according to HRTEM observations. Refractory sulfur-containing compounds of dibenzothiophene (DBT) and 4,6-dimethyldibenzothiophene (4,6-DMDBT) were used as model compounds in HDS studies. The simultaneous HDS of both model compounds was performed at different reaction temperatures from 330??C to 400??C. The effect of the reaction temperature on the WGS reaction in the presence of sulfur-containing model compounds was reported. A kinetic model for HDS reactions was proposed and used in discussing experiment results. The relative HDS reactivity of 4,6-DMDBT to DBT using dispersed Mo sulfide catalyst in in-situ hydrogen was found to be higher than the reported results which were obtained over supported catalysts. Nickel and potassium were introduced into Mo sulfide catalysts as promoters and their effect on the WGS reaction and the HDS reaction were discussed. The simultaneous HDS was carried out in the two different hydrogen sources. The in-situ hydrogen reaction system showed higher conversion and desulfurization results of both sulfur model compounds. This observation has been found to be mainly contributed by the higher activity of the Mo sulfide catalyst prepared in in-situ H2. Strong inhibitive effect of nitrogen-containing compounds, basic quinoline or non-basic carbazole, on the HDS of refractory sulfur model compounds was observed and discussed. Basic quinoline was a much stronger inhibitor than non-basic carbazole. The two HDS reaction pathways were affected by nitrogen-containing compounds to different extents. The HDN of quinoline over the dispersed Mo sulfide catalyst using in-situ hydrogen had been studied extensively by a previous member in our group. In this thesis, the HDN of carbazole was studied. From the identification of HDN products of carbazole, a HDN reaction network was proposed. The HDN of carbazole was processed at different reaction temperatures. The WGS reaction was not inhibited in the presence of carbazole. Comparable reactivity of the two hydrogen sources towards the HDN of carbazole was observed. The presence of 4,6-DMDBT caused significant effect on the HDN of carbazole due to the competitive adsorption on the catalyst surface.