Doctor of Philosophy / Department of Mathematics / Yan Soibelman / Given a quiver Q with/without potential, one can construct an algebra structure on the cohomology of the moduli stacks of representations of Q. The algebra is called Cohomological Hall algebra (COHA for short). One can also add a framed structure to quiver Q, and discuss the moduli space of the stable framed representations of Q. Through these geometric constructions, one can construct two representations of Cohomological Hall algebra of Q over the cohomology of moduli spaces of stable framed representations. One would get the double of the representations of Cohomological Hall algebras by putting these two representations together. This double construction implies that there are some relations between Cohomological Hall algebras and some other algebras. In this dissertation, we focus on the quiver without potential case. We first define Cohomological Hall algebras, and then the above construction is stated under some assumptions. We computed two examples in detail: A₁-quiver and Jordan quiver. It turns out that A₁-COHA and its double representations are related to the half infinite Clifford algebra, and Jordan-COHA and its double representations are related to the infinite Heisenberg algebra. Then by the fact that the underlying vector spaces of these two COHAs are isomorphic to each other, we get a COHA version of Boson-Fermion correspondence.

Doctor of Philosophy / Department of Mathematics / Yan S. Soibelman / The motivic Donaldson-Thomas theory of 2-dimensional Calabi-Yau categories can be induced from the theory of 3-dimensional Calabi-Yau categories via dimensional reduction. The cohomological Hall algebra is one approach to the motivic Donaldson-Thomas invariants. Given an arbitrary quiver one can construct a double quiver, which induces the preprojective algebra. This corresponds to a 2-dimensional Calabi-Yau category. One can further construct a triple quiver with potential, which gives rise to a 3-dimensional Calabi-Yau category. The critical cohomological Hall algebra (critical COHA for short) is defined for a quiver with potential. Via the dimensional reduction we obtain the cohomological Hall algebra (COHA for short) of the preprojective algebra. We prove that a subalgebra of this COHA consists of a semicanonical basis, thus is related to the generalized quantum groups. Another approach is motivic Hall algebra, from which an integration map to the quantum torus is constructed. Furthermore, a conjecture concerning some invariants of 2-dimensional Calabi-Yau categories is made. We investigate the correspondence between the A∞-equivalent classes of ind-constructible 2-dimensional Calabi-Yau categories with a collection of generators and a certain type of quivers. This implies that such an ind-constructible category can be canonically reconstructed from its full subcategory consisting of the collection of generators.