Thesis advisor: Gabor Kalman / Complex plasma systems have been studied for a long time. In this thesis we focus on a quasi-2D layer system. In fact, most experimental studies of complex plasmas are based on 2D systems, because it is easy to use camera to record the in-plane movement of particles. Unfortunately, due to the finite confining strength, the system is not a strictly 2D layer, it is a quasi-2D layer. We firstly studied the density profile of such a quasi-2D system by density functional theory(DFT). From the density profile research result, we found that the system can form a trilayer structure with proper parameters. Then we studied the dynamical properties of a trilayer system, and for simplicity, we only studied an ideal three layer model, both in liquid and lattice case. In lattice case, we firstly searched the stable lattice structure at different inter-layer distance. Then we used lattice sites summation to construct the dynamical matrix and solve the dispersion relation. For liquid case, we did the theoretical prediction for the collective dispersion by quasi localized charge approximation(QLCA), then we extracted the collective mode information from the molecular dynamics(MD) simulation. The QLCA and MD results were compared and discussed. The reason for the previous gap discrepancy problem is discovered. / Thesis (PhD) — Boston College, 2019. / Submitted to: Boston College. Graduate School of Arts and Sciences. / Discipline: Physics.