The study of waves and instabilities in quantum plasmas is of fundamental importance for understanding collective interactions in superdense astrophysical objects, in high intense laser-plasma/solid-matter interactions, in microelectronic devices and metallic nanostructures. In dense quantum plasmas, there are new pressure laws associated with the Fermi-Dirac distribution functions and new quantum forces associated with the quantum Bohm potential and the Bohr magnetization involving electron ½ spin. These forces significantly alter the collective behavior of dense quantum plasmas. This thesis contains six papers, considering several novel collective modes and instabilities at quantum scales. In Paper I, we have used the quantum hydrodynamical (QHD) model for studying the one-dimensional dust-acoustic (DA) waves incorporating the Fermi pressure law and the quantum Bohm potential. The latter modifies the DA wave dispersion relation in a collisional plasma. In Paper II, we have calculated the electrostatic potential of a test charge in an unmagnetized electron-ion quantum plasma. It is found that the Debye-Hückel and oscillatory wake potentials strongly depend upon the Fermi energy at quantum scales. The results can be of interest for explaining the charged particle attraction and repulsion in degenerate quantum plasmas, such as those in semiconductor and microelectronic devices. Paper III presents the parametric study of nonlinear electrostatic waves in two-dimensional collisionless quantum dusty plasmas. A reductive perturbation method has been employed to the QHD equations together with the Poisson equation, obtaining the cylindrical Kadomtsev-Petviashvili (CKP) equations and their stationary localized solutions. We have numerically examined the quantum mechanical and geometrical effects on the profiles of nonplanar quantum dust-ion-acoustic (DIA) and DA solitary waves. The role of static as well as mobile (negatively or positively charged) dust particles on the low-frequency electrostatic waves has also been highlighted for metallic nanostructures. Paper IV introduces the nonlinear properties of the ion-sound waves in a dense electron-ion Fermi magnetoplasma. The computational analysis of the nonlinear system reveals that the Sagdeev-like potential and the ion-sound density excitations are significantly affected by the wave direction cosine and the Mach number at quantum scales. Paper V considers the nonlinear interactions of electrostatic upper-hybrid (UH), ion-cyclotron (IC), lower-hybrid (LH), and Alfvén waves in a quantum magnetoplasma. The nonlinear dispersion relations have been analyzed analytically to obtain the growth rates for both the decay and modulational instabilities involving the dispersive IC, LH, and Alfvén waves. In Paper VI, we have identified a new drift-like dissipative instability in a collisional quantum plasma. The modified unstable drift-like mode can cause cross-field anomalous ion-diffusion at quantum scales.