Spin systems with low dimensionality and geometrical frustration have attracted interest because of the possible emergence of novel magnetic phases at low temperatures by suppressing conventional magnetic order, and allow for a novel spin-disordered ground state, such as quantum spin liquid and glass without any apparent structural disorder. In this thesis, we have discussed magnetic and dielectric measurements on the quasi-two-dimensional triangular antiferromagnet NiGa2S3.85, in order to investigate its magnetic state and dielectric property at low temperatures. In the measurement of susceptibility In the measurement of susceptibility £q(T), we found that the freezing temperature is near 6 K. This suggests that the sample with the sulfur deficiency should be close to the stoichiometric NiGa2S3.85. Under 7 T magnetic filed ZFC(zero-field-cooling) and FC(field-cooling) still bifurcate indicating more magnetic order state than spin-liquid. Temperature-dependent dielectric measurement shows an interesting colossal enhancement of dielectric constant with frequency dispersion and is ascribed to the interfacial polarization (termed as Maxwell-Wagner relaxation) at the interface between adjacent layers. Temperature-dependent dielectric measurement under applied magnetic field (up to 9 T) shows negligible magnetodielectric behavior because of the sample is not magneto resistive.