This thesis demonstrates the advantages of using the Energy-dispersive X-ray fluorescence (ED-XRF) system to quantify the mercury content in fiber banks at first. The Monte Carlo N-Particle (MCNP) code was then used to simulate the XRF system model with suitable parameters such as the input X-ray energy level, the detector material, and the environmental factor (water depth). The SNR results of the mercury spectrum when applying different parameters were obtained. Then, the limit of detection (LOD) and limit of quantification (LOQ) based on the SNR approach are considered. Finally, system parameters were determined in order to obtain more accurate qualitative and quantitative analysis results for future environmental measurements.