The submicron-aperture fiber point-diffraction interferometer (SFPDI) can be applied to realize the measurement of three-dimensional absolute displacement within large range, in which the performance of point-diffraction wavefront and numerical iterative algorithm for displacement reconstruction determines the achievable measurement accuracy, reliability and efficiency of the system. A method based on fast searching particle swarm optimization (FS-PSO) algorithm is proposed to realize the rapid measurement of three-dimensional absolute displacement. Based on the SFPDI with two submicron-aperture fiber pairs, FS-PSO method and the corresponding model of the SFPDI, the measurement accuracy, reliability and efficiency of the SFPDI system are significantly improved, making it more feasible for practical application. The effect of point-diffraction wavefront error on the measurement is analyzed. The error of point-diffraction wavefront obtained in the experiment is in the order of 1x10(-4). (the wavelength. is 532 nm), and the corresponding displacement measurement error is smaller than 0.03 mu m. Both the numerical simulation and comparison experiments have been carried out to demonstrate the accuracy and feasibility of the proposed SFPDI system, high measurement accuracy in the order of 0.1 mu m, convergence rate (similar to 90.0%) and efficiency have been realized with the proposed method, providing a feasible way to measure three-dimensional absolute displacement in the case of no guide rail.