In recent years, deep geothermal energy has been widely exploited through closed-loop borehole heat exchanger system for building heating. In order to precisely evaluate the sustainable heat extraction capacity and the impact of different designs and operating parameters, two heat transfer models are implemented in the open-source scientific software OpenGeoSys (OGS), with respect to the Deep Borehole Heat Exchanger (DBHE) and Enhanced U-tube Borehole Heat Exchanger (EUBHE) system. Besides, three types of boundary conditions are implemented, including the constant inflow temperature, the constant heat extraction rate, and constant building thermal power that integrates the ground source heat pump (GSHP) module. By applying the two BHE models, the influence of different designs and operating parameters on the GSHP system is evaluated. The sustainable heat extraction capacity and efficiency of a deep EUBHE system are predicted. Moreover, its performance and efficiency are further compared against the 2-DBHE array system that has the same total borehole length. It is found that the soil thermal conductivity is the most important parameter in the design of DBHE and EUBHE systems. The sustainable specific heat extraction rate of the EUBHE system is 86.5 W/m higher than an array with 2 DBHEs. Under the building thermal load of 1.225 MW, the total electricity consumed by the EUBHE system is approximately 27 % less than the 2-DBHE array over 10 years. The average Coefficient of System Performance (CSP) value of the EUBHE system is 1.66 higher over 10 heating seasons. The two numerical models implemented in the OpenGeoSys software can be used to predict and optimize the thermal characteristics of the closed-loop DBHE and EUBHE systems in real projects.