Density wave oscillations (DWO) are a class of two-phase flow instabilities which can pose significant safety concerns to boiling water reactors (BWR). During an anticipated transient without scram (ATWS) while operating in the proposed extended operating domain MELLLA+, natural circulation conditions can potentially lead to DWO-type instabilities which have the capability to develop into cycles of fuel surface dryout and rewet, damaging core integrity. In order to provide data on these phenomena, a series of tests were performed at the KATHY facility during which DWO was developed with and without simulated neutronic feedback. In this dissertation, the data provided by these tests is analyzed to determine the onset conditions for DWO. Following this, several models are assessed for their capability in predicting this stability boundary compared to the experimental results. The models were chosen in order to provide a suitably large range of prediction methodologies. Two analytical drift-flux models developed with and without thermal equilibrium are shown, with respective differences compared. A computational model of the full KATHY natural circulation loop is built using the 1D thermal-hydraulics code TRACE. This is adapted with a point-kinetics model for neutronic feedback for experimental comparison. With both the analytical models and the TRACE model, a series of parametric studies are performed showing the effects of inlet/outlet flow restrictions, pressure, channel geometry, and axial power profile on the stability boundary. Finally, two machine learning neural network-based models are developed and trained on various subsets of the experimental data. The results from each model showed certain benefits and drawbacks based on model complexity and physicality. / Doctor of Philosophy / Certain conditions in the core of a boiling water reactor (BWR) can lead to unstable flows due to the high ratio between the power and the coolant flow rate. These instabilities, called density wave oscillations (DWO), have been shown to occur during a specific accident scenario known as an anticipated transient without scram (ATWS) when the reactor is operating in a lower flow domain called MELLLA+. In this accident, pump flow through the core is halted, but the reactor is not shut down. This can lead to serious safety concerns if left unaddressed. To analyze these instabilities, the KATHY facility performed a series of tests with and without power feedback from simulated neutron response. In this dissertation, the onset conditions from these tests are given and compared to several models for predicting the stability boundary. Two analytical models proposed by Ishii and Saha are compared and the effect of certain parameters on the stability is assessed. Next, a model of the KATHY loop is built using the thermal-hydraulics code TRACE both with and without simulated power feedback. Finally, two types of machine learning models are developed to determine their accuracy in predicting the instability conditions. The overall performance of each is compared and their benefits and drawbacks are highlighted.