Docking is a computer simulation method used to predict the preferred orientation of two interacting chemical species that has been successfully applied to numerous macromolecules over the years. However, non-traditional targets have inherent difficulties associated with their screening. Large interfaces, lack of obvious binding sites, and transient pockets are some examples. Additionally, most natural ligands of challenging targets are inadequate models for identifying or designing new ligands. Therefore, it is not surprising that customary techniques of structure-based virtual screening are incompatible with these non-traditional targets. We hypothesized that an integrative virtual screening campaign comprised of docking followed by refinement of best receptor–ligand complexes would effectively identify small-molecule ligands of challenging receptors. We targeted the single-stranded DNA (ssDNA) binding groove of the human RAD52, and a cryptic allosteric pocket of the Helicobacter pylori Glutamate Racemase (GR). In this project, we first determined which docking method was more appropriate for each studied non-traditional target, and then examined how good our two-step docking workflow was in finding novel active ligand scaffolds. This research developed a powerful layered virtual screening workflow for the discovery of lead compounds against challenging protein targets. Furthermore, we successfully applied a statistical analysis method, which used receiver operating characteristic (ROC) curves, to validate the selected docking protocol that would be used in the screening campaigns. Using the validated workflow, we identified a natural compound that competes with ssDNA to bind to RAD52. The performed screening campaigns also provided new insights into the studied binding pockets, as well as structure-activity relationships (SAR) and binding determinants of the ligands. Our achievements reinforce the power of the ROC curve analysis approach in directing the search for the most appropriate docking protocol and helping to speed up drug discovery in pharmaceutical research.