In this dissertation, we present a number of studies that primarily focus on data movement challenges among different types of memories (viz., 3D-DRAM, DDRx DRAM and NVM) employed together as a flat-address heterogeneous memory system. We introduce two different hardware-based techniques for prefetching data from slow off-chip phase change memory (PCM) to fast on-chip memories. The prefetching techniques efficiently fetch data from PCM and place that data into processor-resident or 3D-DRAM-resident buffers without putting high demand on bandwidth and provide significant performance improvements. Next, we explore different page migration techniques for flat-address memory systems which differ in when to migrate pages (i.e., periodically or instantaneously) and how to manage the migrations (i.e., OS-based or hardware-based approach). In the first page migration study, we present several epoch-based page migration policies for different organizations of flat-address memories consisting of two (2-level) and three (3-level) types of memory modules. These policies have resulted in significant energy savings. In the next page migration study, we devise an efficient "on-the-fly'" page migration technique which migrates a page from slow PCM to fast 3D-DRAM whenever it receives a certain number of memory accesses without waiting for any specific time interval. Furthermore, we present a light-weight hardware-assisted address reconciliation process for address management of the migrated pages. Such an on-the-fly page migration with hardware-assisted address reconciliation technique provides significant performance improvement over systems using epoch-based page migration and OS-based address management. Finally, we have developed an analytical model, which employs offline analyses of memory access counts per page and recommends whether an application is migration friendly or not. This can be useful in deciding if page migration (either epoch-based or on-the-fly based) should be used or turned off for a given application. Thus, our data management techniques and model enable significant performance improvements for flat-address heterogeneous memory systems involving NVMs.