Eukaryotic genomes can produce two types of transcripts: protein-coding and non-coding RNAs (ncRNAs). Cryptic ncRNA transcripts are bona fide RNA Pol II products that originate from bidirectional promoters, yet they are degraded by the RNA exosome. Such pervasive transcription is prevalent across eukaryotes, yet its regulation and function is poorly understood. We hypothesized that chromatin architecture at cryptic promoters may regulate ncRNA transcription. Nucleosomes that flank promoters are highly enriched in two histone marks: H3-K56Ac and the variant H2A.Z, which make nucleosomes highly dynamic. These histone modifications are present at a majority of promoters and their stereotypic pattern is conserved from yeast to mammals, suggesting their evolutionary importance. Although required for inducing a handful of genes, their contribution to steady-state transcription has remained elusive. In this work, we set out to understand if dynamic nucleosomes regulate cryptic transcription and how this is coordinated with the RNA exosome. Remarkably, we find that H3-K56Ac promotes RNA polymerase II occupancy at a large number of protein coding and noncoding loci, yet neither histone mark has a significant impact on steady state mRNA levels in budding yeast. Instead, broad effects of H3-K56Ac or H2A.Z on levels of both coding and ncRNAs are only revealed in the absence of the nuclear RNA exosome. We show that H2A.Z functions with H3-K56Ac in chromosome folding, facilitating formation of Chromosomal Interaction Domains (CIDs). Our study suggests that H2A.Z and H3-K56Ac work in concert with the RNA exosome to control mRNA and ncRNA levels, perhaps in part by regulating higher order chromatin structures. Together, these chromatin factors achieve a balance of RNA exosome activity (yin; negative) and Pol II (yang; positive) to maintain transcriptional homeostasis.