Polymer nanocomposite technology has had significant impact on material design. With the environmental advantages of photopolymerization, a research has recently focused on producing nanocomposites utilizing inexpensive clay particles based on in situ photopolymerization. In this research, novel polymerizable organoclays and thiol-ene photopolymerization have been utilized to develop advanced photopolymer clay nanocomposites and to overcome several limitations in conventional free radical photopolymers. To this end, factors important in nanocomposite processes such as monomer composition, clay dispersion, and photopolymerization behavior in combination with the evolution of ultimate nanocomposite properties have been investigated. For monomer-organoclay compositions, higher chemical compatibility of components induces enhanced clay exfoliation, resulting in photopolymerization rate increases due to an amplified clay template effect. Additionally, by affecting the stoichiometric ratio between thiol and acrylate double bond in the clay gallery, thiolated organoclays enhance thiol-ene copolymerization with increased final thiol conversion while acrylated organoclays encourage acrylate homopolymerization. In accordance with the reaction behavior, incorporation of thiolated organoclays makes polymer chains more flexible with decreased glass transition temperature due to higher formation of thio-ether linkages while adding acrylated organoclays significantly increases the modulus. Photopolymer nanocomposites also help overcome two major drawbacks in conventional free radical photopolymerization, namely severe polymerization shrinkage and oxygen inhibition during polymerization. With addition of a low level of thiol monomers, the oxygen inhibition in various acrylate systems can be overcome by addition of only 5wt% thiolated organoclay. The same amount of polymerizable organoclay also induces up to 90% decreases in the shrinkage stress for acrylate or thiol-acrylate systems. However, nonreactive clays do not reduce the stress substantially and even decreases the polymerization rate in air. Additionally, the clay morphology and polymerization behavior are closely related with evolution of ultimate nanocomposite performance. Use of polymerizable organoclay significantly improves overall toughness of nanocomposites by increasing either modulus or elongation at break based on the type of polymerizable organoclay, which demonstrates the promise of this technology as a modulation and/or optimization tool for nanocomposite properties.