The structure and spatial arrangement of grain boundaries strongly affect the properties of polycrystalline materials such as corrosion, creep, weldability, superconductivity, and diffusivity. However, constructing predictive grain boundary structure-property models is taxing, both experimentally and computationally due to the high dimensionality of the grain boundary character space. The purpose of this work is to develop an effective method to infer grain boundary structure-property models from measurement of the effective properties of polycrystals by utilizing the inverse problem theory. This study presents an idealized case in which structure-property models for grain boundary diffusivity are inferred from a noisy simulation. The method presented in this study is derived from a general mathematical expression of inverse problem theory. The derivation of the method is carried step by step by considering diffusivity as the property of interest. The use of the Bayesian probability approach in the inference method makes the uncertainty quantification possible to perform. This study demonstrates how uncertainty quantification for the inferred structure-property models is easily performed within the idealized case framework. The method of quantifying the uncertainty is carried by utilizing the Metropolis-Hastings algorithm and Kernel Density Estimation method. The validation of the method is carried out by considering structure-property models with one, three, and five degrees of freedom. Two- and three-dimensional simulated polycrystals are used in this study to obtain the simulation data. The two-dimensional simulated polycrystals used in this study are generated using grain growth simulation performed using a front-tracking algorithm. The three-dimensional polycrystals used in this study are generated using Neper software resulting in a real-like polycrystals. The structure-property models used in the validation are picked by considering the qualitative features that reflect trends observed in literature. The inference method is performed by ignoring any knowledge about the structure-property model in the process.