The use and application of modeling and simulation (M&S) is pervasive in today's world. A key component in the application of models is to conduct appropriate verification and validation (V&V). V&V is conducted to make sure the model represents reality to the appropriate level of detail based on the questions posed. V&V techniques are well documented within the literature for observable systems, i.e. required data can be collected from the operations of the real system for comparison with the simulation results; however, V&V techniques for non-observable systems are limited to subjective validation. This subjective validation can be applied to the simulation outputs, operational validation, or towards the model development, conceptual validation. Oftentimes subjective operational validation of the simulation is the primary source of validation efforts. It is shown in this thesis that the sole reliance on subjective operational validation of the simulation can easily lead to the inaccurate acceptance of a model. In order to improve M&S practices for the representation of non-observable systems, models must be developed in a methodological manner that provides a traceable and defensible argument behind the model’s representation of reality. Though there is growing discussion within the recent literature, few methods exist on proper conceptual model development and validation. The research objective of this thesis is to identify a methodology to develop a model in a traceable and defensible manner for a system or system of systems that is non-observable. To address this research objective the proposal will address eight aspects of model development. The first is to define a set of terms that are common vernacular in the field of M&S. This is followed by the assessment of what defines a ‘good’ model and how to determine if the model is ‘good’ or not. This leads to a review of V&V and the observation that subjective validation in isolation is not sufficient for model validation. Next, a review of model development procedures is conducted and analyzed against a set of criteria. A selection is made using the Analytic Hierarchy Process (AHP). A procedure developed by Balci in 1986 is selected for the use in development of models for non-observable systems. Specific steps within Balci's 1986 procedure are investigated further to determine appropriate techniques that should be used when developing models of non-observable systems. These steps are system and objective definition, conceptual model, communicative model, and experimental models and results. Five techniques are identified in the literature that can be applied to system and objective definition: Soft Systems Methodology, Requirements Engineering, Unified Modeling Language, Systems Modeling Language, and Department of Defense Architecture Framework. These techniques are reviewed and selection is made using AHP. The System Modeling Language (SysML) is selected as the best technique to perform System an Objective Definition. Significant resources are devoted to the study of conceptual model development. Proposed in this thesis is a process to decompose the impacts of the system and apply subjective weightings in order to identify aspects of the system with significant importance. This approach enables the modeling of the system in question to the appropriate level of fidelity based on the identified importance of the system impacts. Additionally, this process provides traceability and defensibility of the final model form. Communicative model development is rarely addressed in the literature; however, many of the techniques used in system and objective definition can be applied to developing a communicative model. A similar study to the system and objective definition, AHP was utilized to make a selection. It was concluded that the Unified Modeling Language provides the best tool for creating a communicative model. In the final step, experimental models and results, the literature was found to be rich in techniques. A gap was found in the analysis of the outputs of stochastic simulations. Four questions resulted: 'which stochastic measures should be used in analyzing a stochastic simulation?', 'how many replications are required for an accurate estimation of the stochastic measure?', which least squares method should be used in the regression of a stochastic response?, and 'how many replications are required for an accurate regression of a stochastic measure? Heuristics are presented for each of these questions. A proof of concept is provided on the methodology developed within this thesis. The selected scenario is a Humanitarian Aid/Disaster Relief Mission, where the U.S. Navy has been tasked with distributing aid in an effective manner to the affected population. Upon application of the proposed methodology, it was observed that subjective decomposition and weighting of the scenario proved to be a useful tool for guiding and justifying the form of the eventual model. Shortcomings of the methodology were identified. The primary shortcomings identified were the linking of information between the steps of the model development procedure, and the difficulty in correctly identifying the structure of the system impacts decomposition. The primary contribution of this thesis is to the field of M&S. Contributions are made to the practice of conceptual model development, a growing discussion within the literature over the past several years. The contribution to conceptual model development will aid in the development models for non-observable systems. Additional contributions are made to the analysis of stochastic simulations. The methodology presented in this thesis will provide a new and robust method to develop and validate models in a traceable and defensible manner.

Many real-world systems experience deterioration with usage and age, which often leads to low product quality, high production cost, and low system availability. Most previous maintenance and reliability models in the literature do not incorporate condition monitoring information for decision making, which often results in poor failure prediction for partially observable deteriorating systems. For that reason, the development of fault prediction and control scheme using condition-based maintenance techniques has received considerable attention in recent years. This research presents a new framework for predicting failures of a partially observable deteriorating system using Bayesian control techniques. A time series model is fitted to a vector observation process representing partial information about the system state. Residuals are then calculated using the fitted model, which are indicative of system deterioration. The deterioration process is modeled as a 3-state continuous-time homogeneous Markov process. States 0 and 1 are not observable, representing healthy (good) and unhealthy (warning) system operational conditions, respectively. Only the failure state 2 is assumed to be observable. Preventive maintenance can be carried out at any sampling epoch, and corrective maintenance is carried out upon system failure. The form of the optimal control policy that maximizes the long-run expected average availability per unit time has been investigated. It has been proved that a control limit policy is optimal for decision making. The model parameters have been estimated using the Expectation Maximization (EM) algorithm. The optimal Bayesian fault prediction and control scheme, considering long-run average availability maximization along with a practical statistical constraint, has been proposed and compared with the age-based replacement policy. The optimal control limit and sampling interval are calculated in the semi-Markov decision process (SMDP) framework. Another Bayesian fault prediction and control scheme has been developed based on the average run length (ARL) criterion. Comparisons with traditional control charts are provided. Formulae for the mean residual life and the distribution function of system residual life have been derived in explicit forms as functions of a posterior probability statistic. The advantage of the Bayesian model over the well-known 2-parameter Weibull model in system residual life prediction is shown. The methodologies are illustrated using simulated data, real data obtained from the spectrometric analysis of oil samples collected from transmission units of heavy hauler trucks in the mining industry, and vibration data from a planetary gearbox machinery application.

Many real-world systems experience deterioration with usage and age, which often leads to low product quality, high production cost, and low system availability. Most previous maintenance and reliability models in the literature do not incorporate condition monitoring information for decision making, which often results in poor failure prediction for partially observable deteriorating systems. For that reason, the development of fault prediction and control scheme using condition-based maintenance techniques has received considerable attention in recent years. This research presents a new framework for predicting failures of a partially observable deteriorating system using Bayesian control techniques. A time series model is fitted to a vector observation process representing partial information about the system state. Residuals are then calculated using the fitted model, which are indicative of system deterioration. The deterioration process is modeled as a 3-state continuous-time homogeneous Markov process. States 0 and 1 are not observable, representing healthy (good) and unhealthy (warning) system operational conditions, respectively. Only the failure state 2 is assumed to be observable. Preventive maintenance can be carried out at any sampling epoch, and corrective maintenance is carried out upon system failure. The form of the optimal control policy that maximizes the long-run expected average availability per unit time has been investigated. It has been proved that a control limit policy is optimal for decision making. The model parameters have been estimated using the Expectation Maximization (EM) algorithm. The optimal Bayesian fault prediction and control scheme, considering long-run average availability maximization along with a practical statistical constraint, has been proposed and compared with the age-based replacement policy. The optimal control limit and sampling interval are calculated in the semi-Markov decision process (SMDP) framework. Another Bayesian fault prediction and control scheme has been developed based on the average run length (ARL) criterion. Comparisons with traditional control charts are provided. Formulae for the mean residual life and the distribution function of system residual life have been derived in explicit forms as functions of a posterior probability statistic. The advantage of the Bayesian model over the well-known 2-parameter Weibull model in system residual life prediction is shown. The methodologies are illustrated using simulated data, real data obtained from the spectrometric analysis of oil samples collected from transmission units of heavy hauler trucks in the mining industry, and vibration data from a planetary gearbox machinery application.