The use and application of modeling and simulation (M&S) is pervasive in today's world. A key component in the application of models is to conduct appropriate verification and validation (V&V). V&V is conducted to make sure the model represents reality to the appropriate level of detail based on the questions posed. V&V techniques are well documented within the literature for observable systems, i.e. required data can be collected from the operations of the real system for comparison with the simulation results; however, V&V techniques for non-observable systems are limited to subjective validation. This subjective validation can be applied to the simulation outputs, operational validation, or towards the model development, conceptual validation. Oftentimes subjective operational validation of the simulation is the primary source of validation efforts. It is shown in this thesis that the sole reliance on subjective operational validation of the simulation can easily lead to the inaccurate acceptance of a model. In order to improve M&S practices for the representation of non-observable systems, models must be developed in a methodological manner that provides a traceable and defensible argument behind the model’s representation of reality. Though there is growing discussion within the recent literature, few methods exist on proper conceptual model development and validation. The research objective of this thesis is to identify a methodology to develop a model in a traceable and defensible manner for a system or system of systems that is non-observable. To address this research objective the proposal will address eight aspects of model development. The first is to define a set of terms that are common vernacular in the field of M&S. This is followed by the assessment of what defines a ‘good’ model and how to determine if the model is ‘good’ or not. This leads to a review of V&V and the observation that subjective validation in isolation is not sufficient for model validation. Next, a review of model development procedures is conducted and analyzed against a set of criteria. A selection is made using the Analytic Hierarchy Process (AHP). A procedure developed by Balci in 1986 is selected for the use in development of models for non-observable systems. Specific steps within Balci's 1986 procedure are investigated further to determine appropriate techniques that should be used when developing models of non-observable systems. These steps are system and objective definition, conceptual model, communicative model, and experimental models and results. Five techniques are identified in the literature that can be applied to system and objective definition: Soft Systems Methodology, Requirements Engineering, Unified Modeling Language, Systems Modeling Language, and Department of Defense Architecture Framework. These techniques are reviewed and selection is made using AHP. The System Modeling Language (SysML) is selected as the best technique to perform System an Objective Definition. Significant resources are devoted to the study of conceptual model development. Proposed in this thesis is a process to decompose the impacts of the system and apply subjective weightings in order to identify aspects of the system with significant importance. This approach enables the modeling of the system in question to the appropriate level of fidelity based on the identified importance of the system impacts. Additionally, this process provides traceability and defensibility of the final model form. Communicative model development is rarely addressed in the literature; however, many of the techniques used in system and objective definition can be applied to developing a communicative model. A similar study to the system and objective definition, AHP was utilized to make a selection. It was concluded that the Unified Modeling Language provides the best tool for creating a communicative model. In the final step, experimental models and results, the literature was found to be rich in techniques. A gap was found in the analysis of the outputs of stochastic simulations. Four questions resulted: 'which stochastic measures should be used in analyzing a stochastic simulation?', 'how many replications are required for an accurate estimation of the stochastic measure?', which least squares method should be used in the regression of a stochastic response?, and 'how many replications are required for an accurate regression of a stochastic measure? Heuristics are presented for each of these questions. A proof of concept is provided on the methodology developed within this thesis. The selected scenario is a Humanitarian Aid/Disaster Relief Mission, where the U.S. Navy has been tasked with distributing aid in an effective manner to the affected population. Upon application of the proposed methodology, it was observed that subjective decomposition and weighting of the scenario proved to be a useful tool for guiding and justifying the form of the eventual model. Shortcomings of the methodology were identified. The primary shortcomings identified were the linking of information between the steps of the model development procedure, and the difficulty in correctly identifying the structure of the system impacts decomposition. The primary contribution of this thesis is to the field of M&S. Contributions are made to the practice of conceptual model development, a growing discussion within the literature over the past several years. The contribution to conceptual model development will aid in the development models for non-observable systems. Additional contributions are made to the analysis of stochastic simulations.
The methodology presented in this thesis will provide a new and robust method to develop and validate models in a traceable and defensible manner.
| Identifer | oai:union.ndltd.org:GATECH/oai:smartech.gatech.edu:1853/51886 |
| Date | 22 May 2014 |
| Creators | Turner, Andrew J. |
| Contributors | Mavris, Dimitri |
| Publisher | Georgia Institute of Technology |
| Source Sets | Georgia Tech Electronic Thesis and Dissertation Archive |
| Language | en_US |
| Detected Language | English |
| Type | Dissertation |
| Format | application/pdf |
Page generated in 0.0026 seconds