This thesis introduces a model-based technique for the automatic recognition and three-dimensional reconstruction of buildings directly from a single range image or stereo processing of multiple optical views of an urban site. Initially, focus-of-attention regions that are likely to contain buildings are segmented from the scene. A perceptual grouping algorithm detects building boundaries as closed polygons in the optical image. When a digital elevation map (DEM) is the only input source available, building regions are detected through direct analysis of the elevation data. Both methods then utilize the key idea of matching a database of shape models against the DEM using a model-indexing procedure that compares orientation histograms for each parameterized model in the database to a histogram that corresponds to the DEM region. The set of models (surfaces) that most closely match the DEM region are used as the initial estimates in a robust surface fitting technique that refines the model parameters (such as orientation and peak-roof angle) of each hypothesized roof surface. The surface model that converges to the DEM with the lowest residual fit error is retained as the most likely description of the surface. The database of surface models contains a limited number of canonical shapes common to rooftops, such as planes, peaks, domes, and gables. Reconstruction of complex shapes is achieved through a composition of different parameterizations of the canonical shape models. We show how the technique can be recursively applied to a range image to segment and reconstruct buildings as well as rooftop substructure. The ability of the model-indexing technique to separate surface models under different resolutions of the parameter space and different levels of noise in the DEM is studied. The approach is evaluated on several datasets, and we demonstrate that this two-phase reconstruction approach allows robust and accurate reconstruction of a wide variety of building types. The building reconstruction process is at the heart of a general knowledge-driven system called Ascender II that incorporates contextual control of computer vision algorithms comprising a processing library. The system operates in the aerial image domain and is composed of a number of different computer vision algorithms that discriminate object classes based on evidence extracted from the available data. Algorithms are stored in evidence policies that encode contextual information about their data requirements and expected performance. Explicit knowledge about a site is stored in a Bayesian network that is used to fuse information gathered from the execution of a subset of the evidence policies on an image and forms the basis for automatic control of the library of algorithms. Based on the state of the Bayesian network and information encoded in the evidence policies, algorithms are selectively applied to the data in order to segment and recognize different object classes. Using this mechanism, the building reconstruction processes are more likely to be applied to building regions that have already been discriminated from other objects present in an urban area. Our conjecture is that this will lead to significantly better performance of the algorithms (fewer false positives, for example). The Ascender II system is evaluated on three different data sets. Acquired models are evaluated with respect to both geometric and semantic accuracy. Furthermore, the robustness of the system is analyzed with respect to incorrect and incomplete knowledge within the Bayesian network and errors within the vision algorithms. (Abstract shortened by UMI.)