Recent years have seen ambitious robotic exploration missions to other planets and a renewed interest in sending humans beyond low Earth orbit. These activities give rise to a need for autonomous spacecraft operation. Of particular interest here is the ability of a spacecraft to navigate independent of contact with Earth-based resources. Optical navigation techniques are proposed as a solution to the problem of navigating in a planetary system without requiring navigation information from Earth. A detailed discussion of optical sensor hardware and error sources leads to new high fidelity math models for optical sensor performance that may be used in navigation simulations. Algorithms are developed that allow optical data to be used for the estimation of spacecraft position, velocity, and attitude. Sequential measurements are processed using traditional filtering techniques. Additionally, for the case of attitude estimation, a new attitude filter called Sequential Optimal Attitude Routine (SOAR) is presented. The models and techniques developed in this dissertation are demonstrated in two case studies: (1) navigation of a spacecraft performing a planetary fly-by using real images from the June 2007 MESSENGER fly-by of Venus and (2) navigation of a spacecraft in cislunar space on a return trajectory from the Moon. / text