No matter how inspection techniques have been advanced, manual visual inspection is currently still the first and fundamental step in assessing civil infrastructure. If no sign of deterioration has been spotted in manual inspection, any future inspection actions is not necessary to take. However, manual inspection has been identified with several limitations including the qualitative nature of inspection results, the time-consuming inspection process, and the heavy reliance on inspectors' and/or engineers' experience. In order to overcome these limitations, automated visual inspection systems have been proposed to enhance and/or replicate the manual inspection process. A number of image processing methods have been developed in detecting defects (i.e. coating rusts) and damage (i.e. cracks) on civil infrastructure. Their effectiveness has been verified in inspecting structures, such as bridges, underground pipes, and tunnels. Although existing methods are effective in finding defects and damage from digital images, missing two important links limits their application for rapid infrastructure assessment and rehabilitation. The first link is the correlation between the defects/damage and the structural members that the defects/damage lie on. The second link is the relationship between the defects/damage and their impacts on the structural members. The purpose of this research is to investigate the way of establishing these two links. It is focused on the retrieval of critical structural members and defects/damage information from images/videos, and then the utilization of this information for automated and rapid assessment and rehabilitation of civil infrastructure. Specifically, a combination of techniques from the areas of visual pattern recognition, digital filtering, and machine vision have been used in order to develop a set of methods for concrete column recognition, crack properties retrieval, and air pockets and discoloration detection and evaluation. The methods proposed in this research were implemented in a Microsoft Visual Studio environment, and tested on the real images/videos of concrete structures inflicted with cracks, air pockets and discoloration. The test results indicated that the methods could automatically recognize concrete columns, correctly measure the properties of the cracks in a crack map, and accurately evaluate the impacts of air pockets and discoloration on the visual quality of concrete surfaces.