In this paper, we present a structural video for video retrieval with fractal orthogonal bases composed of the five steps: video summarization (extract key-frames from video), normalized group cuts (classify key-frames), temporal graph (according to key-frames time in video), transformation of a directed graph into string (the process of transformation is one-to-one mapping), and comparison of string similarity (contain of sting architecture and content), to establish the framework of the video contents. With the above-mentioned information, the structure of the video and its complementary knowledge can be built up according to main line and branch line. Therefore, users can not only browse the video efficiently but also focus on the structure what they are interest. In order to construct the fundamental system, we employ distortion metric that extract key-frames from video and classify key-frames according to normalized group cuts that shot are linked together based on their content. After constructing the relation graph, the graph is transformed into string that has enriched structure. The result clusters form a directed graph and a shortest path algorithm is proposed to find main structure of video. In string similarity, it divides into string architecture and content. In string architecture, we adopt edit distance in main structure and recursive branch line. After comparison of string similarity in architecture, it gets the high similarity string comparing with fractal orthogonal bases that guarantee the similar index has the similar image the characteristic union support vector clustering. The results demonstrate that our system can achieve better performance and information coverage.