During the Digital Period, the more people using these digital video. When there are more and more users and amount of video data, the management of video data becomes a significant dimension during development. Therefore, there are more and more studying of accomplishing video database system, which provide users to search and get them. In this paper, a novel method for Video Scene Change Detection and video database retrieval is proposed. Uses Fractal orthonormal bases to guarantee the similar index has the similar image the characteristic union support vector clustering, splits a video into a sequence of shots, extracts a few representative frames(key-frames) to take the video database index from each shot. When image search compared to, according to MIL to pick up the characteristic, which images pursues the video database to have the similar characteristic, computation similar, makes the place output according to this.