Malfunctions on loom machines are the main causes of faulty fabric production. An on-loom fabric inspection system is a real-time monitoring device that enables immediate defect detection for human intervention. This dissertation presented a solution for the on-loom fabric defect inspection, including the new hardware design—the configurable contact image sensor (CIS) module—for on-loom fabric scanning and the defect detection algorithms. The main contributions of this work include (1) creating a configurable CIS module adaptable to a loom width, which brings CIS unique features, such as sub-millimeter resolution, compact size, short working distance and low cost, to the fabric defect inspection system, (2) designing a two-level hardware architecture that can be efficiently deployed in a weaving factory with hundreds of looms, (3) developing a two-level inspecting scheme, with which the initial defect screening is performed on the Raspberry Pi and the intensive defect verification is processed on the cloud server, (4) introducing the novel pairwise-potential activation layer to a convolutional neural network that leads to high accuracies of defect segmentation on fabrics with fine and imbalanced structures, (5) achieving a real-time defect detection that allows a possible defect to be examined multiple times, and (6) implementing a new color segmentation technique suitable for processing multi-color fabric defects. The novel CIS-based on-loom scanning system offered real-time and high-resolution fabric images, which was able to deliver the information of single thread on a fabric. The algorithm evaluation on the fabric defect datasets showed a non-miss-detection rate on defect-free fabrics. The average precision of defect existed images reached above 90% at the pixel level. The detected defect pixels' integrity—the recall scored around 70%. Possible defect regions overestimated on ground truth images and the morphologies of fine defects similar to regular fabric pattern were the two major reasons that caused the imperfection in defect pixel locating. The experiments showed the defect areas on multi-color fabrics could be precisely located under the proposed color segmentation algorithm.