Autonomous driving is increasingly popular among people and automotive industries in realizing their presence both in passenger and goods transportation. Safer autonomous navigation might be very challenging if there is a failure in sensing system. Among several sensing systems, image classification plays a major role in understanding the road signs and to regulate the vehicle control based on urban road rules. Hence, a robust classifier algorithm irrespective of camera position, view angles, environmental condition, different vehicle size & type (Car, Bus, Truck, etc.,) of an autonomous platform is of prime importance. In this study, Convolutional Neural Network (CNN) based classifier algorithm has been implemented to ensure improved robustness for recognizing traffic signs. As training data play a crucial role in supervised learning algorithms, there come an effective dataset requirement which can handle dynamic environmental conditions and other variations caused due to the vehicle motion (will be referred as challenges). Since the collected training data might not contain all the dynamic variations, the model weakness can be identified by exposing it to variations (Blur, Darkness, Shadow, etc.,) faced by the vehicles in real-time as a initial testing sequence. To overcome the weakness caused due to the training data itself, an effective augmentation technique enriching the training data in order to increase the model capacity for withstanding the variations prevalent in urban environment has been proposed. As a major contribution, a framework has been developed to identify model weakness and successively introduce a targeted augmentation methodology for classification improvement. Targeted augmentation is based on estimated weakness caused due to the challenges with difficulty levels, only those necessary for better classification were then augmented further. Predictive Augmentation (PA) and Predictive Multiple Augmentation (PMA) are the two proposed methods to adapt the model based on targeted challenges by delivering with high numerical value of confidence. We validated our framework on two different training datasets (German Traffic Sign Recognition Benchmark (GTSRB) and Heavy Vehicle data collected from bus) and with 5 generated test groups containing varying levels of challenge (simple to extreme). The results show impressive improvement by ≈ 5-20% in overall classification accuracy thereby keeping their high confidence.