Yes / Aggressive posts containing symbolic and offensive images, inappropriate gestures along with provocative textual comments are growing exponentially in social media with the availability of inexpensive data services. These posts have numerous negative impacts on the reader and need an immediate technical solution to filter out aggressive comments. This paper presents a model based on a Convolutional Neural Network (CNN) and Binary Particle Swarm Optimization (BPSO) to classify the social media posts containing images with associated textual comments into non-aggressive, medium-aggressive and high-aggressive classes. A dataset containing symbolic images and the corresponding textual comments was created to validate the proposed model. The framework employs a pre-trained VGG-16 to extract the image features and a three-layered CNN to extract the textual features in parallel. The hybrid feature set obtained by concatenating the image and the text features were optimized using the BPSO algorithm to extract the more relevant features. The proposed model with optimized features and Random Forest classifier achieves a weighted F1-Score of 0.74, an improvement of around 3% over unoptimized features.