Image classification has been a pivotal area of research in Deep Learning, with a vast body of literature working to tackle the problem, constantly striving to achieve higher accuracies. This push to reach achieve greater prediction accuracy however, has further exacerbated the black box phenomenon which is inherent of neural networks, and more for so CNN style deep architectures. Likewise, it has lead to the development of highly tuned methods, suitable only for a specific data sets, requiring significant work to alter given new data. Although these models are capable of producing highly accurate predictions, we have little to no ability to understand the decision process taken by a network to reach a conclusion. This factor poses a difficulty in use cases such as medical diagnostics tools or autonomous vehicles, which require insight into prediction reasoning to validate a conclusion or to debug a system. In essence, modern applications which utilize deep networks are able to learn to produce predictions, but lack interpretability and a deeper understanding of the data. Given this key point, we look to decision trees, opposite in nature to deep networks, with a high level of interpretability but a low capacity for learning. In our work we strive to merge these two techniques as a means to maintain the capacity for learning while providing insight into the decision process. More importantly, we look to expand the understanding of class relationships through a tree architecture. Our ultimate goal in this work is to create a technique able to automatically create a visual feature based knowledge hierarchy for class relations, applicable broadly to any data set or combination thereof. We maintain these goals in an effort to move away from specific systems and instead toward artificial general intelligence (AGI). AGI requires a deeper understanding over a broad range of information, and more so the ability to learn new information over time. In our work we embed networks of varying sizes and complexity within decision trees on a node level, where each node network is responsible for selecting the next branch path in the tree. Each leaf node represents a single class and all parent and ancestor nodes represent groups of classes. We designed the method such that classes are reasonably grouped by their visual features, where parent and ancestor nodes represent hidden super classes. Our work aims to introduce this method as a small step towards AGI, where class relations are understood through an automatically generated decision tree (representing a class hierarchy), capable of accurate image classification. / Master of Science / Many modern day applications make use of deep networks for image classification. Often these networks are incredibly complex in architecture, and applicable only for specific tasks and data. Standard approaches use just a neural network to produce predictions. However, the internal decision process of the network remains a black box due to the nature of the technique. As more complex human related applications, such as medical image diagnostic tools or autonomous driving software, are being created, they require an understanding of reasoning behind a prediction. To provide this insight into the prediction reasoning, we propose a technique which merges decision trees and deep networks. Tested on the MNIST image data set we were able to achieve an accuracy over 99.0%. We were also able to achieve an accuracy over 73.0% on the CIFAR-10 image data set. Our method is found to create decision trees that are easily understood and are reasonably capable of image classification.

Barrier islands are dynamic environments facing increasing vulnerability to climate changes, sea level rises, and anthropogenic activities. Hurricane Sandy (October 2012) modified the Atlantic coast of the United States. On Fire Island and Westhampton Island, New York, multiple overwashes and three breaches occurred. The U.S. Army Corps of Engineers filled two breaches, increased dune elevation and stabilized dunes by planting American beachgrass (Ammophila breviligulata). They built two restoration areas to mitigate the impact of an island stabilization project to federally listed breeding piping plovers (Charadrius melodus). The goal of this thesis was to quantify habitat changes after Hurricane Sandy, and assess habitat use of piping plovers specifically in human-created restoration areas. We created land cover maps using an object-based classification method (overall accuracy 85%), and field-collected data from four post-hurricane habitat types. Vegetation cover increased across all habitat types, especially in manipulated (30.1% increase) and natural overwashes (37.9% increase), while dry sand for nesting declined by 8%. Vegetation density indices were higher in natural overwashes than planted engineered dunes, likely a reflection of plant age and establishment. We monitored 83 nests (67 pairs) of piping plovers from 2015–2017. Restoration areas were successful in attracting breeding piping plovers, although pair densities were lowest in this habitat in 2016, and in 2017 plovers selected against the restoration areas (�2 = 29.47, df = 3, p<0.0001). There was no effect of habitat type on reproductive parameters. We suggest vegetation removal may be necessary to maintain early successional habitats for piping plover management. / Master of Science / Barrier islands are thin strips of land, which generally lie parallel to the mainland. They are dynamic environments subject to change from weather conditions and ocean currents. They are vulnerable to future climate changes, sea level rise, and increased human activity. Hurricane Sandy, October 2012, caused major landscape changes to the Atlantic coast. On two barrier islands, Fire Island and Westhampton Island, south of Long Island, the U.S. Army Corps of Engineers (USACE) further modified the post-hurricane habitats. They created two restoration sites designed to offset the impact of an island stabilization project to a federally protected breeding shorebird, the piping plover. Piping plovers nest on unvegetated or sparsely vegetated sand. We created land cover maps from aerial images and collected field data to describe habitat changes after Hurricane Sandy (2013, 2015–2017) in four habitat types creating during or after the hurricane. The vegetation cover and density increased across these four habitats, but especially in the manipulated overwashes and natural overwashes. This is likely due to the age of the plants in each habitat type. As vegetation grew back, areas of sand in each habitat type declined. Human-created restoration sites were used by plovers but in 2017 plovers used them less than expected. We monitored piping plover nests and chicks (April to August 2015–2017), and there were no differences in these measurements among habitat types. The findings from this thesis will assist in piping plover conservation on Fire Island and Westhampton Island.

Yes / In recent years, deep learning has been increasingly used for several applications such as object analysis, feature extraction and image classification. This paper explores the use of deep learning in a flood monitoring application in the context of an EC-funded project, Smart Cities and Open Data REuse (SCORE). IoT sensors for detecting blocked gullies and drainages are notoriously hard to build, hence we propose a novel technique to utilise deep learning for building an IoT-enabled smart camera to address this need. In our work, we apply deep leaning to classify drain blockage images to develop an effective image classification model for different severity of blockages. Using this model, an image can be analysed and classified in number of classes depending upon the context of the image. In building such model, we explored the use of filtering in terms of segmentation as one of the approaches to increase the accuracy of classification by concentrating only into the area of interest within the image. Segmentation is applied in data pre-processing stage in our application before the training. We used crowdsourced publicly available images to train and test our model. Our model with segmentation showed an improvement in the classification accuracy. / Research presented in this paper is funded by the European Commission Interreg project Smart Cities and Open Data REuse (SCORE).

The efficient management of storage units requires a reliable and streamlined move-out process. Manual validation methods are resource intensive. Therefore, the task is to introduce an automated approach that capitalises on modern smartphone capabilities to improve the move-out validation process. Hence, the purpose of this thesis project. The proposed solution is a Proof of Concept (POC) application that utilises the Light Detection and Ranging (LiDAR) sensor and camera of a modern iPhone. This is performed through RoomPlan, a framework developed for real-time, indoor room scanning. It generates a 3D model of the room with its key characteristics. Moreover, to increase the number detectable object categories, the solution is integrated with the image classifier Tiny YOLOv3. The solution is evaluated through a quantitative evaluation in a storage unit. It shows that the application can validate whether the storage unit is empty or not in all the completed scans. However, an improvement of the object detecition is needed for the solution to work in a commercial case. Therefore, further work includes investigation of the possibilities to expand the object categories within the image classifier or creating a similar detection pipeline as RoomPlan adjusted for this specific case. The usage of LiDAR sensors indicated to be a stable object detector and a successful tool for the assignment. In contrast, the image classifier had lower detection accuracy in the storage unit.

Visual recognition tasks have witnessed vast improvements in performance since the advent of deep learning. Despite the gains in performance, image understanding algorithms are still not completely robust to partial occlusion. In this work, we propose a novel object classification method based on compositional modeling and explore its effect in the context of the newly introduced panoptic segmentation task. The panoptic segmentation task combines both semantic and instance segmentation to perform labelling of the entire image. The novel classification method replaces the object detection pipeline in UPSNet, a Mask R-CNN based design for panoptic segmentation. We also discuss an issue with the segmentation mask prediction of Mask R-CNN that affects overlapping instances. We perform extensive experiments and showcase results on the complex COCO and Cityscapes datasets. The novel classification method shows promising results for object classification on occluded instances in complex scenes. / Master of Science / Visual recognition tasks have witnessed vast improvements in performance since the advent of deep learning. Despite making significant improvements, algorithms for these tasks still do not perform well at recognizing partially visible objects in the scene. In this work, we propose a novel object classification method that uses compositional models to perform part based detection. The method first looks at individual parts of an object in the scene and then makes a decision about its identity. We test the proposed method in the context of the recently introduced panoptic segmentation task. The panoptic segmentation task combines both semantic and instance segmentation to perform labelling of the entire image. The novel classification method replaces the object detection module in UPSNet, a Mask R-CNN based algorithm for panoptic segmentation. We also discuss an issue with the segmentation mask prediction of Mask R-CNN that affects overlapping instances. After performing extensive experiments and evaluation, it can be seen that the novel classification method shows promising results for object classification on occluded instances in complex scenes.

Advancements in machine learning and artificial intelligence has created opportunities for vast improvements in the manufacturing sector. This study was conducted at a world-leading manufacturing company with the goal to assist in the development of a framework for application of machine learning in the company's operational workflow. Specifically, with the aim to investigate the potential benefits and pitfalls when utilizing machine learning to supervise a laser soldering process. This thesis analyzes and designs all the required steps for a machine learning approach for this specific manufacturing process. This included (1) image capturing, (2) preprocessing, (3) modelling, (4) testing and (5) functional tool. The thesis also discusses strategies for dealing with limitations posed by the industrial environment, for example unattainable process data and imbalanced datasets. In conclusion, it became evident that for a machine learning approach in an industrial setting it is crucial to understand the underlying process, the importance of a reliable data collection setup as well as the necessity of a proper framework. The thesis also proposes a sliding window approach as a preprocessing method for similar image classification tasks. / Framsteg inom maskininlärning och artificiell intelligens har banat vägen för omfattande förbättringar inom tillverkningsindustrin. Denna studie genomfördes på ett världsledande tillverkningsföretag för att stödja utvecklingen av ett ramverk kring hur maskininlärning kan appliceras inom deras operativa flöde. Syftet var specifikt att undersöka de potentiella fördelarna och fallgroparna med att utnyttja maskininlärning för övervakning av en laserlödningsprocess.  Denna rapport undersökte och konstruerade alla nödvändiga steg för hur maskininlärning kan appliceras på denna specifika produktionsprocess. Detta inkluderar (1) bildtagning, (2) dataförbehandling, (3) modellering, (4) testning och (5) slutgiltig implementation. Rapporten diskuterar även strategier för att hantera de begränsningar som uppstod på grund av den industriella miljön, till exempel otillgänglig processdata och obalanserad data.  Avslutningsvis, så var det uppenbart att förståelsen för den underliggande processen, vikten av en tillförlitlig anordning för datainsamlig samt behovet för ett tillbörligt ramverk för maskininlärning inom företaget är avgörande. Rapporten förslår även en "sliding window" ansats som preprocessering metod för liknade cases inom bildklassificering.

Background: Recently, animal image classification has involved the impracticalityof deep learning models due to high computational demands, limiting their use inwildlife monitoring. This calls for the crucial necessity of lightweight models for real-time animal identification in resource-limited environments like wildlife cameras andmobile devices. Achieving accurate animal classification in these settings becomes acrucial concern for advancing classification methods. Objectives: The goal of this research is to evaluate lightweight transfer learningmodels for classifying animal images while balancing computational efficiency andaccuracy. The objectives include analyzing the models’ performance and providingmodel selection criteria based on performance and efficiency for resource-constraintenvironments. This study contributes to the advancement of machine learning inwildlife preservation and environmental monitoring, which is critical for accuratespecies identification. Methods: The proposed methodology involves conducting a thorough literaturereview to identify lightweight transfer learning models for image classification. TheAnimal-90 dataset was utilized, comprising images of ninety distinct animal species.To train the dataset, selected pre-trained models, MobileNetV2, EfficientNetB3,ShuffleNet, SqueezeNet, and MnasNet were employed with custom classificationheads. A 5-fold Cross-Validation technique was used to validate the model. Acombined metric approach is applied to rank the models based on the results fromthe metrics, Accuracy, Inference time, and number of parameters. Results: The experimental outcomes revealed EfficientNetB3 to be the most ac-curate but also at the same time it has the highest number of parameters amongother models. Friedman’s test has rejected the null hypothesis of models havingsimilar performance. The combined metric approach ranked ShuffletNet as the topmodel among the compared models in terms of performance and efficiency. Conclusions: The research unveiled the commendable performance of all the mod-els in animal image classification, with ShuffleNet achieving the top rank among allother models in terms of accuracy and efficiency. These lightweight models, espe-cially ShuffleNet, show promise in managing limited resources while ensuring accurateanimal classification and confirming their reliability in wildlife conservation

Manual investigation of railway infrastructure is a labor-intensive and time-consuming task, and automating it has become a high priority for railway operators to reduce unexpected infrastructure expenditure. In this thesis, we propose a new image classification approach for classifying defect and non-defective rail joints in image data, based on previous fault detection algorithms using object detection. The rail joints model is to our knowledge a world first, with the vast majority of research into applying computer vision for rail defect detection focusing mainly on the rail tracks and sleepers. Our new image classification models are based on the widely popular Inception ResNet V2, which we fine-tune and compare against a counterpart trained using self-supervision. Additional comparisons are performed against the Faster R-CNN object detector that has had successes with rail tracks and sleepers at the Swiss Federal Railways, as well as against the novel transformer-based DETR architecture. The research has used an in-house object detection annotated dataset from the Swiss railways, recorded in the context of predictive rail maintanance, with rail joints labeled as either defective, or non-defective. Our proposed image classification approach, using either a pre-trained and then fine-tuned, or self-supervised CNN, uses the bounding boxes in a dataset originally intended for object detection, to perform an expanded crop of the images around the rail joint before feeding it to the neural network. Our new image classification approach significantly outperforms object detection neural networks for the task of classifying defective and non-defective rail joints, albeit with the requirement that the rail joint has to be identified prior to classification. Furthermore, our results suggest that the trained models classify defective joints in the test set more consistently than human rail inspectors. The results show that our proposed method can achieve practical performance on unseen data, and can practically be applied to real-life defect detection with high precision and recall, notably on the railways operated by Swiss Federal Railways, SBB CFF FFS. / Manuell inspektion av järnvägsinfrastruktur är en tids- och arbetskrävande uppgift, och automatisering av inspektionerna har på senare tid blivit mer prioriterat av järnvägsoperatörer i syfte att minska oväntade utgifter som uppkommer till följd av undermålig infrastruktur. I det här examensarbetet presenterar vi en ny bildklassificeringsmetod för att klassificera defekta och icke-defekta järnvägsskarvar i bilder tagna från diagnostiska tåg. Modelleringen av järnvägsskarvar som vi har utfört är till vår kännedom något som aldrig gjorts förut för järnvägsinfrastruktur, då majoriteten av forskning inom datorseende för inspektion av järnvägsinfrastruktur historiskt mest har fokuserat på räls och sliprar. Den nya bildklassificeringsmodellen som vi har utvecklat använder den populära arkitekturen Inception ResNet V2, som vi finjusterar och jämför med ett dito som har tränats med självövervakad inlärning. Vidare jämförelser görs mot objektigenkänningsmetoden Faster R-CNN som fungerat väl för sliprar på den schweiziska järnvägen, samt mot den nya transformer-baserade arkitekturen DETR. Forskningen har använt ett dataset annoterat för objektigenkänning från den schweiziska järnvägen, med järnvägsskarvar märkta som defekta, eller icke-defekta. Vår föreslagna bildklassificeringsmetod, med antingen en förtränad och sedan finjusterad CNN, eller en CNN tränad med sjävövervakad inlärning, använder de annoterade boxarna från datasetet för att beskära bilderna runt skarvarna, och sedan klassificera dem. Vår nya metod baserad på bildklassificering presterar väsentligt bättre än neurala nätverk för objektigenkänning, dock med kravet att järnvägsskarven måste ha identifierats i bilden före klassificering. Vidare visar våra resultat att de tränade bildklassificeringsmodellerna klassificerar defekta skarvar i test-setet mer konsekvent än mänskliga järnvägsinspektörer. Resultaten visar att vår nya metod kan användas praktiskt för att upptäcka defekter i verkligheten, med hög precision och recall i data som inte setts under träningen. Specifikt visar vi att de nya modellerna är praktiskt användbara för järnvägen som drivs av Schweiziska Federala Järnvägarna, SBB CFF FFS.

La classification des images est aujourd'hui un défi d'une grande ampleur puisque ça concerne d’un côté les millions voir des milliards d'images qui se trouvent partout sur le web et d’autre part des images pour des applications temps réel critiques. Cette classification fait appel en général à des méthodes d'apprentissage et à des classifieurs qui doivent répondre à la fois à la précision ainsi qu'à la rapidité. Ces problèmes d'apprentissage touchent aujourd'hui un grand nombre de domaines d'applications: à savoir, le web (profiling, ciblage, réseaux sociaux, moteurs de recherche), les "Big Data" et bien évidemment la vision par ordinateur tel que la reconnaissance d'objets et la classification des images. La présente thèse se situe dans cette dernière catégorie et présente des algorithmes d'apprentissage supervisé basés sur la minimisation de fonctions de perte (erreur) dites "calibrées" pour deux types de classifieurs: k-Plus Proches voisins (kNN) et classifieurs linéaires. Ces méthodes d'apprentissage ont été testées sur de grandes bases d'images et appliquées par la suite à des images biomédicales. Ainsi, cette thèse reformule dans une première étape un algorithme de Boosting des kNN et présente ensuite une deuxième méthode d'apprentissage de ces classifieurs NN mais avec une approche de descente de Newton pour une convergence plus rapide. Dans une seconde partie, cette thèse introduit un nouvel algorithme d'apprentissage par descente stochastique de Newton pour les classifieurs linéaires connus pour leur simplicité et leur rapidité de calcul. Enfin, ces trois méthodes ont été utilisées dans une application médicale qui concerne la classification de cellules en biologie et en pathologie. / Image classification becomes a big challenge since it concerns on the one hand millions or billions of images that are available on the web and on the other hand images used for critical real-time applications. This classification involves in general learning methods and classifiers that must require both precision as well as speed performance. These learning problems concern a large number of application areas: namely, web applications (profiling, targeting, social networks, search engines), "Big Data" and of course computer vision such as the object recognition and image classification. This thesis concerns the last category of applications and is about supervised learning algorithms based on the minimization of loss functions (error) called "calibrated" for two kinds of classifiers: k-Nearest Neighbours (kNN) and linear classifiers. Those learning methods have been tested on large databases of images and then applied to biomedical images. In a first step, this thesis revisited a Boosting kNN algorithm for large scale classification. Then, we introduced a new method of learning these NN classifiers using a Newton descent approach for a faster convergence. In a second part, this thesis introduces a new learning algorithm based on stochastic Newton descent for linear classifiers known for their simplicity and their speed of convergence. Finally, these three methods have been used in a medical application regarding the classification of cells in biology and pathology.

L'objectif de cette thèse est d'améliorer les représentations des images et des vidéos dans le but d'obtenir une reconnaissance visuelle accrue, tant pour des entités spécifiques que pour des catégories plus génériques. Les contributions de cette thèse portent, pour l'essentiel, sur des méthodes de description du contenu visuel. Nous proposons des méthodes pour la recherche d'image par le contenu ou par des requêtes textuelles, ainsi que des méthodes pour la reconnaissance et la localisation d'action dans des vidéos. En recherche d'image, les contributions se fondent sur des méthodes `a base de plongements de Hamming. Tout d'abord, une méthode de comparaison asymétrique vecteur-`a-code est proposée pour améliorer la méthode originale, symétrique et utilisant une comparaison code-`a-code. Une méthode de classification fondée sur l'appariement de descripteurs locaux est ensuite proposée. Elle s'appuie sur une classification opérée dans un espace de similarités associées au plongement de Hamming. En reconnaissance d'action, les contributions portent essentiellement sur des meilleures manières d'exploiter et de représenter le mouvement. Finalement, une méthode de localisation est proposée. Elle utilise une partition de la vidéo en super-voxels, qui permet d'effectuer un échantillonnage 2D+t de suites de boîtes englobantes autour de zones spatio-temporelles d'intérêt. Elle s'appuie en particulier sur un critère de similarité associé au mouvement. Toutes les méthodes proposées sont évaluées sur des jeux de données publics. Ces expériences montrent que les méthodes proposées dans cette thèse améliorent l'état de l'art au moment de leur publication. / The subject of this thesis is about image and video representations for visual recognition. This thesis first focuses on image search, both for image and textual queries, and then considers the classification and the localization of actions in videos. In image retrieval, images similar to the query image are retrieved from a large dataset. On this front, we propose an asymmetric version of the Hamming Embedding method, where the comparison of query and database descriptors relies on a vector-to-binary code comparison. For image classification, where the task is to identify if an image contains any instance of the queried category, we propose a novel approach based on a match kernel between images, more specifically based on Hamming Embedding similarity. We also present an effective variant of the SIFT descriptor, which leads to a better classification accuracy. Action classification is improved by several methods to better employ the motion inherent to videos. This is done by dominant motion compensation, and by introducing a novel descriptor based on kinematic features of the visual flow. The last contribution is devoted to action localization, whose objective is to determine where and when the action of interest appears in the video. A selective sampling strategy produces 2D+t sequences of bounding boxes, which drastically reduces the candidate locations. The method advantageously exploits a criterion that takes in account how motion related to actions deviates from the background motion. We thoroughly evaluated all the proposed methods on real world images and videos from challenging benchmarks. Our methods outperform the previously published related state of the art and remains competitive with the subsequently proposed methods.