This thesis describes how we utilize machine learning and image preprocessing to create a system that can extract a license plate number by taking a picture of a car with an Android smartphone. This project was provided by ÅF at the behalf of one of their customers who wanted to make the workflow of their employees more efficient. The two main techniques of this project are object detection to detect license plates and optical character recognition to then read them. In between are several different image preprocessing techniques to make the images as readable as possible. These techniques mainly includes skewing and color distorting the image. The object detection consists of a convolutional neural network using the You Only Look Once technique, trained by us using Darkflow. When using our final product to read license plates of expected quality in our evaluation phase, we found that 94.8% of them were read correctly. Without our image preprocessing, this was reduced to only 7.95%.

In this thesis we present a detect-tracking algorithm (see figure 3.1) that combines the detection robustness of static region proposal based object detectors, like the faster region convolutional neural network (R-CNN) and the region-based fully convolutional networks (R-FCN) model, with the tracking prediction strength of extended Kalman filters, by using, what we have called, a translating and non-rigid user input region of interest (RoI-) mapping. This so-called RoI-mapping maps a region, which includes the object that one is interested in tracking, to a featureless three-channeled image. The detection part of our proposed algorithm is then performed on the image that includes only the RoI features (see figure 3.2). After the detection step, our model re-maps the RoI features to the original frame, and translates the RoI to the center of the prediction. If no prediction occurs, our proposed model integrates a temporal dependence through a Kalman filter as a predictor; this filter is continuously corrected when detections do occur. To train the region proposal based object detectors that we integrate into our detect-tracking model, we used TensorFlow®’s object detection api, with a random search hyperparameter tuning, where we fine-tuned, all models from TensorFlow® slim base network classification checkpoints. The trained region proposal based object detectors used the inception V2 base network for the faster R-CNN model and the R-FCN model, while the inception V3 base network only was applied to the faster R-CNN model. This was made to compare the two base networks and their corresponding affects on the detection models. In addition to the deep learning part of this thesis, for the implementation part of our detect-tracking model, like for the extended Kalman filter, we used Python and OpenCV® . The results show that, with a stationary camera reference frame, our proposed detect-tracking algorithm, combined with region proposal based object detectors on images of size 414 × 740 × 3, can detect and track a small object in real-time, like a tennis ball, moving along a horizontal trajectory with an average velocity v ≈ 50 km/h at a distance d = 25 m, with a combined detect-tracking frequency of about 13 to 14 Hz. The largest measured state error between the actual state and the predicted state from the Kalman filter, at the aforementioned horizontal velocity, have been measured to be a maximum of 10-15 pixels, see table 5.1, but in certain frames where many detections occur this error has been shown to be much smaller (3-5 pixels). Additionally, our combined detect-tracking model has also been shown to be able to handle obstacles and two learnable features that overlap, thanks to the integrated extended Kalman filter. Lastly, our detect-tracking model also was applied on a set of infra-red images, where the goal was to detect and track a moving truck moving along a semi-horizontal path. Our results show that a faster R-CNN inception V2 model was able to extract features from a sequence of infra-red frames, and that our proposed RoI-mapping method worked relatively well at detecting only one truck in a short test-sequence (see figure 5.22).

Multiple object tracking is the process of assigning unique and consistent identities to objects throughout a video sequence. A popular approach to multiple object tracking, and object tracking in general, is to use a method called tracking-by-detection. Tracking-by-detection is a two-stage procedure: an object detection algorithm first detects objects in a frame, these objects are then associated with already tracked objects by a tracking algorithm. One of the main concerns of this thesis is to investigate how different object detection algorithms perform on surveillance video supplied by National Forensic Centre. The thesis then goes on to explore how the stand-alone alone performance of the object detection algorithm correlates with overall performance of a tracking-by-detection system. Finally, the thesis investigates how the use of visual descriptors in the tracking stage of a tracking-by-detection system effects performance.  Results presented in this thesis suggest that the capacity of the object detection algorithm is highly indicative of the overall performance of the tracking-by-detection system. Further, this thesis also shows how the use of visual descriptors in the tracking stage can reduce the number of identity switches and thereby increase performance of the whole system.

A detecção de objetos, pertencentes a uma determinada classe, em vídeos é de uma atividade amplamente estudada devido às aplicações potenciais que ela implica. Por exemplo, para vídeos obtidos por uma câmera estacionária, temos aplicações como segurança ou vigilância do tráfego, e por uma câmera dinâmica, para assistência ao condutor, entre outros. Na literatura, há diferentes métodos para tratar indistintamente cada um dos casos mencionados, e que consideram só imagens obtidas por um único tipo de câmera para treinar os detectores. Isto pode levar a uma baixa performance quando se aplica a técnica em vídeos de diferentes tipos de câmeras. O estado da arte na detecção de objetos de apenas uma classe, mostra uma tendência pelo uso de histogramas, treinamento supervisionado e, basicamente, seguem a seguinte estrutura: construção do modelo da classe de objeto, detecção de candidatos em uma imagem/quadro, e aplicação de uma medida sobre esses candidatos. Outra desvantagem observada é o uso de diferentes modelos para cada linha de visada de um objeto, gerando muitos modelos e, em alguns casos, um classificador para cada linha de visada. Nesta dissertação, abordamos o problema de detecção de objetos, usando um modelo da classe do objeto criada com um conjunto de dados de imagens estáticas e posteriormente usamos o modelo para detectar objetos na seqüência de imagens (vídeos) que foram coletadas a partir de câmeras estacionárias e dinâmicas, ou seja, num cenário totalmente diferente do usado para o treinamento. A criação do modelo é feita em uma fase de aprendizagem off-line, utilizando o conjunto de imagens PASCAL 2007. O modelo baseia-se em uma mistura de modelos baseados em partes deformáveis (MDPM), originalmente proposto por Felzenszwalb et al. (2010b) no âmbito da detecção de objetos em imagens. Não limitamos o modelo para uma determinada linha de visada. Foi elaborado um conjunto de experimentos que exploram o melhor número de componentes da mistura e o número de partes do modelo. Além disso, foi realizado um estudo comparativo de MDPMs simétricas e assimétricas. Testamos esse método para detectar objetos como pessoas e carros em vídeos obtidos por câmera estacionária e dinâmica. Nossos resultados não mostram apenas o bom desempenho da MDPM e melhores resultados que o estado da arte na detecção de objetos em vídeos obtidos por câmeras estacionárias ou dinâmicas, mas também mostram o melhor número de componentes da mistura e as partes para o modelo criado. Finalmente, os resultados mostram algumas diferenças entre as MDPMs simétricas e assimétricas na detecção de objetos em diferentes vídeos. / The problem of detecting objects that belong to a specific class of objects, in videos is a widely studied activity due to its potential applications. For example, for videos that have been taken from a stationary camera, we can mention applications such as security and traffic surveillance; when the video have been taken from a dynamic camera, a possible application is autonomous driving. The literature, presents several different approaches to treat indiscriminately with each of the cases mentioned, and only consider images obtained from a stationary or dynamic camera to train the detectors. These approaches can lead to poor performaces when the tecniques are used in sequences of images from different types of camera. The state of the art in the detection of objects that belong to a specific class shows a tendency to the use of histograms, supervised training and basically follows the structure: object class model construction, detection of candidates in the image/frame, and application of a distance measure to those candidates. Another disadvantage is that some approaches use several models for each point of view of the car, generating a lot of models and, in some cases, one classifier for each point of view. In this work, we approach the problem of object detection, using a model of the object class created with a dataset of static images and we use the model to detect objects in videos (sequence of images) that were collected from static and dynamic cameras, i.e., in a totally different setting than used for training. The creation of the model is done by an off-line learning phase, using an image database of cars in several points of view, PASCAL 2007. The model is based on a mixture of deformable part models (MDPM), originally proposed by Felzenszwalb et al. (2010b) for detection in static images. We do not limit the model for any specific viewpoint. A set of experiments was elaborated to explore the best number of components of the integration, as well as the number of parts of the model. In addition, we performed a comparative study of symmetric and asymmetric MDPMs. We evaluated the proposed method to detect people and cars in videos obtained by a static or a dynamic camera. Our results not only show good performance of MDPM and better results than the state of the art approches in object detection on videos obtained from a stationary, or dynamic, camera, but also show the best number of components of the integration and parts or the created object. Finally, results show differences between symmetric and asymmetric MDPMs in the detection of objects in different videos.

A detecção de objetos é uma área de extrema importância para sistemas de visão computacional. Em especial, dado o aumento constante da utilização de dispositivos móveis, torna-se cada vez mais importante o desenvolvimento de métodos e aplicações capazes de serem utilizadas em tais aparelhos. Neste sentido, neste trabalho propõe-se o estudo e implementação de um aplicativo para dispositivos móveis capaz de detectar, em tempo real, objetos existentes em ambientes internos com uma aplicação para auxiliar um usuário a se localizar dentro do local. O aplicativo depende somente das capacidades do próprio aparelho e, portanto, procura ser mais flexível e sem restrições. A detecção de objetos é realizada por casamento de grafos-chave entre imagens de objetos pré-escolhidas e a imagem sendo capturada pela câmera do dispositivo. Os grafos-chave são uma generalização do método de detecção de pontos-chave tradicional e, por levarem em consideração um conjunto de pontos e suas propriedades estruturais, são capazes de descrever e detectar os objetos de forma robusta e eficiente. Para realizar a localização, optou-se por detectar placas existentes no próprio local. Após cada detecção, aplica-se um simples, mas bastante eficaz, sistema de localização baseado na comparação da placa detectada com uma base de dados de imagens de todo o ambiente. A base foi construída utilizando diversas câmeras colocadas sobre uma estrutura móvel, capturando sistematicamente imagens do ambiente em intervalos regulares. A implementação é descrita em detalhes e são apresentados resultados obtidos por testes reais no ambiente escolhido utilizando um celular Nokia N900. Tais resultados são avaliados em termos da precisão da detecção e da estimativa de localização, bem como do tempo decorrido para a realização de todo o processo. / Object detection is an area of extreme importance for computer vision systems. Specially because of the increasing use of mobile devices, it becomes more and more important to develop methods and applications that can be used in such devices. In this sense, we propose the study and implementation of an application for mobile devices that is able to detect, in real time, existing indoor objects with an application to help a user in localization in the environment. The application depends solely on the device capabilities and hence, it is flexible and unconstrained. Object detection is accomplished by keygraph matching between images of previously chosen signs and the image currently being captured by the camera device. Keygraphs are a generalization of the traditional keypoints method and, by taking into consideration a set of points and its structural properties, are capable of describing the objects robustly and efficiently. In order to perform localization, we chose to detect signs existing in the environment. After each detection, we apply a simple, but very effective, localization method based on a comparison between the detected sign and a dataset of images of the whole environment. The dataset was built using several cameras atop a mobile structure, systematically capturing images of the environment at regular intervals. The implementation is described in details and we show results obtained from real tests in the chosen environment using a Nokia N900 cell phone. Such results are evaluated in terms of detection and localization estimation precision, as well as the elapsed time to perform the whole process.

<p>Quick and accurate segmentation and object detection of the biomedical image is the starting point of most disease analysis and understanding of biological processes in medical research. It will enhance drug development and advance medical treatment, especially in cancer-related diseases. However, identifying the objects in the CT or MRI images and labeling them usually takes time even for an experienced person. Currently, there is no automatic detection technique for nucleus identification, pneumonia detection, and fetus brain segmentation. Fortunately, as the successful application of artificial intelligence (AI) in image processing, many challenging tasks are easily solved with deep convolutional neural networks. In light of this, in this thesis, the deep learning based object detection and segmentation methods were implemented to perform the nucleus segmentation, lung segmentation, pneumonia detection, and fetus brain segmentation. The semantic segmentation is achieved by the customized U-Net model, and the instance localization is achieved by Faster R-CNN. The reason we choose U-Net is that such a network can be trained end-to-end, which means the architecture of this network is very simple, straightforward and fast to train. Besides, for this project, the availability of the dataset is limited, which makes U-Net a more suitable choice. We also implemented the Faster R-CNN to achieve the object localization. Finally, we evaluated the performance of the two models and further compared the pros and cons of them. The preliminary results show that deep learning based technique outperforms all existing traditional segmentation algorithms. </p>

Perceiving the dynamics of moving objects in complex scenarios is crucial for smart monitoring and safe navigation, thus a key enabler for intelligent supervision and autonomous driving. A variety of research has been developed to detect and track moving objects from data collected by optical sensors and/or laser scanners while most of them concentrate on certain type of objects or face the problem of lacking motion cues. In this thesis, we present a data-driven, model-free detection-based tracking approach for tracking moving objects in urban scenes from time sequential point clouds obtained via state-of-art Doppler LiDAR, which can not only collect spatial information (e.g. point clouds) but also Doppler images by using Doppler-shifted frequencies. In our approach, we first use Doppler images to detect moving points and determine the number of moving objects, which are then completely segmented via a region growing technique. The detected objects are then input to the tracking session which is based on Multiple Hypothesis Tracking (MHT) with two innovative extensions. One extension is that a new point cloud descriptor, <i>Oriented Ensemble of Shape Function (OESF)</i>, is proposed to evaluate the structure similarity when doing object-to-track association in MHT. Another extension is that speed information from Doppler images is used to predict the dynamic state of the moving objects, which is integrated into MHT to improve the estimation of dynamic state of moving objects. The proposed approach has been tested on datasets collected by a terrestrial Doppler LiDAR and a mobile Doppler LiDAR <a>separately</a>. The quantitative evaluation of detection and tracking results shows the unique advantages of the Doppler LiDAR and the effectiveness of the proposed detection and tracking approach.<br>

Rip current images are useful for assisting in climate studies but time consuming to manually annotate by hand over thousands of images. Object detection is a possible solution for automatic annotation because of its success and popularity in identifying regions of interest in images, such as human faces. Similarly to faces, rip currents have distinct features that set them apart from other areas of an image, such as more generic patterns of the surf zone. There are many distinct methods of object detection applied in face detection research. In this thesis, the best fit for a rip current object detector is found by comparing these methods. In addition, the methods are improved with Haar features exclusively created for rip current images. The compared methods include max distance from the average, support vector machines, convolutional neural networks, the Viola-Jones object detector, and a meta-learner. The presented results are compared for accuracy, false positive rate, and detection rate. Viola-Jones has the top base-line performance by achieving a detection rate of 0.88 and identifying only 15 false positives in the test image set of 53 rip currents. The described meta-learner integrates the presented Haar features, which are developed in accordance with the original Viola-Jones algorithm. Ada-Boost, a feature ranking algorithm, shows that the newly presented Haar features extract more meaningful data from rip current images than some of the current features. The meta-classifier improves upon the stand-alone Viola-Jones when applying these features by reducing its false positives by 47% while retaining a similar computational cost and detection rate.

In the e-commerce industry, importing data from third party clothing brands require validation of this data. If the validation step of this data is done manually, it is a tedious and time-consuming task. Part of this task can be replaced or assisted by using computer vision to automatically find clothing types, such as T-shirts and pants, within imported images. After a detection of clothing type is computed, it is possible to recommend the likelihood of clothing products correlating to data imported with a certain accuracy. This was done alongside a prototype interface that can be used to start training, finding clothing types in an image and to mask annotations of products. Annotations are areas describing different clothing types and are used to train an object detector model. A model for finding clothing types is trained on Mask R-CNN object detector and achieves 0.49 mAP accuracy. A detection take just above one second on an Nvidia GTX 1070 8 GB graphics card. Recommending one or several products based on a detection take 0.5 seconds and the algorithm used is k-nearest neighbors. If prediction is done on products of which is used to build the model of the prediction algorithm almost perfect accuracy is achieved while products in images for another products does not achieve nearly as good results.

This dissertation develops a novel system for object recognition in videos. The input of the system is a set of unconstrained videos containing a known set of objects. The output is the locations and categories for each object in each frame across all videos. Initially, a shot boundary detection algorithm is applied to the videos to divide them into multiple sequences separated by the identified shot boundaries. Since each of these sequences still contains moderate content variations, we further use a cost optimization-based key frame extraction method to select key frames in each sequence and use these key frames to divide the videos into shorter sub-sequences with little content variations. Next, we learn object proposals on the first frame of each sub-sequence. Building upon the state-of-the-art object detection algorithms, we develop a tree-based hierarchical model to improve the object detection. Using the learned object proposals as the initial object positions in the first frame of each sub-sequence, we apply the SPOT tracker to track the object proposals and re-rank them using the proposed temporal objectness to obtain object proposals tubes by removing unlikely objects. Finally, we employ the deep Convolution Neural Network (CNN) to perform classification on these tubes. Experiments show that the proposed system significantly improves the object detection rate of the learned proposals when comparing with some state-of-the-art object detectors. Due to the improvement in object detection, the proposed system also achieves higher mean average precision at the stage of proposal classification than the state-of-the-art methods.