The objective of this Thesis research is to develop algorithms for temporally consistent semantic segmentation in videos. Though many different forms of semantic segmentations exist, this research is focused on the problem of temporally-consistent holistic scene understanding in outdoor videos. Holistic scene understanding requires an understanding of many individual aspects of the scene including 3D layout, objects present, occlusion boundaries, and depth. Such a description of a dynamic scene would be useful for many robotic applications including object reasoning, 3D perception, video analysis, video coding, segmentation, navigation and activity recognition. Scene understanding has been studied with great success for still images. However, scene understanding in videos requires additional approaches to account for the temporal variation, dynamic information, and exploiting causality. As a first step, image-based scene understanding methods can be directly applied to individual video frames to generate a description of the scene. However, these methods do not exploit temporal information across neighboring frames. Further, lacking temporal consistency, image-based methods can result in temporally-inconsistent labels across frames. This inconsistency can impact performance, as scene labels suddenly change between frames. The objective of our this study is to develop temporally consistent scene descriptive algorithms by processing videos efficiently, exploiting causality and data-redundancy, and cater for scene dynamics. Specifically, we achieve our research objectives by (1) extracting geometric context from videos to give broad 3D structure of the scene with all objects present, (2) Detecting occlusion boundaries in videos due to depth discontinuity, (3) Estimating depth in videos by combining monocular and motion features with semantic features and occlusion boundaries.

With the rapid development of photography, we can easily record the 3D world by taking photos and videos. In traditional images and videos, the viewer observes the scene from fixed viewpoints and cannot navigate the scene or edit the 2D observation afterward. Thus, visual content editing and synthesis become an essential task in computer vision. However, achieving high-quality visual synthesis often requires a complex and expensive multi-camera setup. This is not practical for daily use because most people only have one cellphone camera. But a single camera, on the contrary, could not provide enough multi-view constraints to synthesize consistent visual content. Therefore, in this thesis, I address this challenging single-camera visual synthesis problem by leveraging different regularizations. I study three consistent synthesis problems: time-consistent synthesis, view-consistent synthesis, and view-time-consistent synthesis. I show how we can take cellphone-captured monocular images and videos as input to model the scene and consistently synthesize new content for an immersive viewing experience. / Doctor of Philosophy / With the rapid development of photography, we can easily record the 3D world by taking photos and videos. More recently, we have incredible cameras on cell phones, which enable us to take pro-level photos and videos. Those powerful cellphones even have advanced computational photography features build-in. However, these features focus on faithfully recording the world during capturing. We can only watch the photo and video as it is, but not navigate the scene, edit the 2D observation, or synthesize content afterward. Thus, visual content editing and synthesis become an essential task in computer vision. We know that achieving high-quality visual synthesis often requires a complex and expensive multi-camera setup. This is not practical for daily use because most people only have one cellphone camera. But a single camera, on the contrary, is not enough to synthesize consistent visual content. Therefore, in this thesis, I address this challenging single-camera visual synthesis problem by leveraging different regularizations. I study three consistent synthesis problems: time-consistent synthesis, view-consistent synthesis, and view-time-consistent synthesis. I show how we can take cellphone-captured monocular images and videos as input to model the scene and consistently synthesize new content for an immersive viewing experience.

Land cover information is a key input to many earth system models, and thus accurate and consistent land cover maps are critically important to global change science. However, existing global land cover products show unrealistically high levels of year-to-year change. This thesis explores methods to improve accuracies for global land cover classifications, with a focus on reducing spurious year-to-year variation in results derived from MODIS data. In the first part of this thesis I use clustering to identify spectrally distinct sub-groupings within defined land cover classes, and assess the spectral separability of the resulting sub-classes. Many of the sub-classes are difficult to separate due to a high degree of overlap in spectral space. In the second part of this thesis, I examine two methods to reduce year-to-year variation in classification labels. First, I evaluate a technique to construct training data for a per-pixel supervised classification algorithm by combining multiple years of spectral measurements. The resulting classifier achieves higher accuracy and lower levels of year-to-year change than a reference classifier trained using a single year of data. Second, I use a spatio-temporal Markov Random Field (MRF) model to post-process the predictions of a per-pixel classifier. The MRF framework reduces spurious label change to a level comparable to that achieved by a post-hoc heuristic stabilization technique. The timing of label change in the MRF processed maps better matched disturbance events in a reference data, whereas the heuristic stabilization results in label changes that lag several years behind disturbance events.

Weather forecasting has improved greatly since the middle of the 20th century, thanks to better forecasting models, an evolved weather observing system, and improved ways of assimilating the observation data. However, these large systematical improvements make it difficult to use the weather data for climatological studies. Furthermore, observations are scarce, and they cannot be made everywhere. One way to solve this problem is to produce reanalyses, where a fixed version of a numerical weather prediction (NWP) model is used to produce gridded analysis and forecast data with detailed descriptions of the weather by assimilating observation data for a determined time period. One of the newest regional reanalyses is UERRA (Uncertainties in Ensembles of Regional Re-Analyses), which spans over the time period 1961-2015 and covers the whole Europe. By using a fixed NWP model, the only two factors that might influence the temporal quality of a regional reanalysis dataset are the varying number and quality of weather observations, and the quality of the global driving model which gives information about boundaries and large-scale features. In this report, data from one of the UERRA products has been used with the aim to investigate the temporal consistency of the 30-hour forecast skill regarding three parameters; temperature at 2 meters height (t2m), wind speed at 100 meters height (ws100) and 500 hPa geopotential (Φ500). The work has been focused on only land points over Europe during winters and summers, as this enables to investigate the model behaviour at the lowest and highest temperatures. The 30-hour forecast skill was estimated throughout the time period from how well it performed compared to the 6-hour forecast. Temporal inconsistencies were found throughout the reanalysis, with the largest temporal differences being present for Φ500, followed by ws100. UERRA shifts its global driving model in 1979 from ERA-40 (ECMWF Re-Analysis 40) to ERA-Interim (ECMWF Interim Re-Analysis), which ends up as a significant improvement of forecast skill for all investigated parameters. Furthermore, ws100 also shows a significant skill improvement in wintertime from 1979 onwards, while Φ500 shows a systematical improvement for both seasons. In general, the forecast skill is lower in wintertime than in summertime, which might be a result from higher natural variability of the weather in winters. A quick study of forecast data from ERA-Interim shows that the same improving trend in Φ500 can be seen also in that dataset, while the two model drifts differ completely. It was concluded that the addressed issues with temporal inconsistency should be communicated to end users utilizing the UERRA datasets, as knowledge about this can be greatly beneficial when studying climatological trends and patterns and when using the model to reforecast weather events. / Väderprognostisering har utvecklats betydligt sedan mitten på 1900-talet, tack vare bättre prognosmodeller, fler väderobservationer och förbättrade sätt att samla in och nyttja observationerna. Den snabba utvecklingen gör det dock svårt att på ett tillförlitligt sätt kunna jämföra väderdata från olika tidsperioder med varandra, då det är svårt att säkerställa kvaliteten på observationer från flera decennier tillbaka. Ett sätt att lösa det här problemet är att framställa så kallade återanalyser, vilka använder en enskild väderprognosmodell för att uppskatta vädret historiskt i varje punkt i ett förutbestämt rutnät, som sträcker sig över en enskild kontinent eller hela Jorden. En av de nyaste återanalyserna är UERRA, vilket är en regional återanalys över Europa som sträcker sig över tidsperioden 1961–2015. Då en och samma modell används för att beräkna vädret över hela perioden så påverkas inte kvaliteten på datat av den historiska utvecklingen av prognosmodeller. De enda två faktorerna som kan påverka datakvaliteten är den varierande tillgängligheten till väderobservationer, samt kvaliteten på den globala modellen vilken ger information om vädret utanför Europa. För att undersöka om det finns tidsmässiga skillnader i hur konsistent eller inkonsistent kvaliteten på UERRA-återanalysen är, har väderdatat från denna analyserats med avseende på temperatur, vindstyrka och lufttryckshöjd. Arbetet har fokuserats på enbart landpunkter över Europa för sommar och vinter, då detta möjliggör att kunna se hur bra modellen presterar vid de allra lägsta och högsta temperaturerna. Datat har utvärderats genom att undersöka hur tillförlitlig en prognos för 30 timmar framåt är jämfört med en prognos för 6 timmar framåt. Resultaten visar att kvaliteten på återanalysdatat i UERRA inte är konsistent genom hela tidsperioden, där de största skillnaderna hittades för lufttryckshöjden följt av vindstyrkan. För alla tre parametrar hittades betydande kvalitetsskillnader från vilken typ av global modell som används för att ge väderinformation utanför Europa, då UERRA byter global modell under år 1979. För lufttryckshöjden sågs även att datakvaliteten ökar konsekvent även efter 1979 och framåt, vilket därmed är ett resultat från den ökande mängden väderobservationer. Generellt sågs en högre prognoskvalitet sommartid än vintertid, vilket tros vara ett resultat från att vädret varierar mycket mer vintertid vilket därmed bör göra det mer svårprognostiserat. Dessa skillnader i datakvaliteten bör tydliggöras för alla användare av UERRA-återanalysen, då det är viktigt att ha kännedom om detta före eventuella slutsatser dras från återanalysdatat om hur vädret har varit historiskt sett.

Video editing and object recognition are two significant fields in computer vi- sion: the first has remarkably assisted digital production and post-production tasks of a digital video footage; the second is considered fundamental to image classification or image based search in large databases (e.g. the web). In this thesis, we address two problems, namely we present a novel formulation that tackles video editing tasks and we develop a mechanism that allows to generate more robust descriptors for objects in an image. Concerning the first problem, this thesis proposes two variational models to perform temporally coherent video editing. These models are applied to change an object’s (rigid or non-rigid) texture throughout a given video sequence. One model is based on propagating color information from a given frame (or be- tween two given frames) along the motion trajectories of the video; while the other is based on propagating gradient domain information. The models we present in this thesis require minimal user intervention and they automatically accommodate for illumination changes in the scene. Concerning the second problem, this thesis addresses the problem of affine invariance in object recognition. We introduce a way to generate geometric affine invariant quantities that are used in the construction of feature descrip- tors. We show that when these quantities are used they do indeed achieve a more robust recognition than the state of the art descriptors. i / La edición de vídeo y el reconocimiento de objetos son dos áreas fundamentales en el campo de la visión por computador: la primera es de gran utilidad en los procesos de producción y post-producción digital de vídeo; la segunda es esencial para la clasificación o búsqueda de imágenes en grandes bases de datos (por ejemplo, en la web). En esta tesis se acometen ambos problemas, en concreto, se presenta una nueva formulación que aborda las tareas de edición de vídeo y se desarrolla un mecanismo que permite generar descriptores más robustos para los objetos de la imagen. Con respecto al primer problema, en esta tesis se proponen dos modelos variacionales para llevar a cabo la edición de vídeo de forma coherente en el tiempo. Estos modelos se aplican para cambiar la textura de un objeto (rígido o no) a lo largo de una secuencia de vídeo dada. Uno de los modelos está basado en la propagación de la información de color desde un determinado cuadro de la secuencia de vídeo (o entre dos cuadros dados) a lo largo de las trayectorias de movimiento del vídeo. El otro modelo está basado en la propagación de la información en el dominio del gradiente. Ambos modelos requieren una intervención mínima por parte del usuario y se ajustan de manera automática a los cambios de iluminación de la escena. Con respecto al segundo problema, esta tesis aborda el problema de la invariancia afín en el reconocimiento de objetos. Se introduce un nuevo método para generar cantidades geométricas afines que se utilizan en la generación de descriptores de características. También se demuestra que el uso de dichas cantidades proporciona mayor robustez al reconocimiento que los descriptores existentes actualmente en el estado del arte.

No description available.

Diplomová práce se zabývá porozuměním dynamických scén pro navigaci mobilních robotů. V první části předkládáme nový přístup k "sebe-učícím" modelům - fůzi odhadu úběžníku cesty založeného na frekvenčním zpracování a pravděpodobnostních modelech využívající barvu pro segmentaci. Detekce úběžníku cesty je založena na odhadu dominantních orientací texturního toku, získáného pomocí banky Gaborových vlnek, a hlasování. Úběžník cesty poté definuje trénovací oblast, která se využívá k samostatnému učení barevných modelů. Nakonec, oblasti tvořící cestu jsou vybrány pomocí měření Mahalanobisovi vzdálenosti. Pár pravidel řeší situace, jako jsou mohutné stíny, přepaly a rychlost adaptivity. Kromě toho celý odhad úběžníku cesty je přepracován - vlnky jsou nahrazeny aproximacemi pomocí binárních blokových funkcí, což umožňuje efektivní filtraci pomocí integrálních obrazů. Nejužší hrdlo celého algoritmu bylo samotné hlasování, proto překládáme schéma, které nejdříve provede hrubý odhad úběžníku a následně jej zpřesní, čímž dosáhneme výrazně vyšší rychlosti (až 40x), zatímco přesnost se zhorší pouze o 3-5%. V druhé části práce předkládáme vyhlazovací filtr pro prostorovo-časovou konzistentnost predikcí, která je důležitá pro vyspělé systémy. Klíčovou částí filtru je nová metrika měřící podobnost mezi třídami, která rozlišuje mnohem lépe než standardní Euclidovská vzdálenost. Tato metrika může být použita k nejrůznějším úlohám v počítačovém vidění. Vyhlazovací filtr nejdříve odhadne optický tok, aby definoval lokální okolí. Toto okolí je použito k rekurzivní filtraci založené na podobnostní metrice. Celková přesnost předkládané metody měřená na pixelech, které nemají shodné predikce mezi původními daty a vyfiltrovanými, je téměř o 18% vyšší než u původních predikcí. Ačkoliv využíváme SHIM jako zdroj původních predikcí, algoritmus může být kombinován s kterýmkoliv jiným systémem (MRF, CRF,...), který poskytne predikce ve formě pravěpodobností. Předkládaný filtr představuje první krok na cestě k úplnému usuzování.