In the last decade, the machine learning universe embraced deep neural networks (DNNs) wholeheartedly with the advent of neural architectures such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), transformers, etc. These models have empowered many applications, such as ChatGPT, Imagen, etc., and have achieved state-of-the-art (SOTA) performance on many vision, speech, and language modeling tasks. However, SOTA performance comes with various issues, such as large model size, compute-intensive training, increased inference latency, higher working memory, etc. This thesis aims at improving the resource efficiency of neural architectures, i.e., significantly reducing the computational, storage, and energy consumption of a DNN without any significant loss in performance. Towards this goal, we explore novel neural architectures as well as training algorithms that allow low-capacity models to achieve near SOTA performance. We divide this thesis into two dimensions: \textit{Efficient Low Complexity Models}, and \textit{Input Hardness Adaptive Models}. Along the first dimension, i.e., \textit{Efficient Low Complexity Models}, we improve DNN performance by addressing instabilities in the existing architectures and training methods. We propose novel neural architectures inspired by ordinary differential equations (ODEs) to reinforce input signals and attend to salient feature regions. In addition, we show that carefully designed training schemes improve the performance of existing neural networks. We divide this exploration into two parts: \textsc{(a) Efficient Low Complexity RNNs.} We improve RNN resource efficiency by addressing poor gradients, noise amplifications, and BPTT training issues. First, we improve RNNs by solving ODEs that eliminate vanishing and exploding gradients during the training. To do so, we present Incremental Recurrent Neural Networks (iRNNs) that keep track of increments in the equilibrium surface. Next, we propose Time Adaptive RNNs that mitigate the noise propagation issue in RNNs by modulating the time constants in the ODE-based transition function. We empirically demonstrate the superiority of ODE-based neural architectures over existing RNNs. Finally, we propose Forward Propagation Through Time (FPTT) algorithm for training RNNs. We show that FPTT yields significant gains compared to the more conventional Backward Propagation Through Time (BPTT) scheme. \textsc{(b) Efficient Low Complexity CNNs.} Next, we improve CNN architectures by reducing their resource usage. They require greater depth to generate high-level features, resulting in computationally expensive models. We design a novel residual block, the Global layer, that constrains the input and output features by approximately solving partial differential equations (PDEs). It yields better receptive fields than traditional convolutional blocks and thus results in shallower networks. Further, we reduce the model footprint by enforcing a novel inductive bias that formulates the output of a residual block as a spatial interpolation between high-compute anchor pixels and low-compute cheaper pixels. This results in spatially interpolated convolutional blocks (SI-CNNs) that have better compute and performance trade-offs. Finally, we propose an algorithm that enforces various distributional constraints during training in order to achieve better generalization. We refer to this scheme as distributionally constrained learning (DCL). In the second dimension, i.e., \textit{Input Hardness Adaptive Models}, we introduce the notion of the hardness of any input relative to any architecture. In the first dimension, a neural network allocates the same resources, such as compute, storage, and working memory, for all the inputs. It inherently assumes that all examples are equally hard for a model. In this dimension, we challenge this assumption using input hardness as our reasoning that some inputs are relatively easy for a network to predict compared to others. Input hardness enables us to create selective classifiers wherein a low-capacity network handles simple inputs while abstaining from a prediction on the complex inputs. Next, we create hybrid models that route the hard inputs from the low-capacity abstaining network to a high-capacity expert model. We design various architectures that adhere to this hybrid inference style. Further, input hardness enables us to selectively distill the knowledge of a high-capacity model into a low-capacity model by cleverly discarding hard inputs during the distillation procedure. Finally, we conclude this thesis by sketching out various interesting future research directions that emerge as an extension of different ideas explored in this work.

Modern numerical ordinary differential equation initial-value problem (ODE-IVP) solvers compute a piecewise polynomial approximate solution to the mathematical problem. Evaluating the mathematical problem at this approximate solution defines the defect. Corless and Corliss proposed rigorous defect control of numerical ODE-IVP. This thesis automates rigorous defect control for explicit, first-order, nonlinear ODE-IVP. Defect control is residual-based backward error analysis for ODE, a special case of Wilkinson's backward error analysis. This thesis describes a complete software implementation of the Corless and Corliss algorithm and extensive numerical studies. Basic time-stepping software is adapted to defect control and implemented. Advances in software developed for validated computing applications and advances in programming languages supporting operator overloading enable the computation of a tight rigorous enclosure of the defect evaluated at the approximate solution with Taylor models. Rigorously bounding a norm of the defect, the Corless and Corliss algorithm controls to mathematical certainty the norm of the defect to be less than a user specified tolerance over the integration interval. The validated computing software used in this thesis happens to compute a rigorous supremum norm. The defect of an approximate solution to the mathematical problem is associated with a new problem, the perturbed reference problem. This approximate solution is often the product of a numerical procedure. Nonetheless, it solves exactly the new problem including all errors. Defect control accepts the approximate solution whenever the sup-norm of the defect is less than a user specified tolerance. A user must be satisfied that the new problem is an acceptable model. / Thesis / Master of Science (MSc) / Many processes in our daily lives evolve in time, even the weather. Scientists want to predict the future makeup of the process. To do so they build models to model physical reality. Scientists design algorithms to solve these models, and the algorithm implemented in this project was designed over 25 years ago. Recent advances in mathematics and software enabled this algorithm to be implemented. Scientific software implements mathematical algorithms, and sometimes there is more than one software solution to apply to the model. The software tools developed in this project enable scientists to objectively compare solution techniques. There are two forces at play; models and software solutions. This project build software to automate the construction of the exact solution of a nearby model. That's cool.

Ce travail, à la frontière de l’analyse et des probabilités, s’intéresse à l’étude de systèmes différentiels a priori mal posés. Nous cherchons, grâce à des techniques issues de la théorie des chemins rugueux et de l’étude trajectorielle des processus stochastiques, à donner un sens à de tels systèmes puis à les résoudre, tout en montrant que les notions proposées ici étendent bien les notions classiques de solutions. Cette thèse se décompose en trois chapitres. Le premier traite des systèmes différentiels ordinaires perturbés additivement par des processus irréguliers éventuellement stochastiques ainsi que des effets de régularisation de tels processus. Le deuxième chapitre concerne l’équation de transport linéaire perturbée multiplicativement par des chemins rugueux ; enfin, le dernier chapitre s’intéresse à une équation de la chaleur non linéaire perturbée par un bruit blanc espace-temps, l’équation de quantisation stochastique phi4 en dimension 3. / In this work we investigate a priori ill-posed differential systems from an analytic and probabilistic point of view. Thanks to technics inspired by the rough path theory and pathwise study of stochastic processes, we want to define those ill-posed systems and then study them. The first chapter of this thesis is related to ordinary differential equations perturbed by some irregular (stochastic) processes and the effects induced by the regularization of such processes. The second chapter deals with the linear transport equation multiplicatively perturbed by a rough path. Finally, in the last chapter we investigate the stochastic quantization equation Phi4 in three dimensions.

Ce travail s’inscrit dans un contexte de contrôle de la pollution des ressources en eau. On s’intéresse plus particulièrement à l’impact des engrais d’origine agricole sur la qualité de l’eau, en alliant modélisation économique et hydrogéologique. Pour cela, nous définissons d’une part un objectif économique spatio-temporel prenant en compte le compromis entre l’utilisation d’engrais et les coûts de dépollution. D’autre part, nous décrivons le transport du polluant dans le sous-sol (3D en espace) par un système non linéaire d’équations aux dérivées partielles couplées de type parabolique (réaction-convection-dispersion) et elliptique dans un domaine borné. Nous prouvons l’existence globale d’une solution au problème de contrôle optimal. L’unicité est quant à elle démontrée par analyse asymptotique pour le problème effectif tenant compte de la faible concentration d’engrais en sous-sol. Nous établissons les conditions nécessaires d’optimalité et le problème adjoint associé à notre modèle. Quelques exemples analytiques sont donnés et illustrés. Nous élargissons ces résultats au cadre de la théorie des jeux, où plusieurs joueurs interviennent, et prouvons notamment l’existence d’un équilibre de Nash. Enfin, ce travail est illustré par des résultats numériques (2D en espace), obtenus en couplant un schéma de type Éléments Finis Mixtes avec un algorithme de gradient conjugué non linéaire. / This work is devoted to water ressources pollution control. We especially focus on the impact of agricultural fertilizer on water quality, by combining economical and hydrogeological modeling. We define, on one hand, the spatio-temporal objective, taking into account the trade off between fertilizer use and the cleaning costs. On an other hand, we describe the pollutant transport in the underground (3D in space) by a nonlinear system coupling a parabolic partial differential equation (reaction-advection-dispersion) with an elliptic one in a bounded domain. We prove the global existence of the solution of the optimal control problem. The uniqueness is proved by asymptotic analysis for the effective problem taking into account the low concentration fertilizer. We define the optimal necessary conditions and the adjoint problem associated to the model. Some analytical results are provided and illustrated. We extend these results within the framework of game theory, where several players are involved, and we prove the existence of a Nash equilibrium. Finally, this work is illustrated by numerical results (2D in space), produced by coupling a Mixed Finite Element scheme with a nonlinear conjugate gradient algorithm.

L'objectif de cette thèse est de contribuer à l'optimisation de problèmes dynamiques en présence de retard. Le point de vue qui nous intéressera est celui de Pontryagin qui dans son ouvrage publié en 1962 a donné les conditions nécessaires d'existence de solutions pour ce type de problème. Warga dans son ouvrage publié en 1972 a fait un catalogue des solutions possible, Li et al. ont étudié le cas de contrôle périodique. Notre méthode de démonstration est directement inspirée de la démonstration de P. Michel du cas des systèmes gouvernés par des équations différentielles ordinaires. La principale difficulté pour cette approche est l'utilisation de la résolvante de l'équation différentielle fonctionnelle linéarisée de l'équation différentielle fonctionnelle d'évolution qui gouverne le système. Nous traitons aussi de condition d'Euler-Lagrange dans le cadre d'un problème de calcul variationnel avec retard. / In this thesis, we have attempted to contribute to the optimization of dynamical problems with delay in state space. We are specifically interested in the viewpoint of Pontryagin who outlined in his book published in 1962 the necessary conditions required for solving such problems. In his work published in 1972, Warga catalogued the possible solutions. Li and al. analyzed the case of periodic control. We will treat an optimal control problem governed by a Delay Functional Differential Equation. Our method is close to the one of P. Michel on dynamical system governed by Ordinary Differential Equations. The main problem ariving out in this approach is the use of the resolvent of the Delay Functional Differential Equation. We also consider with Euler-Lagrange condition in the framework of variational problems with delay.

We analyze a number of systems of evolution equations that arise in the study of physical chemistry. First we discuss the well-posedness of a system of mixing compressible barotropic multicomponent flows. We discuss the regularity of these variational solutions, their existence and uniqueness, and we analyze the emergence of a novel type of entropy that is derived for the system of equations. Next we present a numerical scheme, in the form of a discontinuous Galerkin (DG) finite element method, to model this compressible barotropic multifluid. We find that the DG method provides stable and accurate solutions to our system, and that further, these solutions are energy consistent; which is to say that they satisfy the classical entropy of the system in addition to an additional integral inequality. We discuss the initial-boundary problem and the existence of weak entropy at the boundaries. Next we extend these results to include more complicated transport properties (i.e. mass diffusion), where exotic acoustic and chemical inlets are explicitly shown. We continue by developing a mixed method discontinuous Galerkin finite element method to model quantum hydrodynamic fluids, which emerge in the study of chemical and molecular dynamics. These solutions are solved in the conservation form, or Eulerian frame, and show a notable scale invariance which makes them particularly attractive for high dimensional calculations. Finally we implement a wide class of chemical reactors using an adapted discontinuous Galerkin finite element scheme, where reaction terms are analytically integrated locally in time. We show that these solutions, both in stationary and in flow reactors, show remarkable stability, accuracy and consistency. / text

The main motivation behind writing this thesis was to construct numerical methods to approximate solutions to differential equations driven by rough paths, where the solution is considered in the rough path-sense. Rough paths of inhomogeneous degree of smoothness as driving noise are considered. We also aimed to find applications of these numerical methods to stochastic differential equations. After sketching the core ideas of the Rough Paths Theory in Chapter 1, the versions of the core theorems corresponding to the inhomogeneous degree of smoothness case are stated and proved in Chapter 2 along with some auxiliary claims on the continuity of the solution in a certain sense, including an RDE-version of Gronwall's lemma. In Chapter 3, numerical schemes for approximating solutions to differential equations driven by rough paths of inhomogeneous degree of smoothness are constructed. We start with setting up some principles of approximations. Then a general class of local approximations is introduced. This class is used to construct global approximations by pasting together the local ones. A general sufficient condition on the local approximations implying global convergence is given and proved. The next step is to construct particular local approximations in finite dimensions based on solutions to ordinary differential equations derived locally and satisfying the sufficient condition for global convergence. These local approximations require strong conditions on the one-form defining the rough differential equation. Finally, we show that when the local ODE-based schemes are applied in combination with rough polynomial approximations, the conditions on the one-form can be weakened. In Chapter 4, the results of Gyurko & Lyons (2010) on path-wise approximation of solutions to stochastic differential equations are recalled and extended to the truncated signature level of the solution. Furthermore, some practical considerations related to the implementation of high order schemes are described. The effectiveness of the derived schemes is demonstrated on numerical examples. In Chapter 5, the background theory of the Kusuoka-Lyons-Victoir (KLV) family of weak approximations is recalled and linked to the results of Chapter 4. We highlight how the different versions of the KLV family are related. Finally, a numerical evaluation of the autonomous ODE-based versions of the family is carried out, focusing on SDEs in dimensions up to 4, using cubature formulas of different degrees and several high order numerical ODE solvers. We demonstrate the effectiveness and the occasional non-effectiveness of the numerical approximations in cases when the KLV family is used in its original version and also when used in combination with partial sampling methods (Monte-Carlo, TBBA) and Romberg extrapolation.

No description available.

In this thesis, we study the existence of pathwise random periodic solutions to both the semilinear stochastic differential equations with linear multiplicative noise and the semilinear stochastic partial differential equations with linear multiplicative noise in a Hilbert space. We identify them as the solutions of coupled forward-backward infinite horizon stochastic integral equations in general cases, and then perform the argument of the relative compactness of Wiener-Sobolev spaces in C([0, T],L2Ω,Rd)) or C([0, T],L2(Ω x O)) and Schauder's fixed point theorem to show the existence of a solution of the coupled stochastic forward-backward infinite horizon integral equations.

Im Rahmen dieser Dissertation bemühe ich mich, den statistischen Ansatz der zeitkontinuierlichen dynamischen Modellierung, der die Rolle der Zeit explizit berücksichtigt, zu erweitern und praktisch anwendbar zu machen. Diese Dissertation ist so strukturiert, dass ich in Kapitel 1 die Natur dynamischer Modelle bespreche, verschiedene Ansätze zum Umgang mit mehreren Personen betrachte und ein zeitkontinuierliches dynamisches Modell mit Input-Effekten (wie Interventionen) und einem Gaußschen Messmodell detailliert darstelle. In Kapitel 2 beschreibe ich die Verwendung der Software ctsem für R, die als Teil dieser Dissertation entwickelt wurde und die Modellierung von Strukturgleichungen und Mixed-Effects über einen frequentistischen Schätzansatz realisiert. In Kapitel 3 stelle ich einen hierarchischen, komplett Random-Effects beinhaltenden Bayesschen Schätzansatz vor, unter dem sich Personen nicht nur in Interceptparametern, sondern in allen Charakteristika von Mess - und Prozessmodell unterscheiden können, wobei die Schätzung individueller Parameter trotzdem von den Daten aller Personen profitiert. Kapitel 4 beschreibt die Verwendung der Bayesschen Erweiterung der Software ctsem. In Kapitel 5} betrachte ich die Natur experimenteller Interventionen vor dem Hintergrund zeitkontinuierlicher dynamischer Modellierung und zeige Ansätze, die die Art und Weise adressieren, mit der Interventionen auf psychologische Prozesse über die Zeit wirken. Das berührt Fragen, wie: 'Nach welcher Zeit zeigt eine Intervention ihre maximale Wirkung', 'Wie ändert sich die Form des Effektes im Laufe der Zeit' und 'Für wen ist die Wirkung am stärksten oder dauert am längsten an'. Viele Bei-spiele, die sowohl frequentistische als auch bayessche Formen der Software ctsem verwenden, sind enthalten. Im letzten Kapitel fasse ich die Dissertation zusammen, zeige Limitationen der angebotenen Ansätze auf und stelle meine Gedanken zu möglichen zukünftigen Entwicklungen dar. / With this dissertation I endeavor to extend, and make practically applicable for psychology, the statistical approach of continuous time dynamic modelling, in which the role of time is made explicit. The structure of this dissertation is such that in Chapter 1, I discuss the nature of dynamic models, consider various approaches to handling multiple subjects, and detail a continuous time dynamic model with input effects (such as interventions) and a Gaussian measurement model. In Chapter 2, I describe the usage of the ctsem software for R developed as part of this dissertation, which provides a frequentist, mixed effects, structural equation modelling approach to estimation. Chapter 3 details a hierarchical Bayesian, fully random effects approach to estimation, allowing for subjects to differ not only in intercept parameters but in all characteristics of the measurement and dynamic models -- while still benefiting from other subjects data for parameter estimation. Chapter 4 describes the usage of the Bayesian extension to the ctsem software. In Chapter 5 I consider the nature of experimental interventions in the continuous time dynamic modelling framework, and show approaches to address questions regarding the way interventions influence psychological processes over time, with questions such as 'how long does a treatment take to reach maximum effect', `how does the shape of the effect change over time', and 'for whom is the effect strongest, or longest lasting'. Many examples using both frequentist and Bayesian forms of the ctsem software are given. For the final chapter I summarise the dissertation, consider limitations of the approaches offered, and provide some thoughts on possible future developments.