Computer networks are engineered for performance and flexibility, delivering billions of packets per second with high reliability, until they fail. It is during such time of crisis that debugging and troubleshooting come to the forefront, however, the focus on performance results in design tradeoffs that make it challenging to troubleshoot them. This dissertation hypothesizes that a view of the network as a single entity solves the above problems, without compromising either performance or visibility. The primary contributions are 1) a topology oblivious network abstraction for performance monitoring and troubleshooting, 2) transformation of the network abstract query to device local semantics, 3) optimizations for reducing state collection overhead, and 4) global state semantics in the proposed query language easing expression of network queries. Abstracting the entire system as an entity simplifies the debugging process, making possible comprehensive root-cause analysis and exonerating the network administrator from dealing with many devices, delivering gains in productivity and efficiency. By merging network topology information with state collection, this thesis provides a new way to look at the network monitoring and troubleshooting problem. Such an amalgamation allows the translation of a performance query expressed in a domain specific language to small pieces of code operating on different devices in the network collecting necessary state. This merging results in lesser overhead per switch and reduces the strain on devices and provides a simple abstraction to the administrator. / PHD / Computer networks are complex and therefore take a tremendous amount of effort to operate in an efficient and error-free manner. This dissertation presents a mechanism that allows the operator to ignore many intricate details about the network and concentrate on how to fix problems. Such an approach enables automatic methods to facilitate the operator to focus on most relevant information. Saving time, money and resources when operating large networks.

We study distributed estimation of dynamic random fields observed by a sparsely connected network of agents/sensors. The sensors are inexpensive, low power, and they communicate locally and perform computation tasks. In the era of large-scale systems and big data, distributed estimators, yielding robust and reliable field estimates, are capable of significantly reducing the large computation and communication load required by centralized estimators, by running local parallel inference algorithms. The distributed estimators have applications in estimation, for example, of temperature, rainfall or wind-speed over a large geographical area; dynamic states of a power grid; location of a group of cooperating vehicles; or beliefs in social networks. The thesis develops distributed estimators where each sensor reconstructs the estimate of the entire field. Since the local estimators have direct access to only local innovations, local observations or a local state, the agents need a consensus-type step to construct locally an estimate of their global versions. This is akin to what we refer to as distributed dynamic averaging. Dynamic averaged quantities, which we call pseudo-quantities, are then used by the distributed local estimators to yield at each sensor an estimate of the whole field. Using terminology from the literature, we refer to the distributed estimators presented in this thesis as Consensus+Innovations-type Kalman filters. We propose three distinct types of distributed estimators according to the quantity that is dynamically averaged: (1) Pseudo-Innovations Kalman Filter (PIKF), (2) Distributed Information Kalman Filter (DIKF), and (3) Consensus+Innovations Kalman Filter (CIKF). The thesis proves that under minimal assumptions the distributed estimators, PIKF, DIKF and CIKF converge to unbiased and bounded mean-squared error (MSE) distributed estimates of the field. These distributed algorithms exhibit a Network Tracking Capacity (NTC) behavior – the MSE is bounded if the degree of instability of the field dynamics is below a threshold. We derive the threshold for each of the filters. The thesis establishes trade-offs between these three distributed estimators. The NTC of the PIKF depends on the network connectivity only, while the NTC of the DIKF and of the CIKF depend also on the observation models. On the other hand, when all the three estimators converge, numerical simulations show that the DIKF improves 2dB over the PIKF. Since the DIKF uses scalar gains, it is simpler to implement than the CIKF. Of the three estimators, the CIKF provides the best MSE performance using optimized gain matrices, yielding an improvement of 3dB over the DIKF. Keywords: Kalman filter, distributed state estimation, multi-agent networks, sensor networks, distributed algorithms, consensus, innovation, asymptotic convergence, mean-squared error, dynamic averaging, Riccati equation, Lyapunov iterations, distributed signal processing, random dynamical systems.