Scalable system monitoring is a fundamental abstraction for large-scale networked systems. The goal of this dissertation is to design and build a scalable monitoring middleware that provides system introspection for large distributed systems and that will facilitate the design, development, and deployment of distributed monitoring applications. This middleware will enable monitoring applications to flexibly control the tradeoff between result precision and communication cost and to improve result accuracy in the face of node failures, network delays, and system reconfigurations. We present PRISM (PRecision-Integrated Scalable Monitoring), a scalable monitoring middleware that provides a global aggregate view of large-scale networked systems and that can serve as a building block for a broad range of distributed monitoring applications by coordinating views of multiple vantage points across the network. To coordinate a global view for system introspection, PRISM faces two key challenges: (1) scalability to large systems and high data volumes and (2) safeguarding accuracy in the face of node and network failures. To address these challenges, we design, implement, and evaluate PRISM, a system that defines precision as a new unified abstraction to enable scalable monitoring. PRISM quantifies (im)precision along a three-dimensional vector: arithmetic imprecision (AI) and temporal imprecision (TI) balance precision against monitoring overhead for scalability while network imprecision (NI) addresses the challenge of providing consistency guarantees despite failures. Our prototype implementation of PRISM addresses the challenge of providing these metrics while scaling to a large number of nodes and attributes by (1) leveraging Distributed Hash Tables (DHTs) to create scalable aggregation trees, (2) self-tuning AI budgets across nodes in a principled, near-optimal manner to shift precision to where it is useful, (3) pipelining TI delays across tree levels to maximize batching of updates, and (4) applying dual-tree prefix aggregation which exploits symmetry in our DHT topology to drastically reduce the cost of the active probing needed to maintain NI. Through extensive simulations and experiments on four large-scale testbeds, we observe that PRISM provides a key substrate for scalable monitoring by (1) reducing monitoring load by up to two orders of magnitude compared to existing approaches, (2) providing a flexible framework to control the tradeoff between accuracy, bandwidth cost, and response latency, (3) characterizing and improving confidence in the accuracy of results in the face of system disruptions, and (4) improving the observed accuracy by up to an order of magnitude despite churn. We have built several monitoring applications on top of PRISM including a distributed heavy hitter detection service, a distributed monitoring service for Internet-scale systems, and a detection service for monitoring distributed-denial-of-service (DDoS) attacks at the source-side in distributed networked systems. Finally, we demonstrate how the unified precision abstraction enables new monitoring applications by presenting experiences from these applications. / text

<p dir="ltr">The onset of smart manufacturing signifies a crucial shift in the industrial landscape, underscoring the pressing need for systems capable of adapting to and managing the complex dynamics of modern production environments. In this context, the importance of smart monitoring becomes increasingly apparent, serving as a vital tool for ensuring operational efficiency and reliability. Inspired by the critical role of auditory perception in human decision-making, this study investigated the application of machine sound recognition for practical use in manufacturing environments. Addressing the challenge of utilizing machine sounds in the loud noises of factories, the study employed an Internal Sound Sensor (ISS).</p><p dir="ltr">The study examined how sound propagates through structures and further explored acoustic characteristics of the ISS, aiming to apply these findings in machine monitoring. To leverage the ISS effectively and achieve a higher level of monitoring, a smart sound monitoring framework was proposed to integrate sound monitoring with machine data and human-machine interface. Designed for applicability and cost effectiveness, this system employs real-time edge computing, making it adaptable for use in various industrial settings.</p><p dir="ltr">The proposed framework and ISS deployed across a diverse range of production environments, showcasing a leap forward in the integration of smart technologies in manufacturing. Their application extends beyond continuous manufacturing to include discrete manufacturing systems, demonstrating adaptability. By analyzing sound signals from various production equipment, this study delves into developing machine sound recognition models that predict operational states and productivity, aiming to enhance manufacturing efficiency and oversight on real factory floors. This comprehensive and practical approach underlines the framework's potential to revolutionize operational management and manufacturing productivity. The study progressed to integrating manufacturing context with sound data, advancing towards high-level monitoring for diagnostic predictions and digital twin. This approach confirmed sound recognition's role in manufacturing diagnostics, laying a foundation for future smart monitoring improvements.</p>