No description available.

Big data processing frameworks utilizing distributed frameworks to parallelize the computing of datasets have become a staple part of the data engineering and data science pipelines. One of the more known frameworks is Dask, a widely utilized distributed framework used for parallelizing data processing jobs. In Dask, the main component that traverses and plans out the execution of the job is the scheduler. Dask utilizes a centralized scheduling approach, having a single server node as the scheduler. With no failover mechanism implemented for the scheduler, the work in progress is potentially lost if the scheduler fails. As a consequence, jobs that might have been executed for hours or longer need to be restarted. In this thesis, a highly available scheduler is designed, based on Dask. We introduce a highly-available scheduler that replicates the state of the job on a distributed key-value store. The replicated schedulers allow us to design an architecture where the schedulers are able to take over the job in case of a scheduler failure. To reduce the performance overhead of replication, we further explore optimizations based on partitioning typical task graphs and sending each partition to its own scheduler. The results show that the replicated scheduler is able to tolerate server failures and is able to complete the job without restarting but at a cost of reduced throughput due to the replication. This is mitigated by our partitioning, which achieves almost linear performance gains relative to our baseline fault-tolerant scheduler, through the utilization of a parallelized scheduling architecture. / Dataprocesseringsramverk av stora datamängder har blivit en viktig del inom Data Engineering och Data Science pipelines. Ett av de mer kända ramverken är Dask som används för att parallelisera jobb inom data processering. En av huvudkomponenterna i Dask är dess schemaläggare som traverserar och planerar exekveringen av av arbete. Dask använder en centraliserad schemaläggning, med en enda server nod som schemaläggare. Utan en implementerad felhanteringsmekanism innebär det att allt arbete är förlorat ifall schemaläggaren kraschar. I denna uppsats så skapar vi en schemaläggare baserad på Dask. Vi introducerar hög tillgänglighet till schemaläggaren genom att replikera statusen av ett jobb till en distribuerad Key-Value store. För att reducera kostnaden av replikationen så utforskas optimeringar genom att partitionera typiska uppgifts-grafer för att sedan skicka dem till varsin schemaläggare. Resultaten visar att en replikerad schemaläggare tolererar att schemaläggningsservarna kraschar, och att den kan slutföra ett jobb utan att behöva starta om, på en kostnad av reducerad schemaläggningseffektivitet på grund av replikationen. Denna reduktion av effektivitet mitigeras av vår partitioningsstrategi, som genom att använda en paralliserad schemaläggningsarkitektur, uppnår nästan linjära prestandaökningar jämfört med den simpla feltoleranta schemaläggaren.

Monitoring waterbirds have several benefits, including analyzing the number of endangered species, giving a reliable indication of public health, etc. Monitoring waterbirds in their habitat is a challenging task since the location is distant, and the collection of monitoring data requires large bandwidth. A promising technology to tackle these challenges is thought to be Wireless Multimedia Sensor Networks (WMSN). These networks are composed of small energy-constrained IoT devices that communicate together to collect data or monitor a given location. Performances in such networks are impacted by not only upper-layer protocols (transmission, routing, application layer) but also Medium Access Control (MAC) Layer. Therefore, improvement in this layer can increase the performance considerably. Traditional contention-based MAC modes like CSMA have large energy expenditure even though they have a good network performance profile. Energy-constrained devices cannot have a long lifespan with this type of MAC layer technology. Therefore, the IEEE 802.15.4e amendment proposed TSCH MAC mode which takes advantage of time-slotted access and channel hopping techniques. IETF integrated TSCH protocol into IPv6-based wireless sensor networks and standardized it as 6TiSCH which is a unique protocol stack for Low-Power and Lossy Networks (LLN). WMSN applications (e.g. Waterbirds Monitoring Application) generates heterogeneous traffic. Heterogeneous traffic can be defined as a mixture of different traffic types (light: temperature, humidity, etc. and heavy: audio, picture, video, etc.). TSCH-based WMSNs are considered a fit for this kind of traffic since they provide better performance and low power usage.  Yet, the 6TiSCH Working Group left open the scheduling of TSCH communication for industries to make TSCH more easily adaptable to any kind of application. Until now, there have been a huge number of scheduling algorithms from industries and academia. Each scheduling algorithm has a different objective that maximizes the network performance of a specific application. This thesis work studies the most recent state-of-the-art scheduling algorithms (protocols) and compares them in a unique simulation environment with heterogeneous traffic to find out which protocol performs well while maintaining low energy consumption. Particularly, this work studies a new approach in TSCH scheduling which is Reinforcement Learning based scheduling. We implemented one of the state-of-the-art RL-based schedulers in Contiki-NG and included it in our comparison of TSCH schedulers. The experiment results showed that the RL-based scheduler implemented in this work demonstrated better performance in PDR and latency compared to other scheduling protocols. However, it presented high energy usage. On the other hand, Orchestra performed well while keeping the energy expenditure of nodes at a low level.

We study the problem of joint congestion control, routing and MAC layer scheduling in multi-hop wireless mesh networks, where the nodes in the network are subjected to energy expenditure rate constraints. As wireless scenario does not allow all the links to be active all the time, only a subset of given links can be active simultaneously. We model the inter-link interference using the link contention graph. All the nodes in the network are power-constrained and we model this constraint using energy expenditure rate matrix. Then we formulate the problem as a network utility maximization (NUM) problem. We notice that this is a convex optimization problem with affine constraints. We apply duality theory and decompose the problem into two sub-problems namely, network layer congestion control and routing problem, and MAC layer scheduling problem. The source adjusts its rate based on the cost of the least cost path to the destination where the cost of the path includes not only the prices of the links in it but also the prices associated with the nodes on the path. The MAC layer scheduling of the links is carried out based on the prices of the links. The optimal scheduler selects that set of non-interfering links, for which the sum of link prices is maximum. We study the effects of energy expenditure rate constraints of the nodes on the maximum possible network utility. It turns out that the dominant of the two constraints namely, the link capacity constraint and the node energy expenditure rate constraint affects the network utility most. Also we notice the fact that the energy expenditure rate constraints do not affect the nature of optimal link scheduling problem. Following this fact, we study the problem of distributed link scheduling. Optimal scheduling requires selecting independent set of maximum aggregate price, but this problem is known to be NP-hard. We first show that as long as scheduling policy selects the set of non-interfering links, it can not go unboundedly away from the optimal solution of network utility maximization problem. Then we proceed and evaluate a simple greedy scheduling algorithm. Analytical bounds on performance are provided and simulations indicate that the greedy heuristic performs well in practice.