Increasingly, online computer applications rely on large-scale data analyses to offer personalised and improved products. These large-scale analyses are performed on distributed data processing execution engines that run on thousands of networked machines housed within an individual data centre. These execution engines provide, to the programmer, the illusion of running data analysis workflows on a single machine, and offer programming interfaces that shield developers from the intricacies of implementing parallel, fault-tolerant computations. Many such execution engines exist, but they embed assumptions about the computations they execute, or only target certain types of computations. Understanding these assumptions involves substantial study and experimentation. Thus, developers find it difficult to determine which execution engine is best, and even if they did, they become “locked in” because engineering effort is required to port workflows. In this dissertation, I first argue that in order to execute data analysis computations efficiently, and to flexibly choose the best engines, the way we specify data analysis computations should be decoupled from the execution engines that run the computations. I propose an architecture for decoupling data processing, together with Musketeer, my proof-of-concept implementation of this architecture. In Musketeer, developers express data analysis computations using their preferred programming interface. These are translated into a common intermediate representation from which code is generated and executed on the most appropriate execution engine. I show that Musketeer can be used to write data analysis computations directly, and these can execute on many execution engines because Musketeer automatically generates code that is competitive with optimised hand-written implementations. The diverse execution engines cause different workflow types to coexist within a data centre, opening up both opportunities for sharing and potential pitfalls for co-location interference. However, in practice, workflows are either placed by high-quality schedulers that avoid co-location interference, but choose placements slowly, or schedulers that choose placements quickly, but with unpredictable workflow run time due to co-location interference. In this dissertation, I show that schedulers can choose high-quality placements with low latency. I develop several techniques to improve Firmament, a high-quality min-cost flow-based scheduler, to choose placements quickly in large data centres. Finally, I demonstrate that Firmament chooses placements at least as good as other sophisticated schedulers, but at the speeds associated with simple schedulers. These contributions enable more efficient and effective use of data centres for large-scale computation than current solutions.

The scope of this thesis is to design and implement a system for deploying, managing and monitoring a Raspberry Pi cluster using Nix technologies. The thesis describes the benefits of the functional approach of Nix and the subsystems that are based on it. The thesis also results in a supporting web application, providing an intuitive environment for working with cluster configuration deployments and clearly displaying information about the utilization of individual nodes using dashboards. The final part of the thesis is devoted to testing cluster performance using sample distributed computing jobs.

Thesis (MScEng)--Stellenbosch University, 2013. / ENGLISH ABSTRACT: Clustering has emerged as one of the drivers for economic growth and improved competitiveness for specific industries. In order for cluster initiatives to be successful, they need to be managed in a certain way. The importance of excellent cluster management has been increasingly realised in the past years. Certain factors of cluster management are considered to lead to success in the management of cluster initiatives as identified in published research. In the same vain, certain aspects of cluster program management are considered to lead to success. A cluster program run by the Western Cape Provincial Government (WCPG) in South Africa is analysed and compared to the generic success factors in cluster program management. Selected cluster initiatives in the Western Cape are analysed and compared with each other as well as to these success factors in cluster management. The Western Cape Tooling Initiative (WCTI), a cluster initiative in the tool, die and mould making industry, is analysed in more detail, and compared to the other cluster initiatives, all of which but one are part of the cluster program. WCTI is found to not have been following success factors in cluster management, and the rate of progress of the WCTI from when it was started supports that fact. However, recent changes in the leadership of the cluster initiative have brought about changes in WCTI’s cluster management emphasis and signs of increased success can be noticed, even though the full effect will not be felt until after some years as success in cluster management takes years to be realised. / AFRIKAANSE OPSOMMING: Trosvervaardiging het in die onlangse verlede ontwikkel tot een van die drywers vir ekonomiese groei en verhoogde kompeterendheid in spefifieke industrië. Tros inisiatiewe vereis ‘n spesifieke bestuursraamwerk om hulle potensiaal te bereik, en die belangrikheid van uitstekende bestuurspraktyke word al hoe meer beklemtoon. Sekere faktore, beskryf in literatuur, van bestuur bleik te lei na sukses in die bestuur van sulke inisiatiewe. Op ‘n soortgelyke basis is daar aspekte van trosprogram bestuur wat voorvereistes vir sukses is. ‘n Trosprogram onder leiding van die Wes Kaapse Provinsiale Regering in Suid Afrika word analiseer en vergelyk met die generiese sukses faktore van trosprogrambestuur. Verskeie trosinisiatiewe in die Wes Kaap word ondersoek en met mekaar vergelyk asook gemeet aan hierdie suksesfaktore. Meer spesifiek word die “Western Cape Tooling Initiative” (WCTI) in meer detail analiseer, en vergelyk met die ander inisiatiewe wat almal, met een uitsondering, deel is van die trosprogram self. Daar is gevind dat die WCTI nie streng volgens die riglyne en/of binne die raamwerk van suksesvolle trosbestuur opgetree het nie, en dit word gestaaf deur die relatief lae sukseskoers tot dusver. Met die onlangse veranderinge aan die leierskap van die inisiatief bleik dit egter dat meer aandag aan hierdie praktyke gegee word, en daar is tekens van ‘n verbetering in die prestasie van die program. Die volle effek van hierdie veranderinge sal egter ‘n rukkie neem om volledig tot uitwerking te kom.

Esta tese tem por fim facilitar o desenvolvimento da indústria marítima brasileira. Os clusters industriais são amplamente reconhecidos como fontes de promoção da evolução tecnológica e por impactar positivamente em muitos aspectos do desenvolvimento regional. Por outro lado, os estaleiros de construção e reparo naval são tidos como estratégicos a soberania de muitos países, função do largo emprego de mão-de-obra e por produzir bens que possibilitem o fortalecimento das relações comerciais entre estados e países. Sendo a convergência destas duas situações bastante promissoras, decidiu-se focar esforços em propor um modelo de referência que pudesse ser empregado para desenvolvimento de clusters de indústrias marítimas. Estudos empíricos sobre clusters de indústrias relatam algumas externalidades positivas características, observadas em clusters bem-sucedidos. Outros trabalhos desenvolveram estudos sobre alguns fatores que são fundamentais ao sucesso de clusters. Existem ainda aqueles que externam algumas ações que foram tomadas para o desenvolvimento de clusters diversos. Esta tese propõe confluir estes trabalhos, encontrando as relações entre as ações, os fatores que condicionam o sucesso de clusters de indústria marítima e suas externalidades positivas para estabelecer uma proposta de modelo de ações para o desenvolvimento e consolidação de clusters de indústria marítima. Neste trabalho foi feita também, a avaliação dos reflexos das externalidades destes fatores que são fundamentais para o desenvolvimento de clusters nos fatores críticos de sucesso das empresas individuais. Para materialização desta proposta, estudou-se diversos trabalhos teóricos e empíricos sobre clusters industriais, situados tanto em países desenvolvidos como nos ainda em desenvolvimento. A premissa básica do trabalho foi determinar que fatores são determinantes para o desenvolvimento de clusters e quais objetivos estratégicos deveriam ser seguidos para que se pudesse consegui-los. Os esforços de pesquisa resultaram na materialização de uma proposta de modelo estratégico genérico, que, com as devidas adequações as características e necessidades regionais específicas, pode ser empregado para o desenvolvimento de qualquer cluster de indústria marítima. Em uma segunda etapa do trabalho, foi proposta a aplicação do modelo de referência para o desenvolvimento do cluster marítimo da região de Itajaí e Navegantes, em Santa Catarina. Para tanto, foi feita pesquisa de campo em pouco mais de 40 atores regionais, com a finalidade de determinar o estágio de desenvolvimento do cluster e qual nível de evolução e presença dos fatores que levam um cluster a ser bem-sucedido. Após a aplicação hipotética da proposta do modelo, os resultados foram colocados para avaliação de 4 gestores de empresas da região que acenaram positivamente para identificação dos resultados com a realidade regional. O modelo estratégico proposto mostrou-se sólido e abrangente, com boas perspectivas de sucesso na aplicação para o desenvolvimento de clusters de indústrias marítimas. / This thesis aims to facilitate the Brazilian maritime industry development. Industrial clusters are widely recognized as a source of technological evolution promotion as well as for bringing positive impact over a range of aspects in the regional development. On the other hand, construction and repair boatyards are taken as strategic for many countries sovereignty, having the function of widespread hand labor and employment, and goods production that allow strengthening trade relations among states and countries. Considering the convergence of these two situations much promising, it was decided to focus efforts towards a reference model to be used for the development of maritime industries clusters. Industries clusters empirical studies report some characteristic positive externalities observed in successful clusters. Other studies discussed on some fundamental factors for clusters success. There are also those studies that point out some actions taken for the development of a variety of clusters. This thesis propose to converge these works, joining the actions relations, the conditioning factors for the maritime industries clusters and its positive externalities in order to establish an actions model proposal for the maritime industries clusters development and consolidation. This thesis includes also the assessment of the externalities consequences of these factors, which are essential for the clusters development in the critical success factor of individual companies. Several theoretical and empirical studies on industrial clusters, located in developed and underdeveloped countries were studied in order to achieve this proposal. The basic premise of the study was to determine decisive factors for clusters development and the strategic objectives that should be followed for their accomplishment. The research efforts resulted in the materialization of a generic strategic model proposal, which the appropriate adjustments to the specific regional needs and characteristics, can be used for the development of any marine industrial cluster. In a Second stage, it was proposed the reference model to be used to develop the maritime cluster in Itajaí and Navegantes region, state of Santa Catarina. Therefore, a field survey was conducted over a few more than 40 actors, in order to determine the cluster development stage and evaluate the evolution level and presence of the factors that may enable a cluster to be successful. After the hypothetical application of the proposed model, the results were placed for evaluation of 4 region companies\' managers who waved positively to identify the results to the regional reality. The proposed strategic model proved to be solid and comprehensive, with good prospects of application success for the maritime industrial clusters development.

Esta tese tem por fim facilitar o desenvolvimento da indústria marítima brasileira. Os clusters industriais são amplamente reconhecidos como fontes de promoção da evolução tecnológica e por impactar positivamente em muitos aspectos do desenvolvimento regional. Por outro lado, os estaleiros de construção e reparo naval são tidos como estratégicos a soberania de muitos países, função do largo emprego de mão-de-obra e por produzir bens que possibilitem o fortalecimento das relações comerciais entre estados e países. Sendo a convergência destas duas situações bastante promissoras, decidiu-se focar esforços em propor um modelo de referência que pudesse ser empregado para desenvolvimento de clusters de indústrias marítimas. Estudos empíricos sobre clusters de indústrias relatam algumas externalidades positivas características, observadas em clusters bem-sucedidos. Outros trabalhos desenvolveram estudos sobre alguns fatores que são fundamentais ao sucesso de clusters. Existem ainda aqueles que externam algumas ações que foram tomadas para o desenvolvimento de clusters diversos. Esta tese propõe confluir estes trabalhos, encontrando as relações entre as ações, os fatores que condicionam o sucesso de clusters de indústria marítima e suas externalidades positivas para estabelecer uma proposta de modelo de ações para o desenvolvimento e consolidação de clusters de indústria marítima. Neste trabalho foi feita também, a avaliação dos reflexos das externalidades destes fatores que são fundamentais para o desenvolvimento de clusters nos fatores críticos de sucesso das empresas individuais. Para materialização desta proposta, estudou-se diversos trabalhos teóricos e empíricos sobre clusters industriais, situados tanto em países desenvolvidos como nos ainda em desenvolvimento. A premissa básica do trabalho foi determinar que fatores são determinantes para o desenvolvimento de clusters e quais objetivos estratégicos deveriam ser seguidos para que se pudesse consegui-los. Os esforços de pesquisa resultaram na materialização de uma proposta de modelo estratégico genérico, que, com as devidas adequações as características e necessidades regionais específicas, pode ser empregado para o desenvolvimento de qualquer cluster de indústria marítima. Em uma segunda etapa do trabalho, foi proposta a aplicação do modelo de referência para o desenvolvimento do cluster marítimo da região de Itajaí e Navegantes, em Santa Catarina. Para tanto, foi feita pesquisa de campo em pouco mais de 40 atores regionais, com a finalidade de determinar o estágio de desenvolvimento do cluster e qual nível de evolução e presença dos fatores que levam um cluster a ser bem-sucedido. Após a aplicação hipotética da proposta do modelo, os resultados foram colocados para avaliação de 4 gestores de empresas da região que acenaram positivamente para identificação dos resultados com a realidade regional. O modelo estratégico proposto mostrou-se sólido e abrangente, com boas perspectivas de sucesso na aplicação para o desenvolvimento de clusters de indústrias marítimas. / This thesis aims to facilitate the Brazilian maritime industry development. Industrial clusters are widely recognized as a source of technological evolution promotion as well as for bringing positive impact over a range of aspects in the regional development. On the other hand, construction and repair boatyards are taken as strategic for many countries sovereignty, having the function of widespread hand labor and employment, and goods production that allow strengthening trade relations among states and countries. Considering the convergence of these two situations much promising, it was decided to focus efforts towards a reference model to be used for the development of maritime industries clusters. Industries clusters empirical studies report some characteristic positive externalities observed in successful clusters. Other studies discussed on some fundamental factors for clusters success. There are also those studies that point out some actions taken for the development of a variety of clusters. This thesis propose to converge these works, joining the actions relations, the conditioning factors for the maritime industries clusters and its positive externalities in order to establish an actions model proposal for the maritime industries clusters development and consolidation. This thesis includes also the assessment of the externalities consequences of these factors, which are essential for the clusters development in the critical success factor of individual companies. Several theoretical and empirical studies on industrial clusters, located in developed and underdeveloped countries were studied in order to achieve this proposal. The basic premise of the study was to determine decisive factors for clusters development and the strategic objectives that should be followed for their accomplishment. The research efforts resulted in the materialization of a generic strategic model proposal, which the appropriate adjustments to the specific regional needs and characteristics, can be used for the development of any marine industrial cluster. In a Second stage, it was proposed the reference model to be used to develop the maritime cluster in Itajaí and Navegantes region, state of Santa Catarina. Therefore, a field survey was conducted over a few more than 40 actors, in order to determine the cluster development stage and evaluate the evolution level and presence of the factors that may enable a cluster to be successful. After the hypothetical application of the proposed model, the results were placed for evaluation of 4 region companies\' managers who waved positively to identify the results to the regional reality. The proposed strategic model proved to be solid and comprehensive, with good prospects of application success for the maritime industrial clusters development.