The use of internet has increased the amount of data available and gathered. Companies are investing in big data analytics to gain insights from this data. However, the value of the analysis and decisions made based on it, is dependent on the quality ofthe underlying data. For this reason, data quality has become a prevalent issue for organizations. Additionally, failures in data quality management are often due to organizational aspects. Due to the growing popularity of decentralized organizational structures, there is a need to understand how a decentralized organization can improve data quality. This thesis conducts a qualitative single case study of an organization currently shifting towards becoming data driven and struggling with maintaining data quality within the logistics industry. The purpose of the thesis is to answer the questions: • RQ1: What is data quality in the context of logistics data? • RQ2: What are the obstacles for improving data quality in a decentralized organization? • RQ3: How can these obstacles be overcome? Several data quality dimensions were identified and categorized as critical issues,issues and non-issues. From the gathered data the dimensions completeness, accuracy and consistency were found to be critical issues of data quality. The three most prevalent obstacles for improving data quality were data ownership, data standardization and understanding the importance of data quality. To overcome these obstacles the most important measures are creating data ownership structures, implementing data quality practices and changing the mindset of the employees to a data driven mindset. The generalizability of a single case study is low. However, there are insights and trends which can be derived from the results of this thesis and used for further studies and companies undergoing similar transformations. / Den ökade användningen av internet har ökat mängden data som finns tillgänglig och mängden data som samlas in. Företag påbörjar därför initiativ för att analysera dessa stora mängder data för att få ökad förståelse. Dock är värdet av analysen samt besluten som baseras på analysen beroende av kvaliteten av den underliggande data. Av denna anledning har datakvalitet blivit en viktig fråga för företag. Misslyckanden i datakvalitetshantering är ofta på grund av organisatoriska aspekter. Eftersom decentraliserade organisationsformer blir alltmer populära, finns det ett behov av att förstå hur en decentraliserad organisation kan arbeta med frågor som datakvalitet och dess förbättring. Denna uppsats är en kvalitativ studie av ett företag inom logistikbranschen som i nuläget genomgår ett skifte till att bli datadrivna och som har problem med att underhålla sin datakvalitet. Syftet med denna uppsats är att besvara frågorna: • RQ1: Vad är datakvalitet i sammanhanget logistikdata? • RQ2: Vilka är hindren för att förbättra datakvalitet i en decentraliserad organisation? • RQ3: Hur kan dessa hinder överkommas? Flera datakvalitetsdimensioner identifierades och kategoriserades som kritiska problem, problem och icke-problem. Från den insamlade informationen fanns att dimensionerna, kompletthet, exakthet och konsekvens var kritiska datakvalitetsproblem för företaget. De tre mest förekommande hindren för att förbättra datakvalité var dataägandeskap, standardisering av data samt att förstå vikten av datakvalitet. För att överkomma dessa hinder är de viktigaste åtgärderna att skapa strukturer för dataägandeskap, att implementera praxis för hantering av datakvalitet samt att ändra attityden hos de anställda gentemot datakvalitet till en datadriven attityd. Generaliseringsbarheten av en enfallsstudie är låg. Dock medför denna studie flera viktiga insikter och trender vilka kan användas för framtida studier och för företag som genomgår liknande transformationer.

<p>  </p> <p>Most new vehicles manufactured in the last two years are connected vehicles (CV) that transmit back to the original equipment manufacturer at near real-time fidelity. These CVs generate billions of data points on an hourly basis, which can provide valuable data to agencies to improve the overall mobility experience for users. However, with this growing scale of CV big data, stakeholders need efficient and scalable methodologies that allow agencies to draw actionable insights from this large-scale data for daily operational use. This dissertation presents a suite of applications, illustrated through case studies, that use CV data for assessing and managing mobility and safety on surface transportation systems.</p> <p>A systematic review of construction zone CV data and crashes on Indiana’s interstates for the calendar year 2019, found a strong correlation between crashes and hard-braking event data reported by CVs. Trajectory-level CV data analyzed for a construction zone on interstate 70 provided valuable insights into travel time and traffic signal performance impacts on the surrounding road network. An 11-state analysis of electric and hybrid vehicle usage in proximity to public charging stations highlighted regions under and overserved by charging infrastructure, providing quantitative support for infrastructure investment allocations informed by real-world usage trends. CV data were further leveraged to document route choice behavior during active freeway incidents providing stakeholders with a historical record of observed routing patterns to inform future alternate route planning strategies. CV trajectory data analysis facilitated the identification of trip chaining activities resulting in improved outlier curation and realistic estimation of travel time metrics.</p> <p>The overall contribution of this thesis is developing analytical big data procedures to process billions of CV data records to inform engineering and public policy investments in infrastructure capacity, highway safety improvements, and new EV infrastructure. These scalable and efficient analysis techniques proposed in this dissertation will help agencies at the federal, state and local levels in addition to private sector stakeholders in assessing transportation system performance at-scale and enable informed data-driven decision making.</p>

[ES] La investigación llevada a cabo en esta tesis doctoral tiene como objetivo general analizar la importancia del poder transformador de la analítica de big data, a través de las capacidades analíticas de big data en el ecosistema español de las pequeñas y medianas empresas. El contexto de la transformación digital ha remodelado la forma de hacer negocios en las organizaciones debido a la complejidad e incertidumbre del entorno, el surgimiento de empresas nativas digitales, la introducción de nuevas tecnologías e industria 4.0 y el aumento de la competitividad de los mercados. Si bien la implantación de la tecnología e infraestructuras digitales ha sido un tema estudiado en la literatura académica en los últimos años, se divisan retos importantes a nivel humano debido a la modificación del contexto laboral, el liderazgo y las habilidades y competencias necesarias para competir de forma exitosa actualmente. Las personas, y no la tecnología, son el centro de la transformación digital y todo cambio organizativo derivado debe priorizarlas. Muchas de las empresas que invierten en tecnologías como el big data son incapaces de extraer el valor que éste puede ofrecer a través de la analítica de datos, y por tanto, no lo utilizan para tomar decisiones de valor para la organización que lleven a un incremento del desempeño. Tienen poco desarrolladas las capacidades analíticas de big data, necesarias para aprovechar la transformación digital y promover una implementación efectiva de las nuevas tecnologías. De esta problemática derivamos la importancia de conocer cuáles son los antecedentes de las capacidades analíticas de big data y su efecto en ellas, con el objetivo de conseguir un verdadero impacto en el desempeño organizativo. Por una parte, la cultura organizativa se ha identificado como una de las barreras al cambio o un factor impulsor que permite efectuar una transformación digital efectiva. Para ello es necesario implantar nuevas formas de trabajar y adquirir habilidades y conocimientos adecuados que permitan tomar decisiones en base al análisis de los datos. Es por tanto fundamental, que la cultura organizativa promueva e incentive la promoción de capacidades analíticas de big data y la transformación digital. Por otra parte, se destaca el papel del CEO de la organización, y de su visión estratégica orientada al dato para incentivar, liderar y motivar el cambio hacia una transformación digital. El rol del directivo es crucial para motivar un cambio cultural que permita ver la transformación digital y las capacidades analíticas de big data como instrumentos para lograr una mejora de la competitividad, desempeño, creación de valor y aumento de la reputación y satisfacción de las personas. Por tanto, el CEO debe tener un compromiso con la transformación digital tangible y visión estratégica orientada a los datos para tomar decisiones y planificar la estrategia a seguir por toda la organización. Entre las conclusiones del estudio se destaca en primer lugar la relación positiva y significativa de las capacidades analíticas de big data con la transformación digital y el desempeño organizativo a través de la innovación. Además, se pone en valor la importancia de la cultura organizativa y de la orientación al dato, así como de un nivel adecuado de madurez digital, como antecedentes de las capacidades analíticas de big data. Finalmente se analizan los diversos arquetipos culturales para destacar que una cultura digital, jerárquica o adhocrática favorecen la creación de capacidades analíticas y por tanto incrementan el proceso de transformación digital. A partir de las conclusiones se deriva la necesidad de inversión en formación para las personas en capacidades digitales y analíticas y el rol clave del directivo para conseguir una transformación digital exitosa y aprovechar la inversión tecnológica realizada. Por último, se destaca la importancia del diagnóstico cultural y elaboración de un plan de cambio cultural. / [CA] La investigació duta a terme en aquesta tesi doctoral té com a objectiu general analitzar la importància del poder transformador de l'analítica de big data, a través de les capacitats analítiques de big data en l'ecosistema espanyol de les petites i mitjanes empreses. El context de la transformació digital ha remodelat la manera de fer negocis en les organitzacions a causa de la complexitat i incertesa de l'entorn, el sorgiment d'empreses natives digitals, la introducció de noves tecnologies i indústria 4.0 i l'augment de la competitivitat dels mercats. Tot i que la implantació de la tecnologia i infraestructures digitals ha sigut un tema estudiat en la literatura acadèmica en els últims anys, s'albiren reptes importants a nivell humà a causa de la modificació del context laboral, el lideratge i les habilitats necessàries per a competir de manera exitosa actualment. Les persones, i no la tecnologia, són el centre de la transformació digital i tot canvi organitzatiu derivat ha de prioritzar-les. Moltes de les empreses que inverteixen en tecnologies com el big data són incapaços d'extraure el valor que aquest pot oferir a través de l'analítica de dades, i per tant, no l'utilitzen per a prendre decisions de valor per a l'organització que porten a una millora dels resultats. La raó és que tenen poc desenvolupades les capacitats analítiques de big data, necessàries per a aprofitar la transformació digital i promoure una implementació efectiva de les noves tecnologies. D'aquesta problemàtica derivem la importància de conéixer quins són els antecedents de les capacitats analítiques de big data i el seu efecte en elles, amb l'objectiu d'aconseguir un vertader impacte en la millora dels resultats. D'una banda, la cultura organitzativa s'ha identificat com una de les barreres al canvi per a efectuar una transformació digital efectiva. Per aquesta raó cal implantar noves maneres de treballar i adquirir habilitats i coneixements adequats que permeten prendre decisions sobre la base de l'anàlisi de les dades. És per tant fonamental, que la cultura organitzativa promoga i incentive la promoció de capacitats analítiques de big data i la transformació digital. D'altra banda, es destaca el paper del CEO de l'organització, i de la seua visió estratègica orientada a les dades per a incentivar, liderar i motivar el canvi cap a una transformació digital. El paper directiu és crucial per a motivar un canvi cultural que permeta veure la transformació digital i les capacitats analítiques de big data com a instruments per a aconseguir una millora de la competitivitat, acompliment, creació de valor i augment de la reputació i satisfacció de les persones. Per tant, el CEO ha de tindre un compromís amb la transformació digital tangible i una visió orientada a les dades per a prendre decisions i planificar l'estratègia a seguir per tota l'organització. Entre les conclusions de l'estudi es destaca en primer lloc la relació positiva i significativa de les capacitats analítiques de big data amb la transformació digital i la millora dels resultats a través de la innovació. A més, es posa en valor la importància de la cultura organitzativa i de l'orientació a la dada, així com d'un nivell adequat de maduresa digital, com a antecedents de les capacitats analítiques de big data. Finalment s'analitzen els diversos arquetips culturals i es destaca que una cultura digital, jeràrquica o adhocrática afavoreixen la creació de capacitats analítiques i per tant incrementen l'éxit del procés de transformació digital. A partir de les conclusions es deriven algunes implicacions pràctiques com la necessitat d'inversió en formació per a les persones en competències i capacitats digitals i analítiques, el rol clau del directiu per a aconseguir una transformació digital exitosa i aprofitar la inversió tecnològica. Finalment es destaca la importància del diagnòstic cultural i elaboració d'un pla de canvi cultural alineat amb els objectius envers la transformació digital. / [EN] The general objective of the research carried out in this doctoral thesis is to analyse the importance of the transformative power of big data analytics through big data analytical capabilities in the Spanish context of small and medium-sized enterprises. The context of digital transformation has reshaped the way of doing business in organisations due to the complexity and uncertainty of the environment, the emergence of digital native companies, the introduction of new technologies and the increased competitiveness of markets. Whilst the implementation of technology and digital infrastructures has been covered in the academic literature in recent years, there are significant challenges at the human level due to the changing context of work, leadership and the skills needed to compete successfully today. People, not technology, are at the heart of digital transformation and any resulting organisational change must priorise them. Many companies that invest in technologies such as big data are unable to extract the value that big data can offer through data analytics, and therefore do not use it to make valuable decisions for the organisation that lead to increased performance. They have underdeveloped big data analytical capabilities, which are necessary to take advantage of digital transformation and promote the effective implementation of new technologies. From this problem the importance of knowing the background of big data analytical capabilities and their effect on them is derived, in order to achieve a real impact on organisational performance. On the one hand, organisational culture has been identified as a barrier or booster of change for an effective digital transformation. This requires the implementation of new ways of working and the acquisition of appropriate skills and knowledge to enable data-driven decision making. It is therefore essential that the organisational culture promotes and encourages the promotion of big data analytical capabilities and digital transformation. On the other hand, the role of the CEO of the organisation, and his or her data-driven strategic vision to incentivise, lead and motivate change towards digital transformation is highlighted. The role of the top management is crucial to motivate a cultural change that allows to see digital transformation and big data analytics capabilities as instruments to achieve superior outcomes (i.e., improved competitiveness, performance, value creation and increased reputation and people satisfaction). Therefore, the CEO must have a strong commitment to digital transformation and a data-driven orientation to make decisions and settle the strategy for the entire organisation. Among the conclusions of the study, the positive and significant relationship of big data analytics capabilities with digital transformation and organisational performance through innovation are highlighted. This thesis points out the importance of organisational culture and data orientation, as well as an appropriate level of digital maturity, as antecedents to big data analytics capabilities. Finally, the various cultural archetypes are analysed to highlight that a digital, hierarchical or adhocratic culture favours the creation of analytical capabilities and therefore enhances the digital transformation process. From the conclusions, some practical implications are derived, such as the need to invest in training people in digital and analytical skills and capabilities, the key role of the manager in achieving a successful digital transformation and leveraging technological investment. Finally, the importance of cultural diagnosis and the development of a cultural change plan aligned with the strategic objectives for digital transformation is highlighted, and practical recommendations are settled. / Tesis elaborada gracias al apoyo de las Ayudas de Formación del Profesorado Universitario (FPU) otorgadas por el Ministerio de Educación, Cultura y Universidades, del Gobierno de España, y a la Cátedra de Empresa y Humanismo de la Universitat de València. / Orero Blat, M. (2023). Enhancing Big Data Analytics Capabilities: The Influence of Organisational Culture and Data-Driven Orientation [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/191788

資料的快速增長與變化以及分析工具日新月異，增加資料分析的挑戰，本研究希望透過一個完整機器學習流程，提供學術或企業在導入大數據分析時的參考藍圖。我們以Spark作為大數據分析的計算框架，利用MLlib的Spark.ml與Spark.mllib兩個套件建構機器學習模型，解決傳統資料分析時可能會遇到的問題。在資料分析過程中會比較Spark不同分析模組的適用性情境，首先使用本地端叢集進行開發，最後提交至Amazon雲端叢集加快建模與分析的效能。大數據資料分析流程將以微博為實驗範例，並使用香港大學新聞與傳媒研究中心提供的2012年大陸微博資料集，我們採用RDD、Spark SQL與GraphX萃取微博使用者貼文資料的特增值，並以隨機森林建構預測模型，來預測使用者是否具有官方認證的二元分類。 / The rapid growth of data volume and advanced data analytics tools dramatically increase the challenge of big data analytics services adoption. This paper presents a big data analytics pipeline referenced blueprint for academic and company when they consider importing the associated services. We propose to use Apache Spark as a big data computing framework, which Spark MLlib contains two packages Spark.ml and Spark.mllib, on building a machine learning model. This resolves the traditional data analytics problem. In this big data analytics pipeline, we address a situation for adopting suitable Spark modules. We first use local cluster to develop our data analytics project following the jobs submitted to AWS EC2 clusters to accelerate analytic performance. We demonstrate the proposed big data analytics blueprint by using 2012 Weibo datasets. Finally, we use Spark SQL and GraphX to extract information features from large amount of the Weibo users’ posts. The official certification prediction model is constructed for Weibo users through Random Forest algorithm.

Security attacks are becoming more prevalent as cyber attackers exploit system vulnerabilities for financial gain. The resulting loss of revenue and reputation can have deleterious effects on governments and businesses alike. Signature recognition and anomaly detection are the most common security detection techniques in use today. These techniques provide a strong defense. However, they fall short of detecting complicated or sophisticated attacks. Recent literature suggests using security analytics to differentiate between normal and malicious user activities. The goal of this research is to develop a repeatable process to detect cyber attacks that is fast, accurate, comprehensive, and scalable. A model was developed and evaluated using several production log files provided by the University of North Florida Information Technology Security department. This model uses security analytics to complement existing security controls to detect suspicious user activity occurring in real time by applying machine learning algorithms to multiple heterogeneous server-side log files. The process is linearly scalable and comprehensive; as such it can be applied to any enterprise environment. The process is composed of three steps. The first step is data collection and transformation which involves identifying the source log files and selecting a feature set from those files. The resulting feature set is then transformed into a time series dataset using a sliding time window representation. Each instance of the dataset is labeled as green, yellow, or red using three different unsupervised learning methods, one of which is Partitioning around Medoids (PAM). The final step uses Deep Learning to train and evaluate the model that will be used for detecting abnormal or suspicious activities. Experiments using datasets of varying sizes of time granularity resulted in a very high accuracy and performance. The time required to train and test the model was surprisingly fast even for large datasets. This is the first research paper that develops a model to detect cyber attacks using security analytics; hence this research builds a foundation on which to expand upon for future research in this subject area.

Submitted by Gilson de Oliveira Silva Calais (gilcalais@terra.com.br) on 2017-12-18T17:32:40Z No. of bitstreams: 1 MPGPP - Trabalho FInal - SEADE - 01-12-2017.pdf: 2421044 bytes, checksum: c213c0a77d4d53cd5e4ce850af4db68a (MD5) / Approved for entry into archive by Mayara Costa de Sousa (mayara.sousa@fgv.br) on 2017-12-18T22:56:16Z (GMT) No. of bitstreams: 1 MPGPP - Trabalho FInal - SEADE - 01-12-2017.pdf: 2421044 bytes, checksum: c213c0a77d4d53cd5e4ce850af4db68a (MD5) / Made available in DSpace on 2017-12-19T11:39:42Z (GMT). No. of bitstreams: 1 MPGPP - Trabalho FInal - SEADE - 01-12-2017.pdf: 2421044 bytes, checksum: c213c0a77d4d53cd5e4ce850af4db68a (MD5) Previous issue date: 2017-12-01 / Elaborado ao longo de 2017, o presente estudo tem por objetivo a análise organizacional da Fundação Sistema Estadual de Análise de Dados (SEADE), entidade de direito público vinculada à Secretaria Estadual de Planejamento e Gestão, responsável pela produção e disseminação de análises e estatísticas socioeconômicas e demográficas do Estado de São Paulo. Frente ao atual contexto de mudanças do setor, devido aos impactos das novas tecnologias e, em especial, aos efeitos do Big Data Analytics, a gestão focada em processos e uma nova estrutura organizacional, elaborada com base em melhores práticas e em modelagem de processos de referência internacional, mostram-se fundamentais para assegurar os investimentos necessários para manter sua capacidade de produzir e disseminar informações estatísticas em alto nível e adequadas às necessidades e expectativas dos usuários. Para conciliar harmonicamente esses objetivos, recomenda-se um conjunto de ações, de caráter estratégico, que, além de objetivar melhor atendimento àquilo que lhe é prioritário, também implique em mais acessos por parte dos usuários, através de sua principal ferramenta de comunicação com o mercado. Tais achados e recomendações baseiam-se em uma revisão do modelo de negócio subscrito na configuração de seus processos operacionais e de apoio, no arranjo organizacional e na forma de sua comunicação com os usuários, além de responder às crescentes demandas por maior eficiência e transparência na gestão dos recursos públicos. / Elaborated in 2017, the present study aims the organizational analysis of the Fundação Sistema Estadual de Análise de Dados (SEADE), an entity of public law linked to the State Department of Planning and Management, responsible for the production and dissemination of analyzes and socioeconomic and demographic statistics of the State of São Paulo. Given the current context of changes in the industry, due to the impacts of new technologies and the effects of Big Data Analytics, the process-focused management and a new organizational structure, based on best practices and processes modeling of international reference, are essential to ensure the necessary investments to maintain their capacity to produce and disseminate statistical information at a high level and adapted to the needs and expectations of the users. To harmoniously reconcile these objectives, it is recommended a set of actions, of strategic nature, which, in addition to objectifying better attendance to its priority, also implies more access by the users through its main communication tool with the market. These findings and recommendations are based on a review of the business model underwritten in the configuration of its operational and support processes, in the organizational arrangement and in the form of its communication with the users, in addition to responding to the growing demands for greater efficiency and transparency in the management of public resources.

In Ethiopia, agriculture accounts for 85% of the total employment, and the country’s export entirely relies on agricultural commodities. The country is continuously affected by chronic food shortage. In the last 40 years, the country’s population have almost tripled; and more agricultural productivity is required to support the livelihood of millions of citizens. As reported by various research, Ethiopia needs to address a number of policy and strategic priorities to improve agriculture; however, in-efficient agriculture supply chain for the supply of input is identified as one of the significant challenges to develop agricultural productivity in the country. The research problem that interest this thesis is to understand Big Data Analytics’ (BDA) potential in achieving better Agriculture Input Supply Chain in Ethiopia. Based on this, we conducted a basic qualitative study to understand the expectations of Supply Chain Management (SCM) professionals, the requirements for the potential applications of Big Data Analytics - and the implications of applying the same from the perspectives of SCM professionals in Ethiopia. The findings of the study suggest that BDA may bring operational and strategic benefit to agriculture input supply chain in Ethiopia, and the application of BDA may have positive implication to agricultural productivity and food security in the country. The findings of this study are not generalizable beyond the participants interviewed.

About a third of the energy in Sweden is used for production in industry, where a few energy-intensive industries account for a large proportion of the energy use. When energy efficiency takes place here, the positive environmental effects will be substantial. Therefore, there is often great potential to reduce energy use in these industries. The purpose of this study is to investigate the company's energy use in order to be able to present proposals for cost-effective improvement measures that lead to a reduction in the company's energy consumption, which in turn leads to a reduced environmental impact. The study has used sustainable development, total quality management and Kotter’s 8-step process for leading change as a theoretical background. The study is a qualitative and quantitative case study and the data collection was done using semi-structured and unstructured interviews, observations and document analysis. The study’s results show that the company uses unnecessary energy in three energy consumption areas: compressed air, heating and electricity. To reduce energy use, the company needs to put in place a shut-down management, appoint an energy coordinator, prioritize preventive work, repair broken equipment and introduce preventive maintenance, optimize the ovens, involve employees in continuous improvement and to have a committed leadership. By reducing energy consumption, the company can reduce its impact on the environment, and by implementing the improvement measures, the company can save several million SEK each year. / Ungefär en tredjedel av energin i Sverige används för produktion inom industrin, där ett fåtal energiintensiva branscher står för en stor andel av industrins energianvändning. När energieffektivisering sker här blir de positiva miljöeffekterna stora. Därför finns det ofta stor potential att minska energianvändningen i dessa branscher. Syftet med denna studie är att undersöka företagets energianvändning för att sedan kunna presentera förslag på kostnadseffektiva förbättringsåtgärder som leder till en minskning av företagets energiförbrukning, vilket i sin tur leder till en minskad miljöpåverkan. Studien har använt sig av hållbar utveckling, hörnstensmodellen och Kotters 8-stegsmodell för förändringsledning som teoretisk bakgrund. Studien är en kvalitativ och kvantitativ fallstudie och datainsamlingen gjordes med hjälp av semistrukturerade och ostrukturerade intervjuer, observationer och dokumentstudier. Studiens resultat visar på att företaget använder energi i onödan inom tre energiförbrukningsområden: tryckluft, värme och el. För att minska energianvändningen behöver företaget införa avstängningsrutiner, utse en energikoordinator, prioritera förebyggande arbete, reparera trasig utrustning och införa förebyggande underhåll, optimera ugnarna, involvera medarbetarna i förbättringsarbetet och ha ett engagerat och delaktigt ledarskap. Genom att minska energiförbrukningen kan företaget minska sin påverkan på miljön och genom att implementera förbättringsåtgärder kan företaget spara flera miljoner kronor varje år.

<p>The detection of anomalies in real-world networks is applicable in different domains; the application includes, but is not limited to, credit card fraud detection, malware identification and classification, cancer detection from diagnostic reports, abnormal traffic detection, identification of fake media posts, and the like. Many ongoing and current researches are providing tools for analyzing labeled and unlabeled data; however, the challenges of finding anomalies and patterns in large-scale datasets still exist because of rapid changes in the threat landscape. </p><p>In this study, I implemented a novel and robust solution that combines data science and cybersecurity to solve complex network security problems. I used Long Short-Term Memory (LSTM) model, Louvain algorithm, and PageRank algorithm to identify and group anomalies in large-scale real-world networks. The network has billions of packets. The developed model used different visualization techniques to provide further insight into how the anomalies in the network are related. </p><p>Mean absolute error (MAE) and root mean square error (RMSE) was used to validate the anomaly detection models, the results obtained for both are 5.1813e-04 and 1e-03 respectively. The low loss from the training phase confirmed the low RMSE at loss: 5.1812e-04, mean absolute error: 5.1813e-04, validation loss: 3.9858e-04, validation mean absolute error: 3.9858e-04. The result from the community detection shows an overall modularity value of 0.914 which is proof of the existence of very strong communities among the anomalies. The largest sub-community of the anomalies connects 10.42% of the total nodes of the anomalies. </p><p>The broader aim and impact of this study was to provide sophisticated, AI-assisted countermeasures to cyber-threats in large-scale networks. To close the existing gaps created by the shortage of skilled and experienced cybersecurity specialists and analysts in the cybersecurity field, solutions based on out-of-the-box thinking are inevitable; this research was aimed at yielding one of such solutions. It was built to detect specific and collaborating threat actors in large networks and to help speed up how the activities of anomalies in any given large-scale network can be curtailed in time.</p><div><div><div> </div> </div> </div> <br>

De nos jours, il y a une demande croissante pour fournir de l'information temps réel à partir de l'environnement, e.g. état infectieux de maladies, force du signal, conditions de circulation, qualité de l'air. La prolifération des dispositifs de capteurs et la mobilité des personnes font de la Mobile Collaborative Sensing (MCS) un moyen efficace de détecter et collecter l'information à un faible coût. Dans MCS, au lieu de déployer des capteurs statiques dans une zone, les personnes disposant d'appareils mobiles jouent le rôle de capteurs mobiles. En général, une application MCS exige que l'appareil de chacun ait la capacité d'effectuer la détection et retourne les résultats à un serveur central, mais également de collaborer avec d'autres dispositifs. Pour que les résultats puissent représenter l'information physique d'une région cible et convenir, quel type de données peut être utilisé et quel type d'information doit être inclus dans les données collectées? Les données spatio-temporelles peuvent être utilisées par des applications pour bien représenter la région cible. Dans des applications différentes, l'information de localisation et de temps sont 2 types d'information communes, et en les utilisant la région cible d'une application est sous surveillance complète du temps et de l'espace. Différentes applications nécessitent de l'information différente pour atteindre des objectifs différents. E.g. dans cette thèse: i- MCS-Locating application: l'information de résistance du signal doit être incluse dans les données détectées par des dispositifs mobiles à partir d'émetteurs de signaux ; ii- MCS-Prédiction application : la relation entre les cas d'infection et les cas infectés doit être incluse dans les données par les dispositifs mobiles provenant des zones de flambée de la maladie ; iii- MCS-Routing application : l'information routière en temps réel provenant de différentes routes de circulation doit être incluse dans les données détectées par des dispositifs embarqués. Avec la détection de l'information physique d'une région cible, et la mise en interaction des dispositifs, 3 thèmes d'optimisation basés sur la détection sont étudiés et 4 travaux de recherche menés: -Mobile Collaboratif Détection Cadre : un cadre mobile de détection collaborative est conçu pour faciliter la coopérativité de la collecte, du partage et de l'analyse des données. Les données sont collectées à partir de sources et de points temporels différents. Pour le déploiement du cadre dans les applications, les défis clés pertinents et les problèmes ouverts sont discutés. -MCS-Locating : l'algorithme LiCS (Locating in Collaborative Sensing based Data Space) est proposé pour atteindre la localisation de la cible. LiCS utilise la puissance du signal reçu dans tous les périphériques sans fil comme empreintes digitales de localisation pour les différents emplacements. De sorte LiCS peut être directement pris en charge par l'infrastructure sans fil standard. Il utilise des données de trace d'appareils mobiles d'individus, et un modèle d'estimation d'emplacement. Il forme le modèle d'estimation de localisation en utilisant les données de trace pour atteindre la localisation de la cible collaborative. Cette collaboration entre périphériques est au niveau des données et est supportée par un modèle. -MCS-Prédiction: un modèle de reconnaissance est conçu pour acquérir dynamiquement la connaissance de structure de la RCN pertinente pendant la propagation de la maladie. Sur ce modèle, un algorithme de prédiction est proposé pour prédire le paramètre R. i.e. le nombre de reproduction qui est utilisé pour quantifier la dynamique de la maladie pendant sa propagation. -MCS-Routing : un algorithme de navigation écologique ‘eRouting’ est conçu en combinant l'information de trafic temps réel et un modèle d'énergie/émission basé sur des facteurs représentatifs. Sur la base de l'infrastructure standard d'un système de trafic intelligent, l'information sur le trafic est collectée / Nowadays, there is an increasing demand to provide real-time information from the environment, e.g., the infection status of infectious diseases, signal strength, traffic conditions, and air quality, to citizens in urban areas for various purposes. The proliferation of sensor-equipped devices and the mobility of people are making Mobile Collaborative Sensing (MCS) an effective way to sense and collect information at a low deployment cost. In MCS, instead of just deploying static sensors in an interested area, people with mobile devices play the role of mobile sensors to sense the information of their surroundings, and the communication network (3G, WiFi, etc.) is used to transfer data for MCS applications. Typically, a MCS application not only requires each participant's mobile device to possess the capability of performing sensing and returning sensed results to a central server, but also requires to collaborate with other mobile and static devices. In order to make sensed results well represent the physical information of a target region, and well be suitable to a certain application, what kind of data can be used for different applications, and what kind of information needs to be included into the collected sensing data? Spatio-temporal data can be used by different applications to well represent the target region. In different applications, location and time information is two kinds of common information, and by using such information, the target region of an application is under comprehensive monitoring from the view of time and space. Different applications require different information to achieve different sensing purposes. E.g. in this thesis: i- MCS-Locating application: signal strength information needs to be included into the sensed data by mobile devices from signal transmitters; ii- MCS-Prediction application: the relationship between infecting and infected cases needs to be included into the sensed data by mobile devices from disease outbreak areas; iii- MCS-Routing application: real-time traffic and road information from different traffic roads, e.g., traffic velocity and road gradient, needs to be included into the sensed data by road-embedded and vehicle-mounted devices. With sensing the physical information of a target region, and making mobile and static devices collaborate with each other in mind, in this thesis three sensing based optimization applications are studied, and following four research works are conducted: - a MCS Framework is designed to facilitate the cooperativity of data collection, sharing, and analysis among different devices. Data is collected from different sources and time points. For deploying the framework into applications, relevant key challenges and open issues are discussed. - MCS-Locating: an algorithm LiCS (Locating in Collaborative Sensing based Data Space) is proposed to achieve target locating. It uses Received Signal Strength that exists in any wireless devices as location fingerprints to differentiate different locations, so it can be directly supported by off-the-shelf wireless infrastructure. LiCS uses trace data from individuals' mobile devices, and a location estimation model. It trains the location estimation model by using the trace data to achieve collaborative target locating. Such collaboration between different devices is data-level, and model-supported. - MCS-Prediction: a recognition model is designed to dynamically acquire the structure knowledge of the relevant RCN during disease spread. On the basis of this model, a prediction algorithm is proposed to predict the parameter R. R is the reproductive number which is used to quantify the disease dynamics during disease spread. - MCS-Routing: an eco-friendly navigation algorithm, eRouting, is designed by combining real-time traffic information and a representative factor based energy/emission model. Based on the off-the-shelf infrastructure of an intelligent traffic system, the traffic information is collected