Yes / Machine learning (ML) is increasingly employed for automating complex tasks, specifically in autonomous driving. While ML applications bring us closer to fully autonomous systems, they simultaneously introduce security and safety risks specific to safety-critical systems. Existing methods of software development and systems based on ML are fundamentally different. Moreover, the existing certification methods for automotive systems cannot fully certify the safe operation of ML-based components and subsystems. This is because existing safety certification criteria were formulated before the advent of ML. Therefore, new or adapted methods are needed to certify ML-based systems. This article analyses the existing safety standard, ISO26262, for automotive applications, to determine the certifiability of ML approaches used in low-risk automotive applications. This will contribute towards addressing the task of assuring the security and safety of ML-based autonomous driving systems, particularly for low-risk automotive applications, to gain the trust of regulators, certification agencies, and stakeholders.

Yes / This article proposes an approach named SafeML II, which applies empirical cumulative distribution function-based statistical distance measures in a designed human-in-the loop procedure to ensure the safety of machine learning-based classifiers in autonomous vehicle software. The application of artificial intelligence (AI) and data-driven decision-making systems in autonomous vehicles is growing rapidly. As autonomous vehicles operate in dynamic environments, the risk that they can face an unknown observation is relatively high due to insufficient training data, distributional shift, or cyber-security attack. Thus, AI-based algorithms should make dependable decisions to improve their interpretation of the environment, lower the risk of autonomous driving, and avoid catastrophic accidents. This paper proposes an approach named SafeML II, which applies empirical cumulative distribution function (ECDF)-based statistical distance measures in a designed human-in-the-loop procedure to ensure the safety of machine learning-based classifiers in autonomous vehicle software. The approach is model-agnostic and it can cover various machine learning and deep learning classifiers. The German Traffic Sign Recognition Benchmark (GTSRB) is used to illustrate the capabilities of the proposed approach. / This work was supported by the Secure and Safe MultiRobot Systems (SESAME) H2020 Project under Grant Agreement 101017258.

<p>With the fast spread of machine learning techniques, sharing and adopting public deep neural networks become very popular. As deep neural networks are not intuitive for human to understand, malicious behaviors can be injected into deep neural networks undetected. We call it trojan attack or backdoor attack on neural networks. Trojaned models operate normally when regular inputs are provided, and misclassify to a specific output label when the input is stamped with some special pattern called trojan trigger. Deploying trojaned models can cause various severe consequences including endangering human lives (in applications like autonomous driving). Trojan attacks on deep neural networks introduce two challenges. From the attacker's perspective, since the training data or training process is usually not accessible to the attacker, the attacker needs to find a way to carry out the trojan attack without access to training data. From the user's perspective, the user needs to quickly scan the online public deep neural networks and detect trojaned models.</p> <p>We try to address these challenges in this dissertation. For trojan attack without access to training data, We propose to invert the neural network to generate a general trojan trigger, and then retrain the model with reverse-engineered training data to inject malicious behaviors to the model. The malicious behaviors are only activated by inputs stamped with the trojan trigger. To scan and detect trojaned models, we develop a novel technique that analyzes inner neuron behaviors by determining how output activation change when we introduce different levels of stimulation to a neuron. A trojan trigger is then reverse-engineered through an optimization procedure using the stimulation analysis results, to confirm that a neuron is truly compromised. Furthermore, for complex trojan attacks, we propose a novel complex trigger detection method. It leverages a novel symmetric feature differencing method to distinguish features of injected complex triggers from natural features. For trojan attacks on NLP models, we propose a novel backdoor scanning technique. It transforms a subject model to an equivalent but differentiable form. It then inverts a distribution of words denoting their likelihood in the trigger and applies a novel word discriminativity analysis to determine if the subject model is particularly discriminative for the presence of likely trigger words.</p>

Dans cette thèse, les travaux constitutifs demandent "Qu'est-ce qui est gagné ou perdu lorsque nous faisons varier le nombre de paramètres ?". Cette question est de plus en plus pertinente à l'ère de la recherche scientifique où la mise à l'échelle des réseaux est une recette largement utilisée pour garantir des gains de performance, mais où l'on ne comprend pas bien comment les changements de capacité modifient les propriétés de généralisation. Cette thèse mesure la généralisation selon plusieurs dimensions différentes, englobées par des questions telles que \textit{``Certains types d'exemples ou de classes sont-ils affectés de manière disproportionnée par la compression ?''} et \textit{``La variation du nombre de poids amplifie-t-elle la sensibilité aux erreurs corrompues ? contributions?''}. Pour explorer l'impact de la variation du nombre de poids, nous exploitons l'élagage - une classe de techniques de compression largement utilisée qui introduit un niveau de parcimonie des poids. Cela nous permet de faire varier précisément le nombre de poids apprenables dans les réseaux que nous comparons. Nous constatons à la fois dans la computer vision et dans NLP, à travers différents ensembles de données et tâches, que la parcimonie amplifie l'impact disparate sur les performances du modèle entre les sous-groupes de données minoritaires et majoritaires, de sorte que les \textit{les riches deviennent plus riches et les pauvres s'appauvrissent}. Même si l’erreur moyenne globale reste largement inchangée lorsqu’un modèle est compressé, les attributs sous-représentés encourent une part disproportionnée de l’erreur. Les modèles clairsemés \emph{cannibalisent} les performances sur les attributs protégés sous-représentés pour préserver les performances sur les attributs majoritaires. La compression amplifie également la sensibilité à certains types de perturbations. Nous trouvons également quelques mises en garde importantes : dans les contextes à faibles ressources où les données sont très spécialisées et distinctes des tâches en aval, la rareté aide en freinant la mémorisation et en induisant l'apprentissage d'une représentation plus générale. Les travaux inclus dans cette thèse suggèrent qu'il existe des rendements clairement décroissants pour une formule simple de paramètres d'échelle. Nos résultats ont de puissantes implications : la plupart des paramètres apprenables sont utilisés pour apprendre un ensemble de points de données qui portent une part disproportionnée de l'erreur. Nous appelons ces points de données Pruning Identified Exemplars (\texttt{PIEs}). Nous constatons que la majorité des poids sont nécessaires pour améliorer les performances sur ce petit sous-ensemble de la distribution de l’entraînement. Cela suggère qu'une petite fraction de la distribution de formation a un \textit{coût par capacité de données} beaucoup plus élevé. Cohérents dans tous les chapitres de cette thèse, nos résultats soutiennent la recommandation selon laquelle les modèles clairsemés doivent être soigneusement audités avant d'être déployés à grande échelle. L’un des principaux points à retenir de notre travail est que nos algorithmes ne sont pas impartiaux et que certains choix de conception peuvent amplifier les dommages. Il est essentiel de comprendre cet impact disparate compte tenu du déploiement généralisé de modèles compressés dans la nature. Nos résultats soutiennent la recommandation selon laquelle les modèles compressés font l'objet d'un audit supplémentaire avant leur déploiement dans la nature. / In this thesis, the constituent works ask “What is gained or lost as we vary the number of parameters?”. This question is increasingly relevant in an era of scientific inquiry where scaling networks is a widely used recipe to secure gains in performance but where there is not a good understanding of how changes in capacity alter generalization properties. This thesis measures generalization along several different dimensions, encompassed by questions such as “Are certain types of examples or classes disproportionately impacted by compression?” and “Does varying the number of weights amplify sensitivity to corrupted inputs?”. To explore the impact of varying the number of weights, we leverage pruning – a widely used class of compression techniques that introduce weight-level sparsity. We find in both computer vision and language settings, across different datasets and tasks that sparsity amplifies the disparate impact in model performance between minority and majority data subgroups such that the rich get richer and the poor get poorer. While the overall average error is largely unchanged when a model is compressed, underrepresented attributes incur a disproportionately high portion of the error. The work included in this thesis suggests that there are clearly decreasing returns to a simple formula of scaling parameters. Most learnable parameters are used to learn a set of data points which bears a disproportionately high portion of the error. We term these data points Pruning Identified Exemplars (PIEs). This suggests that a small fraction of the training distribution has a far higher per-data capacity cost. Consistent across all chapters of this thesis, our results support the finding that popular compression techniques are not impartial and can amplify harm. Our results support the recommendation that compressed models receive additional auditing scrutiny prior to deployment in the wild.