Poggio and Vetter (1992) showed that learning one view of a bilaterally symmetric object could be sufficient for its recognition, if this view allows the computation of a symmetric, "virtual," view. Faces are roughly bilaterally symmetric objects. Learning a side-view--which always has a symmetric view--should allow for better generalization performances than learning the frontal view. Two psychophysical experiments tested these predictions. Stimuli were views of shaded 3D models of laser-scanned faces. The first experiment tested whether a particular view of a face was canonical. The second experiment tested which single views of a face give rise to best generalization performances. The results were compatible with the symmetry hypothesis: Learning a side view allowed better generalization performances than learning the frontal view.

In this dissertation, we introduce three techniques for network sciences. The first of these techniques is a series of new models for describing network growth. These models, called network specialization models, are built with the idea that networks grow by specializing the function of subnetworks. Using these models we create theoretical networks which exhibit well-known properties of real networks. We also demonstrate how the spectral properties are preserved as the models grow. The second technique we describe is a method for decomposing networks that contain automorphisms in a way that preserves the spectrum of the original graph. This method for graph (or equivalently matrix) decomposition is called an equitable decomposition. Previously this method could only be used for particular classes of automorphisms, but in this dissertation we have extended this theory to work for every automorphism. Further we explain a number of applications which use equitable decompositions. The third technique we describe is a generalization of network symmetry, called latent symmetry. We give numerous examples of networks which contain latent symmetries and also prove some properties about them

Neural networks trained using gradient-based optimization methods exhibit a surprising phenomenon known as mode connectivity, where two independently trained network weights are not isolated low loss minima in the parameter space. Instead, they can be connected by simple curves along which the loss remains low. In case of linear mode connectivity up to permutation, even linear interpolations of the trained weights incur low loss when networks that differ by permutation of their hidden neurons are considered equivalent. While some recent research suggest that this implies existence of a single near-convex loss basin to which the parameters converge, others have empirically shown distinct basins corresponding to different strategies to solve the task. In some settings, averaging multiple network weights naively, without explicitly accounting for permutation invariance still results in a network with improved generalization. In this thesis, linear mode connectivity among a set of neural networks independently trained on labelled datasets, both naively and upon reparameterization to account for permutation invariance is studied. Specifically, the effect of hidden layer width on the connectivity is empirically evaluated. The experiments are conducted on a two dimensional toy classification problem, and the insights are extended to deeper networks trained on handwritten digits and images. It is argued that accounting for permutation of hidden neurons either explicitly or implicitly is necessary for weight averaging to improve test performance. Furthermore, the results indicate that the training dynamics induced by the optimization plays a significant role, and large model width alone may not be a sufficient condition for linear model connectivity. / Neurala nätverk som tränats med gradientbaserade optimeringsmetoder uppvisar ett överraskande fenomen som kallas modeconnectivity, där två oberoende tränade nätverksvikter inte är isolerade lågförlustminima i parameterutrymmet. Istället kan de kopplas samman med enkla kurvor längs vilka förlusten förblir låg. I händelse av linjär mode-anslutning upp till permutation medför även linjära interpolationer av de tränade vikterna låga förluster när nätverk som skiljer sig åt genom permutation av deras dolda neuroner anses vara likvärdiga. Medan en del nyare undersökningar tyder på att detta innebär att det finns en enda nära-konvex förlustbassäng till vilken parametrarna konvergerar, har andra empiriskt visat distinkta bassänger som motsvarar olika strategier för att lösa uppgiften. I vissa inställningar resulterar ett naivt medelvärde av flera nätverksvikter, utan att uttryckligen ta hänsyn till permutationsinvarians, fortfarande i ett nätverk med förbättrad generalisering. I den här avhandlingen studeras linjärmodsanslutningar mellan en uppsättning neurala nätverk som är oberoende tränade på märkta datamängder, både naivt och vid omparameterisering för att ta hänsyn till permutationsinvarians. Specifikt utvärderas effekten av dold lagerbredd på anslutningen empiriskt. Experimenten utförs på ett tvådimensionellt leksaksklassificeringsproblem, och insikterna utökas till djupare nätverk som tränas på handskrivna siffror och bilder. Det hävdas att redogörelse för permutation av dolda neuroner antingen explicit eller implicit är nödvändigt för viktgenomsnitt för att förbättra testprestanda. Dessutom indikerar resultaten att träningsdynamiken som induceras av optimeringen spelar en betydande roll, och enbart stor modellbredd kanske inte är ett tillräckligt villkor för linjär modellanslutning.