In this paper, we try to understand the concept of ’Neural Collapse’ from a mathemati-cal point of view. The survey will be conducted based on [1]. The authors of [1] providea first global optimization landscape analysis of Neural Collapse. Mainly there are threeaspects the authors like to investigate. The first is to add the weight decay on classicalcross-entropy loss to show that the global minimizers are the simplex ETF based onanalysing the Hessian. Secondly, the ’Layer-peeled’ network still preserves the im-portant features of the full network. In other words even simplifying the loss functionthe network does not lose its explainability. Lastly, how the Layer-peeled network canreduce the memory costs and generalization is as good as the full network. Our studydelves into these details on, how the simplified network is defined? How this simplifiednetwork is different from the original network in terms of the loss function, and finallywe understand the theory behind these steps. We also conduct numerical analysis onspecific input, observe and analyze this phenomenon and finally report our results.
| Identifer | oai:union.ndltd.org:UPSALLA1/oai:DiVA.org:umu-193721 |
| Date | January 2022 |
| Creators | Mokkapati, Siva |
| Publisher | Umeå universitet, Institutionen för matematik och matematisk statistik |
| Source Sets | DiVA Archive at Upsalla University |
| Language | English |
| Detected Language | English |
| Type | Student thesis, info:eu-repo/semantics/bachelorThesis, text |
| Format | application/pdf |
| Rights | info:eu-repo/semantics/openAccess |
Page generated in 0.0021 seconds