Speaker: David Schwab, CUNY Graduate Center Title: How noise affects the Hessian spectrum in overparameterized neural networks Abstract: Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks, contributing to their resurgence. While some theoretical progress has been made, it remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks with a degenerate valley in their loss landscape, SGD on average decreases the trace of the Hessian of the loss. We also show that isotropic noise in the non-degenerate subspace of the Hessian decreases its determinant. In addition to explaining SGDs role in sculpting the Hessian spectrum, this opens the door to new optimization approaches that guides the model to solutions with better generalization. We test our results with experiments on toy models and deep neural networks.