Globscope: Toward a Global View of the Loss Landscape

Understanding the global structure of neural network loss landscapes is important for gaining insight into model merging, hyperparameter selection, generalization, and the relationships between distinct solutions. Visualizing the global structure of loss landscapes is very challenging because of the high dimensionality of the parameter space of neural networks. Prior work has primarily focused on visualizing the loss landscape around one single basin, missing how different minima or basins relate to each other. We introduce Globscope, a framework for providing a global view of the loss landscape across multiple solutions or basins. Globscope learns a low-dimensional non-linear manifold of model parameters using an autoencoder framework, enabling both latent-space visualization and reconstruction of full model weights. Then it summarizes the relations among minima and connecting regions on this manifold through topological data analysis. Our framework produces continuous, interpretable visualizations that reveal global connectivity patterns in the landscape. We compare Globscope with kernel-based methods and demonstrate how it performs in preserving the global structure across diverse solutions. We further show how Globscope can be used to analyze two applications: revealing global low-loss solution pathways between distinct solutions using mode connectivity algorithms, and visualizing permutation symmetries of different solutions using re-basin approaches.